Cluster of Alertmanagers#

Principles#

To have Alertmanager running in HA, we have deployed it in cluster mode.

There are therefore more than one Alertmanager members.

In cluster mode :

Silences are duplicated between members
Notifications are sent by only one member
Alerts are not duplicated between members -> Prometheus must therefore send alerts to all members (see diagram below)
Members communicate with each other via the Gossip protocol using port 9094 by default (see diagram below).

We have multiple Alertmanager clusters in a K8s cluster.

Under certain conditions, it may happen that Alertmanager members end up mixed.

For example, instead of having two Alertmanager clusters with two members each, we have one large Alertmanager cluster with 4 members.

To avoid this problem, for each Alertmanager cluster, we use a different port for communication between the different members (see diagram below).

This diagram shows the ports used for the cloud cluster :

And for the infra cluster, we have :