Aller au contenu

AlertmanagerClusterDown#

This alert is triggered when half or more of the Alertmanager instances within the same cluster are down.

To determine if an Alertmanager member is down, the Alertmanager target on Prometheus is checked.

Context#

In Caascad and NGOT, Alertmanager is deployed as a cluster.

There are therefore several instances of Alertmanager.

So for a cluster of two instances, the alert is triggered as soon as an instance is down.

There are two severities:

  • warning: half the members are down
  • critical: more than half the members are down

Troubleshooting Hints#

Tip

Below, <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

  • Notice the namespace label in the alert

Check the node Alertmanager is running on#

Check that the node on which the Alertmanager is up.

Check Alertmanager status#

Check Alertmanager pod: 

kubectl get pod -o yaml <alertmanager_pod> -n <monitoring_namespace>
kubectl describe pod <alertmanager_pod> -n <monitoring_namespace>

Check the Alertmanager logs:

  • kubectl logs <alertmanager_pod> -n <monitoring_namespace> -c alertmanager

  • Go to Grafana. In the Explore tab, you can select Loki datasource and run query {app="alertmanager"}.

Get more information on the Prometheus target#

Do a "port-forward" on the prometheus of the cluster: 

$ kubectl -n monitoring port-forward svc/caascad-prometheus 12345:9090 &

Connect to http://localhost:12345/targets: click on Unhealty button to get the target down. You can see the error message.