Aller au contenu

AlertmanagerMembersInconsistent#

Context#

This alert appears when:

  • at least one of alertmanager cluster members cannot be found
  • there are too many members in the alertmanager cluster

Tip

Below, <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

  • Notice the namespace label in the alert

Warning

This alert may cause a silence to replicate across zones in the same cluster.

Verify the presence of silence on both zones to detect any duplication by running the command:

kubectl -n monitoring-stack-client-obs-<ZONE> exec -it alertmanager-0 -- amtool silence query --alertmanager.url http://localhost:9093

Case 1#

At least one of alertmanager cluster members cannot be found.

Troubleshoot#

Check if IP addresses discovered by alertmanager cluster are the same ones as in alertmanager Service.

Following example show possible inconsistency in Endpoint IP addresses:

$ kubectl describe svc -n <monitoring_namespace> <pod_name>-headless

Name:              <pod_name>-headless
Namespace:         <monitoring_namespace>
...
Port:              http  9093/TCP
TargetPort:        http/TCP
Endpoints:         172.16.0.107:9093,172.16.1.22:9093
Port:              cluster-tcp  29094/TCP
TargetPort:        29094/TCP
Endpoints:         172.16.0.107:29094,172.16.1.22:29094
Port:              cluster-udp  29094/UDP
TargetPort:        29094/UDP
Endpoints:         172.16.0.107:29094,172.16.1.22:29094


$ kubectl get pod -n <monitoring_namespace> -o wide -l app.kubernetes.io/name=alertmanager

NAME                     READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
<pod_name>   2/2     Running   0          4d18h   172.16.0.107   10.0.152.61   <none>           <none>
<pod_name>   2/2     Running   0          4d18h   172.16.1.34    10.0.120.93   <none>           <none>

Solution#

Deleting an incorrect Endpoint should trigger its recreation with a correct IP address: 

kubectl delete endpoints -n monitoring caascad-alertmanager-headless

Case 2#

There are too many members in the alertmanager cluster.

Troubleshoot#

  1. Put Alertmanager in debug mode: 

    kubectl edit statefulset -n monitoring caascad-alertmanager

    Add args --log.level=debug in alertmanager container.

  2. Check alertmanager logs to find ip of members:

    kubectl logs -n <monitoring_namespace> caascad-alertmanager-0 -c alertmanager | grep Notify

    -> 

    level=debug ts=2022-05-05T10:13:03.279Z caller=delegate.go:235 component=cluster received=NotifyJoin node=01G29VAW5C8YYJT3SHERRW4CV8 addr=172.16.1.7:9094
level=debug ts=2022-05-05T10:13:03.338Z caller=delegate.go:235 component=cluster received=NotifyJoin node=01G29V9ZA7S3WFDA0J2NW0DCE0 addr=172.16.1.43:9094
level=debug ts=2022-05-05T10:13:03.338Z caller=delegate.go:235 component=cluster received=NotifyJoin node=01G29PYXQA6R032P714Z2ZD15T addr=172.16.1.33:9094
level=debug ts=2022-05-05T10:13:03.339Z caller=delegate.go:235 component=cluster received=NotifyJoin node=01G29Q05VFQJR5JG3MRYJQ9DCH addr=172.16.1.5:9094

  3. Find ip in cluster to know which pod it corresponds to:

    kubectl get pod -A -o wide | grep 172.16.1.7
kubectl get pod -A -o wide | grep 172.16.1.43
kubectl get pod -A -o wide | grep 172.16.1.33
kubectl get pod -A -o wide | grep 172.16.1.5

Solution#

If legitimate, delete each pod of the alertmanager cluster.

kubectl delete pod -n <monitoring_namespace> <pod_name>
kubectl delete pod -n <monitoring_namespace> <pod_name>
kubectl delete pod -n <monitoring_namespace> <pod_name>
kubectl delete pod -n <monitoring_namespace> <pod_name>