Aller au contenu

KthxbyeNotAbleToAcknowledgeAlertsSilences#

This alert appears when kthxbye is not able to acknowledge alerts silences. This can be due to different reasons:

  • alertmanager misconfiguration
  • alertmanager down
  • etc

Alertmanager's zone is indicated in the values of cc_prom_source of the alarm.

Troubleshooting Hints#

Check logs alertmanager pod for more details.

kubeclt logs <pod_alertmanager> -n <monitoring_namespace>
kubectl get svc -n <monitoring_namespace> | grep alertmanager   # service should be present
kubectl port-forward svc/<svc_alertmanager> -n <monitoring_namespace> 8080:8080 &
curl -sL http://localhost:8080/metrics

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

Where <svc_alertmanager> can be:

  • Zone Cloud: caascad-alertmanager, client-alertmanager or app-alertmanager
  • Zone Infra: caascad-alertmanager or consumption-alertmanager

Where <monitoring_namespace> can be:

  • Notice the namespace label in the alert

Where <svc_alertmanager> can be:

  • Zone NGOT: alertmanager