Aller au contenu

PrometheusErrorSendingAlertsToSomeAlertmanagers#

Prometheus has more than 1% errors on sending alerts to a specific Alertmanager.

Context#

Warning

Some customer have an external Alertmanager, this Alertmanager is not in our scope but we inform customer with their identified support contact.

To find information on external Alertmanager check the additionalAlertManagerConfigs in client_specific_configurations.

In Caascad, Alertmanager is deployed as a cluster.

Prometheus sends alerts to all members of an Alertmanager cluster.

If Prometheus has more than 1% sending errors on two instances of Alertmanager, two alerts are raised. The alertmanager label makes it possible to differentiate them.

If Prometheus has more than 1% sending errors on one instance of Alertmanager, one alert is raised.

Troubleshooting Hints#

Tip

Below, <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

  • Notice the namespace label in the alert

Identify if your Alertmanager is internal or external#

You can use alertmanager label.

Check Prometheus logs#

Check Prometheus logs to see the type of error. 

kubectl logs <prometheus> -n <monitoring_namespace> -c prometheus | grep alertmanager

Internal Alertmanager: Check Alertmanager logs#

For each Alertmanager where there are errors check the Alertmanager logs: 

kubectl logs <alertmanager_pod> -n <monitoring_namespace> -c alertmanager