Aller au contenu

PrometheusErrorSendingAlertsToAnyAlertmanager#

Prometheus has more than 3% errors on sending alerts on any Alertmanager.

Context#

Warning

Some customer have an external Alertmanager, this Alertmanager is not in our scope but we inform customer with their identified support contact.

To find information on external Alertmanagers check the additionalAlertManagerConfigs in client_specific_configurations.

In Caascad, Alertmanager is deployed as a cluster.

Prometheus sends alerts to all members of an Alertmanager cluster.

The alert is therefore triggered when there are more 3% sending errors on Alertmanager regardless of the Alertmanager instance.

Troubleshooting Hints#

Tip

Below, <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

  • Notice the namespace label in the alert

Check Prometheus logs#

Check Prometheus logs to see for which Alertmanager there are failures and for the type of error.

Identify if your Alertmanager is internal or external.

kubectl logs <prometheus> -n <monitoring_namespace> -c prometheus | grep alertmanager

Internal Alertmanager: Check Alertmanager logs#

For each Alertmanager where there are errors check the Alertmanager logs: 

kubectl logs <alertmanager_pod> -n <monitoring_namespace> -c alertmanager