LokiErrorSendingAlertsToAnyAlertmanagers#
Loki has more than 3% errors on sending alerts to each member of Alertmanager cluster.
Context#
Loki sends alerts to all members of an Alertmanager cluster.
The alert is therefore triggered when there are more 3% sending errors on Alertmanager regardless of the Alertmanager instance.
Warning
If this alert is triggered, alerts were not sent.
If label ngot_contract has other value than obs-corp-prd, it means there is a customer impact. In this case, the client needs to be informed about the alerts loss.
Note
It's normal to see fake in the labels or description of the alert: this corresponds to the default name of the tenant in Loki.
Troubleshooting Hints#
Check Loki logs#
Check Loki logs to see the type of error.
kubectl logs statefulset/loki-ruler -n <logging_namespace> | grep alertmanager # Notice the `namespace` label in the alert
Check Alertmanager logs#
Tip
Below, <monitoring_namespace> can be:
monitoring-stack-client-<ngot_contract>
monitoring-stack-<ngot_contract>
Note
The ngot_contract label can be obtained from the alert.
For each Alertmanager where there are errors check the Alertmanager logs:
kubectl logs statefulset/alertmanager -n <monitoring_namespace> -c alertmanager