LokiErrorSendingAlertsToSomeAlertmanagers#
Loki has more than 1% errors on sending alerts to a specific Alertmanager.
Context#
Loki sends alerts to all members of an Alertmanager cluster.
If Loki has more than 1% sending errors on two instances of Alertmanager, two alerts are raised.
The alertmanager label makes it possible to differentiate them.
If Loki has more than 1% sending errors on one instance of Alertmanager, one alert is raised.
Note
It's normal to see fake in the labels or description of the alert: this corresponds to the default name of the tenant in Loki.
Troubleshooting Hints#
Check Loki logs#
Check Loki logs to see the type of error :
kubectl logs statefulset/loki-ruler -n <logging_namespace> | grep alertmanager # Notice the `namespace` label in the alert
Check Alertmanager logs#
Tip
Below, <monitoring_namespace> can be:
monitoring-stack-client-<ngot_contract>
monitoring-stack-<ngot_contract>
Note
The ngot_contract label can be obtained from the alert.
For each Alertmanager identified at the previous step, check the logs :
kubectl logs statefulset/alertmanager -n <monitoring_namespace> -c alertmanager