Aller au contenu

ThanosRuleSenderIsFailingAlerts#

This alert is trigerred when alerts triggered by Thanos Ruler are not being sent to Alertmanager.

This may indicate connection, incompatibility or misconfiguration problems.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

Check Thanos Ruler configuration#

kubectl exec -it -n ${NAMESPACE} thanos-ruler-obs-corp-stg-thanos-ruler-0 -- cat /etc/thanos/config/alertmanager-config/alertmanager-configs.yaml

kubectl exec -it -n ${NAMESPACE} thanos-ruler-obs-corp-prd-thanos-ruler-0 -- cat /etc/thanos/config/alertmanager-config/alertmanager-configs.yaml

Expected result:

alertmanagers:
  - api_version: "v2"
    static_configs: ["alertmanager-0.alertmanager-headless:9093", "alertmanager-1.alertmanager-headless:9093"]
    scheme:  "http"
    timeout: "10s"

Thanos Ruler errors#

Thanos ruler is connected to two Alertmanager instances.

Check Thanos Ruler logs to see the type of error and identify which alertmanager is the problem on:

kubectl get pods -n ${NAMESPACE}
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=thanos-ruler

Alertmanager errors#

Check Alertmanager logs and see errors:

kubectl logs -n ${NAMESPACE} <alertmanager-0|alertmanager-1>