ThanosRuleConfigReloadFailure

This alert is raised when the Thanos Ruler is not able to reload its rules configuration for a certain period of time.

Troubleshooting hints

Note

stagingproduction

• NAMESPACE=monitoring-stack-corp-obs-corp-stg

• NAMESPACE=monitoring-stack-corp-obs-corp-prd

1. Verify the logs of the container config-reloader

kubectl -n ${NAMESPACE} logs -l app.kubernetes.io/name=thanos-ruler -c config-reloader

2. Verify the logs of the container thanos-ruler

   kubectl -n $NAMESPACE logs -l app.kubernetes.io/name=thanos-ruler -c thanos-ruler
   

   In logs, you can see errors lines like:

   component=rules \
   msg="loading groups failed" \
   err="/thanos/data/.tmp-rules/ABORT/etc/thanos/rules/thanos-ruler-obs-corp-stg-thanos-ruler-rulefiles-0/prometheusrules-svc-monitoring-stack-corp-stg-2-blackbox-exporter-d8043a2b-b2ad-47aa-8d7f-7c7f0a34e791.yaml: yaml: unmarshal errors:\n  ..."
   

3. Check if there are any errors in the PrometheusRules.

   kubectl -n prometheusrules get prometheusrules <prometheusrules_name> -o yaml
   

   where prometheusrules_name is retrieved from logs.

   With the example above:

   kubectl -n prometheusrules get prometheusrules svc-monitoring-stack-corp-stg-2-blackbox-exporter -o yaml