Aller au contenu

ThanosRuleHighRuleEvaluationWarnings#

Thanos Ruler has high number of evaluation warnings.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

  • Make a port forward for Thanos Ruler that generated the alarm, and visualise :10902/rules to identify the PrometheusRule that are not in OK state;

    kubectl port-forward -n ${NAMESPACE} svc/thanos-ruler 10902:10902 &

  • Check status/logs of Thanos Ruler

    kubectl get pods -n ${NAMESPACE}
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=thanos-ruler

  • Check the expression of the PrometheusRule by using Explore menu in Grafana staging or production.