Aller au contenu

ThanosRuleRuleEvaluationLatencyHigh#

Thanos Ruler has high rule evaluation latency.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

  • Check status/logs of Thanos Ruler to identify which rules have a high evaluation latency (if this information is logged). Or to identify the reason why rules are evaluating slowly.

    kubectl get pods -n ${NAMESPACE}
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=thanos-ruler

  • Make a port forward for Thanos Ruler that generated the alarm, and visualise :10902/rules to identify the PrometheusRule have a high latency;

    kubectl port-forward -n ${NAMESPACE} svc/thanos-ruler 10902:10902 &

Solutions#

  • If a rule has a too complex expression, it may be simplified or some part of the expression may be evaluated as recording rules.