Aller au contenu

ThanosRulerMetricsMissing#

Alert is triggered when thanos_build_info{job='thanos-ruler'} metric is missing from Prometheus-cluster.

This alert either indicates that:

  • the ruler does not return the thanos_build_info{job='thanos-ruler'} metric
  • or Prometheus-cluster does not scrape ruler's metrics.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

Case: the ruler does not return metric#

To check if ruler metrics are available:

kubectl get svc -n ${NAMESPACE} thanos-ruler # service should be present
kubectl port-forward svc/thanos-ruler -n ${NAMESPACE} 10902:10902 &
curl -sL http://localhost:10902/metrics
curl -sL http://localhost:10902/metrics | grep thanos_build_info

If metrics are not available, try to find the reason by checking status/logs of rulers:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=thanos-ruler
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=thanos-ruler

Case: Prometheus-cluster does not scrape ruler's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE} thanos-ruler

  • check the target, by doing a port-forward on the Prometheus-cluster

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &

    Connect to http://localhost:12345/targets, search thanos-ruler target, and verify error message: targets

    Warning

    Be careful of clear selection before filtering on thanos-ruler.