Aller au contenu

ThanosReceiverMetricsMissing#

Alert is triggered when thanos_build_info{job='thanos-receive-headless'} metric is missing from Prometheus-cluster.

This alert either indicates that:

  • the receiver does not return the thanos_build_info{job='thanos-receive-headless'} metric
  • or Prometheus-cluster does not scrape receiver's metrics.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

Case: the receiver does not return metric#

To check if receiver metrics are available:

kubectl get svc -n ${NAMESPACE} thanos-receive-headless # service should be present
kubectl port-forward svc/thanos-receive-headless -n ${NAMESPACE} 10902:10902 &
curl -sL http://localhost:10902/metrics
curl -sL http://localhost:10902/metrics | grep thanos_build_info

If metrics are not available, try to find the reason by checking status/logs of receivers:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/component=receive
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/component=receive

Case: Prometheus-cluster does not scrape receiver's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE} thanos-receive

    Note

    The service monitor thanos-receive scrapes the receiver and distributor metrics.

  • check the target, by doing a port-forward on the Prometheus-cluster

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &

    Connect to http://localhost:12345/targets, search thanos-receive target, and verify error message: targets

    Warning

    Be careful of clear selection before filtering on thanos-receive.