Aller au contenu

ThanosQuerierProxyMetricsMissing#

Alert is triggered when thanos_build_info{job='thanos-querier', service='thanos-querier-proxy-query'} metric is missing from Prometheus-cluster.

This alert either indicates that:

  • the querier proxy does not return the thanos_build_info{job='thanos-querier', service='thanos-querier-proxy-query'} metric
  • or Prometheus-cluster does not scrape querier proxy's metrics.

Troubleshooting Hints#

Note

  • NAMESPACE=monitoring-stack-corp-obs-corp-stg
  • NAMESPACE=monitoring-stack-corp-obs-corp-prd

Case: the querier proxy does not return metric#

To check if querier proxy metrics are available:

kubectl get svc -n ${NAMESPACE} thanos-querier-proxy-query # service should be present
kubectl port-forward svc/thanos-querier-proxy-query -n ${NAMESPACE} 9090:9090 &
curl -sL http://localhost:9090/metrics
curl -sL http://localhost:9090/metrics | grep thanos_build_info

If metrics are not available, try to find the reason by checking status/logs of querier proxy:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/instance=thanos-monitoring-querier-proxy -l app.kubernetes.io/component=query
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/instance=thanos-monitoring-querier-proxy -l app.kubernetes.io/component=query

Case: Prometheus-cluster does not scrape querier proxy's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE} thanos-querier-proxy-query

  • check the target, by doing a port-forward on the Prometheus-cluster

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &

    Connect to http://localhost:12345/targets, search thanos-querier-proxy-query target, and verify error message: targets

    Warning

    Be careful of clear selection before filtering on thanos-querier-proxy-query.