Aller au contenu

ThanosQueryHttpRequestQueryErrorRateHigh#

This alert is triggered when Thanos Query has more than 5% of http errors 5xx.

These http errors indicate that Thanos Query fails to read data from Sidecar, Store-Gateway (all interrogate Object Store) or Thanos Querier (proxy).

Troubleshooting Hints#

Caascad, NGOT client and NGOT "central" (mon3 and mon4)#

kubectl get pods -n  <monitoring_namespace> | grep thanos  # check querier/store-gateway pods are up and running
kubectl logs <thanos_pods> -n <monitoring_namespace>

kubectl get pods -n  <monitoring_namespace> | grep prometheus  # check 2 pods are up and running
kubectl logs <prometheus_pod> -c thanos-sidecar -n <monitoring_namespace> # check logs of sidecar container

kubectl get pods -n  <thanos_pods> -n <monitoring_namespace> -o yaml | grep bucket  # check bucket name is correct for querier/store-gateway
kubectl get pods <prometheus_pod> -n <monitoring_namespace> -o yaml | grep bucket # check bucket name is correct for sidecar container
kubectl get svc -n  <monitoring_namespace> | grep thanos  # service querier, sidecar, store-gateway should be present

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

To check if bucket name is correct, follow bucket nomenclature documentation.

NGOT "central" (mon3, mon4)#

FIXME PF-3028: remove "mon3, mon4" in the title.

First identify which querier has a high query error rate:

  • Querier for Grafana, namespace grafana-obs-corp-prd/grafana-obs-corp-stg.
  • Querier proxy, namespace monitoring-stack-corp-obs-corp-prd/monitoring-stack-corp-obs-corp-stg, with "proxy" in its name.
  • Querier dedicated to Thanos-Ruler, namespace monitoring-stack-corp-obs-corp-prd/monitoring-stack-corp-obs-corp-stg without "proxy" in its name.

Check the logs:

kubectl -n <namespace> get pod |grep quer
kubectl -n <namespace> logs <thanos_pods>

The logs may help to find out what component fails to request.

It is also possible to get the logs of the components that fails requests to the Querier:

  • Querier for Grafana: check Grafana.
  • Querier proxy: check the querier for Grafana.
  • Querier dedicated to Thanos-Ruler: check Thanos-Ruler.