Aller au contenu

ThanosQueryGrpcClientErrorRate#

This alert is triggered when Thanos Query if failing to send more than 5% of requests towards Thanos Store Gateway, Thanos Sidecar or Thanos Querier (proxy).

Troubleshooting Hints#

kubectl get pods -n  <monitoring_namespace> | grep querier  # check pod is up and running
kubectl logs <thanos_querier> -n <monitoring_namespace>
kubectl get pods -n  <thanos_querier> -n <monitoring_namespace> -o yaml | grep bucket  # check bucket name is correct
kubectl get svc -n  <monitoring_namespace> | grep querier

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

Tip

On NGOT, remember that there are 4 possibles queriers :

  • Querier client: namespace of the client. This is the querier connected to the Grafana of the client.
  • Querier Grafana: namespace grafana-obs-corp-prd/grafana-obs-corp-stg. This is the querier connected to the Grafana for NGOT teams. On the other side, it is connected to Thanos-Querier (proxy).
  • Querier Proxy: namespace monitoring-stack-corp-obs-corp-prd/monitoring-stack-corp-obs-corp-stg. This is the proxy (or dispatcher) to all the Thanos components
  • Querier Ruler: namespace monitoring-stack-corp-obs-corp-prd/monitoring-stack-corp-obs-corp-stg. This is a dedicated querier for Thanos Ruler.