Aller au contenu

ThanosStoreGatewayGrpcErrorRate#

This alert is triggered when Thanos Store Gateway is failing to handle more than 5% of requests.

Troubleshooting Hints#

kubectl get pods -n  <monitoring_namespace> | grep store-gateway  # check pod is up and running
kubectl logs <thanos_store_gateway> -n <monitoring_namespace>
kubectl get svc -n  <monitoring_namespace> | grep store-gateway   # service should be present

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

The logs may help to find out what component fails to request.

It is also possible to get the logs of the components that fail requests to the Store Gateway:

  • Caascad: Check the Thanos Querier and the Prometheus sidecar
  • NGOT in client namespace: Check the Thanos Querier and Prometheus sidecar in the namespaces of the client.
  • NGOT "central" (mon3, mon4): Check all the Thanos queriers (Proxy and Ruler) and Thanos Receive in the same namespace as the Store Gateway.