Aller au contenu

GrafanaRequestsFailing#

Explanations#

Fires when more than 50% of Grafana API requests fail with HTTP code 500.

The alert fires for most queries (see the exception that are hard-coded in the alert expression).

Troubleshooting#

  • Note the API endpoint : it is the handler label in the alert.

  • Check Grafana logs :

    kubectl logs -n <namespace> -l app.kubernetes.io/name=grafana -c grafana
    Where namespace can be :

    • monitoring
    • monitoring-client
    • Notice the namespace label in the alert.
    • monitoring
    • monitoring-app

Solutions#

Handler : /api/datasources/:id/resources/*#

When this alert with this handler is firing, that means there are resquests error with a datasource. Check grafana logs to identify the impacted datasource.

Grafana logs e.g.:

logger=tsdb.prometheus t=2022-11-10T14:22:04.636341257Z level=error msg="Range query failed" query="sum (irate (nginx_ingress_controller_request_size_sum{...}) err="Post \"http://thanos-monitoring-app-querier.monitoring-app.svc.cluster.local:9090/api/v1/query_range\": context canceled"

We then find a problem with thanos-monitoring-app-querier so we can delete this pod.

Note

15/11/2022 : Please update this documentation when this alert is firing with a new handler.