Aller au contenu

GrafanaDown#

Alert is triggered when grafana_http_request_duration_seconds_count metric is missing from Prometheus cloud-caascad.

This alert either indicates that :

  • the Grafana application does not return the grafana_http_request_duration_seconds_count metric
  • or Prometheus does not scrape Grafana's metrics

Troubleshooting Hints#

Identify which Grafana fails to send it's metrics in Prometheus and check it's status/logs.

kubectl get pods -n <monitoring_namespace> | grep grafana
kubectl logs <grafana_pod> -n <monitoring_namespace>

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring or monitoring-client
  • Notice the namespace label in the alert
  • monitoring or monitoring-app.

To check if grafana metrics are available:

kubectl get svc -n <monitoring_namespace> | grep grafana   # service should be present
kubectl port-forward svc/<grafana_service> -n <monitoring_namespace> 8080:80 &
curl -sL http://localhost:8080/metrics
curl -sL http://localhost:8080/metrics | grep grafana_http_request_duration_seconds_count

Note

For Angel platform, there is no Grafana in cloud-client zone (monitoring-client namespace).

A silence is set in place in Karma for this Grafana.

To check Grafana target status in Prometheus:

kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &

Connect to http://localhost:12345/targets.

There are two use cases:

  1. The target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

  2. The target is absent, in this case you need to check why the target is not discovered by Prometheus.

    One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present : 

    kubectl get servicemonitor -n <monitoring_namespace> | grep grafana