Aller au contenu

BlackboxExporterDown#

Alert is triggered when Blackbox-Exporter metrics (blackbox_exporter_build_info) are missing from destination Prometheus.

This alert can indicate one of the following :

  • Blackbox-Exporter application does not expose any metrics
  • Prometheus fails at scrapping the metrics exposed by Blackbox-Exporter.

Troubleshooting Hints#

Availability of Blackbox-Exporter metrics#

Check if Blackbox-Exporter metrics are available : 

kubectl get svc -n <blackbox_namespace> | grep blackbox-exporter # service should be present
kubectl port-forward svc/<svc_blackbox> -n <blackbox_namespace> 9115:9115 &
curl -sL http://localhost:9115/metrics
curl -sL http://localhost:9115/metrics | grep blackbox_exporter_build_info

Where <blackbox_namespace> has the value of the label namespace indicated in the alert.

  • If the metrics are all missing, check why with Blackbox-Exporter logs : 
    kubectl describe pod -l app.kubernetes.io/instance=blackbox-exporter -n  <blackbox_namespace> 
kubectl logs -l app.kubernetes.io/instance=blackbox-exporter -n <blackbox_namespace>
  • If only blackbox_exporter_build_info is missing:
    • also check the logs (see above)
    • check if Blackbox-Exporter version changed recently: the metric name may have changed or the metric may have been removed.

Blackbox-Exporter metrics not scraped by Prometheus#

Check Prometheus troubleshooting.

Known cases#

Absent target#

In this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor: 

kubectl get servicemonitor -n <blackbox_namespace> # servicemonitor should be present

If the servicemonitor is present, the labels could be incorrect: 

kubectl get servicemonitor <blackbox_servicemonitor> -n <blackbox_namespace> -o yaml

Tip

Where the labels need to have one of the following value:

caascad.com/prometheus-monitor: caascad

cloudservicesfactory/managed-by: corp-central

Failed target#

The target is present, but in a failed state. In this case check the logs of Prometheus to find the reason for which the target is failing. 

kubectl logs <prometheus_pod> -n <prometheus_ns>