Aller au contenu

BlackboxMetricsMissing#

The Blackbox exporter conducts tests, known as probes, on endpoints and provides metrics based on the outcomes. This alert is based on those metrics.

This alert is triggered when the metrics for a target of Blackbox-Exporter are missing for more than 5 minutes.

To determine where is the problem, refer to the target label in the alert.

The target label provides you two informations:

  • the failing endpoint (example: thanos-querier-proxy-mon3.obs-corp-stg.csfpriv.com, in this case the issue is Thanos-querier on mon3 zone);
  • the name of the failed module (example: grpc_thanos_querier_proxy_mon3_tls,in this case the issue is GRPC communication with mon3).

About Blackbox-Exporter#

Blackbox-Exporter is used to monitor applications and different monitoring components (commonly named monitoring of monitoring).

Blackbox-Exporter executes probes against endpoints and returns metrics based on the results of those probes.

For more information about how Blackbox-Exporter is deployed here.

Troubleshooting hints#

Do a port-forward on Blackbox-Exporter :

kubectl -n <blackbox_exporter_namespace> port-forward \
    svc/blackbox-exporter \
    9115:9115

kubectl -n <blackbox_exporter_namespace> port-forward \
    svc/blackbox-exporter-<contract_name> \
    9115:9115

Then browse http://localhost:9115 and find your target and its logs.

Depending on the result, you can have 2 options:

Important

In most cases, this issue is not related to Blackbox-Exporter itself. Here are most common use-cases:

  • the ServiceMonitor or the Probe for the target is missing or is broken
  • the ServiceMonitor or the Probe has not been detected by Prometheus
  • Prometheus fails to connect to Blackbox-Exporter

Prometheus#

  • Caascad: this is one of the Prometheus of the Cloud zone. Check the labels of the alert to find out which Prometheus to connect to.
  • NGOT: this is one of the central monitoring stacks (kub-34 or kub-53). Check the labels of the alert to find out which Prometheus to connect to.

Port-forward to Prometheus and check the targets on http://localhost:9090/targets.

This is better explained here

You may also check the logs of Prometheus.

Servicemonitor / Probe#

  • Find out if it is a ServiceMonitor or a Probe

    kubectl get servicemonitor -A | grep blackbox # Because the names of the servicemonitors usually contain blackbox
kubectl get probe -A

    You should find the expected servicemonitor or probe.

    If you find no ServiceMonitor or Probe for the expected target, it may be not deployed. Check why in Gitlab and Envs-ng/Trackbone.

  • In case of a ServiceMonitor: check if it is healthy and contains the expected labels:

    kubectl get servicemonitor -l caascad.com/prometheus-monitor=caascad -n $NAMESPACE

    kubectl get servicemonitor -l cloudservicesfactory/managed-by=corp-cluster -n $NAMESPACE

  • In case of a Probe: check that probe is present on NGOT stacks kub-34 and kub-53:

    kubectl get probe -n monitoring-stack-corp-obs-corp-prd | grep XXX # for obs-pf use pf

    Note

    Probes are defined only on NGOT and only on central stacks (kub-34/kub-53).

Blackbox-Exporter#

Check if Blackbox-Exporter works correctly and probes the good targets. More information here.

Grafana#

In Grafana, check that Blackbox-Exporter metrics probe_success{} are available.

This should not help to debug but the metric should appear when the issue is fixed.

Tip

Don't forget to select the correct Datasource in Grafana.

Solutions#

Known Case: Remote-Write#

When the certificate or credentials secrets have been recently added, modified or deleted, some Probes or Blackbox-exporters modules may be missing.

To redeploy, you can refer to this documentation.

Known Case: Missing informations in Blackbox-exporter cache#

Blackbox-exporter configuration may have missing authentication information to connect to client Prometheus (via an ingress).

To confirm this is the case, connect to the zone on which Blackbox-exporter has a failed probe (idenfied by cluster label in the alert) and check if it's configuration contains authentication information to connect to the customer Prometheus (kub-34/kub-53):

kubectl -n monitoring-stack-corp-obs-corp-prd get secret blackbox-exporter-obs-corp-prd \
    -o json |jq -r '.data["blackbox.yaml"]' | base64 -d|grep $CLIENT

Where $CLIENT is identified with the label obs_client in the alert (example: tryit).

Warning

If the secret contains the right information, you are not in this known case.

If the secret does not contain the right informations, then you will need to update Blackbox-exporter configuration. There are two ways to do this:

Automatically#

At this date (25/09/2024), there is on open incident CAASINC-1866 regarding issues to update the cache of Blackbox-exporter. This method may fail.

The following commands will update the Blackbox-exporter cache and then the configuration of all Blackbox-exporter: 

trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-1 -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-2 --add-services

Manually#

  1. Retrieve client Prometheus ingress username/password from Vault (/secret/zones/fe/svc-monitoring-stack-client-$CLIENT)/prometheus-ingress/prometheus-ingress-XXXXXX-XXXXX).

  2. Add the client Prometheus ingress username/password information to Blackbox-exporter cache in Vault (secrets/secret/show/applications/blackbox-exporter/credentials-cache, prometheus_client field).

  3. Update Blackbox-exporter configuration:

    trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-1 --add-services
trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-2 --add-services

  4. Check that Blackbox-exporter configuration now contains the client authentication information (on kub-34 or kub-53):

    kubectl -n monitoring-stack-corp-obs-corp-prd get secret blackbox-exporter-obs-corp-prd \
-o json |jq -r '.data["blackbox.yaml"]' | base64 -d|grep $CLIENT

The alerts should disappear.