Aller au contenu

BlackboxHTTPDown#

The Blackbox-exporter conducts tests, known as probes, on endpoints and provides metrics based on the outcomes. This alert is based on those metrics.

This alerts happens when a target of Blackbox-Exporter is down.

Important

When this alert happens, THIS IS NOT AN ISSUE WITH BLACKBOX-EXPORTER. This alert cannot appear if Blackbox-Exporter is broken.

This alert is triggered when the metrics for a target of Blackbox-Exporter are missing for more than 5 minutes.

To determine where is the problem, refer to the target label in the alert.

The target label provides you two informations:

  • the failing endpoint (example: alertmanager-mon3.obs-corp-stg.csfpriv.com, in this case the issue is Alertmanager on mon3 zone);
  • the name of the failed module (example: http_2xx_alertmanager_mon3_basicauth, in this case the issue is HTTP return code from mon3).

For more information about how Blackbox-Exporter is deployed here.

This alerts happens when a target of Blackbox-Exporter is down.

Important

When this alert happens, THIS IS NOT AN ISSUE WITH BLACKBOX-EXPORTER. This alert cannot appear if Blackbox-Exporter is broken.

Troubleshooting hints#

Step 1: Identify the target#

The target is defined with the label target of the alert.

You may quick-check directly the target itself at this step if you know it and bypass the next steps. If you cannot find the issue at this step, continue to step 2.

Step 2: Find Blackbox-Exporter#

NGOT#

On NGOT, Blackbox-Exporter only exists on central monitoring stacks. You can find it with these labels:

  • cluster
  • namespace

Check its presence:

kswitch <cluster>
kubectl -n <namespace> get pod -l app.kubernetes.io/instance=blackbox-exporter

Caascad#

On Caascad, Blackbox-Exporter is deployed in the namespace monitoring. The obs_client tells which cluster to connect to.

Check its presence:

kswitch <obs_client>
kubectl -n monitoring get pod -l app.kubernetes.io/instance=blackbox-exporter

Step 3: Check the target in Blackbox-Exporter UI#

Follow this documentation to connect to the Blackbox-Exporter UI. Blackbox-Exporter to connect to was identified at step 2 above.

Find the target (from the target label of the alert) in the UI and inspect the logs.

As described in that documentation, you may want to force the debug mode with an URL ending with &debug=true. Example:

http://localhost:9115/probe?target=<your target here>&module=http_2xx&debug=true
Check the logs of the failed probe to get more information.

If the target contains /federate and the logs contain status_code=401, check authentication known issue.

Alternative with debug mode#

If you need to check the logs of Blackbox-Exporter itself, you will notice they are not very verbose. You may want to enable debug mode globally.

This method is not preferred because it needs global operations on Blackbox-Exporter and will generate lots of logs.

Follow this documentation.

Warning

Don't forget to rollback debug logLevel when the troubleshooting is finished.

Alternative With Loki in Grafana: check the logs#

Connect to the Grafana identified by cluster label in the alert. This is the environement that hosts the Blackbox-Exporter with failed probe.

Then go to Explore, and select datasource Loki -> logs.

Query : {app_kubernetes_io_instance="blackbox-exporter"}|="<target>"

  • If you don't see your target, that probably means you don't have set the correctly Target.
  • If you see your target and you don't see a status_code=200 that mean you have probably a error in your configuration
  • If you don't get a HTTP response your probably have a DNS problem.
  • If you can find a "Probe failed", PROBE with which you are trying to dialog is not available. Make sure this service are in state running

Known cases#

Authentication with Federated Prometheus (HTTP code 401)#

This issue may arise when the target label of the alert contains /federate.

In Prometheus federation, there is a source Prometheus (which exposes its API) and a destination Prometheus (which connects to the source Prometheus to collect a predefined list of metrics). There is a schema in this doc.

The Blackbox-exporter checks the HTTP return code from the source Prometheus. To do this, the Blackbox-exporter must authenticate to the source Prometheus (via ingress). If the authentication information has changed on the source Prometheus side, and the Blackbox-exporter is not updated with the new credentials, it will fail with a 401 error.

To diagnose this issue:

  1. Authenticate to Vault: 

    LINE=prd # or stg
export VAULT_ADDR=https://vault.infra-$LINE.caascad.com/
vault token lookup > /dev/null 2>&1 || vault login -method oidc

  2. Check the Blackbox-exporter authentication cache in Vault: 

    vault read secret/applications/blackbox-exporter/credentials-cache -format=json | jq -r .data.federate_prometheus_client

  3. Verify the federated client Prometheus authentication password in Vault: 

    vault read secret/zones/fe/svc-monitoring-stack-client-${CLIENT}/prometheus-federate-auth -format=json | jq -r .data.password

  4. Compare the passwords retrieved in steps 2 and 3. If they differ, the issue is likely due to outdated credentials in the Blackbox-exporter cache.

Information

The Prometheus password can change, for example, if someone runs the following command: 

trackbone apply -z svc-monitoring-stack-client-$CLIENT -c kube-prometheus-stack -t bootstrap=true

To resolve the issue, update the Blackbox-exporter cache with the new client Prometheus password. There are 2 ways to do this:

Automatically

Warning

At this date (27/09/2024) the refresh_cache option does not work for Blackbox-exporter.

To do this: 

trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-1 -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-2 --add-services

Manually

Follow these steps:

  1. Retrieve the federated Prometheus password: 

    vault read secret/zones/fe/svc-monitoring-stack-client-${CLIENT}/prometheus-federate-auth -format=json | jq -r .data.password

  2. Update the password in the Blackbox-exporter cache:

    • Path: applications/blackbox-exporter/credentials-cache → federate_prometheus_client section → password field.

    Note

    If the password contains special characters (like :, $, etc.), enclose it in single quotes ('password').

  3. Redeploy the Blackbox-exporter configuration in central zones: 

    trackbone apply -z svc-monitoring-stack-corp-prd-1 -c blackbox-exporter-core
trackbone apply -z svc-monitoring-stack-corp-prd-2 -c blackbox-exporter-core

After completing these steps, the alerts should disappear.

While resolving the Blackbox-exporter authentication issue, you must also ensure that the metrics are properly copied from the source Prometheus to the destination Prometheus.

To do this:

  1. Identify the destination client Prometheus: 

    cd zones/ngot-zones
rg -A10 federate_zones:

  2. Redeploy the authentication information on the destination client Prometheus: 

    trackbone apply -z svc-monitoring-stack-client-${DEST_CLIENT} -c client-federate-metrics-servicemonitor

  3. Verify the metrics on the Grafana Explore tab of the destination client environment. Ensure that metrics with the label obs_client=sourcePrometheus are present (example: obs_client=test05).

IPv6#

Natively, the Blackbox-Exporter will prefer an IPv6 address if one is returned by DNS.

You can adjust this behaviour by adding preferred_ip_protocol: "ip4" to the module configuration if you are not using IPv6.