Aller au contenu

Troubleshooting Blackbox-Exporter#

Interactive Blackbox-exporter#

This method usually helps to have more information easily about why a specific target of Blackbox-Exporter fails.

Do a port-forward on Blackbox-Exporter :

kubectl -n <blackbox_exporter_namespace> port-forward \
    svc/blackbox-exporter \
    9115:9115

kubectl -n <blackbox_exporter_namespace> port-forward \
    svc/blackbox-exporter-<contract_name> \
    9115:9115

Then browse http://localhost:9115 and find your target and its logs.

You can also test interactively your target with debug mode there. Add &debug=true at the end of the URL. Example : http://localhost:9115/probe?target=<your target here>&module=http_2xx&debug=true

Blackbox-Exporter Logs#

This method usually helps when either all Blackbox-Exporter are failing or when you search more detailed information why a Blackbox-Exporter target failed.

Logs can be found :

  • with kubectl logs
  • in Grafana. Select datasource Loki for the Blackbox-Exporter in cloud zone. Select datasource Loki-client for the Blackbox-Exporter in client zones.

If needed, Blackbox-Exporter can also be redeployed in debug mode. Edit envs-ng/blackbox-exporter-core.cue and add a section like this :

envs: [string]: configurations: [=~"^blackbox-exporter-core-"]: {
    helm: values: "prometheus-blackbox-exporter": {
        extraArgs: [
            "--log.level=debug",
        ]
    }
}

Then redeploy with Trackbone as usual.

cd contexts/<env>
trackbone apply -c blackbox-exporter-core -z $SOURCE

Note

On NGOT environments, $SOURCE can take one of the following values svc-monitoring-stack-corp-<production_line>-{1,2}.

Where production_line can be either prd or stg.

Warning

Debug mode is very verbose and this needs a lot of resources (network and storage). Do not forget to rollback after the debugging session.