Aller au contenu

BlackboxHttpSlowProbe#

This warning mean the HTTP request took more than 3 average sec. That can from multiple root cause.

Troubleshooting hints#

Important

In most cases, this issue is not related to Blackbox-Exporter itself. Here are most common use-cases:

  • the target application is running slowly
  • the network path between Blackbox-Exporter and the application is degraded
  • the DNS answers slowly (to resolve the target IP)

Caascad and NGOT#

Blackbox-Exporter UI#

Grafana#

In Grafana, check some metrics to find out when the problem started and, in some cases, the reason.

The target label below comes from the alert labels.

Here are the metrics to check:

  • probe_duration_seconds{target="xxx"}: this is the metric used for the alert.
    • it should show when the problem started, if it often happens or not...
    • it may show some differences between our Blackbox-Exporters. If some significative differences appear, the problem may be linked to one Blackbox-Exporter or to the stack and cluster underneath.
  • probe_http_duration_seconds{target="xxx"}: notice the phase label. If values are significantly higher for a specific phase, this may help to find the root cause.

Caascad#

  • Check the dashboard Blackbox-Exporter ( panel HTTP Duration, DNS Lookup and Probe Duration ) on Grafana.
  • If value are increase in HTTP Duration, that mean it's a HTTP problem.
  • If value are increase in DNS Lookup, that mean it's a DNS problem.
  • If value avec increase in Probe Duration, that mean it's a Probe problem.
  • That can response for next questions :
    • Is this a global linear increase over time. If the answer is yes, since when ?
    • Is it a spontaneous and fixed increase ?
  • By answering these questions, possible hints to explore for the root causes may be an overall increase traffic usage, a network problem, or an application problem.