Aller au contenu

PrometheusRemoteWriteBehind#

When Prometheus (source) sends data to Prometheus (destination) with Remote Write, if the destination has a problem, the source will retry to send the data.

This alert show that those data are accumulating on Prometheus (source).

Warning

Prometheus can accumulate metrics for 1 hour. Any data older than 1 hour will be definitively lost if they could not be sent.

Troubleshooting Hints#

  • Check the logs of both Prometheus
    • Prometheus (destination) is probably the one not working well
    • Prometheus (source) can give clues why it cannot send the data
  • Check the URL of the Remote Write configuration in Prometheus configuration (in the url label of the alert; it can also be found in /etc/prometheus/config_out/prometheus.env.yaml in the prometheus pod/container)
  • Test that URL (without a Bearer token, a 401 error is expected)
  • Check that Rancher is working
  • Check the load of both Prometheus (CPU/Mem...)
  • The Grafana dashboard Prometheus / Remote Write may help to find any issue.

Tip

In most cases, there is no problem on Prometheus (source). Prefer investigating on Prometheus (destination) first.

Prometheus Remote Write known cases#

Some solutions are described here.