Aller au contenu

PrometheusRemoteStorageFailures#

Some samples (aka metrics) failed to be sent over Remote Write.

Warning

The other alert PrometheusAllRemoteStorageFailures may be firing too. In this case,

  1. we are loosing metrics!
  2. PrometheusRemoteStorageFailures is also firing (or pending) for all Remote-Write endpoints.

If PrometheusAllRemoteStorageFailures is not firing at the same time, we are loosing metrics, but not on all Remote-Write endpoints. High Availability saves us.

Troubleshooting Hints#

In most cases, this alert comes with errors HTTP 4xx : the source Prometheus failed to connect to the destination Prometheus.

First checks (possible known cases)#

Check this first:

  1. If one of PrometheusContainerReachedMaxAllowedOpenSockets or PrometheusContainerTooManyOpenSockets also occurs, first resolve it then check if PrometheusRemoteStorageFailures still happens.

  2. If Promitor is also hosted on a cluster where the alert PrometheusRemoteStorageFailures happens, and if PrometheusRemoteStorageFailures appears only on this cluster, we noticed that restarting the 2 pods of Promitor, then restarting the 2 pods of Prometheus-Cluster would fix the problem.

    Note

    14/10/2024: This is a workaround. We do not know the root case yet. Any investigation on why this happens is welcome.

    However this case is known and the CAASINC may not be kept open because the root cause is not known. This is an exception.

    Instead, you will explain that Promitor was also running on that cluster and the alert was limited to that cluster.

  3. If all the solutions (including restarting all Prometheus-cluster) have already be applied (see below), and PrometheusRemoteStorageFailures are still remaining on many clusters, and it seems to be linked to Promitor*, then restart the 2 pods of Promitor. Then restart again the Prometheus-cluster.

    Note

    14/10/2024: This is a workaround. We do not know the root case yet. Any investigation on why this happens is welcome.

    However this case is known and the CAASINC may not be kept open because the root cause is not known. This is an exception.

    Instead, you will explain that Promitor was also running on a list of cluster (specify the list) and that you restarted all Prometheus-cluster after restarting Promitor.

Generic troubleshooting hints#

  • Check the logs on both Prometheus
  • Check the URL of the Remote Write configuration in Prometheus configuration (in the url label of the alert; it can also be found in /etc/prometheus/config_out/prometheus.env.yaml in the prometheus pod/container)
  • Test that URL (without a Bearer token, a 401 error is expected)
  • Check that Rancher is working
  • Check that Prometheus has the correct token to connect to the URL
  • Check the load of both Prometheus (CPU/Mem...)
  • The Grafana dashboard Prometheus / Remote Write may help to find any issue.

Known cases#

Duplicate samples#

Example of logs in the Prometheus destination :

ts=2022-08-18T12:07:54.802Z caller=write_handler.go:102 level=error component=web msg="Out of order sample from remote write" err="duplicate sample for timestamp" series="{__name__=\"job:loki_request_duration_seconds_bucket:sum_rate\", cc_client=\"ocb-test05\", cc_prom=\"cloud-caascad\", cc_prom_source=\"ocb-test05\", cluster=\"ocb-test05\", job=\"distributor\", le=\"0.01\", prometheus=\"monitoring/caascad-prometheus\", prometheus_replica=\"prometheus-caascad-prometheus-1\"}" timestamp=1660824473868

This means that the destination Prometheus receives the same sample at least twice.

To verify the sample sent, you can request the sample(s) (by removing labels like prometheus_replica and prometheus) in the source Prometheus (with Explore).

For our example :

{__name__="job:loki_request_duration_seconds_bucket:sum_rate", job="distributor", le="0.01"}

Result is :

job:loki_request_duration_seconds_bucket:sum_rate{cc_client="ocb-test05", cc_prom="cloud-caascad", cc_prom_source="cloud-caascad", job="distributor", le="0.01", prometheus="monitoring/caascad-prometheus"}
job:loki_request_duration_seconds_bucket:sum_rate{cc_client="ocb-test05", cc_prom="cloud-caascad", cc_prom_source="cloud-client", job="distributor", le="0.01", prometheus="monitoring/caascad-prometheus"}

We see here that there are two samples and the only difference is the label cc_prom_source.

For this example, the problem is that when writing with Remote Write, the cc_prom_source is rewritten (with the value ocb-test05). Which implies that two samples are sent but on reception it gives the same sample.

Prometheus Remote Write known cases#

Some solutions are described here.