PrometheusRemoteStorageUnexpectedlyDropped

Some samples (aka metrics) are dropped without having been sent over Remote Write.

Warning

The other alert PrometheusAllRemoteStorageUnexpectedlyDropped may be firing too. In this case,

1. we are loosing metrics!
2. PrometheusRemoteStorageUnexpectedlyDropped is also firing (or pending) for all Remote-Write endpoints.

If PrometheusAllRemoteStorageUnexpectedlyDropped is not firing at the same time, we are loosing metrics, but not on all Remote-Write endpoints. High Availability saves us.

Troubleshooting Hints

First check the reason label.

Note

The reason=dropped_series label pair exists but is normal. This should not be seen in this alert.

Known case: reason=too_old

This case is due to the parameter sample_age_limit that forces dropping samples that are too old.

The parameter sample_age_limit is a protection against metrics remaining too long in Prometheus.

The problem there is not why we are loosing metrics but why aren't they sent in the sample_age_limit delay ?

On 04/11/2024 we don't know exactly why this is happening. Is it a network limitation or too many metrics to send in a short time ?

The solution there is to restart all Prometheus-Clusters to empty their queue. This documentation explains how to do it.