PrometheusAllRemoteStorageFailures

Some samples (aka metrics) failed to be sent over Remote Write.

Because we are loosing data (the failed samples) this alert is critical. Its criticity depends on how many data we are loosing.

This alert has few differences with PrometheusRemoteStorageFailures:

• PrometheusAllRemoteStorageFailures fires when we are loosing data when sending on All Remote-Write endpoints. For this reason, its severity is critical. Moreover, the labels remote_name and url are missing because all of them are impacted. This alert is pending for a short time before firing (for=5m).
• PrometheusRemoteStorageFailures fires for each Remote-Write endpoint where we are loosing data. Because other Remote-Write endpoints may have no problem, the severity is only warning. Most of the time, the problem repairs itself so the alert is pending for a long time before firing (for=30m).

Troubleshooting and solutions are the same as PrometheusRemoteStorageFailures.