Aller au contenu

PrometheusRemoteWriteDesiredShards#

Before Prometheus sends data with Remote Write protocol, it accumulates in a place named shards. Prometheus will fill them simultaneously. Under unexpected load, Prometheus can create additional shards.

In the Prometheus configuration, there is a min and a max value for the number of shards. When this alert fires, it means that Prometheus needs more shards than the max value.

Troubleshooting Hints#

  • Check the Prometheus logs
  • Check the Prometheus load (CPU/Mem...)
  • Check if there is any problem with the remote Prometheus
  • The Grafana dashboard Prometheus / Remote Write may help to find any issue
  • Advanced hints : check the queueConfig parameters. They should fit most needs but in some extreme configurations, some tuning may be needed.

Prometheus Remote Write known cases#

Some solutions are described here.