LokiRemoteWriteDesiredShards

Before Loki Ruler sends data with Remote Write protocol, it accumulates in a place named shards. Loki Ruler will fill them simultaneously. Under unexpected load, Loki can create additional shards.

In the Loki configuration, there is a min and a max value for the number of shards. When this alert fires, it means that Loki needs more shards than the max value.

Note

If the problem is temporary, with the retry mechanisms in the Remote Write protocol, no metrics are lost.

On the other hand, if the retries persist for more than one hour, it can lead to loss of metrics.

But in this case, another alert, with critical severity, will also indicate it.

Note

It's normal to see fake in the labels or description of the alert: this corresponds to the default name of the tenant in Loki.

Troubleshooting Hints

• Check the logs of both

  • Prometheus

    Tip

    Below, <monitoring_namespace> can be:

    ClientCorp

    monitoring-stack-client-<ngot_contract>

    monitoring-stack-<ngot_contract>

  Note

  The `ngot_contract` label can be obtained from the alert.
  
  For each Prometheus where there are errors check the Prometheus logs:
  

  console kubectl logs statefulset/prometheus-<ngot_contract>-prometheus -n <monitoring_namespace> -c prometheus

  • Loki Ruler

    kubectl logs statefulset/loki-ruler -n <logging_namespace>  # Notice the `namespace` label in the alert
    

• Check the load of Prometheus (CPU/Mem...)