Aller au contenu

LokiIngesterActiveMissing#

A hash Ring need to contain three Loki Ingesters to work correctly.

This alert is trigered when the hash Ring contains less than three Loki Ingester in ACTIVE state.

The criticity of this alert depends on the number of Ingester still in ACTIVE state in hash Ring:

  • warning : the hash Ring still contains 2 ACTIVE Loki Ingesters
  • critical : the hash Ring contains one or less ACTIVE Loki Ingester

This is a degradated state, and it can imply:

  • if there are two Loki Ingesters still in an ACTIVE state, there is not data loss
  • if there is only one Loki Ingester still in ACTIVE state, in-memory data might be lost as each time series is replicated on two ingesters at the same time
  • if there is no more Loki Ingester in ACTIVE state, in-memory data is lost

More information regarding Cortex hash ring can be found here.

Troubleshooting hints#

Check the state of the ingesters

  • Port-forward on the distributor 
    kubectl -n <namespace> port-forward svc/distributor 3100:3100 &

  • Connect to Cortex Ring UI: http://localhost:3100/ring and check the state and the number of Ingesters not in ACTIVE state.

  • Check the state of ingester pod

  • Check logs of ingester pod

Solutions#

Use the following procedure to manage the Loki Ingesters.