LokiIngesterNotActive

This alert is triggered when a Loki Ingester is in a state different than ACTIVE for too long.

This can happen when a new Ingester has been started correctly, but the leaving one is blocked in a non-ACTIVE state (Unhealthy, Pending, Joining, Leaving). This means that we can have for example three ACTIVE Loki Ingesters and one Pending Loki Ingester.

The evaluation of the impact and required actions to fix the issue, depends on the number of Loki Ingesters still in ACTIVE state:

• if there are two Loki Ingesters still in ACTIVE state, there is not data loss
• if there is only one Loki Ingester still in ACTIVE state, in-memory data might be lost as each time series is replicated on two ingesters at the same time
• if there is no more Loki Ingester in ACTIVE state, in-memory data is lost

More information regarding Cortex hash ring can be found here.

Troubleshooting hints

• Check the state of the ingesters : kubectl -n <namespace> port-forward svc/distributor 3100:3100 & and connect to http://localhost:3100/ring
• Check the state of the ingester/distributor pods : kubectl get pods -n <namespace>
• Check the logs of the ingester/distributor pods : kubectl logs <ingester_pod> -n <namespace>
• If you can spend time on investigation : https://github.com/cortexproject/cortex/blob/master/docs/guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over

Solutions

Use the following procedure to manage the Loki Ingesters.