LokiUnhealthyIngesterAutomaticallyDeleted

This alert is triggered when the Loki feature autoforget_unhealthy has automatically removed at least one ingester in unhealthy state from the hash Ring.

The criticity of this alert depends on the number of unhealthy ingester removed from the hash Ring in the last 12 hours:

• info : one Unhealthy Ingester has been automatically removed from the hash Ring in the last 12 hours
• warning : at least three Unhealthy Ingester have been automatically removed from the hash Ring in the last 12 hours

Note

The alert with criticity info is visible in Karma only when another alert is in firing state on the same namespace.

If it's the case, check all other alerts on the same namespace in order to get more information regarding the problems that cause ingesters become unhealthy.

Multiple ingesters in unhealthy state indicate there are some issues at Loki ingester level and it need a further investigation.

More information regarding Cortex hash ring can be found here.

Troubleshooting hints

Check the state of the ingesters :

• Port-forward on the distributor

  kubectl -n <namespace> port-forward svc/distributor 3100:3100 &
  

  !!! tip Where <logging_namespace> can be: === "Caascad" - logging - logging-client, === "NGOT" - Notice the namespace label in the alert

• Connect to Cortex Ring UI: http://localhost:3100/ring and check the state and the number of Ingesters not in ACTIVE state.

• Check the state of ingester pod:

  • Check the start time of the pod:

    • Has it rebooted recently?

    • If it is the case, check if it is not the node reboot that caused the ingester pod reboot.

    • If yes, you should investigate why the node has rebooted.

  • Check the logs of the ingester pod:

    • Identify when the problem has started.

    • Is the issue still present?

    • Are there any visible problems (like timeouts, etc.)?

    The logs of the ingester can indicate a connection problem:

    level=error ts=2022-11-24T06:28:13.385261296Z caller=client.go:242 msg="error getting path" key=collectors/ring err="Get \"http://consul:8500/v1/kv/collectors/ring?consistent=&index=6245926&wait=10000ms\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    level=error ts=2022-11-24T06:28:18.110675282Z caller=client.go:156 msg="error getting key" key=collectors/ring err="Get \"http://consul:8500/v1/kv/collectors/ring\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    

    In that case, check the logs of the Consul pod.