Aller au contenu

LokiFlushQueueNotEmpty#

This alert appears when Loki is not able to push logs in S3 bucket.

It indicates that the buffers (flush queue) start to fill up with logs that can not be pushed to storage.

It appears in one of the following situations:

  • network problem
  • bad S3 link configured in Loki configmap
  • bad AK/SK
  • etc

Troubleshooting#

Check logs ingester pods for more details#

kubectl get pods -n  <logging_namespace> | grep ingester  # get name of ingester pods
kubectl logs <ingester-name> -n <logging_namespace>

Tip

Where <logging_namespace> can be:

  • logging
  • logging-client,
  • Notice the namespace label in the alert

Tip

If you see logs like : 

msg=\"failed to flush user\" err=\"RequestError: send request failed\\ncaused by: Put \\\"http://oss.eu-west-0.prod-cloud-ocb.orange-business.com/loki-cloud-client-ocb-test05/fake/fcbc108991757af8%3A17ffacacf83%3A17ffb01d13e%3A4808e0ce\\\...
You can check that the error does not occur for the same chunk to qualify the criticality of the incident.

Connect to the pod#

kubectl exec -it <ingester-name> -n <logging_namespace>  -c ingester -- sh
$ ps -ef    #check config file location in command line
$ cat /etc/loki/config.yaml | grep s3 #check s3 bucket URL
Where <logging_namespace> can be logging or logging-client.

Solution#

In order to clear the loki flush queue you can delete the ingester

Warning

if more than 1 ingester are impacted do not delete them at the same time, we might lose logs

kubectl delete <ingester-name> -n <logging_namespace>
sometimes the old ingester are still in the ring, you can remove it with this script

git clone git@git.corp.caascad.com:caascad/applications/caascad-loki.git && cd src/loki_tools/
./loki_remove_ghosts_ingesters_from_ring.sh <logging_namespace>