Aller au contenu

LokiFlushQueueFillingUp#

This alert appears when Loki is not able to push logs in S3 bucket.

It indicates that the buffers (flush queue) start to fill up with logs that can not be pushed to storage.

It appears in one of the following situations:

  • network problem
  • bad S3 link configured in Loki configmap
  • bad AK/SK
  • etc

Troubleshooting#

Check logs ingester pods for more details.

kubectl get pods -n  <logging_namespace> | grep ingester  # get name of ingester pods
kubectl logs <ingester-name> -n <logging_namespace>
kubectl exec -it <ingester-name> -n <logging_namespace>  -c ingester -- sh
$ ps -ef    #check config file location in command line
$ cat /etc/loki/config.yaml | grep s3   #check s3 bucket URL

Tip

Where <logging_namespace> can be:

  • logging
  • logging-client,
  • Notice the namespace label in the alert

Solution#

In order to clear the loki flush queue you can delete the ingester

Warning

if more than 1 ingester are impacted do not delete them at the same time, we might lose logs

kubectl delete <ingester-name> -n <logging_namespace>
sometimes the old ingester are still in the ring, you can remove it with this script

git clone git@git.corp.caascad.com:caascad/applications/caascad-loki.git && cd src/loki_tools/
./loki_remove_ghosts_ingesters_from_ring.sh <logging_namespace>