Aller au contenu

LokiFailedToPushLogs#

Refs#

Alert context#

The alert is based on a custom metric loki_error_push_logs_custom which persists for 20 minutes : if there is no modification of the metric for 20 minutes, the metric disappears.

This metric is incremented when in the Loki logs there are error logs failed to flush user or failed to upload dbs.

Alert parameters#

  • For : 20 minutes
  • Expression : sum (increase(loki_error_push_logs_custom[20m])) by (cc_prom_source, namespace, pod) > 5

Case where the alert is triggered#

Over a period of 20 minutes:

  • if only one error log: the expression is false -> no increase
  • if two error logs: the expression is false -> increase of 1 ...

  • if seven error logs: the expression becomes true -> increase of 6

  • if 20 minutes later :

    • another error log in this period -> the alert goes into firing
    • no more error logs -> the metric no longer exists -> the alert goes resolved

We chose to check for an increase of 5, this is an arbitrary number.

For the same chunk, a retry is made every 30 secondes, so in 20 minutes, there are 40 failures.

For the same index table, a retry is made every 1 minute, so in 20 minutes, there are 20 failures.

Which implies that the number must be between 0 and 20 (we didn't put zero to avoid false positives).