Aller au contenu

LokiRequestLatency#

The alert expression means that there are too many slow requests. See the details here.

Troubleshooting hints#

  • Check if some other alerts may be linked to LokiRequestLatency. Search for LokiRequest400Errors or any alert related to network, cpu or other slowness problems.

  • Check these labels :

    • job : this is the name of the implicated Loki component (for example : querier, table-manager, ...)
    • route : this is the API route (e.g. URL path) that shows some latency.

  • Check the logs of the impacted component of Loki (see the job label of the alert to know the component).

  • Check the metric loki_request_duration_seconds_bucket and filter on status_code!="200" to check if we have only latency problems or if we also have another problem at the same time.

Solutions#

  • If another alert happens at the same time, first fix the other alerts. It may fix the LokiRequestLatency alert at the same time.

Known cases#

job=querier route=loki_api_v1_label#

Note

This case is not well known. On 15-03-2022, we started to study this problem but the incident suddenly stopped. We could not investigate more.

Warning

If this incident happens again, please note your investigation and update this doc.

This alert appeared on 15-03-2022 and the logs and metrics showed many requests to the Loki Querier. The client has logs with high cardinality of labels, which explains why the route loki_api_v1_label was impacted.

Here is how to see the size of the indexes :

kubectl -n <logging_namespace> exec -it svc/querier -c querier -- du -h /loki/index
kubectl -n <logging_namespace> exec -it svc/querier -c querier -- du -h /loki/index

Tip

Where <logging_namespace> can be:

  • logging
  • logging-client,
  • Notice the namespace label in the alert

High cardinality of labels is bad. It should be explained. On 15-03-2022, it could be explained because of many cronjobs leading to many pods with their own labels.

The alert stopped before we could investigate more.

When this alert happened, there are 2 questions we had not time and knowledge to answer :

  • is the client feeling latencies ?
  • what can we measure to know where the problem comes from ?

Without answers to these questions, on 15-03-2022 we could not fix the problem.