Aller au contenu

LokiRequestLatency#

Refs#

TL;DR#

The alert expression is based on a recording rule and on histogram_quantiles.

The meaning of the alert is that 1% of the requests are slower than the value (in the message of the alert). The alert happens when that value is higher than the threshold (1 second).

In short, there are too many slow requests.

Expressions#

alert: "LokiRequestLatency"#

namespace_job_route:loki_request_duration_seconds:99quantile{route!~\"(?i).*tail.*\"} > 1

Unit : seconds

record: namespace_job_route:loki_request_duration_seconds:99quantile#

histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m]))
        by (le, namespace, job, route, cc_prom_source))
  • Unit : number
  • Important labels :
    • le: contains a number in seconds.
    • route : Loki route for which we will investigate on latencies
    • namespace, job, cc_prom_source: common labels for the resolution of the problem.

Explanation#

Part 1 : the recording rule#

A loki_request_duration_seconds_bucket is a set of metrics with different le durations.

Note

Notes about le label :

  • le means Lower or Equal.
  • possible values : [0.0100, 0.0250, 0.0500, 0.100, 0.250, 0.500, 1, 2.50, 5, 10, 25, 50, 100, ∞]
  • unit : seconds

Each value of loki_request_duration_seconds_bucket is the number of requests that took less or equal time than the le value.

namespace_job_route:loki_request_duration_seconds:99quantile finds the 99th quantile for loki_request_duration_seconds_bucket : it takes the values of metric and combines it with le :

  • it finds the value for which 99% were lower (faster in our case) and 1% were higher (slower)
  • it finds the le interval for this value
  • it computes a better approximation inside the le interval. This is the result, in seconds.

Example :

  • If the metric shows that there is a value V where
    • 99% of the metrics values are smaller than V for labels 0.0100, 0.0250, 0.0500 and 0.100,
    • and the remaining 1% are for the other labels,
  • then the quantile is between 0.100 and 0.250.
  • Prometheus will then do another operation and find a better value in the interval [0.100, 0.250], for example 0.123.
  • The recording rule will then have its value as 0.123, meaning that
    • 99% of the requests are faster than 0.123 seconds
    • and 1% are slower than 0.123 seconds.

Part 2 : the alerting rule#

When namespace_job_route:loki_request_duration_seconds:99quantile is generated for all labels, we can have a look on the alert expression. It alerts when the label route does not contain tail.

When it alerts, it means that 1% of the requet duration was slower than the threshold. In our case, the threshold is 1 second.

In the alert, besides the namespace, job and cc_prom_source, the most important label is the route. It should give a precision on which requests are slower (examples: api_prom_push, metrics, ready, /logproto.Querier/Query...)