Aller au contenu

PrometheusTSDBBlocksLoadedLow#

Refs :#

TL;DR#

This alert happens when the number of loaded blocs in Prometheus is lower than it should be.

Expression#

For Caascad the expression is like this:

1
2
3
prometheus_tsdb_blocks_loaded{namespace!='caascad-monitoring'}<=2
and on(namespace,pod) delta(prometheus_tsdb_blocks_loaded[130m])<=0
and on(namespace,pod) max by(namespace,pod) (min_over_time((increase(prometheus_tsdb_head_samples_appended_total[20h]))[24h:5m]))>0

Where

  • <=2 (line 1) is the main threshold (and differs on Caascad and NGOT)
  • 20h (line 3) may change depending on how far in the past we should detect changes

For Ngot, the expression is like this: 

prometheus_tsdb_blocks_loaded <=12
and on(namespace,pod,cluster,instance) delta(prometheus_tsdb_blocks_loaded[130m])<=0

Explanation#

Main expression#

The main expression is :

prometheus_tsdb_blocks_loaded <= <threshold>

We want to alert when the number of loaded blocs in Prometheus is too low.

  • Caascad : warning when threshold is 2; critical when threshold is 1
  • NGOT : warning when we do not have all the blocs (13) and critical when we have lower than 6 blocs.

Use case : the number of blocs is growing#

There are cases when the number of blocs is growing. Examples :

  • we had an incident. It is resolved and the number of blocs is growing. However, on NGOT, with the high threshold, the expected number can be reached only after 24 hours.
  • Prometheus was just deployed. We have no blocs yet. But every 2 hours, we will have one more bloc.
  • Prometheus has been used a long time ago, but not recently, until very recently. With the new metrics coming, some cleanup was done and old blocs were removed. However, new blocs will be created every 2 hours with the new metrics coming.

We do not want to alert when the number of blocs is growing. We disable the alert with this expression : 

and on(namespace,pod,cluster,instance) delta(prometheus_tsdb_blocks_loaded[130m])<=0

When the number of blocs is higher at the end of the interval of 2 hours than at the beginning, the delta is positive. So we alert only if that delta is negative (loss of blocs) or null (no blocs being created).

We put 130m instead of 2h because the "2 hours interval" is bigger than 2 hours :

  • t+0 : a new bloc is created
  • t+2h : cleanup is done
  • t+2h+some minutes : a new bloc is created

After "t+2h" and before "t+2h+some minutes", there is a little time when the delta is nul and when we expect an increment. When we specify 130m instead of 2h we take that "some minutes" interval into account and it works.

Use case : new metrics after a long time without metrics#

Note

This use case is only valid for Caascad : in Ngot, Prometheus have metrics all the time.

There is a case when no metrics are sent to Prometheus for a long time (more than 24 hours in most cases) then new metrics are coming.

When no metrics are coming, Prometheus no more cleans up blocs. The number of blocs is usually correct at that time. And that number will not change.

When new metrics are coming, Prometheus will restart cleanup. All the old blocs will be removed at that time. The consequence is that prometheus_tsdb_blocks_loaded will update to 0 and generate an alert.

This use case may be a problem with having no metrics. It is not a problem with having no blocs. So we want to disable the alert in this case, with this expression :

and on(namespace,pod) max by(namespace,pod) (min_over_time((increase(prometheus_tsdb_head_samples_appended_total[20h]))[24h:5m]))>0

First we check if we have metrics in the last 24 hours.

  • Caascad critical : prometheus_tsdb_blocks_loaded <=1
    • we ignore the last 2 hours (prometheus_tsdb_blocks_loaded is 0)
    • we ignore the last 2 more hours (prometheus_tsdb_blocks_loaded is 1)
    • total : we ignore 4 hours
    • with an offset of 4 hours, the interval of 24h becomes 24h-4h = 20h.
  • Caascad warning : prometheus_tsdb_blocks_loaded <=2
    • this is one more bloc than the critical alert
    • we ignore 2 more hours (prometheus_tsdb_blocks_loaded is 2)
    • total : we ignore 6 hours
    • with an offset of 6 hours, the interval of 24h becomes 24h-6h = 18h.

Use case : disk space too low#

When the excepted number of blocs cannot be written on disk, for any reason, mostly when the disk space is too low, we should be alerted.

prometheus_tsdb_blocks_loaded <= <threshold>
This expression should be true because there is a proble to write blocs on the disk.

and on(namespace,pod,cluster,instance) delta(prometheus_tsdb_blocks_loaded[130m])<=0

Because new blocs are not written on disk, the delta is not positive. This expression should be true.

and on(namespace,pod) max by(namespace,pod) (min_over_time((increase(prometheus_tsdb_head_samples_appended_total[20h]))[24h:5m]))>0

When we have blocs and problems to write blocs, we are in a case where we have metrics. So this expression should be true.

The result is the alert firing, which is what we want in this case.

Important note : with this use case, we need to evaluate the expression on another Prometheus that does Monitoring of Monitoring.

Explanation : when there is a problem to write blocks on the disk, we may have no history and some parts of the expression will not give expected results. However, we need history to make the difference between the case when there were no metrics earlier and the case when there were metrics but they were lost due to the problem of writing blocs on the disk.

For this reason, the rules must be evaluated on another Prometheus. A Prometheus that does Monitoring of Monitoring. Here is the deployment architecture for this rule :

  • Caascad
    • infra-caascad : will be evaluated at least on infra-consumption
    • infra-consumption : will be evaluated on infra-caascad
    • cloud-caascad : will be evaluated on cloud-client
    • cloud-client : will be evaluated on cloud-caascad
    • cloud-app : will be evaluated on cloud-caascad
  • NGOT
    • cluster : will be evaluated on monitoring-stack central
    • monitoring-stack central : will be evaluated on cluster
    • monitoring-stack client : will be evaluated on cluster

Because one Prometheus may evaluate the rules on its own metrics, the alert may fire from both itself and the Prometheus that watches it. However, the only warranty is from the Prometheus that watches it.