Aller au contenu

PrometheusMissingRuleEvaluations#

Context#

This alert is triggered when Prometheus is not able to evaluate rules at the rate it is being asked to, and can lead to alerting rules not being evaluated.

Danger

This is a dangerous situation because if rules are not being evaluated, alarms can not be triggered.

Known cases#

To identify if one of the known use cases corresponds to the incident, check logs of the prometheus pod:

  • use case 1: msg="Error on ingesting out-of-order result from rule evaluation"
  • use case 2: msg="Evaluating rule failed"

Use case 1#

Complex prometheus rules take a long time to be evaluated by Prometheus.

Troubleshooting hints use case 1#

  1. Identify the rules that take a long time to be evaluated (this is in general the case of complex recording rules)

    • this information is available in Grafana : prometheus_rule_group_last_duration_seconds metric (label rule_group)

  2. Check Prometheus default evaluation interval (this represents the frequency at which Prometheus starts a new evaluation of the rules, despite the fact it did not finish the previous evaluation):

kubectl exec -it <prometheus_pod> -n <namespace> -- grep evaluation_interval /etc/prometheus/prometheus.yml`

Important

If evaluation_interval(step 2) is larger than the time necessary to evaluate a rule (step 1), the root cause of the incident is the one described in use case 1.

Solutions use case 1#

  • find solution to simplify the recording rules that take a long time to be evaluated (identified at step 1)

  • increase the evaluation interval for the rule that is taking a long time.

For this action, there are two possible ways:

  1. By modifing prometheus-rules and follow the procedure to deploy the new values everywhere. Procedure in caascad-prometheus-rules repository.

Add/modify rule evaluation interval in the file containing the rule in applications/caascad-prometheus-rules/rules.

Example:

    ConcoursePipelineErrorPostgresStatus: {
        interval: "5m"
        rules: [
            {
                    alert: "ConcoursePipelineErrorPostgresStatus"
  1. Manually add/modify the evaluation interval for the faulty prometheus rule.

Example:

kubectl -n prometheusrules edit prometheusrules infra-concourse
The modification consists in adding the interval of 5m. This action will replace the default evaluation interval for that specific rule : 
  - interval: 5m
    name: ConcoursePipelineErrorPostgresStatus
    rules:
    - alert: ConcoursePipelineErrorPostgresStatus

Warning

If the incident appears during NBH and the issue is urgent, perform one of the following two actions :

  • delete the prometheus rule that is causing the issue: kubectl delete prometheusrules <prometheus-rule_name> -n prometheusrules

The global problem will be checked during BH by monitoring team.

Use case 2#

Bad metrics that Prometheus fails to handle.

Example: metrics build with many-to-many aggregation type.

Troubleshooting hints use case 2#

  • Identify the rule that causes the problem and the reason for which it fails to be evaluated (in Grafana prometheus_rule_group_last_duration_seconds metric)

  • check the logs of the prometheus pod

Solutions use case 2#

  • fix the rule (the missing/bad aggregation) by ensuring that the mandatory labels are specified. More information here.

Other cases#

If none of the above use case corresponds to the issue, the standard solutions are to be applied:

  • check prometheus logs

  • identify the rule that is causing issues

  • fix faulty rule

  • increase evaluation interval for the faulty rule