Aller au contenu

PrometheusRuleFailures#

Prometheus fails to evaluate prometheusrules, which prevents it to trigger alerts.

There are two type of prometheusrules:

  • custom (prometheusrules namespace);
  • upstream (prometheusrules-dashboards-upstream namespace).

Important

In the case of NGOT, this alert can have two different severities, depending on zone type of the Prometheus that fails to evaluate the prometheusrule:

  • warning - Client Prometheus;
  • critical - Central Prometheus.

Troubleshooting Hints#

  • Make a port forward for Prometheus that generated the alarm, and visualise :9090/rules to identify the prometheusrule that is in ERR state;
  • Check the expression of the prometheusrule by using Grafana (Explore menu).

Client prometheusrules#

Client prometheusrules and their errors can be visualised also in client Grafana (Alerting > Alert Rules).

If the alert is triggered due to a client prometheusrule, you just need to identify the faulty prometheusrule and inform the Customer in order to fix it.

Client prometheusrules are deployed in the follwing namespaces:

  • rules-obs-${CLIENT} - NGOT.
  • monitoring-app - Caascad.