Aller au contenu

Writing Prometheus Rules#

Introduction#

Rules are PromQL expressions that will either be recorded as new timeseries (recording rules) or generate alerts (alerting rules).

On Caascad and NGOT, the link between the metrics origin, the rules definitions and the Prometheus that evaluates the rules it not obvious.

On Caascad:

  • Cluster and Apps metrics from Infra zone : they are scrapped and evaluated on Prometheus "infra-caascad".
  • Billing metrics from all zones : they are scrapped by Prometheus "client", federated by Prometheus "cloud-client" and federated again by Prometheus "infra-consumption". Rules are evaluated on Prometheus "infra-consumption".
  • Cluster and Apps metrics from Cluster zone : they are scrapped and evaluated on Prometheus "cloud-caascad".
  • Cluster and Caascad Apps metrics from Client zone : they are scrapped on Prometheus "client" and federated on Prometheus "cloud-client". Rules are evaluated on Prometheus "cloud-client".
  • All metrics from Client zone : they are (also) scrapped on Prometheus "client" and federated on Prometheus "cloud-app". Rules are evaluated on Prometheus "cloud-app". Only recording rules (for dashboards for our clients) may be evaluated there (because alerting is visible to our clients : they evaluate their rules there).

On NGOT

  • Cluster and cluster-centric apps metrics are scrapped and evaluated on Prometheus "cluster"
  • App metrics from service zones are scrapped and evaluated on Prometheus "cluster"
  • Metrics from Blackbox-Exporter and Cloudeye-Exporter (on "central" service zones) are scrapped and evaluated on Prometheus "central"

Monitoring of monitoring is tricky and is not described above.

PrometheusRules location#

Alerts are centralized because they need the validation of an Operator before being commited and deployed.

Get a clone of the alerts repository

Note

26/05/2023 : rules for NGOT are mixed in the repo above, even if we can read caascad-prometheus-rules in its name. This repo should be renamed one day.

Follow the documentation in README.md file and in the documentation below.

Rules are located in the rules/rules/ subdirectory.

PrometheusRules theory#

Simple example of a configuration file for the kubernetes prometheusRule object:

  • alert: PodCPUUsage
  • description: Container CPU usage is above 80%

Cue file name#

The name of the files that define rules is not important. In most cases, you will name it the same as the apps name (see below).

Cue file format#

Here is the definition of a cue file for alerting rules :

package rules

rulesets: [RulesetName= "infra" | "cloud-caascad" | "cluster-ngot"]: apps: "your app here": groups: {
  <groups of alerts>
}

Ruleset name#

RulesetName defines the Prometheus (and by extension, the Alertmanager) where you want to deploy your alert. RulesetName is one of:

  • infra (Caascad) : use with metrics from infra zone exporters
  • infra-consumption (Caascad) : use with metrics from client zone exporters
  • cloud-caascad (Caascad) : use with metrics from zone cloud exporters
  • cloud-client (Caascad) : use with metrics from zone client exporters
  • cluster-ngot (NGOT) : use with metrics from cluster zone exporters.
  • service-ngot (NGOT) : use with metrics from service zone exporters.

Examples:

  • RulesetName= "infra" | "cloud-caascad" will deploy rules for Infra and Cloud zones
  • RulesetName= "infra" will deploy rules for Infra zone only
  • RulesetName= "infra" | "cloud-caascad" | "cloud-client" will deploy rules for Infra, Cloud and Client zones
  • RulesetName= "infra" | "infra-consumption" will deploy rules for Infra and specific cases on infra-consumption alertmanager
  • RulesetName= "cloud-caascad" | "cluster-ngot" will deploy rules for Cloud zone (Caascad) and Cluster zone (NGOT).

Ruleset apps#

Apps is the name of your application and will be used for the definition of the PrometheusRules Kubernetes object name.

On all Caascad apps and on NGOT cluster-ngot apps, the apps name has no other meaning. Rules are always deployed.

On NGOT service-ngot apps, the name is also linked to the service configuration (in envs-ng/contexts/ngot/envs.cue). The service-ngot rulesets are deployed when their apps name is set in the service configuration, in _rule_apps (list).

Example of a ruleset for Grafana: 

rulesets: ["service-ngot"]: apps: "grafana": groups: {
    .....
}
In this example, the rules defined in this group will be deployed only if the configuration prometheus-rules defines _rule_apps with grafana inside (like below).

Example for the grafana services in envs-ng/contexts/ngot/envs.cue: 

#NgotServiceZones: "grafana": {
    zone:           _
    infra_zone:     _
    admin_zone:     _
    configurations: close({
        .....
        "prometheus-rules": _ & {_rule_apps: [
                    "grafana",
                    "thanos-query",
        ]}
        .....
    })
}
In this example, rulesets with apps grafana and thanos-query will be deployed on the clusters where the grafana services are deployed.

Note

  • a cue file can containe more than one "application". However you do not want this except if all the "applications" have a strong link together (like "prometheus" and "prometheus-down").

Alert groups format#

Alerts are defined inside groups. Here is an example :

PodCPUUsage: {
    rules: [
        {
            <definition of the alert here (see below)>
        },
    ]
}

Alert groups definition do not follow the PrometheusRules format. However, they are in the same spirit : a group may contain one or several rules.

Warning

In most cases, you will define one rule per group and set the same name of the group as the rule it contains.

Groups are made for sending groups of alerts when they fire together. However we usually define rules that are not supposed to fire together with other rules. This is why you should set one rule per group (except when you know what you are doing).

Alert format#

alert: "PodCpuUsage"
expr: """
      (sum(rate(container_cpu_usage_seconds_total{image!="", container_name!=""}[3m])) BY (instance, name, pod_name, container_name, cc_prom_source) * 100) > 80
      """
for: "5m"
labels: severity: "warning"
annotations:
  message: "Container CPU usage (instance {{ $labels.instance }})"
  description: "Container CPU usage is above 80%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }} "
  • alert: "<alert name>" : alert name is in CamelCase (with first letter uppercase too)
  • expr: "< expr is the expression that we want to install evaluate at regular intervals >". You can put it inside double quotes "<expr>" (and escape any double quotes : "<expr with"double quotes">"). Or use the triple double-quotes """ like above.
  • for "<duration>" : initial duration while the alert is not shown/notified. This prevents sending notifications if the problem solved itself in a short time. Don't set for if lower than 30s.
  • labels "< label set >" : two alerts cannot have the same label set (labels come from the metrics) or if they both fire, the second one will be considered as a repetition of the first one and will not fire. This is where you can distinguish two alerts with distinct labels. This is also where you can set the severity label.
  • annotations "< label set >" : additional labels that do not identify the alert (opposite of labels). Should include at least message.

Warning

Alert name, expression and all values should be defined with double quotes. This is a cue requirement.

Note

in cue, you can either write 

labels: severity: "warning"
or 
labels: {
    severity: "warning"
}
The one-liner is easier to read. You may prefer it.

Labels in alert message, summary and description#

You can use labels in annotations to display any label you want to show. You can see those labels in Grafana.

There are 2 kinds of labels :

  • metrics labels : they are part of the metric. Use {{ $labels.thelabelname }} to display it.
  • external labels : they are defined in Prometheus configuration and will be added after the metrics are retrieved. Use {{ $externalLabels.thelabelname }} to display it.

Today, the defined external labels are:

  • Caascad: cc_prom and cc_client.
  • NGOT: cluster

Mandatory annotations for the rules#

  • The system can work without annotations. However, message was made mandatory at Caascad for homogeneization purposes.
  • Because we want to have more information on alerts, you have to add message (and optionaly some of summary, description and help_alerts). There is no clear explanation of those labels and what they mean. Here is an interpretation :
    • message : this appears in Karma, mail and other notification systems. This is where the Operator should have all the information (s)he needs to investigate. Be short, but give all the information
    • summary : not used in Karma. This is a summary. If you use it, be very short and consider that the Operator may copy/paste the summary into the title of a Jira ticket. Important note : the summary should be removable without any loss of information about the problem.
    • description : this appears in Karma (like a duplicate of message). This is where you can specify a full and long description of the alert. You probably don't want to write a description as it will duplicate help_alerts.md. Important note : the summary should be removable without any loss of information about the problem.
    • help_alerts : link where is the documentation on the alerts. See below best practices for writing the alert documentation.

Warning

annotations.help_alerts is automatically set when the PrometheusRules are generated. In most cases, it is a bad idea to overwrite it. Don't set it. Or set it if you know what you are doing.

Recording rules : for more complex and expensive alert#

Sometimes alert expressions can be complex and expensive.

Recording rules allow you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series.

We can use it as metrics in alert expression.

It also make the alert more visible.

In this example (based on a real PrometheusRules Kubernetes object), the alerting rule uses 2 recording rules :

spec:
  groups:
  - name: recording_rules
    rules:
    - expr: max(max_over_time(timestamp(changes(concourse_builds_finished{pipeline="k8s-functional-tests",status!="succeeded"}[2m])
        > 0)[24h:1s]))
      record: k8s_functionnal_tests_not_succeeded:concourse_builds_finished:max_over_time_timestamp_changes
    - expr: max(max_over_time(timestamp(changes(concourse_builds_finished{pipeline="k8s-functional-tests",status="succeeded"}[2m])
        > 0)[24h:1s]))
      record: k8s_functionnal_tests_succeeded:concourse_builds_finished:max_over_time_timestamp_changes
  - name: alerting_rules
    rules:
    - alert: K8sFunctionalTestFailed
      annotations:
        summary: k8s mininimal functional test failed
        message: Pipeline {{ $labels.pipeline }} has failed on Job {{ $labels.exported_job
          }}
      expr: k8s_functionnal_tests_succeeded:concourse_builds_finished:max_over_time_timestamp_changes - k8s_functionnal_tests_not_succeeded:concourse_builds_finished:max_over_time_timestamp_changes < 0
      for: 1m
      labels:
        severity: warning

Recording rules should be of the general form level:metric:operations.

  • level represents the aggregation level and labels of the rule output.
  • metric is the metric name and should be unchanged other than stripping _total off counters when using rate() or irate().
  • operations is a list of operations that were applied to the metric, newest operation first.

More information and examples on https://prometheus.io/docs/practices/rules/

Note

You can also use the recording rules when writing dashboards.

Alert labels#

Mandatory labels#

Caascad mandatory labels :

  • cc_client
  • cc_prom_source
  • cc_prom

NGOT mandatory labels :

  • obs_client
  • cluster (in some cases, it can be set automatically)
  • severity
  • namespace
  • ngot_contract (only for service zone type)
  • ngot_service (only for service zone type)

Suggested labels :

  • instance
  • service
  • container

These labels give information about the problem. The more labels you have the easier to debug it is.

Note

NGOT zones :

  • Label cluster : when the rule is evaluated on the Prometheus of a cluster zone, it will be set automatically. You do not need to set it. You must not set it on absent(...) expressions.
  • All mandatory labels should be already set on metrics. This doc shows how to do it easily. However, you must set them explicitely when you use absent() in your rule expression.

Warning

Ensure that aggregations will not remove the mandatory labels. See below.

Setting and removing labels#

Labels from the PromQL expression are reported in the alert. There is nothing to do.

You can explicitely keep or remove labels using aggregations :

  • sum by(...) or sum without(...)
  • min by(...) or min without(...)
  • max by(...) or max without(...)
  • avg by(...) or avg without(...)
  • count by(...) or count without(...)
  • group_right(...)
  • group_left(...)
  • ...

When you use by(...) or group_xxxx(...), add the labels you want to keep. All other labels will be removed.

Warning

When using an aggregation with by(...) or group_xxxx(...), ensure you specified the mandatory labels explicitely or they will be removed too. You probably expect something like by (..., cc_prom_source).

Special case : absent()#

PromQL expressions with absent() function need special care. Because the alert will fire when the metric is missing, the labels cannot come from the metric.

You must explicitely set the mandatory and other wanted labels.

Rules with absent() are not as easy as other rules to define. Consider looking for existing rules and copying some existing code.

Warning

Rules that are evaluated on NGOT cluster zones should not set cluster with absent(). This label is an external label and will automatically be set when Prometheus outputs the rule (remote-write for recording rules, Alertmanager for alerting rules). In Prometheus, this label is not set, so if you evaluate rules with absent() and this label, the alert will always fire, which is not what you want.

Testing the expression (check mandatory labels)#

We can test that the alert expression does not remove the cc_prom_source mandatory label by going to the explore tab in Grafana.

For the example of PodCpuUsage, the expression is :

(sum(rate(container_cpu_usage_seconds_total{image!="", container_name!=""}[3m])) BY (instance, name, pod_name, container_name, cc_prom_source) * 100) > 80

In Grafana, your expression (like above) will probably render an empty result. You will only see metric values that match the condition and you usually check when the conditions of firing alerts are not met.

You have to change the expression either by inverting the test (< 80 above instead of > 80) or by removing the test (remove > 80). Example (we removed > 80) :

(sum(rate(container_cpu_usage_seconds_total{image!="", container_name!=""}[3m])) BY (instance, name, pod_name, container_name, cc_prom_source) * 100)

And check the presence of the cc_prom_source label.

See grafana-infra.infra-stg.caascad.com/explore

Troubleshooting#

cue eval#

When Cue does not generate the wanted result, or when it fails with errors, you can try this:

cd rules/rules
cue eval -e rulesets -t zone_name=<zone name>

Example: 

cue eval -e rulesets -t zone_name=ocb-testzone01

Pre-defined zones can be found in rules/zones_default.cue.

You can also eval other objects than rulesets.

cue generate and cue validate#

The command lines cue ... generate and cue ... validate can be found in the README.md file.

The command cue ... validate will generate the rules and launch promtool check rules. This helps to

  • validate the PromQL thanks to promtool check rules
  • inspect specific rules because they are generated in many but short files (in rules/rules/promtool_out/ dir)

The command cue ... generate will generate the rules in one file (in rules/rules/values_out/ dir) that is exactly the file that will be provided to Helm as a values.yaml file. This helps to

  • see the output provided to Helm
  • see the context of the rule you want to debug

trackbone shell#

When trackbone shell does not plan or deploy the expected rules, you can try this:

trackbone shell -z xxxxx -c prometheus-rules

Inside the trackbone shell here are the usual debugging tasks :

  • cat rules/zones.yaml to check that the parameters are set as you want
  • cat plan.sh to generate the zone.cue file then generate the rules
  • cd rules/rules followed by cue eval -e rulesets -t zone_name=xxxx, cue ... generate or any other cue commands to debug.

Using trackbone shell is also a good way to temporarily hack the rules and drop all the hacks without cleaning aniything when you leave the trackbone shell.