Aller au contenu

Inhibition of severity info alerts#

Principles#

Severity info alerts are especially useful for debugging a potential incident.

So the goal is to inhibit info severity alerts when no other alerts appear in the same namespace.

Inhibited alerts are not visible in Karma and no notification is sent.

How it works#

InfoInhibitor rule#

We deploy an InfoInhibitor upstream alert.

This alert fires when in the same namespace :

  • there is an severity=info alert (pending or firing) ;
  • and there are no severity=warning or severity=critical alerts (firing).

Alertmanager configuration#

Inhibit rules#

We configure Alertmanager so that when the InfoInhibitor alert is firing, severity info alerts are inhibited if the namespace label is the same :

inhibit_rules:
- equal:
  - namespace
  source_matchers:
  - alertname = InfoInhibitor
  target_matchers:
  - severity = info

InfoInhibitor rule routing#

The InfoInhibitor alert should be routed to a null receiver :

  • to not send notification about this alert ;
  • to not see this alert in Karma.

This is a severity=none alert and we route all alerts with this severity to a null receiver : 

- continue: false
  matchers:
    - severity = none
      receiver: "null"