Aller au contenu

Mute alert#

Goal: to be able to mute an alert during the desired time period for a specific zone (or zones).

Two steps :

  • changes the alert expression to be parameterized per zone : add extra expression
  • set the extra expression for the desired zone

Change the alert expression#

Go to caascad-prometheus-rules.

In the file where the alert is present :

  • add import strings

  • modify your expression :

    expr: "<base_expression>"

    becomes

    expr: strings.TrimSpace("<base_expression> (zone.rules_extra_expr.<alertname>)"

Example for PromitorScraperScrapeError : 

strings.TrimSpace("avg_over_time(promitor_scrape_error{obs_client=\"\(zone.obs_client)\"}[10m]) == 1 \(zone.rules_extra_expr.promitor_scraper_scrape_error)")

In zones_default.cue, add alertname in rules_extra_expr.

Set the extra expression#

Go to envs-ng.

  • modify tag version in prometheus_rules.cue
  • add extra expression in ngot-zones for alert

Example for mute PromitorScraperScrapeError alert for Chanel zone :

zones/ngot_zones/client-chanel.cue
"prometheus-rules": {
   rules_extra_expr: {
     promitor_scraper_scrape_error: "and ON() (0 < day_of_week() < 6) and ON() (7 < hour() < 17)"
   }
}

  • 0 < day_of_week() < 6 : mute the alert on weekends
  • 7 < hour() < 17 : mute the alert at night