Aller au contenu

Alertmanager Notifications#

A notification represents a group of one or more alerts, and it can be sent by the Alertmanager to different receivers via routes.

This procedure describes how to configure different notification receivers and routes.

Alertmanager uses these types of notification receivers:

  • webhook (mostly for Rocket.chat, alertinghub and Xymon)
  • email
  • slack (for Mattermost)

Definition of these receivers is made:

Caascad#

Notification Receivers#

Webhook Receiver#

In Caascad Alertmanager notification implementaion, there are different webhooks defined:

  • Rocket.Chat
  • slack

The webhook receiver configuration has the following formats.

Rocket.Chat example: 

receivers: {
    rocket_staging: #ReceiverWebhookConfig & {
        name: "rocketchat"
        webhook_configs: [{
            send_resolved: true
            url:           "https://..."
        }]
    }
}
Slack example: 
receivers: {
    rocket_staging: #ReceiverSlackConfig & {
        name: "mattermost"
        slack_configs: [{
            send_resolved: true
            api_url:       "https://..."
        }]
    }
}

The default Alertmanager notification configuration, can be overwritten in envs-ng/client_specific_configurations.cue configuration file.

To obtain the webhook url for Rochet.Chat, see here.

Email Receiver#

The email receiver configuration is made in envs-ng/contexts/caascad/alertmanager.cue and has the following format:

receivers: {
        email: #ReceiverEmailConfig & {
        name: "email"
        email_configs: [{
            to:            "support@..."
            send_resolved: true
            smarthost:     _smtp.smarthost
            from:          _smtp.from
            auth_username: _smtp.auth_username
            auth_password: _smtp.auth_password
        }]
    }
}

Notification Routes#

Alertmanager send notifications to different receivers via routes.

These routes are defined in envs-ng/pkg/alertmanager/routes.cue and have different format depending on the receiver.

Rocket.Chat Routes#

routes: {
    rocket_staging: {
        receiver: receivers.rocket_staging.name
                match: severity: "critical"
        continue: true
    }
}

Email Routes#

routes: {
     email: {
        receiver:        receivers.email.name
        continue:        true
        repeat_interval: "5m"
     }
}

NGOT#

NGOT integrates some notifications targets directly in the client definitions in NGOT zones.

They are defined in svc-monitoring-stack-client-<xxx>.parameters.monitoring-stack.alertmanager.notifications_targets.

Supported targets are:

  • alertinghub,
  • xymon,
  • e-mail,
  • mattermost (based on slack, but slack is not defined yet),
  • rocketchat

Notifications targets description#

Common format#

The format of a notification target is: 

parameters: {
    "monitoring-stack": {
        ...
        alertmanager: {
            notifications_targets: [{
                name: "<name>"
                secret_creds: "<name of creds in Vault>" // optional, see below
                receiver: { ... // optional }
                route: { ... // optional }
            }]
        }
    }
}

Targets that need secrets refer to a secret in Vault path secret/applications/alertmanager/receivers. The name of the secret is the name of the notification target (except for e-mails, see below).

Receivers can be customized following this documentation. The only default value is send_resolved: true (explicitely defined even if upstream, the default value is also true).

Most receiver configuration may need no specific parameters. However, to set send_resolved: false, here is an example:

{
    name: "<name>"
    receiver: {
        send_resolved: false
        ...
    }
}

Note

The notification targets xymon and alertinghub are predefined webhooks where you cannot override send_resolved.

Routes can also be customized following this documentation. There is no default value. Here is an example of route:

{
    name: "<name>"
    ...
    route: {
        matchers: ["severity = critical"]
        repeat_interval: "5m"
    }
}

Xymon#

Xymon notification target is fully automatized.

Enable xymon notification target with this:

{
    name: "xymon"
}

The name must be xymon.

E-mail#

E-mail notification target needs some parameters. The configuration reference is here.

Here is an example:

{
    name: "mail-xxx"
    secret_creds: "mail-zzz"
    receiver: {
        to:          "<to_email_adress>"
        require_tls: false
        from:        "<from_email_adress>"
        smarthost:   "<smarthost_adress>[:<smarthost_port>]"
   }
}
  • name: must begin with mail-.
  • secret_creds: optional. It overrides the name of the secret in Vault. The use-case is when more than one e-mail share the same smarthost with the same credentials.
  • receiver.to: destination e-mail address.
  • receiver.from: sender e-mail address.
  • receiver.require_tls: optional. Specifies if TLS is required on the smarthost.
  • receiver.smarthost: smarthost address. Alertmanager configuration documentation does not specify if the port is mandatory or if there is a default value. Specifying the port is recommended.

A secret must be provisionned in Vault with the name of name or secret_creds and with these keys:

  • username: user name or access key
  • password: password or secret key

Mattermost#

Mattermost notification target is based on Slack configuration and needs some parameters. Here is an example:

{
    name: "mattermost-xxx"
    receiver: {
        username: "Alertmanager Notification"
        icon_url: "https://raw.githubusercontent.com/prometheus/prometheus.github.io/master/assets/prometheus_logo-cb55bb5c346.png"
    }
}
  • name: must begin with mattermost-.
  • receiver.username: example parameter for Mattermost username
  • receiver.icon_url: example parameter for Mattermost icon url.

A secret must be provisionned in Vault with the name of name and with this key:

  • url: will be used as receiver.api_url

Rocket.chat#

Rocket.chat notification target is based on Webhook configuration and needs some parameters. Here is an example:

{
    name: "rocketchat-xxx"
}
  • name: must begin with rocketchat-.
  • receiver.xxx parameters may be specified.

A secret must be provisionned in Vault with the name of name and with this key:

  • url: will be used as receiver.url

Alertinghub#

Alertinghub notification target is fully automatized.

Alertinghub is a little different from other targets.

Enable alertinghub notification target with this:

parameters: {
    "monitoring-stack": {
        ...
        alertmanager: {
            notifications_targets: [#MonitoringStackNotificationsTargetAlertinghub]
        }
    }
}

Operations#

Add a new notification target#

Adding a notification target consists in:

  • adding the new notification target as a parameter in the client definitions file in the "monitoring-stack-client" zone;
  • adding a secret in Vault.

The deployment of the new notification target follows the usual workflow for changes.

Modify a notification target#

Modifying a notification target consists in:

The update of the notification target follows the usual workflow for changes.

Remove a notification target#

Removing a notification target consists in:

The removal of the old notification target follows the usual workflow for changes.

Set time intervals for notifications#

In Alertmanager it is possible to activate/deactivate sending the notifications and can take a different configuration for each defined notification target. The parameter that allows us to do this is time_intervals and you can find more details here.

Note

This type of change is usually being done at the customer's request.

In order to implement this capability, change the client custom configuration file (in zones/ngot_zones directory) as follows:

  • add one or more time_intervals definitions, each one identified by a name;
  • add the name of time_intervals definition in desired notification_targets.
Add time_intervals definition#

The time_intervals definition needs to be done in the following section:

  • _zones
    • svc-monitoring-stack-client-xxxx
      • alertmanager

Configuration example: 

_zones: {
    "svc-monitoring-stack-client-xxxx": {
        alertmanager: {
            let week_hours_notif = "week_hours_notif"
            time_intervals: [{
                name: week_hours_notif
                time_intervals: [{
                    times: [{
                        start_time: "06:00"
                        end_time:   "18:00"
                    }]
                    location: "Europe/Paris"
                    weekdays: ['monday:friday']
                    }]
                }]
        }
    }
}
In this example, we define a time_intervals called week_hours_notif which limits the time interval to working days from 6AM to 6PM.

Configure time_intervals per notification target#

We can define a different time_intervals for each notification_targets and it needs to be done in zones/ngot_zones/client-xxxx.cue (where xxxx is the customer name) as follows:

  • _zones
    • svc-monitoring-stack-client-xxxx
      • alertmanager
        • notifications_targets
          • route

Configuration example to activate notifications: 

_zones: {
        "svc-monitoring-stack-client-xxxx": {
            alertmanager: {
                notifications_targets: [{
                    name: "rocketchat-xxxx"
                        route: {
                            active_time_intervals: [week_hours_notif]
                        }
                }]
            }
        }
}
In this example, we activate the alert notifications for rocketchat-xxxx notification target only during the time interval defined in time_intervals called week_hours_notif.

Configuration example to mute notifications: 

_zones: {
        "svc-monitoring-stack-client-xxxx": {
                alertmanager: {
                        notifications_targets: [{
                                name: "rocketchat-xxxx"
                                        route: {
                                                mute_time_intervals: [week_hours_notif]
                                        }
                        }]
                }   
        }
}
In this example, we mute the alert notifications for rocketchat-xxxx notification target only during the time interval defined in time_intervals called week_hours_notif. Notifications will be sent outside the specified time interval.

Note

At the end of a mute interval, all alert notifications that are still in firing state, will be sent.

Check alertmanager notification configuration#

In order to visualise alertmanager notification configuration, you can do it directly on the alertmanager pod:

kubectl exec -it <alertmanager_pod> -n <namespace> sh

Alertmanager configuration file can be found with this command:

ps -ef|grep alertmanager | cat

The configuration file is contained in the command line of Alertmanager. It is

  • either in /etc/alertmanager/config/alertmanager.yaml,
  • or in /tmp/alertmanager/alertmanager.yml.

Configuration example:

global:
  resolve_timeout: 5m
  smtp_auth_password: PASSWORD_MAIL_SERVER
  smtp_auth_username: USER
  smtp_from: noreply@caascad.com
  smtp_smarthost: email-smtp.us-east-1.amazonaws.com:25
receivers:
- name: rocketchat
  webhook_configs:
  - send_resolved: true
    url: https://chat.corp.caascad.com/hooks/***
- email_configs:
  - send_resolved: true
    to: astreinte.support@orange.com
  name: email
route:
  group_by:
  - cc_prom_source
  - cc_prom
  - alertname
  group_interval: 5m
  group_wait: 30s
  receiver: rocketchat
  repeat_interval: 5m
  routes:
  - continue: true
    receiver: rocketchat
  - continue: true