Aller au contenu

AlertmanagerFailedToSendAlerts#

At least one instance is unable to route alert to the corresponding integration.

Troubleshooting hints#

Tip

Below, <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

  • Notice the namespace label in the alert

  1. Connect to grafana and check the metrics alertmanager_notifications_total using Thanos datasource. 

    rate(alertmanager_notifications_total{integration=~"webhook|email"}[5m])
    The graph shows how long alertmanager has been unable to send alert to the notifications channel.

  2. Check the logs of the container alertmanager to see where the errors come from. 

    kubectl logs <alertmanager_pod> -n <monitoring_namespace> -c alertmanager

Known Cases#

Alertinghub/webhook code 401#

The APIKey used by Alertmanager is not active.

  1. Check Alertreceiver configuration

Go to the MIT website and open the Alert receiver Payload Validation page.

Warning

To access the MIT website, you must set the following proxy: http://proxypac.si.francetelecom.fr:8080/

Example using Chromium:

chromium --proxy-pac-url="http://proxypac.si.francetelecom.fr:8080/" "https://mit.equant.com/MIT/consultation_alert_receiver_payload_validation.php"
  1. Retrieve the internal API key from Vault
  • Go to Vault and navigate to the path: /secrets/secret/list/applications/nginx-alertreceiver/${client}/. Replace ${client} with the appropriate client name, e.g., aed.
  • Copy the API key value and paste it into the API Key search bar on the MIT page to check its APIKey active status.

mit

If the Active field is set to NO, you must open a SWAN ticket to request activation.

  1. Create a SWAN ticket

  • Follow the same steps as when requesting a new API key (excluding the dictionary part), as discribed here and update the Operation description field as follows:

    Operation description
    Hello,

Can you please change the APIKey status to YES?
As shown in the attached screenshot, it is currently set to NO.
Project name: XXX

Thank you.
    Make sure to attach the screenshot in the "Attachments" tab.