AlertmanagerClusterFailedToSendAlerts

All instances of the cluster failed to send notification to an integration.

Troubleshooting hints

Tip

Below, <monitoring_namespace> can be:

CaascadNGOT

• Zone Cloud: monitoring, monitoring-client or monitoring-app
• Zone Infra: monitoring or monitoring-consumption

More information in Caascad namespaces naming.

• Notice the namespace label in the alert

1. Check if all alertmanager pod are running

   kubectl get pods -n <monitoring_namespace> -l app.kubernetes.io/name=alertmanager
   kubectl describe pod <alertmanager_pod> -n <monitoring_namespace>
   

2. Check the logs of the pods alertmanager

   kubectl logs <alertmanager_pod> -n <monitoring_namespace> -c alertmanager
   

3. Check the alertmanager configuration file

   kubectl exec -it -n <monitoring_namespace> <alertmanager_pod> -c alertmanager -- cat /etc/alertmanager/alertmanager.yml
   

4. Check if the url of the webhook or the email adress is correct. See the alerts mail_notifications

Known Cases

Alertinghub/webhook code 499

1. Grafana Verification

Navigate to: Connect to Grafana prdcasa → Explore → Loki-clients

Run the following query:

{namespace="nginx-alertreceiver", cc_prom_source="prdcasa", alerts_source="CUSTOMER_NAME"} | json | line_format `{{ .request_body  }}` | json | status_extracted=`firing` | json labels=`alerts[0]["labels"]` | line_format `{{ .resp_body }}`  | line_format `{"labels":{{ .labels  }}, "message":"{{ .message }}" }`

The status field will show a 499 error

Technical Explanation

The 499 error is specific to Nginx and indicates that the client closed the socket. In this case, the client is Alertmanager.

Likely scenario:

Alertmanager sends an alert to Nginx-Alertreceiver.

Nginx-Alertreceiver attempts to forward it to the host (e.g., 57.203.253.34).

No response is received from the host (e.g., 57.203.253.34) within the expected timeframe.

Alertmanager considers the delay too long and closes the socket → error 499.

Nginx-Alertreceiver stops the connection attempt, logging:

Operation timed out

Action

⚠️ Apparently, when this occurs, the issue does not originate from our infrastructure.

Required actions:

• Apply a silence on the alert.
• Open an incident/ticket.see here
• Notify the client by email about the impacted period use Msurvey

Exemple Client notification email template:

Bonjour,
Vous utilisez la fonction Alerting hub pour le routage des alertes NGOT en direction de Canopsis.
Nous avons eu un incident de routage aujourd'hui le XX/XX/2025 de 03hXX à 08hXX.
Durant cette période aucun routage n'a pu être effectué en direction de Canopsis.
Nous sommes désolés pour le désagrément occasionné et nous restons à votre disposition pour toute information complémentaire via notre ServiceDesk.
Cordialement,
L'équipe Support Ngot.

Alertinghub/webhook code 404

• Connect to Grafana prdcasa
• Go to explore tab and select the code mode
• Select the datasource Loki-client
• Find out the CUSTOMER_NAME for the logQL below. For example, for the namespace monitoring-stack-client-obs-osp, the CUSTOMER_NAME is osp

• Retrieve the error message and the list of labels of the customer alert. Insert the following code into Log browser and replace CUSTOMER_NAME with the client name:

  {namespace="nginx-alertreceiver", cc_prom_source="prdcasa", alerts_source="CUSTOMER_NAME"} | json | line_format `{{ .request_body  }}` | json | status_extracted=`firing` | json labels=`alerts[0]["labels"]` | line_format `{{ .resp_body }}` | json | code=`404` | line_format `{"labels":{{ .labels  }}, "message":"{{ .message }}" }`
  

  Example with osp:

  {namespace="nginx-alertreceiver", cc_prom_source="prdcasa", alerts_source="osp"} | json | line_format `{{ .request_body  }}` | json | status_extracted=`firing` | json labels=`alerts[0]["labels"]` | line_format `{{ .resp_body }}` | json | code=`404` | line_format `{"labels":{{ .labels  }}, "message":"{{ .message }}" }`
  

• Click on Prettify JSON (optional)

• The message should explain the problem. If a label is missing, the list of labels should show that it is missing.

  Verify that all the labels from the client's dictionary (dictionary customers) appear in the log. If some are missing, the client needs to be contacted to add or fix the missing labels.

  Note

  If something goes wrong with the LogQL above, you can try a shorter logQL:

  {namespace="nginx-alertreceiver", cc_prom_source="prdcasa"} |= `CUSTOMER_NAME`
  

  Example for osp:

  {namespace="nginx-alertreceiver", cc_prom_source="prdcasa"} |= `osp`
  

  • Search label resp_body with code=404 and a message. Double check that message.

  • Search label request_body beginning with

    "{\"alerts\":[{\"status\":\"firing\",\"fingerprint......'
    

  • Put it locally on your laptop to explore it with the following command:

    cat body.json | jq -r . | fx
    

  • Find the labels to check in alerts[].labels. Then check them with the client's dictionary as described above.

  • When the message is similar to this (notice the mc_host field does not exist in enrichment file, please viery your CMDB declaration) :

    time=2026-06-25T16:54:08.710Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed"
    component=dispatcher num_alerts=1 err="alertinghub/webhook[0]: notify retry canceled due to unrecoverable
    error after 1 attempts: unexpected status code 404: \n{\"code\":404,\"message\":\"Assets name provided
    for mc_host field does not exist in enrichment file, please verify your CMDB declaration (DNT\\/2IM)\"}" 
    

    1. Connect to the client grafana https://grafana.obs-<client>.cloudservicesfactory.com
    2. In alerting > silences : put a silence on the alert with a wrong CSU. Duration: 3 months.
    3. Send a message to the client using this template :

       Objet : [NGOT]["TO UPDATE"] CSU "TO-UPDATE"
       
       Bonjour,
       
       Vous utilisez le routage des alertes Ngot en direction de TrueSight mais depuis un petit moment nous observons un dysfonctionnement avec l'usage d'un CSU non présent coté TrueSight.
       
       Vous mentionnez le CSU "TO_UPDATE" dans les alertes, cependant, d'après les outils dont nous disposons pour valider les CSU, celui-ci ne figure pas.
       
       Pouvez-vous vérifier de votre côté la correction du label CSU, que ce soit depuis Ngot ou côté TrueSight, afin d'assurer la cohérence avec le même CSU ?
       
       Afin de ne pas solliciter inutilement l'astreinte, un silence de 3 mois a été mis en place sur les alertes associées au CSU "TO_UPDATE" visible [ici|https://grafana.obs-"TO_UPDATE".cloudservicesfactory.com/alerting/silences]. Lorsque vous aurez corrigé le problème, libre à vous de supprimer ce silence.
       
       Nous restons disponibles pour toute information complémentaire.
       
       
       Cordialement,
       Support NGOT
       

Alertmanager with code 401

When this error happens, first check the configuration of nginx-alertreceiver.

kswitch prdcasa

kubectl -n nginx-alertreceiver exec -it svc/nginx-alertreceiver-ingress-nginx-controller -- cat /etc/nginx/nginx.conf > /tmp/nginx.conf

Le fichier /tmp/nginx.conf correspond à la configuration de NGinx.

⚠️ Ce fichier contient les APIKeys. ⚠️ Ne pas divulguer ce fichier. Penser à le supprimer après usage ! ⚠️

grep -C5 <nom du client> /tmp/nginx.conf

Cela devrait afficher au moins 2 lignes, accompagnée de leur contexte. Le résultat attendu a cette forme mais, pour le client air, on observe le problème : la clé manque (apikey= et $code="").

            set $code "<redacted>";
                set $cc_client "sha";
                set $resp_body "";
                set $ngot true;
            }
            if ($request_uri = "/alertreceiver/air?apikey=") {
                set $code "";
                set $cc_client "air";
                set $resp_body "";
                set $ngot true;
            }
            if ($request_uri = "/alertreceiver/cug?apikey=<redacted>") {
                set $code "<redacted>";
--
                set $code "<redacted>";
                set $cc_client "sha";
                set $resp_body "";
                set $ngot true;
            }
            if ($request_uri = "/alertreceiver/air?apikey=") {
                set $code "";
                set $cc_client "air";
                set $resp_body "";
                set $ngot true;
            }
            if ($request_uri = "/alertreceiver/cug?apikey=<redacted>") {
                set $code "<redacted>";

Résolution : dans ce cas, il suffit de relancer Trackbone :

cd contexts/pf
trackbone apply -z prdcasa -c nginx-alertreceiver --add-services

L'origine d'un tel incident, s'il ne concerne qu'un seul client, peut venir d'un problème lors d'un déploiement précédent. Cette hypothèse reste à valider si le problème revient trop souvent.

⚠️ Supprimer le fichier /tmp/nginx.conf s'il ne sert plus. ⚠️

Alertmanager with code 406

The error may come from a problem:

• with client alerts
• or with dictionary configuration

Steps for identify problem

1. Check pods are running

   kubectl get pods -n <monitoring_namespace>
   

2. Check logs in Alertmanager 0 and 1

   kubectl logs <alertmanager_pod> -n <monitoring_namespace>
   

   If you have found logs something like this:

   err="alertinghub/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 406: https://alertreceiver.prdcasa.caascad.com/alertreceiver/XXXX?apikey=XXXXXXXXXXXXXXXX: {\"code\":406,\"message\":\"Mandatory dictionary fields are missing : alerts.labels.csu,alerts.labels.instance, please check the API documentation\"}"
   

3. Check Truesight dictionary

   Follow the verification documentation.

   If you do not have access rights, you need to check if you have a good proxy.

   If you do not have access, you can check Check rules costumer.

4. Check Grafana

   You need to go to the Grafana customer website: Home > Alerting > Groups, then choose 'Alertmanager' > 'alertmanager'.

   This step will help you verify the customer label.

Solution for problem with client alerts

1. Do a temporary fix:

   • add a silence on alert

   • or edit the Prometheus rules:

     kubectl edit prometheusrules -n rules-ocb-<xxxx> -o yaml <alert_name>.alert
     

     Add the missing label.

     Warning: this change is temporary and will be overwritten on the next deployment.

2. Send an email to the customer (see template below)

Solution for problem with dictionary configuration

Create a SWAN ticket.

Alertmanager with code 403

Symptom

Alertmanager receives an alert, but the receiver, such as Alerta, rejects the notification with an HTTP 403 error.

time=2026-07-24T09:59:32.520Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=4 err="webhook-dtfb-alerta/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 403: {\"code\": 403, \"errors\": null, \"message\": \"[POLICY] Alert environment does not match one of poc, itg, dev, prp, prod, pra, unknown\", \"requestId\": \"1e56fe8d-f8a7-4853-99a7-df9fdfd18893\", \"status\": \"error\"}"

Root cause

An alerting rule usually inherits the labels from the metric returned by its PromQL expression.

Example:up{job="node"} == 0

If the up metric contains a wrong label value, the generated metric will also contains this label. Alertmanager then forwards it to the receiver.

The receiver rejects the notification because one of it's labels that does not have an accepted value.

Diagnosis

1. Identify alerts with invalid environment values.

   In Grafana client instance (https://grafana.obs-\<client>.cloudservicesfactory.com/), identify the alerts that do not have the expected value as indicated in alertmanager logs (exemple:[POLICY] Alert environment does not match one of poc, itg, dev, prp, prod, pra, unknown\")

2. Check the affected metric in Grafana client instance (https://grafana.obs-\<client>.cloudservicesfactory.com/): up{<label>="<unaccepted_value"}.

   This confirms whether the invalid label is already present on the metric.

Resolution

Ask the client to fix the label value and set a silence of 1 year in the client Grafana instance (https://grafana.obs-\<client>.cloudservicesfactory.com/).

Appendix

Template for sending to a client:

Bonjour à tous,

Nous prenons contact avec vous aujourd'hui pour vous informer d'un problème identifié concernant une des prometheus-rules déployées dans votre système Prometheus.

Suite à notre analyse, nous avons constaté que les notifications vers alertinghub pour une de vos règles Prometheus ne fonctionnent pas correctement.

Il s'agit de l'alerte <le nom de l'alerte> dans la prometheusrule <le nom de l'objet prometheusrules qui regroupe plusieurs alertes> :

<La rule qui pose problème>

Avec le dictionnaire de translation pour alertinghub mis en place, le label `service` est nécessaire. Ce qui n'est pas le cas avec cette alerte.

Pour corriger vous pouvez :

<Proposition de solution>

En attendant, nous avons <modifié directement l'alerte|mis un silence sur votre alerte>. Mais ce changement est temporaire et sera écrasé au prochain déploiement.

Serait-il possible pour vous de revoir l'alerte et de procéder aux ajustements nécessaires en conséquence.

Nous restons à votre entière disposition pour vous aider dans ce processus et répondre à toutes vos questions.

Cordialement,

Support Team