Aller au contenu

NginxAlertreceiverCustomMetricMissing#

This alert is triggered when NGINX ingress custom metric nginx_alertreceiver_requests is missing for more than 5 minutes.

It is a metric generated by Promtail from the logs.

Troubleshooting hints#

Servicemonitors#

  • Check that servicemonitor is present. 
    kubectl get servicemonitor -n caascad-logging | grep promtail
  • Check if the servicemonitor is healthy and contains the good labels. 
    kubectl get servicemonitor -l caascad.com/prometheus-monitor=caascad -n caascad-logging

  • Check that Promtail metrics are available in Grafana.

    Tip

    Don't forget to select the correct Datasource in Grafana: Thanos-client.

  • Check if the metrics from Promtail are available 

    kubectl port-forward svc/caascad-promtail-headless -n caascad-logging 3101:3101 &
curl -sL http://localhost:3101/metrics # if there are no metrics available, the problem is on Promtail side

  • Check if Promtail custom metric are available in Prometheus 

    kubectl port-forward svc/caascad-prometheus -n caascad-monitoring 9090:9090 &
    Access Prometheus URL (http://localhost:9090) and check:

  • In the main menu, metrics for Promtail (example promtail_build_info metric), should be available.

  • In Status > Targets menu, Promtail target should be up. If it's not, there are two use cases:

    1. If the target is present but in failure state (click on Unhealthy button), check the logs to find the reason for which the target is failing.

    2. The target is absent, in this case you need to check why the target is not discovered by Prometheus.

Logs#

  • Check the logs of the pods. We must have logs in this format : 
    {"time":"2023-01-06T15:38:27+00:00","ingress_name":"nginx-alertreceiver","cc_client":"ocb-test05","alerts_source":"ocb-test05","remote_addr":"","x_forwarded_for":"172.16.0.49","request_id":"23d993e016aa32e5a68a772263a4181a","remote_user":"","bytes_sent":"325","request_time":"0.063","status":"401","vhost":"alertreceiver.prdcasa.caascad.com","request_proto":"HTTP/1.1","path":"/v1/test","request_query":"apikey=VEP8wbYbf2S1GGn5mMkxzvftIM4Li3au","request_length":"1175","duration":"0.063","method":"POST","http_referrer":"","http_user_agent":"Alertmanager/0.23.0","request_body":"{\"receiver\":\"nginx-alertreceiver\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"TestAlert\",\"cc_client\":\"ocb-test05\",\"cc_prom\":\"cloud-app\",\"host\":\"test\",\"parameter\":\"test\",\"prometheus\":\"monitoring-app/app-prometheus\",\"severity\":\"test\"},\"annotations\":{\"description\":\"Ceci est un test\"},\"startsAt\":\"2022-12-26T10:28:13.225Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://app-prometheus.monitoring-app:9090/graph?g0.expr=vector%281%29\\u0026g0.tab=1\",\"fingerprint\":\"16b142006458929f\"}],\"groupLabels\":{\"alertname\":\"TestAlert\",\"cc_prom\":\"cloud-app\"},\"commonLabels\":{\"alertname\":\"TestAlert\",\"cc_client\":\"ocb-test05\",\"cc_prom\":\"cloud-app\",\"host\":\"test\",\"parameter\":\"test\",\"prometheus\":\"monitoring-app/app-prometheus\",\"severity\":\"test\"},\"commonAnnotations\":{\"description\":\"Ceci est un test\"},\"externalURL\":\"http://app-alertmanager-1:9093\",\"version\":\"4\",\"groupKey\":\"{}/{}:{alertname=\\\"TestAlert\\\", cc_prom=\\\"cloud-app\\\"}\",\"truncatedAlerts\":0}\n","resp_body":"{\"code\":401,\"message\":\"APIKey provided in Auhtorization header is not authorised, please check the API documentation\"}"}

Client communication#

If the custom metrics are missing due to an unavailability of NGINX ingress, a client communication needs to be done.

If NGINX ingress fails to forward alerts to AlertReceiver(CASA), impacted client needs to be informed. Detailed procedure regarding client contacts/communication here.

If NGINX ingress fails to forward alerts due to AlertReceiver(CASA) issue, it is necessary to contact the team in charge of this server via an OCEANE/SWAN ticket. Detailed procedure regarding ticket creation here.