NginxAlertreceiverMetricsMissing

This alert is triggered when NGINX ingress metrics are missing for more than 5 minutes.

NGINX Ingress Controller documentation here.

Troubleshooting hints

Servicemonitors

• Check that servicemonitor is present.

  kubectl get servicemonitor -n nginx-alertreceiver
  

• Check if the servicemonitor is healthy and contains the good labels.

  kubectl get servicemonitor -l caascad.com/prometheus-monitor=caascad -n nginx-alertreceiver
  

• Check that NGINX ingress metrics are available in Grafana.

  Tip

  Don't forget to select the correct Datasource in Grafana: Thanos-client.

  • Check if the metrics from NGINX ingress are available

    kubectl get service -n nginx-alertreceiver | grep metrics # get the name of the service
    kubectl port-forward svc/nginx-alertreceiver-ingress-nginx-controller-metrics -n nginx-alertreceiver 10254:10254 &
    curl -sL http://localhost:10254/metrics # if there are no metrics available, the problem is on NGINX ingress side
    

  • Check if NGINX ingress metrics are available in Prometheus

    kubectl port-forward svc/caascad-prometheus -n caascad-monitoring 9090:9090 &
    

    Access Prometheus URL (http://localhost:9090) and check:

• In the main menu, metrics for NGINX ingress (example nginx_alertreceiver_requests metric), should be available.

• In Status > Targets menu, NGINX ingress target should be up. If it's not, there are two use cases:

  1. If the target is present but in failure state (click on Unhealthy button), check the logs to find the reason for which the target is failing.

  2. The target is absent, in this case you need to check why the target is not discovered by Prometheus.

  One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present :

  kubectl get servicemonitor -n nginx-alertreceiver | grep nginx
  

Client communication

If the custom metrics are missing due to an unavailability of NGINX ingress, a client communication needs to be done.

If NGINX ingress fails to forward alerts to AlertReceiver(CASA), impacted client needs to be informed. Detailed procedure regarding client contacts/communication here.

If NGINX ingress fails to forward alerts due to AlertReceiver(CASA) issue, it is necessary to contact the team in charge of this server via an OCEANE/SWAN ticket. Detailed procedure regarding ticket creation here.