Aller au contenu

IngressControllerMetricsMissing#

NGOT#

Alert is triggered when nginx_ingress_controller_leader_election_status metric is missing from Prometheus-cluster.

This alert either indicates that:

  • Ingress-controller does not return the nginx_ingress_controller_leader_election_status metric
  • Or Prometheus-cluster does not scrape ingress-controller's metrics.

Warning

To manage the high volume of metrics generated by the Ingress Controller:

  • Metrics Dropped: We have intentionally dropped all nginx_ingress_controller_.* metrics during the remote write process to the central Prometheus zones.
  • Reason: These metrics can create a large number of time series, which can potentially overload the monitoring system and complicate data management.
  • Scraping Status: Note that these metrics are still scraped by the Prometheus clusters but are excluded from being written to the central Prometheus zones.
  • Impact: As a result, these metrics are not visible in Grafana or other centralized monitoring tools.

Troubleshooting Hints#

The value for NAMESPACE can be obtained from the label associated with the alert.

Tip

NAMESPACE can be:

  • ingress-nginx-v2-private
  • ingress-nginx-v2-public
  • ingress-nginx-v2-proxy
  • ingress-nginx-v2-loki
  • ingress-nginx-v2-signon
  • ingress-nginx-v2-signon-stg
  • ingress-nginx-v2-signon-test
  • ingress-nginx-v2-billing
  • ingress-nginx-v2-bastion

Case: Ingress-controller does not return metrics#

To check if ingress-controller metrics are available:

kubectl get svc -n ${NAMESPACE} # service should be present
kubectl port-forward svc/ingress-nginx.* -n ${NAMESPACE} 10254:10254 &
curl -sL https://localhost:10254/metrics
curl -sL https://localhost:10254/metrics | grep -o '"Name":"nginx_ingress_controller_leader_election_status"'

If metrics are not available, try to find the reason by checking status/logs of ingress-nginx:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=ingress-nginx
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=ingress-nginx

Case: Prometheus-cluster does not scrape ingress-controller's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE}

  • check the target, by doing a port-forward on the Prometheus-cluster

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &

    Connect to http://localhost:12345/targets, search ingress-nginx target, and verify error message.

Caascad#

Alert is triggered when nginx_ingress_controller_leader_election_status metric is missing from Caascad-Prometheus.

This alert either indicates that:

  • Ingress-controller does not return the nginx_ingress_controller_leader_election_status metric
  • Or Caascad-Prometheus does not scrape ingress-controller's metrics.

Troubleshooting Hints#

The value for NAMESPACE can be obtained from the label associated with the alert.

  • infra Zone
NAMESPACE=ingress-nginx-v2-private
  • Cloud Zones (Cluster version < 1.23)
NAMESPACE=ingress-nginx-private
NAMESPACE=ingress-nginx-public
  • Cloud Zones (Cluster version ≥ 1.23)
NAMESPACE=ingress-nginx-v2-private
NAMESPACE=ingress-nginx-v2-public

Case: Ingress-controller does not return metrics#

To check if ingress-controller metrics are available:

kubectl get svc -n ${NAMESPACE} # service should be present
kubectl port-forward svc/ingress-nginx.* -n ${NAMESPACE} 10254:10254 &
curl -sL https://localhost:10254/metrics
curl -sL https://localhost:10254/metrics | grep -o '"Name":"nginx_ingress_controller_leader_election_status"'

If metrics are not available, try to find the reason by checking status/logs of ingress-nginx:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=ingress-nginx
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=ingress-nginx

Case: Caascad-Prometheus does not scrape ingress-controller's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE}

  • check the target, by doing a port-forward on the Caascad-Prometheus

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/caascad-prometheus 12345:9090 &

    Connect to http://localhost:12345/targets, search ingress-nginx target, and verify error message.