Aller au contenu

CertManagerMetricsMissing#

Alert is triggered when certmanager_controller_sync_call_count metric is missing for more than 5 minutes.

This alert either indicates that:

  • Cert-manager does not return the certmanager_controller_sync_call_count metric

  • Or Prometheus-cluster does not scrape cert-manager's metrics.

Troubleshooting Hints#

The value for NAMESPACE can be obtained from the label associated with the alert.

Case: Cert-manager does not return metrics#

To check if cert-manager metrics are available:

kubectl get svc -n ${NAMESPACE} # service should be present
kubectl port-forward svc/cert-manager -n ${NAMESPACE} 9402:9402 &
curl -sL https://localhost:9402/metrics
curl -sL https://localhost:9402/metrics | grep -E '^certmanager_controller_sync_call_count'

If metrics are not available, try to find the reason by checking status/logs of cert-manager:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=cert-manager
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=cert-manager

Case: Prometheus does not scrape cert-manager's metrics#

  • check status/logs of Prometheus

Tip

kubectl get pods -n monitoring | grep prometheus-caascad
kubectl logs <prometheus-caascad_pod> -n monitoring

kubectl get pods -n monitoring | grep prometheus-cluster
kubectl logs <prometheus-cluster_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE} cert-manager

  • check the target, by doing a port-forward on the Prometheus

Tip

kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl port-forward svc/caascad-prometheus -n monitoring 9090:9090 &

kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl port-forward svc/cluster-prometheus -n ${NAMESPACE} 9090:9090 &

Connect to http://localhost:9090/targets, search cert-manager target, and verify error message.

Warning

Be careful of clear selection before filtering on cert-manager.