Aller au contenu

KubeStateMetricsWatchErrors#

This alert triggered when Kube-State-Metrics fails to watch resources.

Troubleshooting Hints#

kubectl get pods -n  <monitoring_namespace> | grep kube-state-metrics  # check pod is up and running
kubectl logs <kube-state-metrics> -n <monitoring_namespace>

Tip

Where <monitoring_namespace> can be:

  • Zone Infra/Cloud : monitoring
  • Zone Client : caascad-monitoring
  • monitoring

We can identify in the logs which resource is involved, so we can do:

kubectl get RESOURCETYPE --watch OBJECTNAME

Known cases#

Here an example related to Ingress (resources). So it can happen for any other resource.

Failed to watch *v1.Ingress: failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:monitoring:caascad-kube-state-metrics" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope

We clearly can see that the user "system:serviceaccount:monitoring:caascad-kube-state-metrics" has not the right to watch ingress.

So we can check the kube-state-metrics clusterrole ingress authorizations.