PrometheusOperatorNotReady

Prometheus-Operator is not ready to reconcile some resources.

Known cases

Rancher / kub-8 / kub-10008

You are in this case if these 3 conditions are filled

1. Rancher is also running on this cluster (e.g. kub-8 and kub-10008)
2. In Prometheus-Operator logs, there are messages like this:

   2025-03-25T10:55:01.173988311+01:00 stdout F ts=2025-03-25T09:55:01.173923224Z level=info caller=/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:561 msg="pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list *v1.Alertmanager: the server could not find the requested resource (get alertmanagers.monitoring.coreos.com)"
   

   where the important part is the server could not find the requested resource (get alertmanagers.monitoring.coreos.com)
3. the alertmanager.monitoring.coreos.com is present on the server:

   kubectl get crd | grep alertmanager
   

Additionnal check:

kubectl -n monitoring get pod -l app.kubernetes.io/instance=prometheus-operator -o yaml | grep startTime:

The start time of Prometheus-Operator is posterior to the deployment of the alertmanager CRD. Otherwise, you are probably in this case but more investigating is needed to understand.

How to fix?

1. Delete the CRD: kubectl delete crd alertmanagers.monitoring.coreos.com
2. Restart Prometheus-Operator

Explanation:

Rancher seems to need the alertmanagers.monitoring.coreos.com CRD and is able to deploy it itself.

When this CRD is deployed, Prometheus-Operator seem to have more needs (we have not investigated this yet) and will stay unready.