Aller au contenu

PrometheusBadConfig#

This alert is triggered when Prometheus is misconfigured.

Troubleshooting Hints#

kubectl get pods -n  <monitoring_namespace> | grep prometheus  # check pod is up and running
kubectl logs <prometheus> -n <monitoring_namespace> -c prometheus

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

When this happens on client clusters, in most cases, it is a bad configured serviceMonitor or podMonitor. The investigation should lead to identify the bad serviceMonitor or podMonitor. The solution is to fix the serviceMonitor or the podMonitor.

Danger

When the problem is a bad serviceMonitor or podMonitor deployed by the client, we also need to fix :

  • try to fix the problem with kubectl edit
  • when kubectl edit won't fix, delete the serviceMonitor/podMonitor

In all cases, the client must be notified as soon as possible.

Known cases#

scrape timeout greater than scrape interval#

Here is the log :

level=error ts=2021-11-26T15:01:06.200Z caller=main.go:608 msg="Error reloading config" err="couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: scrape timeout greater than scrape interval for scrape config with job name \"somenamespace/someservicemonitor/0\""

In most cases, the job name is of the form namespace/servicemonitor/X (where X is a number).

You can try one of

kubectl get servicemonitor -A | grep someservicemonitor
kubectl -n somenamespace get servicemonitor someservicemonitor

This will identify the servicemonitor and its namespace.

Now check its contents :

kubectl -n somenamespace get servicemonitor someservicemonitor -o yaml

You can identify the problem : the scrapeTimeout value is greater than the interval which is bad.

If the serviceMonitor is owned by Caascad or NGOT team, just fix the serviceMonitor and check that the fix solved the issue.

If the serviceMonitor is owned by a client :

  • contact the client ASAP and explain the problem
  • if the client does not fix the problem quickly, update the servicemonitor (kubectl edit servicemonitor) and set the interval as the same value as the scrapeTimeout
  • communicate the operation to the client

Note

Prometheus fails to reload the new configuration. Here are the impacts :

  • Prometheus continues to run with the older correct configuration (e.g. no direct impact)
  • if Prometheus ends, it will not be able to reload its new configuration (e.g. loss of data : collection of metrics is broken). This can happen at any moment in Kubernetes.