Aller au contenu

PrometheusDown#

Triggers when Prometheus it not running.

Troubleshooting Hints#

Prometheus state#

kubectl get pods -n  <monitoring_namespace> | grep prometheus  # check pod is
kubectl logs prometheus-<name>-prometheus-0  -n <monitoring_namespace>

Tip

Where <monitoring_namespace> and <name>can be:

Notice the namespace and pod label in the alert

  • monitoring_namespace:caascad-monitoring name: caascad
  • monitoring_namespace:monitoring name: caascad
  • monitoring_namespace:monitoring-client name: client
  • monitoring_namespace:monitoring-app name: app
  • monitoring_namespace:monitoring name: caascad
  • monitoring_namespace:monitoring-consumption name:consumption

Check if the pod is up and running or if it is restarting in loop (check the containers restart number) and see its state. Usually a

state: Terminated 
reason: error 
exit code: 137

indicates an OOMKILLED in which case you should check the limit requests CPU/MEM.

Federation problems (NGOT)#

This alert checks the metric prometheus_build_info. This alert is not sent from Prometheus (both "cluster" and "client") but is federated from central Prometheus on kub-53/mon3 and kub-34/mon4 (or, on Staging, kub-10002/mon3 and kub-10010/mon4).

Checks:

  • Check if Prometheus (cluster) in monitoring namespace is running.
  • Check if Prometheus (central) in monitoring-stack-corp-obs-corp-prd (or monitoring-stack-corp-obs-corp-stg) on kub-53/mon3 and kub-34/mon4 (or kub-10002/mon3 and kub-10010/mon4) are running.
  • Check in Prometheus (central) the targets (do a port forward, more information here). Check if the target exists. Check its state. If the target is unhealthy, Prometheus may provide an explanation.
  • Check if the ScrapeConfig federate-metrics-mom that defines the federation exists and is correctly defined.
  • Check the Secret federate-metrics-mom that defines the credentials to connect to Prometheus (cluster): is it correctly defined ? Are these the correct credentials ?

Tip

The secret is defined in Vault in secret/applications/prometheus/cluster-federate.