TargetDown#
Check if job metrics are scraped by prometheus
Note
When this alert happens, at this step,you don't know yet in which case you are:
- Prometheus fails to gather metrics on the target (the target may still be running, **no client impact**)
- the target is down (the service is down, **client impact**)
Troubleshooting hints#
Step 1 : identify if you are in a known situation#
job=federate-\[cluster-id\>: FederatePrometheusDown
If you are not in a known situation, you need to go on with this generic procedure.
Step 2 : identify the target#
Identify on which prometheus the target is.
If label cc_prom_source is set, it is on Caascad.
If label cluster=kub-xxx is set, it is on NGOT.
Caascad#
| cc_prom | cc_prom_source | Prometheus |
|---|---|---|
| infra-XXX | whatever | on the infra-XXX Prometheus : see cc_prom |
| cloud-caascad | whatever | cloud-caascad |
| cloud-client | on the client cluster : see cc_prom_source | |
| cloud-app | this should not happen | this should not happen |
First check cc_prom. If the problem is in infra zone or cloud zone, you do not need cc_prom_source to identify which Prometheus failed to join the target.
In cases where the problem is on Client Clusters, the cc_prom is cloud-client and the cc_prom_source should not be missing. In that case, the problem is on the cc_prom_source Prometheus.
NGOT#
The main information are the labels cluster, namespace and job.
Step 3 : troubleshoot#
Go on troubleshooting with the documentation Troubleshooting Prometheus target.