FederatePrometheusDown#

This alerts appears when the federation on one of the prometheus is no longer functional.

Several use cases:

federation beetwen prometheus cloud-client of cloud zone and prometheus of client zone: identify by cc_prom_source and exported_job
federation beetwen prometheus cloud-app of cloud zone and prometheus of client zone : identify by cc_prom_source and exported_job
federation beetwen prometheus infra-consumption of infra zone and prometheus cloud-client of cloud-zone : identify by cc_prom_source and exported_job

Troubleshooting#

In the target zone, check that Prometheus is not in a terminating state.

In the target zone, check Prometheus log. Example for client zones :

kubectl logs -n caascad-monitoring prometheus-caascad-prometheus-0

If you see these kind of messages:

2022-01-10 13:53:08 level=error ts=2022-01-10T12:53:08.254Z caller=federate.go:192 component=web msg="federation failed" err="write tcp 172.16.0.221:9090->172.16.0.1:42361: write: broken pipe"

Verify scrape_duration_seconds{job="federate-<cc_prom_source>"} metric in Thanos-client datasource and compare with scrapeTimeout in prometheus-federation.cue.

On the Prometheus that perform the federation, check the error message for the failed federation target.
Note

The zone that performs the federation (zone_prometheus_that_federates) can be one of the following types:
```
- cloud-client
- cloud-app
- infra
- infra-consumption.
```
In order to do this :
```
kswitch <zone_prometheus_that_federates>
kubectl port-forward svc/<prometheus_service> -n <prometheus_namespace> 9090:9090 &
```
Where prometheus_service can take one of the following values :
- client-prometheus for cloud-client zones ;
- app-prometheus for cloud-app zones ;
- caascad-prometheus for infra zones ;
- consumption-prometheus for infra-consumption zones.
And prometheus_namespace can take one of the following values :
- monitoring-client for cloud-client zones ;
- monitoring-app for cloud-app zones ;
- monitoring for infra zones ;
- monitoring-consumption for infra-consumption zones.
Go to http://localhost:9090/targets, select Unhealthy targets and click on the endpoint that is in state DOWN and check the error.

If the error indicates a network issue, check known casses section below. Example :
```
{
  "status": "Failure",
  "message": "error trying to reach service: dial tcp x.y.z.w:9090: connect: no route to host",
  "code": 500
}
```
Check that the IP indicated in the error message is the one of the target Prometheus :
```
kswitch <target_zone>
kubectl get pods -A -o wide | grep x.y.z.w
```
Note

The zone whose metrics are being federated target_zone can be one of the following types:
- cloud-client ;
- client.
Check that Rancher is healthy. If it is not healthy, please check known casses section below.
Go to check TargetDown.

Solutions#

If the target Prometheus is not in a correct state, a solution may be to delete the pod and let the StatefulSet regenerate it : kubectl delete pod -n caascad-monitoring --force --grace-period 0 prometheus-caascad-prometheus-0
If the scrapping duration exceeds the timeout, modify scrapeTimeout in prometheus-federation.cue for the zone that makes the federation.
Warning

scrapeTimeout must always be smaller than the interval. Which implies that the interval must be changed as well. Example :
```
_prometheusFederationEndpoint: {
interval:      "30s"
scrapeTimeout: "25s"
}
```

Known cases#

If at troubleshooting point 3 you obtain errors pointing network issues (like no route to host), it can be due to Rancher agent on the target cluster.

In this case the resolution consist to make a rollout of the Rancher agent :

kswitch <target_zone>
kubectl -n cattle-system rollout restart deployment.apps/cattle-cluster-agent

Then check that the pods in cattle-system namespace have started correctly.