Prometheus#

Prometheus is the monitoring tool. It will scrape everywhere it can. Then store the metrics and check rules for alerting.

Deployment method#

Deploy the helm chart caascad-kube-prometheus-stack with Trackbone.

Then Prometheus-Operator will notice the new Prometheus CRD and deploy Prometheus.

Delete a prometheus pod and it will respawn. The data are on the pv/pvc and will not be lost. Thanos sidecar will also restart without any problem.

If one pod remains, the service is up with no data loss.

If all pods are stopped at the same time :

data is not collected (can be considered as a data loss because there will be a hole in the graphs)
no data loss (data on the pv/pvc and in S3 is not lost)
grafana cannot show recent metrics (older metrics are available thanks to Thanos)