PrometheusTSDBWALCorruptions

The counter is only increased when tsdb.Open() is called. Prometheus calls it only once in its lifetime (when it initializes). If the corruption can't be recovered, tsdb.Open() returns an error and Prometheus exits. Hence the metric is either 0 (no corruption) or 1 (corruption detected and repaired).

Solutions

• First restart Prometheus pod (with kubectl delete)
• Then, if Prometheus fails at repairing the WAL (after a restart), you have to delete the entire segment causing the problem from the WAL file (before compression). The easier way to do it is to delete the PV/PVC itself.
  • Delete the pod,pvc,pv
  • When Prometheus restarts automatically, it may want to use the old pv/pvc (because Kubernetes is not as fast as you) and fail. Delete again the pod Prometheus and it will restart automatically and create new PV/PVC.

Note : You can safely delete the PV/PVC for 2 reasons :

• there is another Prometheus pod running, so the data is not lost as soon as the other Prometheus pod is running.
• data is "backuped" every 2 hours thanks to Thanos that copy it to the S3 bucket. It's better to restart Prometheus asap and have fresh data than trying to recover the non-corrupted data of the past 2 hours.