Aller au contenu

ThanosPVCNotAutoCleaned#

This alert is triggered when the persistent volume attached to thanos-compactor use more than 100MB for more than 10 hours.

Context#

After every operations (compaction, downsampling and expired metrics deletion) are performed, Thanos delete all the data available in the PVC.

Troubleshooting Hints#

With command line#

kubectl get pods -n  <monitoring_namespace> | grep compact  # check pod is up and running
kubectl logs <thanos_compact> -n <monitoring_namespace>

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

With Grafana#

Connect to the Grafana, go to dashboard -> Kubernetes -> persistent volume, and select datasource Thanos by specifying the target namespace and the persistent volume claim.

  • If the used space is above 100MB for more than 10 hours, that probably means the thanos-compactor does not work correctly.
  • If the used space is above 100MB for less than 10 hours, that probably because thanos is performing operational task.
  • If the used space is below 100MB, that means thanos-compactor works perfectly.

Solution#

Delete the PVC attached to Thanos#

kubectl -n <monitoring_namespace>  delete pvc -l app.kubernetes.io/component=thanos-compactor &
kubectl -n <monitoring_namespace>  delete pod thanos-<monitoring_namespace>-compactor-0

Check#

Check if the pod thanos-compactor and the PVC are running correctly.

kubectl -n <monitoring_namespace>  get pvc -l app.kubernetes.io/component=thanos-compactor
kubectl -n <monitoring_namespace>  get pod thanos-<monitoring_namespace>-compactor-0