Aller au contenu

ThanosPVCNotAutoCleaned#

This alert is triggered when the persistent volume attached to thanos-compactor use more than 100MB for more than 10 hours.

Context#

After every operations (compaction, downsampling and expired metrics deletion) are performed, Thanos delete all the data available in the PVC.

Troubleshooting Hints#

With command line#

kubectl get pods -n  <monitoring_namespace> | grep compact  # check pod is up and running
kubectl logs <thanos_compact> -n <monitoring_namespace>

Tip

Where <monitoring_namespace> can be:

  • Zone Cloud: monitoring, monitoring-client or monitoring-app
  • Zone Infra: monitoring or monitoring-consumption
  • Notice the namespace label in the alert

With Grafana#

Connect to the Grafana, go to dashboard -> Kubernetes -> persistent volume, and select datasource Thanos by specifying the target namespace and the persistent volume claim.

  • If the used space is above 100MB for more than 10 hours, that probably means the thanos-compactor does not work correctly.
  • If the used space is above 100MB for less than 10 hours, that probably because thanos is performing operational task.
  • If the used space is below 100MB, that means thanos-compactor works perfectly.

Solution#

Delete the PVC attached to Thanos#

kubectl -n <monitoring_namespace>  delete pvc -l app.kubernetes.io/component=thanos-compactor &
kubectl -n <monitoring_namespace>  delete pod thanos-<monitoring_namespace>-compactor-0

Check#

Check if the pod thanos-compactor and the PVC are running correctly.

kubectl -n <monitoring_namespace>  get pvc -l app.kubernetes.io/component=thanos-compactor
kubectl -n <monitoring_namespace>  get pod thanos-<monitoring_namespace>-compactor-0

If the alert persists#

There is one known case where the alert persists. It happens when the following sequence of events occurs:

  1. The PVC is deleted as describe above
  2. The pod is restarted. The compactor fills quickly the PVC above the threshold
  3. The pod is killed again.

This can be seen in Grafana with these PromQL expressions:

kube_pod_container_info{pod="thanos-monitoring-compactor-0"}

kubelet_volume_stats_used_bytes{job="kubelet",namespace=~"monitoring.*",persistentvolumeclaim=~"thanos-compactor.*",instance="xxxxx"}

Note

  • First expression: Change the pod name if not in monitoring namespace (it has another name in other namespaces)
  • Second expression: change the namespace to narrow the search of the correct instance. Then set the instance to the one that corresond to the pod found above (it is not the same label value, this is normal)

Explanation: in that case, in the expression of the alert with kubelet_volume_stats_used_bytes, there is neither "before" nor "after" low values and min_over_time() is always above the threshold.

Solutions:

  • put a silence on the alert for 26h after the pod restarted again (you need to know when the pod was killed with the expressions above).
  • wait less than 26h and check the alert presence again. It should disapper after the metrics are dropped in Prometheus (retention: 1d)

Note

The alert should be reworked to aggregate the kubelet_volume_stats_used_bytes metric by some labels. If the problem appears again, please open a ticket to the monitoring team to rework the alert.