Thanos troubleshooting

Access a PVC from another pod

Warning

You need to stop the pod that access the PVC/PV.

This procedure uses thanos-monitoring-compactor-0 in namespace monitoring as an example.

Feel free to adapt to any other pod.

NAMESPACE=monitoring

kubectl -n ${NAMESPACE} get pod thanos-monitoring-compactor-0 -o yaml

Note the spec.volumes.persistentVolumeClaim.claimName of the volume to debug.

kubectl -n ${NAMESPACE} edit sts thanos-monitoring-compactor
# Change the replicas number :    s/replicas: 1/replicas: 0/

If you forgot to note the Claim Name, you can get it here too : get the volumeClaimTemplate.metadata.name of the volume. The Claim Name will be that name followed with the pod name.

Create now a new pod : edit the following in a file /tmp/debug-deploymemt.yaml

---
kind: Pod
apiVersion: v1
metadata:
  name: volume-debugger
spec:
  volumes:
    - name: volume-to-debug
      persistentVolumeClaim:
        claimName: <CLAIM NAME HERE>
  containers:
    - name: debugger
      image: busybox
      command: ['sleep', '3600']
      volumeMounts:
        - mountPath: "/data"
          name: volume-to-debug

Then create the pod and get inside it

kubectl -n ${NAMESPACE} create -f /tmp/debug-deployment.yaml
kubectl -n ${NAMESPACE} get pod volume-debugger # wait until it's running
kubectl -n ${NAMESPACE} exec -it volume-debugger -- sh

Now debug in /data.

Rollback after debugging

kubectl -n ${NAMESPACE} delete pod volume-debugger
kubectl -n ${NAMESPACE} edit sts thanos-monitoring-compactor
# Change the replicas number :    s/replicas: 0/replicas: 1/