PodInTerminatingState#

Fires when a pod remains blocked in Terminating state for more than 5 minutes.

Info

This alert is trigered only for cluster versions higher than 1.17.

If this alert is raised for a customer pod, we need to check if the problem is caused by the Kubernetes cluster.

Troubleshooting hints for customer pods#

Get on the cluster and run kubectl -n <namespace> describe pod <pod_in_terminating_state>
Get on the cluster and run kubectl -n <namespace> logs <pod_in_terminating_state>
Check that all pods that are in Terminating state are not on the same node. If all pods are on the same node we need to check the node state:
```
kubectl get pod <pod_in_terminating_state> -n <namespace> -o wide # get node ip
kubectl node-shell <node_ip>
sudo systemctl status kubelet #check kubelet is running 
```
If there are kubelet issue on the node, kubelet needs to be restarted:
```
sudo systemctl restart kubelet
```
If no issue was identified at Kubernetes level, you should ignore the alert (for example : put a ACK; limit the ACK to the client namespace).

Get on the cluster and run kubectl -n <namespace> describe pod <pod>
- the name of the terminating pod is in the message and in the alert tags.
- the namespace of the terminating pod is in the message and in the alert tags.
- The reason why it failed may appear in the Events: section at the end.
- Check if Termination Grace Period is different than the default value of 30s. This could explain the issue.
Check the node health:
- with kubectl describe node
- connect to the node using ssh and use dmesg and similar tools to get more informations regarding the node state.

When Termination Grace Period is larger than 30s, you may try to kill the pod with kubectl delete --force.

Warning

Hard deleting the pod may cause data loss, because higher value for Termination Grace Period may be there for a good reason.

If it is necessary to reboot the node, follow the node reboot procedure