NodeLoadState#

This alert indicates that the node load is too high.

The criticity of this alert can be warning ou critical, depending on the node load average level over the last 5 minutes.

Troubleshooting hints#

kubectl describe node <node> : check the pods usage; check the resources. Identify if there is a specific pod that is using a lot of memory or if it is another cause.
If a specific pod is using too much CPU, check its resource cpu limits (but think before adding a limit; it may not be a good idea if it has no limit)
Check the evolution of the CPU usage (cpu spikes, I/O latencies, any other short term reason, or long term augmentation ?)

Tip

To check and identify if a specific pod is the cause of the incident, you may use Grafana and these dashboards:

first inspect the node with the dashboards Kubernetes / OBS / Kubernetes / Compute Resources / Node (Pods)
then, if a pod was identified, check it with Kubernetes / OBS / Kubernetes / Compute Resources / Pod

First, check known cases below.

There is no unique solution. It will depend on the cause. Here are some solutions:

rebalance: reschedule the pod
```
NODE_NAME=<node>
POD_NAME=<pod>
POD_NAMESPACE=<namespace>
kubectl cordon "$NODE_NAME"
kubectl delete pod "$POD_NAME" -n "$POD_NAMESPACE"
kubectl uncordon "$NODE_NAME"
```
Tip

You may follow this documentation to rebalance. However pay attention :
- You are working on an incident. Don't launch a chaos monkey or similar like kube-rebalancer to rebalance.
- Using k9s should work. But for traceability, prefer kubectl delete pod command.
fix the cpu limit (but think before adding a limit; it may not be a good idea if it has no limit)
spawn a new node. Check the autoscaler before:
- it could be disabled. In this case, check if the client accepts a new node;
- it could be stuck, which would be the root cause of the problem;
- its limits (number of nodes) could also be reached...
change the node flavour with more cpu or faster disks.