NodeMemFreeTooLow#

This alerts indicates that the free memory on the node is too low.

Troubleshooting hints#

kubectl describe node <node> : check the pods usage; check the resources. Identify if there is a specific pod that is using a lot of memory or if it is another cause.
If a specific pod is using too much memory, check its resource memory limits
Check the evolution of the memory (memory spike or long term augmentation ?).

Tip

To check and identify if a specific pod is the cause of the incident, you may use Grafana and these dashboards:

first inspect the node with the dashboards Kubernetes / OBS / Kubernetes / Compute Resources / Node (Pods)
then, if a pod was identified, check it with Kubernetes / OBS / Kubernetes / Compute Resources / Pod

There is no unique solution. It will depend on the cause. Here are some solutions:

rebalance: reschedule the pod
```
NODE_NAME=<node>
POD_NAME=<pod>
POD_NAMESPACE=<namespace>
kubectl cordon "$NODE_NAME"
kubectl delete pod "$POD_NAME" -n "$POD_NAMESPACE"
kubectl uncordon "$NODE_NAME"
```
Tip

You may follow this documentation to rebalance. However pay attention :
- You are working on an incident. Don't launch a chaos monkey or similar like kube-rebalancer to rebalance.
- Using k9s should work. But for traceability, prefer kubectl delete pod command.
fix the memory limit
spawn a new node. Check the autoscaler before:
- it could be disabled. In this case, check if the client accepts a new node;
- it could be stuck, which would be the root cause of the problem;
- its limits (number of nodes) could also be reached...
change the node flavour with more memory
change the rollout strategy (Recreate ?)
disable the alert for that pod (as explained in this documentation)