Manage CCE Cluster's Node Number#
This documentation attempts to explain how to manage nodes on CCE clusters.
When NOT to add a node#
Warning
ALL infra and cloud zones should have node pools and cluster autoscalers configured. Manually adding worker nodes to these clusters should be avoided.
There is no need to add nodes to clusters having cluster autoscaling configured. This information is found in envs-ng/cce.cue. Clusters using the autocscaler have a cce_node_pools definition. Additionally, autocaler pods should be running on the cluster.
$ kubectl -n kube-system get pods|grep auto
cluster-autoscaler-699c878d76-bzdlj 1/1 Running 0 52d
cluster-autoscaler-699c878d76-pr2v5 1/1 Running 0 52d
The infrakaas/support teams should be alerted if there is a cce_node_pools section in cce.cue but no cluster-autocaler is running in k8s because this is not an acceptable configuration.
When an additional node might be necessary (client zones)#
Client zones without autoscaling might require additional cluster nodes. The following sections will describe the alerts and what can be done. Since we are dealing with client zones, the client should be informed and involved in the decision process.
When using static node pools#
Adding nodes#
Clusters using node pools without autoscaler have a cce_node_pools_static definition. For those kind of node the scaling is done manualy by setting the desired value in cce.cue (envs-ng) configuration:
envs: ["client-env"]: configurations: cce: tfvars: {
...
cce_node_pools_static: "node-pool": desired_nodes: 2
}
Deleting node#
To delete a node we can use os cli to do so:
os cce cluster node delete <cluster_id> <node_id>
Then we have to set number of nodes in cce.cue configuration and apply it.
Relevant karma alert messages#
Karma alerts listed below can be used to decide whether to add an additionnal node
NodeOutOfMemory#
This alert is raised for nodes experiencing high memory pressure. More information should be gathered about the pods running on the node and their resource requirements.
Usually there are 1-2 pods which use up a lot of the node's resources. These "large" pods could be moved to other nodes, depending on the resources available, thus rebalancing the cluster.
A new node should be added if rebalancing the cluster does not yield a satisfactory result (i.e. we are moving the problem to other nodes). The cluster should be rebalanced after adding a new node)
KubeletTooManyPods#
This alert is raised when a node hosts too many pods. The cluster should be rebalanced by scheduling pods on other nodes. A new node should be added if the rebalacing of the cluster fails.
KubeCPUOvercommit, KubeMemoryOvercommit and KubeMemOvercommit#
This alert does not necessarily require an increase in cluster size as it primarily deals with resource requests and most of the times tells us that the values for the resource requests are too high. kubectl top nodes can be used to check the actual node resource usage vs. the resource requets which can be ontained using kubectl describe nodes
The proper way of fixing this alert is to have the client adjust the resource requests for the containers. For various reasons this might not be feasable in which case a new node could be added with the client's go-ahead.
KubeContainerWaiting#
This alert is related to the alerts above and it is raised when k8s is no longer able to schedule pods because of insufficient resources.
In this case a new node should be added at the client's go-ahead or the client should check the existing resource requirements of their k8s workloads.
Other informations#
[kubectl commands]#
The following commmand gives hints on node health:
kubectl describe node <node>
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 29 Apr 2021 11:09:23 +0200 Fri, 19 Mar 2021 20:05:22 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 29 Apr 2021 11:09:23 +0200 Fri, 19 Mar 2021 20:05:22 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 29 Apr 2021 11:09:23 +0200 Fri, 19 Mar 2021 20:05:22 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 29 Apr 2021 11:09:23 +0200 Fri, 19 Mar 2021 20:05:22 +0100 KubeletReady kubelet is posting ready status