CCE Changing Node Pool

This document describes the procedure for modifying the node pool flavors or limit size of our CCE clusters.

Our CCE clusters make use of node pools to manage worker nodes.

Node Flavor is an immutable field of the node pool specification. Modifying the flavor of an existing node pool managed by terraform will trigger a replacement of the pool (existing Kubernetes worker nodes will be destroyed and new ones will be created). This operation has an impact on the applications running on the cluster.

Also min_nodes, desired_nodes, max_nodes parameters are not properly taken into account by CCE, particularly in the case of a reduction of desired nodes.

When node flavor or limit size modifications are required, a better approach would be to add new node pools to the cluster, switch all the workloads to the new nodes and, finally, remove the old node pools from the configuration.

Procedure

All of the operations are performed from within the envs-ng directory:

cd <envs-ng directory>
nix-shell
export ZONE_NAME=<name of the target zone>
git checkout master
git pull

Create a new branch to work on

git checkout -b "${ZONE_NAME}_nodepool_flavor_change"

Identify the nodes belonging to the targeted node pool

We need to know which cluster's nodes belong to the targeted node pool. We will use inttac as an example.

Connect to the cluster and identify the node:

kswitch inttac
kubectl get nodes -o custom-columns="NAME":".metadata.name","NODEPOOL":".metadata.labels['caascad\.io/nodepool']"

This will show a list of nodes and the matching value for label caascad.io/nodepool. We can grep for the name of target node pool. pod-tac is used as an example

kubectl get nodes -o custom-columns="NAME":.metadata.name,"LABELS":.metadata.labels['caascad\.io/nodepool'] | grep pod-tac

This will show all nodes with a label containing the string pod-tac. Double check the correctness of the displayed values and make a note of the node names.

Modify the zones file

The zones files are located in the zones directory:

cd zones

Depending on the cluster we will enter either the caascad_zones or the ngot_zones subdirectory. For this example, we will use caascad_zones:

cd caascad_zones
vi zones.cue

Inside the zones.cue file, locate the section describing our target cluster.

inttac is used in the example below. The node pool section will look something like this:

parameters: kubernetes: pools: {
                        "application-pool": {
                                labels: "caascad.io/nodepool": "application-pool"
                                availability_zone: null
                                max_nodes:         100
                                min_nodes:         2
                                desired_nodes:     2
                                flavor_id:         "s3.xlarge.2"
                        }
                        "pod-tac": {
                                labels: "caascad.io/nodepool": "pod-tac"
                                availability_zone: "eu-west-0c"
                                max_nodes:         5
                                min_nodes:         1
                                desired_nodes:     1
                                flavor_id:         "s6.4xlarge.2"
                        }
                }

There are two node pools visible: application-pool and pod-tac. If we want to change the flavor for the pod-tac node pool to s6.4xlarge.4, we will add a new section with the new node pool definition. The new file will look like this:

parameters: kubernetes: pools: {
                        "application-pool": {
                                labels: "caascad.io/nodepool": "application-pool"
                                availability_zone: null
                                max_nodes:         100
                                min_nodes:         2
                                desired_nodes:     2
                                flavor_id:         "s3.xlarge.2"
                        }
                        "pod-tac": {
                                labels: "caascad.io/nodepool": "pod-tac"
                                availability_zone: "eu-west-0c"
                                max_nodes:         5
                                min_nodes:         1
                                desired_nodes:     1
                                flavor_id:         "s6.4xlarge.2"
                        }
                        "pod-tac2": {
                                labels: "caascad.io/nodepool": "pod-tac"
                                availability_zone: "eu-west-0c"
                                max_nodes:         5
                                min_nodes:         1
                                desired_nodes:     1
                                flavor_id:         "s6.4xlarge.4"
                        }
                }

Note that the flavor_id has changed but we kept the same value for caascad.io/nodepool label (this label might be used by the client to select the nodepool where to schedule specific workloads).

Compile the zones file

The modified zones.cue file needs to be compiled:

cue fmt zones.cue
generate-static-zones-files

Apply the changes using trackbone

Go to the envs-ng directory to apply the configuration. As an example, for caascad, this would be envs-ng/contexts/caascad. Apply the new CCE configuration:

trackbone apply -z <zone_name> -c cce

Trackbone (terraform) will display the configuration changes. Inspect the output and make sure it is correct. Accept the changes and wait for the run to finish.

Drain nodes belonging to the old node pool

Cordon the nodes in the list of nodes identified earlier in order to make them unavailable for scheduling.

Drain the nodes. It is a good idea to drain them node-by-node and to make sure all the pods are in a Running state.

Remove the old node pool from the zones file

Return the zones directory. As an example, for caascad, this would be zones/caascad_zones. Edit the zones.cue file and remove the old node pool. The end result will look something like this:

parameters: kubernetes: pools: {
                        "application-pool": {
                                labels: "caascad.io/nodepool": "application-pool"
                                availability_zone: null
                                max_nodes:         100
                                min_nodes:         2
                                desired_nodes:     2
                                flavor_id:         "s3.xlarge.2"
                        }
                        "pod-tac2": {
                                labels: "caascad.io/nodepool": "pod-tac"
                                availability_zone: "eu-west-0c"
                                max_nodes:         5
                                min_nodes:         1
                                desired_nodes:     1
                                flavor_id:         "s6.4xlarge.4"
                        }
                }

Please note that the old pod-tac section has completely dissapeared. This will remove the node pool the next time the CCE trackbone configuration is applied.

Compile the zones file (again)

The modified zones.cue file needs to be compiled:

cue fmt zones.cue
generate-static-zones-files

At this point, commit all the changed files to git and create a MR in gitlab. The MR has to be approved by the automation team.

Apply the changes using trackbone (again)

Go to the envs-ng directory to apply the configuration. As an example, for caascad, this would be envs-ng/contexts/caascad. Apply the new CCE configuration:

trackbone apply -z <zone_name> -c cce

Trackbone (terraform) will display the configuration changes. Inspect the output and make sure it is correct. Accept the changes and wait for the run to finish.

At this point, the node of the old node pool should no longer be visible from the command line:

kubectl get nodes

Note that the nodes of the old node pool have dissapeared.

Merge changes into the master branch

In gitlab, perform a trackbone plan and verify the changes applied. There should be no changes because they were already applied from the command line.

Finally, merge the change into the master branch by clicking the Merge button and update your envs-ng copy on your work station:

git checkout master
git pull