FE - Cluster Autoscaler Node Management User's Guide#
This guide will explain how to modify the number of nodes for CCE clusters which use the cluster autoscaler.
Parameters at Different Levels#
The cluster autoscaler parameters can be modified either at a global level (i.e. the autoscaler deployment itself) and/or at the node pool level. Each cluster can have one or more node pools with its own autoscaler parameters.
The global parameters (those set for the autoscaler application itself) can be tought of as absolute limits cluster wide (max number of nodes, max number of cores, max GB of RAM etc).
The node pool parameters handle impose restrictions at a node pool level if they don't conflict with the limits configured globally. For example it is impossible to have a max_node limit of 100 configured for a node pool if, globally, the autoscaler's max node limit is 30. In this case, the autoscaler will never exceed 30 nodes withing the node pool even though the node pool's configuration states that the limit is 100. The autoscaler will complain about this conflict in its log file which can be analyzed either using kubectl logs or via grafana (app=cluster-autoscaler).
Caascad Configuration Complexities#
We use two configuration files for managing the cluster autoscalers: caascad-zones and cce.cue in envs-ng. All configuration pertaining to node pools is done in caascad-zones. The cluster-autoscaler application is configured inside the cce.cue file (in the envs-ng repo).
This dual setup is a sure-fire of making errors when configuring the node pools controled by the cluster autoscaler.
A typical example will be detailed in the following sections:
Scenario: Change the node pool configuration in order to always have 4 servers running#
prdtac will be used as an example. Inside caascad-zones, open the zones.cue file. The section pertaining to prdtac looks like this before the change:
parameters: kubernetes: pools: {
"ingress-pool": {
availability_zone: null
max_nodes: 100
min_nodes: 2
desired_nodes: 4
flavor_id: "s3.xlarge.2"
labels: ingress: "true"
taints: [
{
key: "ingress"
value: "true"
effect: "NoExecute"
},
]
}
"application-pool": {
availability_zone: null
max_nodes: 100
min_nodes: 2
desired_nodes: 2
flavor_id: "s3.xlarge.2"
}
Please note: Changes to desired_nodes has no effect when applying the trackbone/terraform configuration because the variable is ignored. In fact, FE changes this value in order perform scale-ups/downs.
By chaging the min_nodes value to 4, we obtain the following configuration for prdtac
parameters: kubernetes: pools: {
"ingress-pool": {
availability_zone: null
max_nodes: 100
min_nodes: 4
desired_nodes: 4
flavor_id: "s3.xlarge.2"
labels: ingress: "true"
taints: [
{
key: "ingress"
value: "true"
effect: "NoExecute"
},
]
}
"application-pool": {
availability_zone: null
max_nodes: 100
min_nodes: 2
desired_nodes: 2
flavor_id: "s3.xlarge.2"
}
Remember, we are working inside zones.cue and we need to merge our changes into caascad-zones master branch via a MR in order to make trackbone implement our changes.
Merging caascad-zones is an exercise in passion#
The workflow to follow is pretty much the standard workflow we observe: create a MR, and merge it into master.
Once our MR is merges into master, a CI pipeline will be triggered which will perform changes in trackbone's configuration such that it points to the new version of caascad-zones. Currently this CI pipeline takes about 16 minutes to complete. A new MR "ROBOT" will be visible in the envs-ng repository if the pipeline succeeds. We are free to merge the MR and proceed to apply the requested changes using trackbone.
Should the pipeline fail to create the ROBOT MR in envs-ng, we must implement the dirty work-around.
Dirty work around for speeding up trackbone's caascad-zones configuration update#
The procedure described below can be implemented either if the required change needs to be applied urgently or when the caascad-zones pipeline fails to create the ROBOT MR in envs-ng.
Inside the envs-ng directory, there is a configuration file used by the nix-shell to provide trackbone with the necessary caascad-zone version. This file is stored in nix/sources.json:
{
"trackbone": {
"branch": "master",
"repo": "https://git.corp.caascad.com/caascad/applications/trackbone",
"rev": "bce9d7e8874e26a415c918436af682ec38e8fe77",
"tag": "2.0.1",
"type": "git"
},
"zones": {
"branch": "master",
"repo": "https://git.corp.caascad.com/caascad/caascad-zones",
"rev": "2fd2a6b89d1659249b53dc124c093a559e24bdc3",
"type": "git"
}
}
niv update zones -a branch=master
Because this command will change the rev value of the zones section, we will have to push the changes into a new envs-ng MR which, in turn, will have to be accepted and merged into the envs-ng' master branch.
Once the changes have been merged into the master branch, we can git pull the master branch and re-launch nix-shell in order to load the new configuration for trackbone.
Finally after all this effort, we can apply the changes using trackbone.
Traps#
It is possible that the autoscaler blocks the creation of additional nodes in some cases, even though we set a high max_nodes limit for the node groups. This happens because of the limitations imposed in the cluster-autoscaler confuguration which resides in the cce.cue file in envs-ng.
In order to fix this, we have to modify the cluster-autoscaler's configuration. Take prdtac as an example. As their application needs grew, the horizontal pod autoscaler started spawning lots of new pods which in turn demanded the deployment of new cluster nodes. The node group was configured for a maximum of 100 nodes. As the cluster-autoscaler continually added nodes to the node group, the node number reached 30 nodes. At this point the cluster-autoscaler refused to add any additional nodes.
By looking at the autoscaler's log file, we could identify a message saying that the maximum limit has been exceeded for cpu. By looking at the cluster-autoscaler's configuration in cce.cue we could see:
#CCEAutoscalerAddon: #CCEAddon & {
name: "autoscaler"
values: {
basic: {
rbac_enabled: true
swr_addr: "100.125.0.94:20202"
swr_user: "hwofficial"
}
custom: {
coresTotal: *120 | int
maxEmptyBulkDeleteFlag: *10 | int
maxNodesTotal: *1000 | int
memoryTotal: *256 | int
scaleDownDelayAfterAdd: *10 | int
scaleDownDelayAfterDelete: *10 | int
scaleDownDelayAfterFailure: *3 | int
scaleDownEnabled: *true | bool
scaleDownUnneededTime: *10 | int
scaleDownUtilizationThreshold: *0.5 | float
scaleUpCpuUtilizationThreshold: *1 | int
scaleUpMemUtilizationThreshold: *1 | int
scaleUpUnscheduledPodEnabled: *true | bool
scaleUpUtilizationEnabled: *true | bool
unremovableNodeRecheckTimeout: *5 | int
}
flavor: {
description: "Has 2 instances"
name: "HA"
replicas: 2
}
}
}
Thus, in order to alleviate the problem, we have to bump the coresTotal option to a more sensible value like 99999 in order to remove this global limit. cce.cue has been modified as follows to unblock the cient's cluster:
envs: ["prdtac"]: configurations: cce: tfvars: addons: autoscaler: values: custom: {
//CW-1728
coresTotal: 99999
memoryTotal: 99999
}
Since this change has been performed in envs-ng, we can follow the standard MR workflow.
After these changes, the customer's clusters has scaled up to 60 nodes and later scaled back down to 4 nodes as expected.
On-duty (Astreinte) Emergencies#
If any of this seems dauntingly complex and there is a pressing urgency to treat the issue immediately we can fall back to using the WebUI and manually change all these numbers for the CCE cluster's cluster-autoscaler's add-on and node pools.
Should we go this route, please inform everyone with a big fat message on #operations_et_incidents to suspend all MES/MEPs until the trackbone configuration has been synchronized with the configuration changes in CCE.