How to rebuild a node

Synopsis

On some circumstances, a node rebuild can be necessary. We'll see here how to rebuild a node in 2 scenarios:

• clusters composed of static nodes
• clusters managed via nodepools

Cluster with static nodes

At the current time we use static nodes only with FE. AWS has this kind of technologies via ec2 but we do not use it. To rebuild a static node the operator should drain it first, then use terraform to rebuild it:

The <zone> and <node_ip> (as shown by kubectl get nodes) are known:

  # generate a trackbone shell for cce
  trackbone shell -c cce -z <zone>

  # get cluster name
  jq '.cluster_name' < terraform.tfvars.json

  # identify <node_name> via <node_ip>
  os switch <zone>
  os cce cluster node list <cluster_name> | grep <node_ip>

  # identify the node <terraform_resource>
  terraform init
  terraform state show | grep <node_name>

  # drain the node
  kubectl drain <node_ip> --ignore-daemonsets --delete-local-data

  # taint the resource
  terraform taint <terraform_resource>

  # check terraform wants to rebuild that node and only it
  bash plan.sh

  # rebuild the node by applying terraform
  bash apply.sh

Cluster with node pools

In AWS there is no convenient ways to rebuild a kubernetes node via EKS or EC2. Because every nodepool are managed by an autoscaler the only way to go is to delete the node.If the remaining resources are insufficient for workloads, the autoscaler will add new nodes by itself.

FE offers a way to reset a node to be consistent with AWS we chose to not use this feature.

The ID of the node to be deleted can be found this way:

kubectl get nodes <node> -o=jsonpath='{.spec.providerID}'

For FE this ID can be used with the cluster name to delete the node

  # generate a trackbone shell for cce
  trackbone shell -c cce -z <zone>

  # get cluster name
  jq '.cluster_name' < terraform.tfvars.json

  # delete the node
  os cce cluster node delete <cluster_name> <node_uuid>

Please note that the node has a différent UUID in ECS.

For AWS the UUID is the EC2 instance id. So to delete the node:

ZONE=<zone name>
export AWS_DEFAULT_REGION=$(sd get zones | jq -r --arg ZONE "${ZONE}" '.[$ZONE].provider.region')
eval $(vault read aws/sts/operator_${ZONE} -format=json| jq -r '.data|"export AWS_ACCESS_KEY_ID=\(.access_key)\nexport AWS_SECRET_ACCESS_KEY=\(.secret_key)\nexport AWS_SESSION_TOKEN=\(.security_token)"')
aws ec2 terminate-instances --instance-ids <node_id>