Aller au contenu

How to reboot a node#

Important#

There are several ways to reboot nodes. A reboot will break the data plane present on the node, thus it is necessary to drain the node before reboot.

Prerequisites#

For all scripts below we will use 2 variables:

  • ZONE_NAME: shell variable for the zone name
  • NODE_NAME: shell variable for the node name

The ZONE_NAME is trivial.

The NODE_NAME can be retrieved from the kubectl command:

kubectl get nodes
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-10-220.eu-west-3.compute.internal   Ready    <none>   47h   v1.18.9-eks-d1db3c
ip-10-0-20-68.eu-west-3.compute.internal    Ready    <none>   47h   v1.18.9-eks-d1db3c
ip-10-0-30-24.eu-west-3.compute.internal    Ready    <none>   47h   v1.18.9-eks-d1db3c

You will also need the node-shell kubectl plugins. To get it, please install the following package: 

toolbox install kubectl-with-plugins

Drain the node#

This is how we drain the node. This operation MUST be done before a node reboot is done.

kubectl drain $NODE_NAME --ignore-daemonsets --delete-local-data

Reboot#

Reboot via kubectl#

kubectl node-shell $NODE_NAME -- reboot
reboot

Reboot via ssh#

ssh bst.${ZONE_NAME}.caascad.com ssh $NODE_NAME reboot

Reboot via the cli interface#

Flexible engine case#

  • first switch to the zone 
    os switch $ZONE_NAME

  • get the node id With the name of the node (private ip) you can get the id of the openstack server instance 

    os server list | grep $NODE_NAME
Using https://vault.infra-stg.caascad.com
Looking for riker secrets
+--------------------------------------+---------------+-----------+--------------+------------+----------+
| ID                                   | name          | status    | flavor       | version    | endpoint |
+--------------------------------------+---------------+-----------+--------------+------------+----------+
| 09e6d912-f274-11ea-9479-0255ac101f69 | caascad-riker | Available | cce.s2.small | v1.15.6-r1 | None     |
+--------------------------------------+---------------+-----------+--------------+------------+----------+

  • reboot the node

os server reboot <server_id>

AWS Case#

export VAULT_ADDR=https://vault.infra-stg.caascad.com
ACCOUNT_ID=$(curl -s https://git.corp.caascad.com/caascad/caascad-zones/raw/master/zones.json | jq --arg ZONE $ZONE_NAME -r '.| to_entries[]| select(.key==$ZONE).value.provider.account_id')


eval $(vault read aws/sts/operator_${ACCOUNT_ID} -format=json| jq -r '.data|"export AWS_ACCESS_KEY_ID=\(.access_key)\nexport AWS_SECRET_ACCESS_KEY=\(.secret_key)\nexport AWS_SESSION_TOKEN=\(.security_token)"')

INSTANCE_ID=$(aws ec2 describe-instances | jq --arg NODE $NODE_NAME -r '.Reservations[].Instances[]| select(.PrivateDnsName==$NODE).InstanceId')

aws ec2 reboot-instances --instance-ids $INSTANCE_ID