Aller au contenu

Automation#

Prerequisites#

Warning

Check whether the client wants to keep the cluster (case 1) or delete everything (case 2)

  • check what is there that was not deployed or provisioned by Caascad 
    kswitch $ZONE_NAME
kubectl get all -A

Silences#

Amtool#

During decommissioning operations, silences need to be set in place for affected zones.

Format: 

amtool-caascad silence add <labels_key=labels_value> \
     --author <nom_auteur> --duration <durée> --comment <ticket_id or comment> \
     --alertmanager.url <alertmanager_url>
Example: 
amtool-caascad --alertmanager.url ocb-test06/client silence add \
    --author automation_team --duration 5h --comment decom

More information about amtool here.

Karma#

Silences can also be set directly in Karma:

  • label : cc_prom_source=<zone name>
  • duration : 1 hour
  • Author : Caascad Team
  • comment : ACK! Decom <zone name>

Rancher#

Remove the target from rancher: 

trackbone destroy -z {ZONE_NAME} -c blackbox-exporter-servicemonitor-rancher-client
trackbone destroy -z {ZONE_NAME} -c rancher2-client-cattle-agent
trackbone destroy -z {ZONE_NAME} -c rancher2-client

Case 1: the client wants to keep the cluster#

Remove caascad objects 

trackbone destroy -c eventrouter -c docker_exporter -c metrics_server -c bastion_monitoring -z ${ZONE_NAME}
kubectl delete ns caascad-system

Then move directly to the VPN part.

Case 2: the client wants to wipe the cluster#

Backup PVC List#

Before destroying the cluster, save the PVC list: 

kubectl get pvc -A -o=jsonpath='{range .items[*]}{.spec.volumeName}{"\n"}{end}' > /tmp/${ZONE_NAME}.pvc.list

Backup nodes volume list#

Before destroying the cluster, save volumes id attached to the nodes:

os switch ${ZONE_NAME}
(kubectl get nodes  -o json | jq -r '.items[].status.nodeInfo.machineID' | xargs -I{} os server show {} -f json | jq -r '.volumes_attached[].id' )> /tmp/${ZONE_NAME}.volumes.list

aws ec2 describe-volumes | jq -r '.Volumes[].Attachments' > /tmp/${ZONE_NAME}.volumes.list

Destroy bastion monitoring#

Remove servicemonitor object deployed in the cloud zone 

trackbone destroy -z "$ZONE_NAME" -c bastion_monitoring

Delete everything#

kubectl get ns -o name |grep -v "namespace/default" | grep -v "namespace/kube-" | xargs -I{} kubectl delete {}
trackbone destroy -z "$ZONE_NAME" -c cce -c eks -c aks -t purge=true

Warning

This step takes a while and it is possible for terraform to time out while nodes are being destroyed.

In case of timeouts, re-run the command. Check also if all node volumes have been properly destroyed (next section).

Volume Removal#

Note

On Fe envs only

Note

Switch on Fe Envs

os refresh
os switch ${ZONE_NAME}
os network list

Delete node volumes#

Info

If you had problems while deleting the cluster and its nodes some volumes might not have been properly deleted.

If you didn't experience any issue you can skip this step.

VOLUMES=""
while read volume; do 
  VOLUMES+=$volume
done < /tmp/${ZONE_NAME}.volumes.list
os volume delete $VOLUMES

Delete pod volumes#

Delete all the remaining PVC entries from the list saved in the previous step

VOLUMES="" 
while read volume; do  
  VOLUMES+=$volume 
done < /tmp/${ZONE_NAME}.pvc.list 
os volume delete $VOLUMES

At this point, the cluster has been destroyed and its volumes have been deleted.

VPN#

Remove the bastion IP from the VPN configuration. Refer to the documentation in: https://git.corp.caascad.com/caascad/ansible/asa/blob/master/README.md

The changes have to be commited to the master branch before running ansible.

Bastion#

In the envs-ng repo:

trackbone destroy -z "$ZONE_NAME" -c fe_bastion

trackbone destroy -z "$ZONE_NAME" -c aws_instances

trackbone destroy -z "$ZONE_NAME" -c azure_instances

Cluster proxy (frp)#

In the envs-ng repo: 

trackbone destroy -z "$ZONE_NAME" -c fe_cluster_proxy_client -c aws_cluster_proxy_client -c azure_cluster_proxy_client
trackbone destroy -z "$ZONE_NAME" -c cluster_proxy_server

Networking#

Warning

Make sure we don't use an existing VPC.

Fixme

Handle existing VPC case properly.

In the envs-ng repo:

trackbone destroy -z "$ZONE_NAME" -c fe_base

trackbone destroy -z "$ZONE_NAME" -c aws_network
trackbone destroy -z "$ZONE_NAME" -c aws_base

trackbone destroy -z "$ZONE_NAME" -c azure_network
trackbone destroy -z "$ZONE_NAME" -c azure_base

Warning

This step might hang if someone added network resources without using terraform. These resources will have to be removed (via the web console) before re-running the command above.

Check loadbalancers if the customer has deployed an ingress controller.

Update envs-ng#

It is possible to have zone-specific configurations in envs-ng which have to be removed.

In the envs-ng repo:

grep -r "$ZONE_NAME" *.cue # only in files managed by infrakaas team

If some files were found,

  • remove the references to the zone in the matching lines
  • create a MR and ask for validation
  • merge it into master branch

Remove zone secrets#

export VAULT_ADDR="https://vault.infra-prd.caascad.com"
vault login -method oidc

vault delete "secret/zones/fe/$ZONE_NAME/api"

Update envs-ng#

It is possible to have zone-specific configurations in envs-ng which have to be removed.

In the envs-ng repo:

grep -r "$ZONE_NAME" *.cue # in files managed by any teams: every remaining references to the zone should be removed

# Note : also remove the reference to the zone in the ignore_zones list in ci.cue

If some files were found,

  • remove the references to the zone in the matching lines
  • create a MR and ask for validation
  • merge it into master branch

Remove zone from caascad-zones and merge#

The zone should be removed from caascad-zones.

Warning

Once the change in caascad-zone has been merged, an MR should appear in the envs-ng repo to update the caascad-zones revision used in the master branch.

Open it, review changes and merge

Apply the karma configuration on the infra zone#

This will remove the karma alerts for the zone 

trackbone apply -z ${INFRA_ZONE_NAME} -c karma -t karma_refresh_cache=true