Aller au contenu

Automation#

Cloud Zone Decom Procedure#

Start from the bottom of the MEP Cloud Zones (more or less).

amtool-caascad silence add obs_client="${ZONE_NAME}" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url ${ZONE_NAME}/caascad
amtool-caascad silence add obs_client="${ZONE_NAME}" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url ${ZONE_NAME}/client
amtool-caascad silence add alertmanager="${ZONE_NAME}" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url infra-prd/caascad
amtool-caascad silence add alertmanager="${ZONE_NAME}-client" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url infra-prd/caascad
amtool-caascad silence add obs_client="${ZONE_NAME}" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url infra-prd/infra-caascad
amtool-caascad silence add cc_consumption_client="${ZONE_NAME}" --author "Caascad Team" --duration "48h" --comment "Decom : ${ZONE_NAME}" --alertmanager.url infra-prd/infra-consumption

1. Remove Rancher#

First we remove the configuration: 

trackbone destroy -z ${ZONE_NAME} -c rancher2-cloud

Finally we remove rancher itself: 

trackbone destroy -z ${ZONE_NAME} -c rancher2-helm -t purge=true

2. Cluster proxy (frp)#

In the envs-ng repo: 

trackbone destroy -z "$ZONE_NAME" -c fe_cluster_proxy_client -c aws_cluster_proxy_client -c azure_cluster_proxy_client
trackbone destroy -z "$ZONE_NAME" -c cluster_proxy_server

3. Remove Object Storage buckets#

First the buckets have to be reconfigured with force_destroy=true and apply terraform configuration

trackbone destroy -z ${ZONE_NAME} -c fe_buckets -t purge=true

trackbone destroy -z ${ZONE_NAME} -c aws_buckets -t purge=true

trackbone destroy -z ${ZONE_NAME} -c azure_storageaccounts

Any buckets which raise errors during this step need to be deleted manually.

4. Remove vault/keycloak#

trackbone destroy -z ${ZONE_NAME} -c vault_aws_sts_cloud
trackbone destroy -z ${ZONE_NAME} -c keycloak-vault-cloud
Observation! permission denied errors during the removal of keycloak-vault-cloud are to be expected and can be ignored.

trackbone destroy -z ${ZONE_NAME} -c keycloak -t purge=true
trackbone destroy -z ${ZONE_NAME} -c vault -t purge=true
trackbone destroy -z ${ZONE_NAME} -c vault-cloud-kms -t purge=true

5. Remove ingresses#

To remove unwanted DNS records we need to delete every ingress object: 

 kubectl get ingress -A -o name | xargs -I{} kubectl delete {}

6. Remove RDS#

trackbone destroy -z ${ZONE_NAME} -c fe_rds
trackbone destroy -z ${ZONE_NAME} -c fe_rds_provisioning

7. Remove ingress-controllers#

trackbone destroy -z ${ZONE_NAME} -c ingress_controller_public -c ingress_controller_private -c ingress_controller_v2_public -c ingress_controller_v2_private -t purge=true

8. Remove Cluster#

trackbone destroy -z ${ZONE_NAME} -c cce -t purge=true

trackbone destroy -z ${ZONE_NAME} -c eks -t purge=true

trackbone destroy -z ${ZONE_NAME} -c aks -t purge=true

9. Remove the bastion#

Record the bastion's EIP. It is displayed by trackbone when running destroy.

trackbone destroy -z ${ZONE_NAME} -c fe_bastion

trackbone destroy -z ${ZONE_NAME} -c aws_instances

trackbone destroy -z ${ZONE_NAME} -c azure_instances

10. Remove the load balancers#

Record the public IP used by the load balancer. The IP is shown by trackbone.

trackbone destroy -z ${ZONE_NAME} -c fe_loadbalancer -c fe_loadbalancer_v3

trackbone destroy -z ${ZONE_NAME} -c aws_ingress

trackbone destroy -z ${ZONE_NAME} -c azure_ingress

11. Remove remaining volumes#

os s ${ZONE_NAME}
eval $(os p -e -u)
openstack volume list -c ID -f value | xargs -I{} openstack volume delete {}

12. Remove VPC configuration and base#

trackbone destroy -z ${ZONE_NAME} -c fe_base

trackbone destroy -z ${ZONE_NAME} -c aws_network
trackbone destroy -z ${ZONE_NAME} -c aws_base

trackbone destroy -z ${ZONE_NAME} -c azure_network
trackbone destroy -z ${ZONE_NAME} -c azure_iam_cloud
trackbone destroy -z ${ZONE_NAME} -c azure_base

13. Remove remaining project config#

trackbone destroy -z ${ZONE_NAME} -c fe_bootstrap_project

trackbone destroy -z ${ZONE_NAME} -c vault_aws_sts_roles
trackbone destroy -z ${ZONE_NAME} -c aws_caascad_roles
trackbone destroy -z ${ZONE_NAME} -c aws_account_bootstrap
trackbone destroy -z ${ZONE_NAME} -c aws_account_create
trackbone destroy -z ${ZONE_NAME} -c aws_ses_rules

Nothing to do

14. Update envs-ng#

It is possible to have zone-specific configurations in envs-ng which have to be removed.

In the envs-ng repo:

grep -r "$ZONE_NAME" *.cue # in files managed by any teams: every remaining references to the zone should be removed

# Note : also remove the reference to the zone in the ignore_zones list in ci.cue

If some files were found,

  • remove the references to the zone in the matching lines
  • create a MR and ask for validation
  • merge it into master branch

15. Remove zone from caascad-zones and merge#

Follow the procedure for removing the zone from caascad zones.

Warning

Once the change in caascad-zone has been merged, an MR should appear in the envs-ng repo to update the caascad-zones revision used in the master branch.

Open it, review changes and merge

16. Modify Whitelist#

Record the natgw IP address from fe_whitelist.cue for the zone to be decomissioned and remove it. When finished: 

trackbone apply -z infra-prd -z infra-stg -c fe_whitelist
Changes to fe_whitelist.cue should be merged into the master branch.

17. Apply the karma configuration on the infra zone#

This will remove the karma alerts for the zone 

trackbone apply -z ${INFRA_ZONE_NAME} -c karma -t karma_refresh_cache=true

18. Remove Angel Customer Project on FE Console.#

Check on FE Console, on the domain that hosted the NGoT Angel zone for the customer. Use IAM page, and go through "projects", search for customer angel zone name : eu-west-0_{ZONE_NAME}

Delete the project.