Aller au contenu

Migration Of Client Zone To Another Cloud Zone#

The migration of a client zone from one cloud zone to another is documented below. bravo, a zone originally attached to ocb-test01 is reattached to ocb-test02.

ZONE_NAME=bravo
PARENT_ZONE_OLD=ocb-test01
PARENT_ZONE_NEW=ocb-test02
INFRA_ZONE_NAME_OLD=infra-stg
INFRA_ZONE_NAME_NEW=infra-stg

ocb-test02 is a cloud zone hosted on AWS. ocb-test01 is a zone hosted on Huawei Cloud (FE).

Impacted entities#

  • caascad-zones
  • rancher
  • councourse workers
  • monitoring

Migration procedure#

caascad-zones#

Caascad zones has been modified as below: 

bravo: {
        parent_zone_name: "ocb-test02"
        provider: #ProviderFE & {
            domain: name: "OCB0003065"
        }
    }
The provider section needs to be explicitly specified in this case because the target cloud zone (ocb-test02) is hosted on a different cloud provider than the client zone.

This configuration should be pushed to a separate branch (Ex: MVPKAAS-558)

rancher#

Uninstall the rancher configuration for bravo while it is still attached to ocb-test01.

On envs-ng branch master: 

$ nix-shell
$ git pull
$ trackbone destroy -z bravo -c rancher2-client-cattle-agent
$ trackbone destroy -z bravo -c rancher2-client
$ trackbone destroy -z bravo -c fe_cluster_proxy_client -c aws_cluster_proxy_client -c azure_cluster_proxy_client
$ trackbone destroy -z bravo -c cluster_proxy_server
$ trackbone apply -z ocb-test01 -c azure_iam_cloud

Configure rancher for the target cloud zone ocb-test02. Rancher needs to use the updated caascad zones files from the dedicated caascad zones branch (i.e. MVPKAAS-558). A new branch will be created in envs-ng as well. 

$ nix-shell
$ git pull
$ git checkout -b MVPKAAS-558
$ niv update zones -a branch=MVPKAAS-558
When finished exit the nix shell, otherwise the following steps will not work. The next step should be performed on the newly created branch (i.e. MVPKAAS-558) 
$ nix-shell
$ trackbone apply -z ocb-test02 -c azure_iam_cloud
$ trackbone apply -z bravo -c azure_caascad_iam
$ trackbone apply -z bravo -c rancher2-client -c cluster_proxy_server -c fe_cluster_proxy_client
$ trackbone apply -z bravo -c rancher2-client-cattle-agent -t bootstrap=true
The client cluster should now be visible in the target cloud zone's rancher web ui.

A bug in the concourse provider forces to remove the team corresponding to the client zone from the state. Inside the dedicated branch (i.e. MVPKAAS-558) still: 

$ nix-shell
$ trackbone shell -z bravo -c concourse_provisioning_client
terraform init
terraform state rm 'concourse_team.teams["india"]'
terraform apply

Merging the branches#

caascad-zones#

Changes to caascad-zones should be merged into the master branch

envs-ng#

There is nothing to merge. The development branch can be discarded.

Monitoring#

Warning

Monitoring operations should be done after nix/sources.json is updated with new caascad-zones in envs-ng.

Functional tests#

cd caascad/pipelines/monitoring-functional-tests
git pull
git checkout -b "update_${ZONE_NAME}_functional-test"
nix-shell --run "niv update zones"
nix-shell
cd ci

# Set INFRA_ZONE_NAME_OLD again (nix-shell may not know it)
fly switch -z "${INFRA_ZONE_NAME_OLD}" -n test
fly -t "${INFRA_ZONE_NAME_OLD}.infra" login -n test
cue -t infra_zone="${INFRA_ZONE_NAME_OLD}" apply-pipelines

# If old and new infra zones differ, run this again :
# Set INFRA_ZONE_NAME_NEW again (nix-shell may not know it)
fly switch -z "${INFRA_ZONE_NAME_NEW}" -n test
fly -t "${INFRA_ZONE_NAME_NEW}.infra" login -n test
cue -t infra_zone="${INFRA_ZONE_NAME_NEW}" apply-pipelines

Prometheus rules#

For infra_zone, old and new cloud zones, redeploy prometheus rules :

trackbone apply -z ${INFRA_ZONE_NAME_OLD} -z ${INFRA_ZONE_NAME_NEW} -z ${PARENT_ZONE_OLD} -z ${PARENT_ZONE_NEW} -c prometheus-rules

Prometheus configurations#

Remove the configurations on the old zone :

rswitch ${PARENT_ZONE_OLD}
helm -n monitoring-client uninstall monitoring-cloud-client-federate-${ZONE_NAME}
helm -n monitoring-client uninstall recordingrule-cluster-info-${ZONE_NAME}
helm -n monitoring-app uninstall monitoring-cloud-app-federate-${ZONE_NAME}

Tip

The commande trackbone destroy would not work here because trackbone destroy -z ${ZONE_NAME} would impact the new cloud zone ${PARENT_ZONE_NEW}, not the old one ${PARENT_ZONE_OLD}.

Add the federations on the new zones :

trackbone apply -z ${ZONE_NAME} -c prometheus-federation-cloud-client
trackbone apply -z ${ZONE_NAME} -c prometheus-federation-cloud-app
trackbone apply -z ${ZONE_NAME} -c cluster-info-recordingrule

Lancer les tests fonctionnels#

Sur l'interface Concourse stg ou prd, relancer les tests fonctionnels :

  • pour la zone cliente (${ZONE_NAME})
  • pour les deux zones cloud, l'ancienne et la nouvelle (${PARENT_ZONE_OLD} et ${PARENT_ZONE_NEW})

Tip

Avant de lancer les tests fonctionnels, vérifier leur état. S'ils sont rouges, voir le résumé à la fin du job afin d'identifier ce qui dysfonctionne avant. Sur les zones de Production, les tests fonctionnels doivent revenir au vert.

Logging#

Warning

Monitoring operations should be done after nix/sources.json is updated with new caascad-zones in envs-ng.

Promtail#

Redeploy Promtail to connect it to the new zone :

trackbone apply -z ${ZONE_NAME} -c promtail

Lancer les tests fonctionnels#

Sur l'interface Concourse stg ou prd, relancer les tests fonctionnels :

  • pour la zone cliente (${ZONE_NAME})
  • pour les deux zones cloud, l'ancienne et la nouvelle (${PARENT_ZONE_OLD} et ${PARENT_ZONE_NEW})

Tip

Avant de lancer les tests fonctionnels, vérifier leur état. S'ils sont rouges, voir le résumé à la fin du job afin d'identifier ce qui dysfonctionne avant. Sur les zones de Production, les tests fonctionnels doivent revenir au vert.

Align everything#

Align other changes: 

trackbone apply -z ${ZONE_NAME} -z ${PARENT_ZONE_OLD} -z ${PARENT_ZONE_NEW}