Automation#

Prerequisites#

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="ocb-..."
export DOMAIN_NAME="caascad.com" # ou cloudservicesfactory.com
export INFRA_ZONE_NAME="infra-..."
export BRANCH_NAME="pf-..."

Warning

During this procedure, several steps require restarts of different components in cloud zones. Before each restart:

check there is no CAASCHR that is planned on the same component at the same time
warn supervsion team on #supervision about the component/zone that is going to be restarted

envs-ng#

Tip

Many steps will consist of running trackbone apply -z "$ZONE_NAME" -c ...

Reusing the shell setup here is a good idea!

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$BRANCH_NAME"
nix-shell

1 - Requirements#

Domain existence#

A CI job will create the new cloud zone's domain. Therefore, it should not be present on the provider prior to the bootstrap step.

FE

Warning

A bug prevents proper zone deletion. If needed, a zone can be renamed in order to allow the CI to work.

Quotas#

Various quotas should be checked. However they may not be available until the domain is created. In that case postpone the verification.

FEAWS / Azure

Service	Ressource Type	Free Quota
Cloud Container Engine	Cluster	1
Elastic Load Balancer	Listener	6
Identity and Access Management	Project	1
User Groups	Domain	100

To be determined

2 - Declare zone in zones module (envs-ng)#

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$ZONE_NAME"
nix-shell

Edit zone file regarding the type of zone you want to add

zones/caascad_zones/zones.cue
zones/caascad_zones/zones_cloudavenue.cue
zones/ngot_zones/values.cue

To declare the new zone. Use any ocb-... zone as an example.

Generate static files and commit your changes

# Run script to generate static files from zones
generate-static-zones-files
# Add all modifications to the commit
git add zones/
git add gen/
git commit -m "Add $ZONE_NAME zone"
# Create target branch
git push -u origin "$BRANCH_NAME"

Create and submit your MR

Create the MR with Draft status, then update status to Ready. (this will bypass mr-plan pipeline)
Comment your MR with message : check zones to trigger check-zones pipeline and validate the impacts of your changes on environments.

Info

Please wait for the CI job completion before asking a validation.

Info

Beware of client zones needing to talk to cloud zones via vpc endpoints. In such case the cloud zone should be filled with the private_link parameter listing details on the transit subnet.

Info

When zones medatada:shared is true then parameters: apps: quay and parameters: apps: harbor must be set to false

3 - Deploy quay robot#

Because some configurations will require a Vault secret generated by the quay-robots, we will start the pipeline manually

cd ${GIT}/caascad/quay-robots
git switch master
git reset master --hard
git pull
fly switch -z ${INFRA_ZONE_NAME} -n deploy
fly set-pipeline -p quay-robots -c pipeline.yml -v infra_line=infra-prd -v zones_branch=${BRANCH_NAME}
fly jobs -p quay-robots
fly trigger-job -j quay-robots/sync-quay-robots
fly watch -j quay-robots/sync-quay-robots

And rollback:

fly set-pipeline -p quay-robots -c pipeline.yml -v infra_line=infra-prd -v zones_branch=master

4 - optional but RECOMMENDED#

This section is intended for users who would like to deploy automatically. If you choose to run each steps individually then skip this part and go to the next section

If you want to try the automatic way:

trackbone apply -z "${ZONE_NAME}" -c fe_bootstrap_project -c fe_base -t bootstrap=true --non-interactive
# add nat gw eip address to ${ENVS_NG}/context/ipam.cue
trackbone apply -z "${INFRA_ZONE_NAME}" -c fe_whitelist --non-interactive
trackbone apply -z "${ZONE_NAME}" -t bootstrap=true --non-interactive -n 4

Some configuration may fail during the process, interrupting the whole workflow. In such case you need to fix the configuration(s) that has failed and restart the workflow from there.

trackbone apply -z "${ZONE_NAME}" -c "${FAILED_CONFIG_1}" -c "${FAILED_CONFIG_2}" ... --cascade -t bootstrap=true --non-interactive -n 4

When it is finished you can jump to the functional tests section

4 - Bootstrap zone project or account#

Go to the context directory where your new zone is handled, this could be: - contexts/caascad - contexts/pf - contexts/ngotalpha - contexts/ngot

and run trackbone apply commands.

FEAWSAzure

Apply fe_bootstrap_project configuration:

trackbone apply -z "$ZONE_NAME" -c fe_bootstrap_project

Apply aws_account_create configuration:

trackbone apply -z "$ZONE_NAME" -c aws_account_create

Nothing to do

Provision secrets for an optional transit subnet

If the customer zone needs to talk to the cloud zone via a private link (vpc endpoints) we need to add another secret to the vault infra to allow our configurations to provision vpc endpoints on a remote subnet. Such a subnet is called the transit subnet. It can belong to another fe domain/project and so requires different api keys to provision objects on.

    === "FE"
        Write API secret in vault for customer project:
        ```sh
        # Create a JSON file with API secrets

        export DOMAIN="<domain OCB...>"                    # the customer FE domain
        export PASSWORD="<support.caascad password>"       # API password of the support.caascad user
        export PROJECT_NAME="<eu-west-0 or eu-west-0_...>" # name of the project that was created in the customer domain
        cat << EOF > /tmp/transit-api-secret.json
        {
            "auth_url": "https://iam.eu-west-0.prod-cloud-ocb.orange-business.com/v3",
            "domain_name": "${DOMAIN}",
            "password": "${PASSWORD}",
            "region": "eu-west-0",
            "tenant_name": "${PROJECT_NAME}",
            "username": "support.caascad"
        }
        EOF

        # Write secret in vault

        export VAULT_ADDR="https://vault.${INFRA_ZONE_NAME}.caascad.com"
        vault login -method oidc
        vault write secret/zones/fe/transit-${ZONE_NAME}/api @/tmp/transit-api-secret.json
        vault read secret/zones/fe/transit-${ZONE_NAME}/api

        rm /tmp/transit-api-secret.json
        ```

5 - Provision base infra#

In the envs-ng repo:

Apply configurations:

FEAWSAzure

trackbone apply -z "$ZONE_NAME" -c fe_base

trackbone apply -z "$ZONE_NAME" -c aws_base
trackbone apply -z "$ZONE_NAME" -c aws_network

trackbone apply -z "$ZONE_NAME" -c azure_base
trackbone apply -z "$ZONE_NAME" -c azure_iam_cloud
trackbone apply -z "$ZONE_NAME" -c azure_network

Store the nat_gw_eip_address:
```
export NAT_EIP="..."
```

Report the nat_gw_eip_address of the zone in ipam.cue:

--- a/ipam.cue
+++ b/ipam.cue
@@ -15,6 +15,7 @@ ipam: {ac
                "ocb-sbmaeg":      ["90.84.192.160"]
                "ocb-corp":        ["90.84.186.239"]
                "ocb-fh":          ["90.84.178.28"]
+               "ocb-ogsb":        ["90.84.177.238"]

                "infra-stg":  ["90.84.196.30"]
                "ocb-demo":   ["90.84.199.202"]

Create a MR on envs-ng. See README for commit message format.

Warning

The trackbone apply and merge must be done in a later step. (See last step)

Apply instances configuration:

FEAWSAzure

trackbone apply -z "$ZONE_NAME" -c fe_bastion

trackbone apply -z "$ZONE_NAME" -c aws_instances

trackbone apply -z "$ZONE_NAME" -c azure_instances

Store the bastion EIP:

export BST_EIP="$(dig +short bst.${ZONE_NAME}.${DOMAIN_NAME})"

6 - Setup VPN for the bastion#

Temporarily add the route locally:

sudo ip route add "${BST_EIP}/32" dev "$(ip -4 -o a | awk '/10.254.255/ {print $2}')"

Add the route to the VPN:

Repository: caascad/ansible/vpn
1. Check the requirements
2. Follow the usage section of the documentation
Warning

Don't forget to commit and merge your changes.

7 - Provision the bastion#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c bastion_provisioning -t bootstrap=true

8 - Provision RDS databases (FE only)#

In the envs-ng repo:

FE

Apply fe_rds configuration:

trackbone apply -z "$ZONE_NAME" -c fe_rds -t bootstrap=true

Apply fe_rds_provisioning configuration:

trackbone apply -z "$ZONE_NAME" -c fe_rds_provisioning -t bootstrap=true

9 - Provision Kubernetes Cluster#

In the envs-ng repo:

FEAWSAzure

Apply cce configuration:

trackbone apply -z "$ZONE_NAME" -c cce -t bootstrap=true

Check that the nodes are ready:

Fixme

Add check in post_apply
```
kswitch $ZONE_NAME
kubectl get nodes
```

Apply eks configuration:
```
trackbone apply -z "$ZONE_NAME" -c eks
```
Check that the nodes are ready:

Fixme

Add check in post_apply
```
kswitch $ZONE_NAME
kubectl get nodes
```

Apply aks configuration:
```
trackbone apply -z "$ZONE_NAME" -c aks
```
Check that the nodes are ready:
```
kswitch $ZONE_NAME
kubectl get nodes
```

10 - Deploy quay robot accounts secrets in cluster & config_sync#

Note

Dockerhub secret is now deploy with config-sync namespace creation When config-sync (kubed) will be deployed on all namespace matching the labels

Check dockerhub secret replication kubectl get secret -A --field-selector='metadata.name==dockerhub-registry-creds-incident

Config-sync-provision#

Note

this create config-sync ns and create Kubernetes secrets

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c config_sync_provision -t bootstrap=true

Config-sync (Kubed)#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c config_sync -t bootstrap=true

!!! warning
    Config-sync use a specific label on namespace to deploy secret but this label self-managed by OPA.
    You must apply this label on your namespace before OPA is deployed in the cluster.
    `kubectl label ns my-namespace config-sync.caascad.com/secret=true`

10 - Deploy custom storage classes#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c storageclasses -t bootstrap=true

11 - Deploy CRDs#

In the envs-ng repo:

prometheus-operator CRDs:

trackbone apply -z "$ZONE_NAME" -c prometheus-operator-crds

velero CRDs: !!! warning Skip this deployment when metadata : shared : true is configured on the cloud zone.
```
trackbone apply -z "$ZONE_NAME" -c velero_crd
```

12 - Deploy LB and ingress#

Loadbalancers deployment#

In the envs-ng repo:

FEAWSAzure

Apply fe_loadbalancer configuration:

trackbone apply -z "$ZONE_NAME" -c fe_loadbalancer

Apply aws_ingress configuration (Elastic IP deployment):
```
trackbone apply -z "$ZONE_NAME" -c aws_ingress
```

Apply azure_ingress configuration:

trackbone apply -z "$ZONE_NAME" -c azure_ingress

Ingress-controller deployment#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c ingress_controller_private -c ingress_controller_public -t bootstrap=true

Whitelists deployment#

In the envs-ng repo:

FEAWS / Azure

trackbone apply -z "$ZONE_NAME" -c fe_whitelist
trackbone apply -z "$INFRA_ZONE" -c fe_whitelist

trackbone apply -z "$INFRA_ZONE" -c fe_whitelist

If the zone has the private_links parameters a bunch of configuration must be added:

FE

trackbone apply -z "${ZONE_NAME}" -c vpc_endpoint_tlsrouter
trackbone apply -z "${ZONE_NAME}" -c vpc_endpoint_dns_2
trackbone apply -z "${ZONE_NAME}" -c fe_vpcep_service_2
trackbone apply -z "${ZONE_NAME}" -c fe_vpcep_endpoint_2

13 - Setup VPN for the loadbalancer#

Add the route to the VPN

Repository: caascad/ansible/vpn
1. Check the requirements
2. Follow the usage section of the documentation
Warning

Don't forget to commit and merge your changes.

14 - Deploy external-dns#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c external_dns -t bootstrap=true

15 - Deploy cert-manager#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c cert-manager -t bootstrap=true

16 - Deploy cluster-proxy#

Server

trackbone apply -z "$ZONE_NAME" -c cluster_proxy_server

Client

trackbone apply -z "$ZONE_NAME" -c fe_cluster_proxy_client

17 - Deploy OPA#

OPA#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c opa -t bootstrap=true

18 - Deploy Vault/Keycloak#

Info

This step must be completed in less than 5 hours after deploying Vault, because the Vault root token is set to expire after this duration. If the token has expired, you won't be able to do the provisioning part of this step, and you will need to chose one of these solutions:

Getting a recovery key
Reinstalling Vault

Vault KMS#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c vault-cloud-kms -t bootstrap=true

Deploy Vault#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c vault -t bootstrap=true

Deploy Keycloak#

Install postgresql-operator (pre-requisite for Keycloak)

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c postgres_operator_crd -t bootstrap=true
trackbone apply -z "$ZONE_NAME" -c postgres_operator -t bootstrap=true

Install Keycloak:

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c keycloak -t bootstrap=true

Provision Keycloak

In the envs-ng repo:
```
trackbone apply -z "$ZONE_NAME" -c keycloak-vault-cloud -t bootstrap=true
```
At this point it should be possible to login into vault using OIDC. Also OIDC clients are provisioned. Test it via vault.$ZONE_NAME.$DOMAIN_NAME.

Deploy rds log#

Apply rds-log configuration:

trackbone apply -z "$ZONE_NAME" -c rds-log -t bootstrap=true

Sync Vault Docker Secrets#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c vault_sync_quay_robots -t bootstrap=true

19 - Deploy S3 Buckets#

In the envs-ng repo:

FEAWSAzure

trackbone apply -z "$ZONE_NAME" -c fe_buckets

trackbone apply -z "$ZONE_NAME" -c aws_buckets

trackbone apply -z "$ZONE_NAME" -c azure_storageaccounts

20 - Deploy Rancher#

Deployment#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c rancher2-helm -t bootstrap=true

Provisioning#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c rancher2-cloud -t bootstrap=true

Warning

Some HTTP 502 errors can occur. Restart the trackbone command.

Note

To get the billing token which will be used to deploy the cloud-client monitoring stack:

export VAULT_ADDR=https://vault.$ZONE_NAME.caascad.com
vault list secret/concourse-infra/global/
vault read secret/concourse-infra/global/kubernetes-${USER}-${ZONE_NAME}
# eg. vault read secret/concourse-infra/global/kubernetes-billing-ocb-demo

21 - Deploy eventrouter#

In envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c eventrouter

22 - Deploy bastion monitoring#

In envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c bastion_monitoring

Then deploy blackbox servicemonitor for ssh monitoring

trackbone apply -z "$ZONE_NAME" -c blackbox_exporter_servicemonitor_cloud_bst

25 - Deploy oauth2-proxy#

!!! warning
    Skip this deployment when `metadata : shared : true` is configured on the cloud zone.
    Skip this deployment when `metadata : cloudavenue : true` is configured on the cloud zone.

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c oauth2-proxy

26 - Deploy Velero#

!!! warning
    Skip this deployment when `metadata : shared : true` is configured on the cloud zone.

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c velero

27 - Deploy Docker exporter#

!!! warning
    Skip this deployment when `metadata : shared : true` is configured on the cloud zone.

In the envs-ng repo:

FE

trackbone apply -z "$ZONE_NAME" -c docker_exporter

28 - Update pipeline k8s-functional-tests#

Now that the cluster is ready, you can update the pipeline k8s-functional-tests.

Get the repository
git clone git@git.corp.caascad.com:caascad/applications/k8s-functional-tests.git
Before loading nix-shell, modify it, and set the branch to the new zone (cf. sample in shell.nix file)
load nix shell
```
nix-shell
```
update pipeline (cf. README.md)

29 - Check everything is ok#

Check that trackbone plan triggers no modifications

Automation#

Prerequisites#

Inputs#

envs-ng#

1 - Requirements#

Domain existence#

Quotas#

2 - Declare zone in zones module (envs-ng)#

3 - Deploy quay robot#

4 - optional but RECOMMENDED#

4 - Bootstrap zone project or account#

5 - Provision base infra#

6 - Setup VPN for the bastion#

7 - Provision the bastion#

8 - Provision RDS databases (FE only)#

9 - Provision Kubernetes Cluster#

10 - Deploy quay robot accounts secrets in cluster & config_sync#

Config-sync-provision#

Config-sync (Kubed)#

10 - Deploy custom storage classes#

11 - Deploy CRDs#

12 - Deploy LB and ingress#

Loadbalancers deployment#

Ingress-controller deployment#

Whitelists deployment#

VPC Endpoints services and related binaries#

13 - Setup VPN for the loadbalancer#

14 - Deploy external-dns#

15 - Deploy cert-manager#

16 - Deploy cluster-proxy#

17 - Deploy OPA#

OPA#

18 - Deploy Vault/Keycloak#

Vault KMS#

Deploy Vault#

Deploy Keycloak#

Deploy rds log#

Sync Vault Docker Secrets#

19 - Deploy S3 Buckets#

20 - Deploy Rancher#

Deployment#

Provisioning#

21 - Deploy eventrouter#

22 - Deploy bastion monitoring#

25 - Deploy oauth2-proxy#

26 - Deploy Velero#

27 - Deploy Docker exporter#

28 - Update pipeline k8s-functional-tests#

29 - Check everything is ok#