Aller au contenu

Automation#

Prerequisites#

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="..." # also known as <cluster_ID>
export CLOUD_ZONE_NAME="ocb-..."
export INFRA_ZONE_NAME="infra-..."
export CONTEXT="caascad" # where context can be caascad/ngotalpha/pf/...

Warning

During this procedure, several steps require restarts of different components in cloud zones. Before each restart:

  • check there is no CAASCHR that is planned on the same component at the same time
  • warn supervsion team on #supervision about the component/zone that is going to be restarted

envs-ng#

Tip

Many steps will consist of running trackbone apply -z "$ZONE_NAME" -c ...

Reusing the shell setup here is a good idea!

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$ZONE_NAME"
nix-shell

1 - Requirements#

Various informations should be checked.

  1. Check if support.caascad can access the zone's domain as administrator
  2. Contact caascad support team to set the API password if the zone is in a new domain
  3. If a new project is required in the domain, create it as support.caascad
  4. Check the quotas (pie chart icon on the top right in the web console):
    1. Cloud Container Engine
    2. Elastic IP (at least 2 available)
    3. User Groups should allow for one RDS service account per zone (recommended value 100 if possible)
    4. Volumes, etc...
  5. Enable CCE via the web console and click on "Agree"
  6. If the network was provided by the customer (custom networks in caascad zones) and the zone uses private_links, ensure the subnets used have nameservers filled with the VPC endpoint DNS IP
  7. If the network is provided by us (managed networks in caascad zones) and the zone uses private_links, ensure the nameservers attribute in caascad zones is filled with the VPC endpoint DNS IP
  1. Declare information about the AWS account
  1. If the new zone is a production zone, ask the client to delegate access to his subscription using the azure-caascad-iam stack. Before he can execute it, we need to give him the inputs which are located on the Infra Vault under the secret/zones/\(parent_zone.provider.type)/\(parent_zone.name)/azure_delegation_info path.

2 - Declare zone in zones module (envs-ng)#

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$ZONE_NAME"
nix-shell
  1. Edit zone file regarding the type of zone you want to add
  • zones/caascad_zones/zones.cue
  • zones/caascad_zones/zones_cloudavenue.cue
  • zones/ngot_zones/values.cue

To declare the new zone. Use any client zone as an example.

  1. Generate static files and commit your changes
# Run script to generate static files from zones
generate-static-zones-files
# Add all modifications to the commit
git add zones/
git add gen/
git commit -m "Add $ZONE_NAME zone"
# Create target branch
git push -u origin "$ZONE_NAME"
  1. Create and submit your MR
  • Create the MR with Draft status, then update status to Ready. (this will bypass mr-plan pipeline)

  • Comment your MR with message : check zones to trigger check-zones pipeline and validate the impacts of your changes on environments.

    Info

    Please wait for the CI job completion before asking a validation.

    Info

    Beware of client zones needing to talk to cloud zones via vpc endpoints. In such case the cloud zone should be filled with the private_link parameter listing details on the transit subnet.

    Info

    When zones medatada:shared is true then parameters: apps: quay and parameters: apps: harbor must be set to false

3 - Bootstrap zone project/account#

Go to the context directory where your new zone is handled, this could be: - contexts/caascad - contexts/pf - contexts/ngotalpha - contexts/ngot

and run trackbone apply commands.

  1. Follow the right section depending on the kind of client zone you want to create.

    Caascad zones are used by caascad only (no real client).

    In this case, we need to create a project/account

    Apply fe_bootstrap_project configuration:

    trackbone apply -z "$ZONE_NAME" -c fe_bootstrap_project

    Follow run configurations to create and initialize the account

    Customer zones are zones which belong to a real client, co-managed by Caascad.

    In this case, the client already has a project/account, so we mustn't create another one.

    Write API secret in vault for customer project: 

    # Create a JSON file with API secrets

export DOMAIN="<domain OCB...>"                    # the customer FE domain
export PASSWORD="<support.caascad password>"       # API password of the support.caascad user
export PROJECT_NAME="<eu-west-0 or eu-west-0_...>" # name of the project that was created in the customer domain

cat << EOF > customer-api-secret.json
{
    "auth_url": "https://iam.eu-west-0.prod-cloud-ocb.orange-business.com/v3",
    "domain_name": "${DOMAIN}",
    "password": "${PASSWORD}",
    "region": "eu-west-0",
    "tenant_name": "${PROJECT_NAME}",
    "username": "support.caascad"
}
EOF

# Write secret in vault

export VAULT_ADDR="https://vault.infra-prd.caascad.com"
vault login -method oidc
vault write secret/zones/fe/${ZONE_NAME}/api @customer-api-secret.json
vault read secret/zones/fe/${ZONE_NAME}/api

rm customer-api-secret.json

    Follow run configurations to initialize the account

4 - Provision base infra#

In the envs-ng repo:

  1. Apply base configuration:

    trackbone apply -z "$ZONE_NAME" -c fe_base

    trackbone apply -z "$ZONE_NAME" -c aws_base

    1. Make sure IAM for the cloud zone is setup 

      trackbone apply -z "$CLOUD_ZONE_NAME" -c azure_iam_cloud

      Warning

      If the cloud zone is not an Azure zone, uncomment the configuration in envs.cue before executing it.

      Info

      TODO: find a way to only execute the configuration when the cloud zone has one or more Azure child zone(s).

    2. If the new zone is a caascad staging zone, setup delegation 

      trackbone apply -z "$ZONE_NAME" -c azure_caascad_iam

    3. Deploy base infrastructure 

      trackbone apply -z "$ZONE_NAME" -c azure_base
trackbone apply -z "$ZONE_NAME" -c azure_network

  2. Apply instances configuration:

    trackbone apply -z "$ZONE_NAME" -c fe_bastion

    trackbone apply -z "$ZONE_NAME" -c aws_instances

    trackbone apply -z "$ZONE_NAME" -c azure_instances

  3. Store the bastion EIP:

    export BST_EIP="$(dig +short bst.${ZONE_NAME}.caascad.com)"

5 - Setup VPN for the bastion#

  1. Temporarily add the route locally:

    sudo ip route add "${BST_EIP}/32" dev "$(ip -4 -o a | awk '/10.254.255/ {print $2}')"

  2. Add the route to the VPN:

    Repository: caascad/ansible/vpn

    1. Check the requirements
    2. Follow the usage section of the documentation

6 - Provision the bastion#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c bastion_provisioning -t bootstrap=true

7 - Provision Kubernetes Cluster#

In the envs-ng repo:

  1. Edit cce.cue if you need a specific configuration. Otherwise, you can keep the defaults.

  2. Apply cce configuration:

    trackbone apply -z "$ZONE_NAME" -c cce -t bootstrap=true

  1. Edit eks.cue if you need a specific configuration. Otherwise, you can keep the defaults.

  2. Apply eks configuration:

    trackbone apply -z "$ZONE_NAME" -c eks -t bootstrap=true

  1. Edit aks.cue if you need a specific configuration. Otherwise, you can keep the defaults.

  2. Apply aks configuration:

trackbone apply -z "$ZONE_NAME" -c aks -t bootstrap=true

  1. Check that the nodes are ready:

    kswitch "$ZONE_NAME"
kubectl get nodes

8 - Deploy the CRDs manifests for prometheus:#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c prometheus-operator-crds

9 - Deploy quay robot accounts secrets in cluster#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c config_sync_provision -t bootstrap=true

10 - Add client cluster to rancher#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c rancher2-client
trackbone apply -z "$ZONE_NAME" -c rancher2-client-cattle-agent -t bootstrap=true
trackbone apply -z "$ZONE_NAME" -c blackbox-exporter-servicemonitor-rancher-client

Note

To get the tokens from vault (new way to get tokens post rancher2-v2.7.0 release):

export VAULT_ADDR=https://vault.${CLOUD_ZONE}.caascad.com
vault list secret/concourse-infra/global/
vault read secret/concourse-infra/global/kubernetes-${USER}-${ZONE}
# eg. vault read secret/concourse-infra/global/kubernetes-monitoring-delta

11 - Provision Keycloak resources#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c keycloak-vault-client

12 - Deploy metrics server#

Warning

Only on AWS:

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c metrics_server

13 - Deploy cluster_autoscaler#

Warning

Only on AWS: In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c cluster_autoscaler

14 - Install eventrouter#

In the envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c eventrouter

15 - Deploy bastion monitoring#

In envs-ng repo:

trackbone apply -z "$ZONE_NAME" -c bastion_monitoring

16 - Deploy Docker exporter#

In the envs-ng repo:

Warning

Only on FE: 

trackbone apply -z "$ZONE_NAME" -c docker_exporter

17 - Deploy vault injector#

Warning

Only if the feature is asked by customer 

trackbone apply -z "$ZONE_NAME" -c vault_injector

17 - Gitea Provisioning#

trackbone apply -z $ZONE_NAME -c gitea_provision

18 - Concourse Provisioning#

trackbone apply -z $ZONE_NAME -c concourse_provisioning_client
trackbone apply -z $ZONE_NAME -c concourse_provisioning_infra

Verify the proper execution and destruction all of resources in kubernetes.

Check if all the desired workers are available:

fly switch -z $CLOUD_ZONE_NAME -n $ZONE_NAME -c
fly workers

Tip

You need to be admin on Concourse in order to list the workers. If you have a blank page or a 401 error, please follow the Troobleshooting documentation.

19 - Update pipeline k8s-functional-tests#

Now that the cluster is ready, you can update the pipeline k8s-functional-tests.

  • Get the repository
    git clone git@git.corp.caascad.com:caascad/applications/k8s-functional-tests.git

  • Before loading nix-shell, modify it, and set the branch to the new zone (cf. sample in shell.nix file)

  • load nix shell 

    nix-shell

  • update pipeline (cf. README.md)