Deployment and upgrade of Thanos components

Caascad and client stacks NGOT

The method of deploying Thanos components for NGOT and Cascaad requires Trackbone, without any particular difficulty.

Central stack "mon3/mon4"

Compactor

Thanos compactor is deployed with Trackbone :

trackbone apply -z <zone> -c thanos-store

Note

Despite its name, the thanos-store command does not deploy the Store on the central stacks "mon3" and "mon4"; it only deploys the Compactor. This behavior is specific to client stack deployments, whereas both components are deployed using the same command.

Receive (and Distributor), Store and Querier (rules)

Note

• Distributor component is just another Receiver with a particular configuration (no storage but with a mechanism to transfer data to other Receivers).
• The Receivers, Distributors, Store and Queriers (rules) component are deployed at the same time with the same Helm chart.

1. Variables:

   prd/kub-34prd/kub-53stg/kub-10002stg/kub-10010

   LINE=prd
   MON=mon4
   CLUSTER=kub-34
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2
   

   LINE=prd
   MON=mon3
   CLUSTER=kub-53
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1
   

   LINE=stg
   MON=mon3
   CLUSTER=kub-10002
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1
   

   LINE=stg
   MON=mon4
   CLUSTER=kub-10010
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2
   

2. Get into the git repo:

   git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
   cd helm/thanos-v3
   

3. Log in Vault:

   export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com/
   vault token lookup > /dev/null 2>&1 || vault login -method oidc
   

4. Retrieve secrets into xxx-credentials.yaml:

   ./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml
   

   Note

   The ${LINE}/${MON}-credentials.yaml is in .gitignore file so you will not be able to see it with git status.

5. Check the contents of xxx-credentials.yaml:

   cat ${LINE}/${MON}-credentials.yaml
   

   Ensure that:

   • the file is not empty
   • the file contains secrets (e.g. the secrets are not empty)

6. Verify the new configuration:

   kswitch "${ZONE_NAME}"
   helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-${LINE}" "thanos-monitoring" . \
       -f values-stack.yaml \
       -f ${LINE}/${MON}.yaml \
       -f ${LINE}/${MON}-credentials.yaml
   

   Check if there are any unwanted difference in the output.

   In case of unexpected difference, you may try with the option --three-way-merge.

7. If you want to follow the rolling out of the deployment, use one of the following methods:

   with K9Swithout k9s

   • Run K9s
   • change namespace to one of
     • monitoring-stack-corp-obs-corp-stg
     • monitoring-stack-corp-obs-corp-prd
   • observe the thanos-receive-X pods

   • Run this command:

     watch -n 2 "kubectl -n monitoring-stack-corp-obs-corp-${LINE} get pod |grep receive"
     

   • do not pay attention to the "distributor" pods.

8. Scale down (upgrade only)

   Before upgrading, scale down the Receivers and Distributors to speed up the upgrade:

   kubectl -n "monitoring-stack-corp-obs-corp-${LINE}" \
       scale deployment thanos-receive-distributor \
       --replicas=0
   
   kubectl -n "monitoring-stack-corp-obs-corp-${LINE}" \
       scale statefulset thanos-receive \
       --replicas=0
   

   Then redeploy as fast as possible to prevent the Auto-scaler from removing nodes.

9. Deploy

   Ensure that you ran helm diff and check the results (because it is so easy to confuse yourself with kswitch and $MON variable).

   helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-${LINE}" --atomic --wait --skip-crds "thanos-monitoring" . \
       -f values-stack.yaml \
       -f ${LINE}/${MON}.yaml \
       -f ${LINE}/${MON}-credentials.yaml \
       --timeout 20m
   

   Note

   You can safely ignore this line:

   coalesce.go:237: warning: skipped value for thanos-v3.thanos.objstoreConfig: Not a table.
   

The operation may be long, mostly because some new nodes may be spawned by the auto-scaler. The Thanos-Receive pods need to be all deployed; then the distributors will restart before the helm upgrade command is finished. Waiting is normal. You may wait watching the k9s or watch command you prepared earlier.

Querier (proxy)

Querier (proxy) deployment is a manual deployment (with Helm) different from other deployments. The first steps are nevertheless common.

1. Variables:

   prd/kub-34prd/kub-53stg/kub-10002stg/kub-10010

   LINE=prd
   MON=mon4
   CLUSTER=kub-34
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2
   

   LINE=prd
   MON=mon3
   CLUSTER=kub-53
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1
   

   LINE=stg
   MON=mon3
   CLUSTER=kub-10002
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1
   

   LINE=stg
   MON=mon4
   CLUSTER=kub-10010
   CONTRACT=obs-corp-${LINE}
   ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2
   

2. Get into the git repo:

   git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
   cd helm/thanos-v3
   

3. Log in Vault:

   export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com/
   vault token lookup > /dev/null 2>&1 || vault login -method oidc
   

4. Retrieve secrets into xxx-credentials.yaml:

   ./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml
   

   Note

   The ${LINE}/${MON}-credentials.yaml is in .gitignore file so you will not be able to see it with git status.

5. Check the contents of xxx-credentials.yaml:

   cat ${LINE}/${MON}-credentials.yaml
   

   Ensure that:

   • the file is not empty
   • the file contains secrets (e.g. the secrets are not empty)

6. Test the deployment

   kswitch "${ZONE_NAME}"
   helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-$LINE" "thanos-monitoring-querier-proxy" . \
       -f values-querier-proxy.yaml \
       -f $LINE/$MON-querier-proxy.yaml
   

7. Deploy

   helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-$LINE" --atomic --wait --skip-crds "thanos-monitoring-querier-proxy" . \
       -f values-querier-proxy.yaml \
       -f $LINE/$MON-querier-proxy.yaml \
       --timeout 20m
   

Ruler

Thanos Ruler is deployed with Prometheus-Operator, itself deployed with Trackbone.

trackbone apply -z "${ZONE_NAME}" -c kube-prometheus-stack

Check the deployment

In order to test, go to test directory of Thanos repository:

git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
cd tests
nix-shell
kswitch "${ZONE_NAME}"
./check_thanos_central_v3.sh "${ZONE_NAME}"