Aller au contenu

Increase Thanos volume#

This procedure explains how to increase the size of the Thanos volume and it can be applied for thanos-receive and thanos-storegateway of the new monitoring stack (mon3/mon4).

In order to identify which Thanos component requires a volume increase, check the information provided by the label persistentvolumeclaim in the alert.

Thanos Storegateway#

Prerequisites#

  1. Variables:

    LINE=prd
MON=mon4
CLUSTER=kub-34
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

    LINE=prd
MON=mon3
CLUSTER=kub-53
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon3
CLUSTER=kub-10002
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon4
CLUSTER=kub-10010
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

  2. Get into the git repo:

    git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
cd helm/thanos-v3

  3. Log in Vault:

    export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com/
vault token lookup > /dev/null 2>&1 || vault login -method oidc

  4. Retrieve secrets into xxx-credentials.yaml:

    ./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml

    Note

    The ${LINE}/${MON}-credentials.yaml is in .gitignore file so you will not be able to see it with git status.

  5. Check the contents of xxx-credentials.yaml:

    cat ${LINE}/${MON}-credentials.yaml

    Ensure that:

    • the file is not empty
    • the file contains secrets (e.g. the secrets are not empty)

Edit the configuration#

  1. Increase the volume size

    • edit the file ${LINE}/${MON}.yaml,
    • set the new size in the section thanos.storegateway.persistence.size.

  2. Verify the new configuration:

    kswitch "${ZONE_NAME}"
helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-${LINE}" "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml

    Check if there are any unwanted difference in the output.

    In case of unexpected difference, you may try with the option --three-way-merge.

Redeploy#

  1. Remove the statefulset and the pvc:

    COMPONENT=thanos-storegateway

    kubectl get sts -n monitoring-stack-corp-obs-corp-$LINE | grep ${COMPONENT}
kubectl delete sts ${COMPONENT} -n monitoring-stack-corp-obs-corp-$LINE
kubectl delete pvc data-${COMPONENT}-0 -n monitoring-stack-corp-obs-corp-$LINE

  2. Redeploy:

    Ensure that you ran helm diff and check the results (because it is so easy to confuse yourself with kswitch and $MON variable).

    helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-${LINE}" --atomic --wait --skip-crds "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml \
    --timeout 20m

    Note

    You can safely ignore this line: 

    coalesce.go:237: warning: skipped value for thanos-v3.thanos.objstoreConfig: Not a table.

  3. Avoid drifts. Check with helm diff on all stack: replay first steps including the (helm diff upgrade command) on all stacks:

    • stg/mon3,
    • stg/mon4,
    • prd/mon3,
    • prd/mon4.

  4. Merge MR for Thanos helm chart.

Thanos Receive#

Apply this procedure only if you know what you are doing. In most case you want to increase the number of Thanos Receive replicas instead.

Caution

Do not increase volume size for Thanos Receive without validation from Monitoring team.

This is not an usual operation, because the sizing of memory and volume is optimised for small replicas and any modification will break this optimisation.

If you are sure you want to increase the volume size of Thanos receive replicas, follow the same procedure as above for Thanos Storegateway.

The differences are:

  1. Prerequisites:

    COMPONENT=thanos-receive # instead of thanos-storegateway

  2. Increase Thanos Receiver volume size in helm chart:

    Edit thanos.receive.persistence.size section in $LINE/$MON.yaml (instead of thanos.storegateway.persistence.size section).

  3. If you want to follow the rolling out of the deployment, use one of the following methods:

    • Run K9s
    • change namespace to one of
      • monitoring-stack-corp-obs-corp-stg
      • monitoring-stack-corp-obs-corp-prd
    • observe the thanos-receive-X pods
    • Run this command: 
      watch -n 2 "kubectl -n monitoring-stack-corp-obs-corp-${LINE} get pod |grep receive"
    • do not pay attention to the "distributor" pods.

  4. When applying the configuration, remove the statefulset and all the pvc before applying the helm chart:

    kswitch "${ZONE_NAME}"
kubectl get sts -n monitoring-stack-corp-obs-corp-$LINE | grep ${COMPONENT}
kubectl delete sts ${COMPONENT} -n monitoring-stack-corp-obs-corp-$LINE
kubectl delete pvc data-${COMPONENT}-0 -n monitoring-stack-corp-obs-corp-$LINE
kubectl delete pvc data-${COMPONENT}-1 -n monitoring-stack-corp-obs-corp-$LINE
...
kubectl delete pvc data-${COMPONENT}-XX -n monitoring-stack-corp-obs-corp-$LINE

Do not forget to check the drifts, get the MR validated and merged.

Note

This step should be performed fast in order to avoid the autoscaler to downscale the K8s nodes.

Warning

In case of problem (rollback,...), downscale the replicas to 0 to speed-up the redeployment of the receivers: 

kubectl scale statefulset -n monitoring-stack-corp-obs-corp-${LINE} thanos-receive --replicas=0
Redeploy ASAP to avoid the autoscaler to downscale the K8s nodes.

Warning

If you have a lot of alerts in Karma after applying this configuration, you will probably need to restart all the Prometheus cluster by following the next procedure.