Aller au contenu

Scale Thanos Receiver#

Introduction#

If you encounter memory or volume usage issues concerning Thanos Receiver, you need to scale the number of Receiver replicas.

Warning

Never update the memory R/L or volume size of a thanos-receiver-X pod/Statefulset. Instead, follow the procedure below.

Steps#

Prerequisites#

  1. Variables:

    LINE=prd
MON=mon4
CLUSTER=kub-34
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

    LINE=prd
MON=mon3
CLUSTER=kub-53
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon3
CLUSTER=kub-10002
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon4
CLUSTER=kub-10010
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

  2. Get into the git repo:

    git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
cd helm/thanos-v3

  3. Log in Vault:

    export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com/
vault token lookup > /dev/null 2>&1 || vault login -method oidc

  4. Retrieve secrets into xxx-credentials.yaml:

    ./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml

    Note

    The ${LINE}/${MON}-credentials.yaml is in .gitignore file so you will not be able to see it with git status.

  5. Check the contents of xxx-credentials.yaml:

    cat ${LINE}/${MON}-credentials.yaml

    Ensure that:

    • the file is not empty
    • the file contains secrets (e.g. the secrets are not empty)

Modify the configuration#

  1. Change the number of replicas in the configuration:

    • edit the file ${LINE}/${MON}.yaml,
    • in the section thanos.receive.replicaCount set the new value (increment/reduce by 1 if you don't know how much).

  2. Verify the new configuration:

    kswitch "${ZONE_NAME}"
helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-${LINE}" "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml

    Check if there are any unwanted difference in the output.

    In case of unexpected difference, you may try with the option --three-way-merge.

Prepare to watch the deployment progress (optional)#

  1. If you want to follow the rolling out of the deployment, use one of the following methods:

    • Run K9s
    • change namespace to one of
      • monitoring-stack-corp-obs-corp-stg
      • monitoring-stack-corp-obs-corp-prd
    • observe the thanos-receive-X pods
    • Run this command: 
      watch -n 2 "kubectl -n monitoring-stack-corp-obs-corp-${LINE} get pod |grep receive"
    • do not pay attention to the "distributor" pods.

Scale down Thanos-receive statefulset#

This step is optional, but highly recommended because it allows you to reduce the downtime.

Warning

In the deployment procedure below, you will scale down the pod. E.g. remove all the receivers. This is unusual but you must do it.

Explanation: during this procedure, Thanos can be considered as broken. If you scale down before, the deployment will be done fast, in parallel. If you don't scale down before, the deployment procedure will replace each pod one by one and it takes a lot more time.

kubectl -n "monitoring-stack-corp-obs-corp-${LINE}" \
    scale statefulset thanos-receive \
    --replicas=0

The pods will change their state to Terminating. Wait until they all disappeared.

Deploy the new configuration#

The following command applies the new number of Thanos Receiver replicas:

Ensure that you ran helm diff and check the results (because it is so easy to confuse yourself with kswitch and $MON variable).

helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-${LINE}" --atomic --wait --skip-crds "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml \
    --timeout 20m

Note

You can safely ignore this line: 

coalesce.go:237: warning: skipped value for thanos-v3.thanos.objstoreConfig: Not a table.

Note

The operation may be long, mostly because some new nodes may be spawned by the auto-scaler. The Thanos-Receive pods need to be all deployed; then the distributors will restart before the helm upgrade command is finished. Waiting is normal. You may wait watching the k9s or watch command you prepared earlier.

Verify#

  • Run the functional test:

    git clone git@git.corp.caascad.com:caascad/applications/poc_check_functional_tests_monitoring.git && cd poc_check_functional_tests_monitoring/ngot
export NB_THANOS_RECEIVER_PODS=<the nb of wanted Thanos-Receiver pods>
./func_tests_svc_monitoring_stack_corp.sh "${ZONE_NAME}"

    Note

    The repo above is named with a prefix poc. This used to be a POC but with time being, it is now used in Production (and should be renamed one day).

  • In case of alert ContainerOutOfMemory:

    • Check the memory usage in Grafana:

      sum(container_memory_working_set_bytes{cluster="kub-34",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-prd", pod=~"thanos-receive-[0-9]*",container="receive"}) by (pod,container)

      sum(container_memory_working_set_bytes{cluster="kub-53",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-prd", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

      sum(container_memory_working_set_bytes{cluster="kub-10002",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-stg", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

      sum(container_memory_working_set_bytes{cluster="kub-10010",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-stg", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

    • The alert should disappear.

Warning

A new alert PrometheusTSDBBlocksLoadedLow may appear after the new deployment.

You can safely ignore it and silence it for 6 hours.

Warning

Restarting thanos-receive will produce a Remote-Write incident. Use the following documentation to fix it.

You can identify the incident with:

  • alerts in Karma

  • with 2 expressions (use both, one is not sufficient) in Grafana:

    count(rate(prometheus_remote_storage_samples_dropped_total{namespace="monitoring",reason!="dropped_series",remote_name=~"prometheus-mon[34]"}[1m]) > 0) by (reason,remote_name)

    count(rate(prometheus_remote_storage_samples_failed_total{remote_name=~"prometheus-mon[34]"}[1m]) > 0) by (remote_name)

    The expected result of these expressions is no data.

Merge#

Do not forget to add your modifications into a Merge Request, and validate/merge it following the usual workflow.