Aller au contenu

Generate new secret for Thanos ingress#

Important

This procedure applies only to Thanos central monitoring stack (mon3/mon4)

Prerequisites#

  1. Variables:

    LINE=prd
MON=mon4
CLUSTER=kub-34
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

    LINE=prd
MON=mon3
CLUSTER=kub-53
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon3
CLUSTER=kub-10002
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-1

    LINE=stg
MON=mon4
CLUSTER=kub-10010
CONTRACT=obs-corp-${LINE}
ZONE_NAME=svc-monitoring-stack-corp-${LINE}-2

  2. Get into the git repo:

    git clone git@git.corp.caascad.com:caascad/applications/caascad-thanos.git /tmp/thanos-v3 && cd /tmp/thanos-v3
cd helm/thanos-v3

  3. Log in Vault:

    export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com/
vault token lookup > /dev/null 2>&1 || vault login -method oidc

  4. Retrieve secrets into xxx-credentials.yaml:

    ./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml

    Note

    The ${LINE}/${MON}-credentials.yaml is in .gitignore file so you will not be able to see it with git status.

  5. Check the contents of xxx-credentials.yaml:

    cat ${LINE}/${MON}-credentials.yaml

    Ensure that:

    • the file is not empty
    • the file contains secrets (e.g. the secrets are not empty)

Create a new secret and push it into Vault#

DATE=$(date +%Y%m%d-%H%M%S)

SECRET_USERNAME=user
SECRET_PASSWORD="$(pwgen -sy -r '\\"' 32 1)"
SECRET_HASH=$(htpasswd -nbB -C 5 "${SECRET_USERNAME}" "${SECRET_PASSWORD}")

export VAULT_ADDR=https://vault.infra-${LINE}.caascad.com; vault token lookup || vault login -method oidc
vault write "secret/zones/fe/${ZONE_NAME}/prometheus-ingress/prometheus-ingress-${DATE}" hash="${SECRET_HASH}" username="${SECRET_USERNAME}" password="${SECRET_PASSWORD}"

Regenerate secret#

cd /tmp/thanos-v3/helm/thanos-v3
./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml

Check the contents of xxx-credentials.yaml

cat ${LINE}/${MON}-credentials.yaml

Ensure that:

  • the file is not empty
  • the file contains secrets (e.g. the secrets are not empty)

Redeploy Thanos Receive with new secret#

  1. Verify the new configuration:

    kswitch "${ZONE_NAME}"
helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-${LINE}" "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml

    Check if there are any unwanted difference in the output.

    In case of unexpected difference, you may try with the option --three-way-merge. Ensure that you ran helm diff and check the results (because it is so easy to confuse yourself with kswitch and $MON variable).

helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-${LINE}" --atomic --wait --skip-crds "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml \
    --timeout 20m

Note

You can safely ignore this line: 

coalesce.go:237: warning: skipped value for thanos-v3.thanos.objstoreConfig: Not a table.

Redeploy all Cluster Prometheus that send metrics to these central Prometheus#

trackbone apply -s zone_type=cluster -t prod -c kube-prometheus-stack --non-interactive -n 5

trackbone apply -s zone_type=cluster -t staging -c kube-prometheus-stack --non-interactive -n 5

Clean-up#

After making sure that the new secret works,you can delete the old secret in Vault. The secret is stored in the following path: secrets/secret/zones/fe/<service_zone_name>/prometheus-ingress/prometheus-ingress-YYYYMMDD-HHMMSS, where YYYYMMDD-HHMMSS indicates the date and hour when the secret has been generated.

Redeploy Thanos Receive without old secret#

cd /tmp/thanos-v3/helm/thanos-v3
./generate_creds_values.sh ${ZONE_NAME} > ${LINE}/${MON}-credentials.yaml

Check the contents of xxx-credentials.yaml

cat ${LINE}/${MON}-credentials.yaml

  1. Ensure that:

    • the file is not empty
    • the file contains secrets (e.g. the secrets are not empty)

  2. Verify the new configuration:

    kswitch "${ZONE_NAME}"
helm diff upgrade --namespace "monitoring-stack-corp-obs-corp-${LINE}" "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml

    Check if there are any unwanted difference in the output.

    In case of unexpected difference, you may try with the option --three-way-merge. Ensure that you ran helm diff and check the results (because it is so easy to confuse yourself with kswitch and $MON variable).

helm upgrade --install --namespace "monitoring-stack-corp-obs-corp-${LINE}" --atomic --wait --skip-crds "thanos-monitoring" . \
    -f values-stack.yaml \
    -f ${LINE}/${MON}.yaml \
    -f ${LINE}/${MON}-credentials.yaml \
    --timeout 20m

Note

You can safely ignore this line: 

coalesce.go:237: warning: skipped value for thanos-v3.thanos.objstoreConfig: Not a table.

Redeploy Blackbox-Exporter, because it uses Prometheus secrets to monitor it#

trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-1 -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-prd-2 --add-services

trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-stg-1 -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -c blackbox-exporter-core -z svc-monitoring-stack-corp-stg-2 --add-services