Aller au contenu

Monitoring#

Preparation#

export ZONE_NAME="ocb-..."
export INFRA_ZONE_NAME="infra-..."

Infra zone redeployments#

Warning

Most operations cannot be done before the zone is removed from caascad-zones in Automation part. You cannot start before this removal.

Delete Prometheus federation for billing#

kswitch ${INFRA_ZONE_NAME}
helm uninstall -n monitoring-consumption monitoring-infra-consumption-federate-${ZONE_NAME}

Update Monitoring functional tests#

Redeploy functional tests without the zone

Create a new git branch 

cd ~/git/caascad/pipelines/monitoring-functional-tests
git switch -c "decom_${ZONE_NAME}_functional-test"
grep "$ZONE_NAME" ci/functional_tests.cue

If some files were found, remove the references to the zone in the matching lines.

Then update the nix pointer to Caascad-zones and redeploy the functional tests. Reference procedure is in README.md file of the functional tests repository.

nix-shell --run "
  niv update zones
  cd ci
  fly switch -z "${INFRA_ZONE_NAME}" -n test;
  fly -t "${INFRA_ZONE_NAME}.infra" login -n test
  fly -t "${INFRA_ZONE_NAME}.infra" destroy-pipeline -p "f-tests-monitoring-${ZONE_NAME}"
"

Check on the CI (Staging or Production - change for the cloud zone being removed) that the zone was removed from the functional tests.

Tip

This operation will be the same for Logging. Both operations can be mutualized. The MR below can also be mutualized.

At last,

  • create a MR and ask for validation
  • merge it into master branch

Remove the history of the zone in the functional tests :

kswitch ${INFRA_ZONE_NAME}
kubectl -n functional-tests-monitoring delete pod -l app=prometheus-pushgateway,release=functional-tests
sleep 2
kubectl -n functional-tests-monitoring get pod -l app=prometheus-pushgateway,release=functional-tests

Ensure the pushgateway was restarted.

Trackbone configurations#

cd contexts/caascad
trackbone apply -z ${INFRA_ZONE_NAME} -c prometheus-rules
trackbone apply -z ${INFRA_ZONE_NAME} -c kube-prometheus-stack-infra-consumption 
trackbone apply -z ${INFRA_ZONE_NAME} -c karma -t karma_refresh_cache=true

Envs-ng configuration update#

It is possible to have zone-specific configurations in envs-ng which have to be removed.

Tip

This operation will be the same for Logging. Both operations can be mutualized. The MR below can also be mutualized.

cd caascad/terraform/envs-ng
grep -r "$ZONE_NAME" *.cue */*.cue */*/*.cue # only in files managed by monitoring team
git switch -c "decom_${ZONE_NAME}_monitoring"

If some files were found,

  • remove the references to the zone in the matching lines
  • create a MR and ask for validation
  • merge it into master branch

Checkpoints#

The points to verify are defined in the Jira template. Here are some details: 

No references to zones:

* (on) in envs-ng  
* (on) in Vault  
** (on) path: zones/fe  
* (on) in the documentation  

In the FE console, no resources linked to zones and the cluster:

* (on) VMs  
* (on) S3 buckets  
* (on) ELB  
* (on) CCE  

Karma:

* (on) No alerts for the zone 
* (on) No alerts in infra-prd for the zone
* (on) No silences for the zone