Monitoring#

Prerequisites#

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="..."
export CLOUD_ZONE_NAME="ocb-..."
export INFRA_ZONE_NAME="infra-..."
export CONTEXT="caascad" # where context can be caascad/ngotalpha/pf/...

Warning

During this procedure, several steps require restarts of different components in cloud zones. Before each restart:

check there is no CAASCHR that is planned on the same component at the same time
warn supervsion team on #supervision about the component/zone that is going to be restarted

Important

Next steps should be done after the zone has been removed from zones.cue (by Automation team).

Prometheus federation#

eval "$(rswitch login ${CLOUD_ZONE_NAME} -e)"
helm uninstall -n monitoring-client monitoring-cloud-client-federate-${ZONE_NAME}
helm uninstall -n monitoring-client recordingrule-cluster-info-${ZONE_NAME}
helm uninstall -n monitoring-app monitoring-cloud-app-federate-${ZONE_NAME}

Prometheus rule#

eval "$(rswitch login ${CLOUD_ZONE_NAME} -e)"
helm uninstall -n monitoring-client recordingrule-cluster-info-${ZONE_NAME}

Update Monitoring functional tests#

It is possible to have zone-specific configurations in Monitoring functional tests (switch to the parent cloud zone) that have to be removed.

cd ${GIT_PATH}/caascad/pipelines/monitoring-functional-tests
git checkout -b "decom_${ZONE_NAME}_functional-test"
grep "$ZONE_NAME" ci/functional_tests.cue

If some files were found, remove the references to the zone in the matching lines.

Then update the nix pointer to Caascad-zones and redeploy the functional tests. Reference procedure is in README.md file of the functional tests repository.

nix-shell --run "
  niv update zones
  cd ci
  fly switch -z "${INFRA_ZONE_NAME}" -n test
  fly -t "${INFRA_ZONE_NAME}.infra" login -n test
  cue -t infra_zone="${INFRA_ZONE_NAME}" -t zone="${CLOUD_ZONE_NAME}" apply-pipelines
"

Check on the CI (Staging or Production) - change for the cloud zone of the zone being removed) that the zone was removed from the functional tests.

At last,

create a MR and ask for validation
merge it into master branch

Remove the history of the zone in the functional tests :

kswitch ${INFRA_ZONE_NAME}
kubectl -n functional-tests-monitoring delete pod -l app=prometheus-pushgateway,release=functional-tests
sleep 2
kubectl -n functional-tests-monitoring get pod -l app=prometheus-pushgateway,release=functional-tests

Ensure the pushgateway was restarted.

Trackbone configurations#

Run trackbone jobs:

cd ${GIT_PATH}/caascad/terraform/envs-ng/
nix-shell
cd contexts//${CONTEXT}/
trackbone apply -z ${CLOUD_ZONE_NAME} -c prometheus-rules
trackbone apply -z ${INFRA_ZONE_NAME} -c prometheus-rules
trackbone apply -z ${CLOUD_ZONE_NAME} -c kube-prometheus-stack-cloud-client
trackbone apply -z ${CLOUD_ZONE_NAME} -c kube-prometheus-stack-cloud-app

Verifications#

Functional tests#

Launch automatized functional tests:

fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${CLOUD_ZONE_NAME}/${stack}-${CLOUD_ZONE_NAME}"
done

Check results in Concourse UI or with this:

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra jobs -p "f-tests-${stack}-${CLOUD_ZONE_NAME}"
done

Prometheus rules#

Parent Cloud zone:

eval "$(rswitch login ${CLOUD_ZONE_NAME} -e)"
kubectl get prometheusrules -n prometheusrules -o yaml | grep ${ZONE_NAME}

Expected result : no line.

eval "$(rswitch login ${CLOUD_ZONE_NAME} -e)"
kubectl -n monitoring-client get prometheusrules cluster-info-${ZONE_NAME}

Expected result : no line.

eval "$(rswitch login ${CLOUD_ZONE_NAME} -e)"
helm list -A -a | grep ${ZONE_NAME}

Expected result : no line.

Infra zone:

eval "$(rswitch login ${INFRA_ZONE_NAME} -e)"
kubectl get prometheusrules -n prometheusrules -o yaml | grep ${ZONE_NAME}

Expected result : no line.

Important

If the prometheus rules of the decommissioned zone are still present in parent cloud zone and infra zone, probably souces.json file does not point toward the good commit id in zones.cue. If it's the case, contact Automation team.

Karma#

In Karma, browse alerts with filter cc_prom_source=<zone name> (all alerts, not only active ones). There should be no alerts left.

In Karma, browse silences with cc_prom_source=<zone name>:

there should be no alerts any more for the remaining silences
remove all silences (at least those that expire in more than 15 minutes)

Note

If some alert remains, you can probably wait a few minutes and it will disappear automatically.

Envs-ng configuration update#

It is possible to have zone-specific configurations in envs-ng which have to be removed.

cd ${GIT_PATH}/caascad/terraform/envs-ng
egrep -rwo "$ZONE_NAME" # only in files managed by monitoring team

If some files were found,

remove the references to the zone in the matching lines
create a MR and ask for validation
merge it into master branch

Merge to master#

The decomissioning of a client zone should have the following MR:

functional tests
envs-ng : if custom values are set in this repository

Ask for approval
Merge into master

Prometheus Request Limit update#

Important

This step must be executed at least 5 days after zone decommissioning.

When decommissioning a client zone, it is necessary to update R/L of Prometheus of the cloud zone to which the client zone was attached to:

Prometheus cloud-client;
Prometheus cloud-app.

In order to evaluate the new values to be set in place:

Go to the Grafana interface of the cloud zone (https://grafana-infra.ocb-.caascad.com/):
- Kubernetes/OBS/Kubernetes/Compute Resources/Pod dashboard
  - datasource Thanos
    - namespace monitoring-client/monitoring-app
      - pod prometheus-client-prometheus-{0,1}
        
        CPU Usage/Memory Usage(WSS) tables.
Divide the last two days memory usage per .75 (75%) to have an idea about the limit value to be set;
Find out the average memory usage for the last two days to have an idea about the request value to be set.

If the obtained target values are lower than the ones by default, just delete the specific parameters in envs-ng/contexts/caascad/kube-prometheus-stack.cue to fall back to the default values.

This change must follow the usual workflow.

Checkpoints#

The points to verify are defined in the Jira template. Here are some details:

(x) No references to zones:  
* (x) in envs-ng  
* (x) in Vault  
** (x) path: zones/fe  
* (x) in the documentation  

(x) In the FE console, no resources linked to zones and the cluster:  
* (x) VMs  
* (x) S3 buckets  
* (x) ELB  
* (x) CCE  

(x) No alerts/silences in Karma