Aller au contenu

Monitoring#

Prerequisites#

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="..." # also known as <cluster_ID>
export CLOUD_ZONE_NAME="ocb-..."
export INFRA_ZONE_NAME="infra-..."
export CONTEXT="caascad" # where context can be caascad/ngotalpha/pf/...

Warning

During this procedure, several steps require restarts of different components in cloud zones. Before each restart:

  • check there is no CAASCHR that is planned on the same component at the same time
  • warn supervsion team on #supervision about the component/zone that is going to be restarted

envs-ng#

Tip

Many steps will consist of running trackbone apply -z "$ZONE_NAME" -c ...

Reusing the shell setup here is a good idea!

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$ZONE_NAME"
nix-shell

Add silences in Alertmanager#

For this deployment, it is necessary to put silences on the entire client cluster zone:

DURATION=2h # Duration for the silence (expected MEP max duration)

amtool-caascad --alertmanager.url ${CLOUD_ZONE_NAME}/cloud-client silence add cc_prom_source=${ZONE_NAME} --author "Caascad" --duration "${DURATION}" --comment "MEP new client zone not finished yet"

Monitoring components (automated parts using Trackbone)#

cd contexts/${CONTEXT}/
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-operator
trackbone apply --non-interactive -z ${ZONE_NAME} -c node-exporter
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-state-metrics
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-prometheus-stack-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c cluster-info-recordingrule
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-federation-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-federation-cloud-app # No changes for shared cloud zones
trackbone apply --non-interactive -z ${CLOUD_ZONE_NAME} -c prometheus-rules
trackbone apply --non-interactive -z ${INFRA_ZONE_NAME} -c prometheus-rules
trackbone apply --non-interactive -z ${ZONE_NAME} -c blackbox-exporter-core-client # No changes for shared cloud zones
trackbone apply --non-interactive -z ${ZONE_NAME} -c blackbox-exporter-servicemonitor-karma-client # No changes for shared cloud zone
trackbone apply --non-interactive -z ${ZONE_NAME} -c blackbox-exporter-servicemonitor-grafana-client # No changes for shared cloud zone

Update Monitoring functional tests#

Update the nix pointer to Caascad-zones and redeploy the functional tests. Reference procedure is in README.md file of the functional tests repository.

cd ${GIT_PATH}/caascad/pipelines/monitoring-functional-tests
git pull
git checkout -b "update_${ZONE_NAME}_functional-test"
nix-shell --run "niv update zones"
nix-shell
cd ci
# Set INFRA_ZONE_NAME again (nix-shell may not know it)
fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test
cue -t infra_zone="${INFRA_ZONE_NAME}" -t zone="${CLOUD_ZONE_NAME}" apply-pipelines

Check on the CI (Staging or Production - change for the cloud zone of the zone being deployed) that the zone was added to the functional tests for monitoring.

Then

  • create a MR and ask for validation
  • merge it into master branch

Verifications#

Monitoring stack (cloud-client)#

  • "consumption" metric federation

    • Grafana URL: https://grafana-infra.${INFRA_ZONE_NAME}.caascad.com/

    • Datasource: thanos-consumption

    • Check: the following metric should be present and has the correct labels : machine_cpu_cores{cc_consumption_client="${CLOUD_ZONE_NAME}", cc_prom_source="${ZONE_NAME}"}

  • Client Grafana contains metrics coming from new client cluster (no Grafana client for shared zones)

    • Grafana URL: https://grafana.${CLOUD_ZONE_NAME}.caascad.com/

    • Dashboard: Kubernetes/Nodes (for example).

    • Datasource : Thanos

    • Cluster: choose the new client cluster

    • Check: there should be metrics for each node (choose several nodes in the drop-down menu of instances).

  • Karma alerts

    • Karma URL: https://karma-infra.${INFRA_ZONE_NAME}.caascad.com/
    • Check: there should not be any alerts regarding the new cluster. If any alert is present, it should be fixed before going further.

Monitoring stack (cloud-app)#

Important

This chapter is to be ignored on cloud shared zones, because there is no monitoring-app namespace deployed on this type of zones.

  • Client Grafana contains metrics coming from new client cluster

    • Grafana URL: https://grafana.${CLOUD_ZONE_NAME}.caascad.com/

    • Dashboard: Kubernetes/Nodes (for example).

    • Datasource : thanos-app

    • Cluster: choose the new client cluster

    • Check: there should be metrics for each node (choose several nodes in the drop-down menu of instances).

  • Karma RW alerts

    • Karma URL: https://karma.${CLOUD_ZONE_NAME}.caascad.com/
    • Check: there should not be any alerts regarding the new cluster. If any alert is present, it should be fixed before going further.

Functional tests#

Connect to Concourse

fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test

Launch automated functional tests :

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${CLOUD_ZONE_NAME}/${stack}-${CLOUD_ZONE_NAME}"
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${CLOUD_ZONE_NAME}/${stack}-${ZONE_NAME}"
done

Check results in Concourse UI or with this :

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra jobs -p "f-tests-${stack}-${CLOUD_ZONE_NAME}"
done

Check both CLOUD_ZONE_NAME and ZONE_NAME in the output.

Monitoring of monitoring#

Script to run from caascad/applications/metricscli.

./metricscli.sh ${CLOUD_ZONE_NAME} cloud-caascad "up{job=~'federate-${ZONE_NAME}',service='client-prometheus'}"
./metricscli.sh ${CLOUD_ZONE_NAME} cloud-caascad "up{job=~'federate-${ZONE_NAME}',service='app-prometheus'}" # No metrics for shared cloud zone

Expected result : some lines.