Monitoring#

Prerequisites#

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="ocb-..."
export DOMAIN_NAME="caascad.com" # ou cloudservicesfactory.com
export INFRA_ZONE_NAME="infra-..."
export BRANCH_NAME="pf-..."

Warning

During this procedure, several steps require restarts of different components in cloud zones. Before each restart:

check there is no CAASCHR that is planned on the same component at the same time
warn supervsion team on #supervision about the component/zone that is going to be restarted

envs-ng#

Tip

Many steps will consist of running trackbone apply -z "$ZONE_NAME" -c ...

Reusing the shell setup here is a good idea!

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$BRANCH_NAME"
nix-shell

Deployment of monitoring components#

Deploy Alertmanager#

trackbone apply --non-interactive -z ${ZONE_NAME} -c alertmanager-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c alertmanager-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c alertmanager-cloud-app # No changes for shared cloud zones

Add silences#

Add silences on all the zone to prevent MEP alerts during the deployment.

DURATION=2h # Duration for the silence (expected MEP max duration)

amtool-caascad --alertmanager.url ${ZONE_NAME}/cloud-caascad silence add cc_client=${ZONE_NAME} --author "Caascad" --duration "${DURATION}" --comment "MEP new cloud zone not finished yet"
amtool-caascad --alertmanager.url ${ZONE_NAME}/cloud-client silence add cc_client=${ZONE_NAME} --author "Caascad" --duration "${DURATION}" --comment "MEP new cloud zone not finished yet"
amtool-caascad --alertmanager.url ${ZONE_NAME}/cloud-caascad silence add cc_client=${ZONE_NAME} alertname=PrometheusNotIngestingSamples --author "Caascad" --duration "2d" --comment "ACK! MEP no client zone has been deployed yet"
amtool-caascad --alertmanager.url ${ZONE_NAME}/cloud-caascad silence add cc_client=${ZONE_NAME} alertname=PrometheusTSDBBlocksLoadedLow --author "Caascad" --duration "6h" --comment "ACK! MEP no blocks yet written in blocks"

Warning

The ack alertname=PrometheusNotIngestingSamples must be set to prevent alert PrometheusNotIngestingSamples to be risen on monitoring-app. You can set it with Karma while amtool-caascad is broken.

The ack alertname=PrometheusTSDBBlocksLoadedLow must be set to prevent alert PrometheusTSDBBlocksLoadedLow. During the first hours the metrics are not yet written in blocks.

Deploy other components#

generate-static-zones-files

trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-operator
trackbone apply --non-interactive -z ${ZONE_NAME} -c blackbox-exporter-core-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c blackbox-exporter-core-cloud-client # No changes for shared cloud zones
trackbone apply --non-interactive -z ${ZONE_NAME} -c grafana-cloud-caascad -t bootstrap=true
trackbone apply --non-interactive -z ${ZONE_NAME} -c grafana-cloud-client -t bootstrap=true # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c kthxbye-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c kthxbye-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c kthxbye-cloud-app # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c node-exporter
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-state-metrics
trackbone apply --non-interactive -z ${ZONE_NAME} -c mom-client-servicemonitor-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c mom-caascad-servicemonitor-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c mom-caascad-servicemonitor-client-federate
trackbone apply --non-interactive -z ${ZONE_NAME} -c mom-caascad-servicemonitor-app # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c mom-caascad-servicemonitor-app-federate # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c s3usage-exporter
trackbone apply --non-interactive -z ${ZONE_NAME} -c s3usage-exporter-servicemonitor
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-rules
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-rules-dashboards-upstream-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-rules-dashboards-upstream-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-rules-dashboards-upstream-cloud-app # No changes for shared cloud zones
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-prometheus-stack-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-prometheus-stack-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c kube-prometheus-stack-cloud-app # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c prometheus-federation-infra-consumption
trackbone apply --non-interactive -z ${INFRA_ZONE_NAME} -c prometheus-rules
trackbone apply --non-interactive -z ${INFRA_ZONE_NAME} -c karma -t karma_refresh_cache=true
trackbone apply --non-interactive -z ${ZONE_NAME} -c thanos-cloud-caascad
trackbone apply --non-interactive -z ${ZONE_NAME} -c thanos-cloud-client
trackbone apply --non-interactive -z ${ZONE_NAME} -c thanos-cloud-app
trackbone apply --non-interactive -z ${ZONE_NAME} -c cluster-info-recordingrule
trackbone apply --non-interactive -z ${ZONE_NAME} -c karma # No changes for shared cloud zones 
trackbone apply --non-interactive -z ${ZONE_NAME} -c fe_service_account_cloudeye_exporter #FE only
trackbone apply --non-interactive -z ${ZONE_NAME} -c cloudeye-exporter # FE only

Note

If you tried to deploy Cloudeye-Exporter on a non-FE cluster, you can ignore the errors.

Monitoring functional tests#

Update#

Update the nix pointer to Caascad-zones and redeploy the functional tests. Reference procedure is in README.md file of the functional tests repository.

cd ${GIT_PATH}/caascad/pipelines/monitoring-functional-tests
git checkout master
git pull --rebase
git checkout -b "update_${ZONE_NAME}_functional_test"
nix-shell --run "niv update zones"
nix-shell
cd ci
# Set INFRA_ZONE_NAME again (nix-shell may not know it)
fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test
cue -t infra_zone="${INFRA_ZONE_NAME}" -t zone="${ZONE_NAME}" apply-pipelines

Check on the CI (Staging or Production - change for the cloud zone being deployed) that the zone was added to the functional tests for monitoring.

Then

create a MR and ask for validation
merge it into master branch

Launch#

Launch automated functional tests :

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${ZONE_NAME}/${stack}-${ZONE_NAME}"
done

Verification#

For all zones#

Verification : dashboards#

Use check_dashboards :
```
./check_dashboards.sh ${ZONE_NAME} caascad obs-kubernetes-resources-cluster && echo OK
```
Note

However, this test checks only the presence of the dashboard. It does not test if it contains metrics.
Connect to Grafana (cloud zone) and check that everything is displayed as expected for at least one of the dashboards.

Caascad: http://grafana-infra.${ZONE_NAME}.caascad.com/
NgotAlpha: http://grafana-infra.${ZONE_NAME}.cloudservicesfactory.com/

Verification : PrometheusRules#

kswitch ${ZONE_NAME}
kubectl -n prometheusrules get prometheusrules # ensure the rules are deployed (pay attention to `cloud-caascad-blackbox-exporter` and `cloud-client-blackbox-exporter`)

Verification : Connection alertmanager (cloud-caascad and cloud-client) to Karma#

Connect to http://karma-infra.<zone infra>.caascad.com/, click on "silences" in karma and check in the list of alertmanagers, that the alertmanagers ${ZONE_NAME} and ${ZONE_NAME}-client are present.

Verification : Billing PrometheusRules#

kswitch ${INFRA_ZONE_NAME}
kubectl get prometheusrules -n prometheusrules -o yaml consumption-billing | grep ${ZONE_NAME}

Karma Infra#

Connect to Karma:

Caascad: http://karma-infra.${INFRA_ZONE_NAME}.caascad.com/
NgotAlpha: http://karma-infra.${INFRA_ZONE_NAME}.cloudservicesfactory.com/

Do a visual check by removing the filter on active alerts.

Checks with functional tests#

Automated functional tests in Concourse#

Connect to Concourse

fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test

Check results in Concourse UI or with this :

for stack in monitoring; do
  fly -t ${INFRA_ZONE_NAME}.infra jobs -p "f-tests-${stack}-${ZONE_NAME}"
done

For not shared zones#

Verification : dashboards#

Use check_dashboards :
```
./check_dashboards.sh ${ZONE_NAME} client obs-kubernetes-resources-cluster && echo OK
```
Note

However, this test checks only the presence of the dashboard. It does not test if it contains metrics.
Connect to Grafana (client zone) and check that everything is displayed as expected for at least one of the dashboards:

Caascad: http://grafana.${ZONE_NAME}.caascad.com/
NgotAlpha: no Grafana client

Check values of datasources for Grafana client (only for Caascad zones).

Note

This test can be automated by requesting the Grafana API directly.

Go to "Explore tab": choose the Thanos datasource.

Take a metric, for example prometheus_build_info, and check on the two grafana that we have the same values.
Check values of datasources for Grafana cloud:

Caascad: http://grafana-infra.${ZONE_NAME}.caascad.com/
NgotAlpha: http://grafana-infra.${ZONE_NAME}.cloudservicesfactory.com/)

Go to "Explore tab", choose the Thanos-client datasource and check also the presence of the metric prometheus_build_info.

Verification : Billing PrometheusRules#

kswitch ${INFRA_ZONE_NAME}
kubectl get prometheusrules -n prometheusrules -o yaml consumption-billing | grep ${ZONE_NAME}