Aller au contenu

Logging#

Deployment of promtail agent on client zone to send logs to loki ("loki-client" datasource) on cloud zone.

Note

In case of error, you can replay the kubectl and helm commands. With some exceptions (like pv), this corrects the errors.

Prerequisites#

The following components need to be already deployed at this step:

  • Monitoring stack: because promtail uses servicemonitors.

Inputs#

export GIT_PATH="$HOME/git/"
export ZONE_NAME="..." # also known as cluster_ID
export CLOUD_ZONE_NAME="ocb-..."
export INFRA_ZONE_NAME="infra-..."
export CONTEXT="..." # where context can be caascad/ngotalpha/pf/...

Deploy promtail#

envs-ng#

Repository: caascad/terraform/envs-ng

# git clone git@git.corp.caascad.com:caascad/terraform/envs-ng.git "$GIT_PATH/caascad/terraform/envs-ng"
cd "$GIT_PATH/caascad/terraform/envs-ng"
git pull
git switch -c "$ZONE_NAME"
nix-shell

Deployment#

cd context/${CONTEXT}
trackbone apply --non-interactive -z ${ZONE_NAME} -c promtail

Verification logging stack#

Automated functional tests in Concourse#

Connect to Concourse

fly switch -z "${INFRA_ZONE_NAME}" -n test
fly -t "${INFRA_ZONE_NAME}.infra" login -n test

Launch automated functional tests :

for stack in logging; do
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${CLOUD_ZONE_NAME}/${stack}-${CLOUD_ZONE_NAME}"
  fly -t ${INFRA_ZONE_NAME}.infra trigger-job -j "f-tests-${stack}-${CLOUD_ZONE_NAME}/${stack}-${ZONE_NAME}"
done

Check results in Concourse UI or with this :

for stack in logging; do
  fly -t ${INFRA_ZONE_NAME}.infra jobs -p "f-tests-${stack}-${CLOUD_ZONE_NAME}"
done

Check both CLOUD_ZONE_NAME and ZONE_NAME in the output.

Non-automated tests#

  • Connect to : https://grafana-infra.ocb-${CLOUD_ZONE_NAME}.caascad.com/

Menu Explore, check the following logs :

  1. Datasource Loki-client, search {cc_prom_source="$ZONE_NAME",job="kubelet"}. Logs should be displayed.

  2. Datasource Loki, search {container="nginx",namespace="logging-client"}|=" 204 ". There should be many lines mentioning the http code 204.

  3. Datasource Loki, search {container="nginx",namespace="logging-client"}|=" 400 ". There should be some errors in GET and POST, but not in large number. Wait 5 minutes before testing, at first there are some 400 errors due to kubelet log, not related to logging stack deployment.

  4. Datasource Loki-client, search {app="eventrouter", cc_prom_source="${ZONE_NAME}"}. Eventrouter logs should be present. These logs show that eventrouter is running correctly.