Aller au contenu

Re-inject old logs still on nodes in Loki#

Loki reconfiguration#

Edit configmap of loki#

kswitch ${CLOUD_ZONE}

kubectl -n ${NAMESPACE} edit cm loki

Change configuration#

reject_old_samples_max_age = xxxh

where xxx is the period of time when we want to retrieve the logs.

Warning

If xxx is bigger than the retention of Loki, you may notice :

  • a critical error from the alert LokiFailedToPushLogs
  • lines like msg="failed to flush user" err="unconfigured table loki_index_from_20201016_18576" (with a date in the table name older than the retention of Loki)

This is expected and normal.

Delete loki pods#

kubectl -n ${NAMESPACE} delete pod --force -l app.kubernetes.io/instance=xxx,app.kubernetes.io/name=loki

with app.kubernetes.io/instance=logs or app.kubernetes.io/instance=logs-client

Run the script so that the ingesters become ready#

cd caascad/applications/caascad-loki
cd src/loki_tools/

Wait until the distributors are running.

./loki_remove_ghosts_ingesters_from_ring.sh logging-client
No unhealthy members. Run with "--force" to force removing members when pod do not exist

Tip

You may notice that nothing happen. Then try again !

./loki_remove_ghosts_ingesters_from_ring.sh logging-client
Checking ingester-585fc79688-7fsk5 ... not found... Sent a request to forget it
Checking ingester-585fc79688-krkkw ... found
Checking ingester-585fc79688-lr4n8 ... not found... Sent a request to forget it
Checking ingester-585fc79688-rmsb4 ... not found... Sent a request to forget it
Checking ingester-585fc79688-xhv8h ... found
Checking ingester-585fc79688-zjbn2 ... found

Verify all pods are running#

kubectl -n ${NAMESPACE} get pod -l app.kubernetes.io/instance=xxx,app.kubernetes.io/name=loki | wc -l # expected value : 11
kubectl -n ${NAMESPACE} get pod -l app.kubernetes.io/instance=xxxx,app.kubernetes.io/name=loki

with app.kubernetes.io/instance=logs or app.kubernetes.io/instance=logs-client.

Start the scripts:

cd caascad/applications/caascad-loki
cd tests
nix-shell
./check_loki.sh "${NAMESPACE}"

Restart of promtails#

Concept#

The idea here is to reset the positions.yaml file and restart Promtail. Because Promtail may overwrite that file when running, we need to stop it before resetting the positions. Then restart it. The easiest way is to remote it, reset the positions and redeploy it as soon as possible.

Remove the daemonset#

If Promtail is running on Client zone :

kswitch ${CLUSTER_ID}

kubectl -n caascad-logging delete daemonset caascad-promtail
kubectl -n caascad-logging get pod # check that there are no more promtail pods

If Promtail is running on Infra or Cloud zone :

kswitch ${CLUSTER_ID}

kubectl -n logging delete daemonset promtail
kubectl -n logging get pod # check that there are no more promtail pods

Deleting Position Files in cluster nodes#

For all nodes in cluster running Promtail :

ssh <node>
sudo rm -f /run/promtail/positions.yaml

Redeploy Promtail#

Redeploy Promtail as usual (Trackbone).

Checks#

Verification in Grafana#

On Grafana, check the appearance of logs dating back to period of time when we want.

Verification in Promtail#

Check the logs of each Promtail.

  • logs like "timestamp too old" (if they're older than late period of time when we want, don't worry)
  • logs like "retry"

Check on each node (positions)#

On each node, run this command:

sudo cat /run/promtail/positions.yaml | grep log | sed -e 's#/hostlog/#/#g' | while read fo so; do f=$(echo $fo | sed -e 's/://g'); s=$(echo $so | sed -e 's/"//g'); echo $f; rs=$(sudo stat -L $f --format "%s"); [ "$s" = "$rs" ] || echo "FAIL: $f $s $rs"; done
It compares the position in the /run/promtail/positions.yaml file with the size of the files.

Make sure that there are no FAIL lines or, if there are any, that the difference between the displayed values (position vs file size) is small.

Exemple of correct result:

/var/log/messages
FAIL : /var/log/messages 106762622 106769677
/var/log/pods/00d04cf3-1840-11eb-89b6-fa163efca7bf/cluster-register/0.log
FAIL : /var/log/pods/00d04cf3-1840-11eb-89b6-fa163efca7bf/cluster-register/0.log 17262110 17281316
/var/log/pods/1e702097-0d32-11eb-966f-fa163e804513/keycloak/0.log
/var/log/pods/1e702097-0d32-11eb-966f-fa163e804513/keycloak/1.log
/var/log/pods/7bb9d5ae-d724-11ea-966f-fa163e804513/nginx-ingress-controller/7.log
FAIL : /var/log/pods/7bb9d5ae-d724-11ea-966f-fa163e804513/nginx-ingress-controller/7.log 21120651 21123072
/var/log/pods/7bb9d5ae-d724-11ea-966f-fa163e804513/nginx-ingress-controller/8.log
FAIL : /var/log/pods/7bb9d5ae-d724-11ea-966f-fa163e804513/nginx-ingress-controller/8.log 42217796 42220145

Functional check#

Start the scripts:

cd caascad/applications/caascad-promtail
cd tests
nix-shell

kswitch ${CLUSTER_ID}
./check_promtail.sh "${NAMESPACE}"

cd caascad/applications/caascad-loki
cd tests
nix-shell
./check_loki_latest_logs.sh "${CLOUD_ZONE}" "${NAMESPACE}"

Return of Loki to nominal state#

There is no good way to know when all the logs have been transferred. When the previous tests are validated, and the situation is stabilized, we can put Loki back in his nominal state.

kswitch ${CLOUD_ZONE}
kubectl -n ${NAMESPACE} edit cm loki
Look for the line reject_old_samples_max_age; Reset the initial value 168h.

kubectl -n logging-client delete pod --force -l app.kubernetes.io/instance=xxx,app.kubernetes.io/name=loki

with app.kubernetes.io/instance=logs or app.kubernetes.io/instance=logs-client.

Wait for a "distributor" pod to reappear

cd caascad/applications/caascad-loki
cd src/loki_tools/
./loki_remove_ghosts_ingesters_from_ring.sh ${NAMESPACE}

Wait for all pods to return

kubectl -n ${NAMESPACE} get pod -l app.kubernetes.io/instance=xxxx,app.kubernetes.io/name=loki | wc -l  # expected value : 11
kubectl -n ${NAMESPACE} get pod -l app.kubernetes.io/instance=xxx,app.kubernetes.io/name=loki

with app.kubernetes.io/instance=logs or app.kubernetes.io/instance=logs-client.

Functional check#

Start the scripts:

cd caascad/applications/caascad-loki
cd tests
nix-shell
./check_loki.sh ${NAMESPACE}
./check_loki_latest_logs.sh "${CLOUD_ZONE}" "${NAMESPACE}"

Tip

You may prefer to launch the functional tests from the full automated stack in Concourse. Yes, it's better. Do it !