Aller au contenu

ThanosCompactionHalted#

This alert is triggered when Thanos Compact has stopped working.

Troubleshooting Hints#

kubectl get pods -n  <monitoring_namespace> | grep compact  # check pod is up and running
kubectl logs <thanos_compact> -n <monitoring_namespace>
kubectl get svc -n  <monitoring_namespace> | grep compact  # service should be present

  • For Caascad zone:

    • Zone Cloud: monitoring, monitoring-client or monitoring-app
    • Zone Infra: monitoring or monitoring-consumption

  • For NGOT zone:

    • Notice the namespace label in the alert

You may try this to find out if this is a known case:

kubectl logs <thanos_compact> -n <monitoring_namespace> | grep "halt"

Solutions#

Case "postings offset table size exceeds 4 bytes" or "symbol table size exceeds 4294967295 bytes"#

Warning

In most cases the procedure must be executed multiple times (one iteration is not enough).

Pre-requisites#

LogQL query (at least one of them):

  • {pod="thanos-<namespace>-compactor-0"} |= "postings offset table" |= "compaction: group
  • {pod="thanos-<namespace>-compactor-0"} |= "symbol table size exceeds" |= "compaction: group

You can find logs like this :

level=error ts=2025-02-24T09:52:36.920872047Z caller=compact.go:487 msg="critical error detected; halting" err="compaction: group 0@16913680754407745960: compact blocks [/data/compact/0@16913680754407745960/01JFKMKE2YSD4RKGKS3YAT885Q /data/compact/0@16913680754407745960/01JFRSCMFDG50JZWE3YN6380CS /data/compact/0@16913680754407745960/01JFXY689AV4G28PTHNG1A49G2 /data/compact/0@16913680754407745960/01JG32ZP99TER0QN4W7J4DATQ2 /data/compact/0@16913680754407745960/01JG87RH6ZP65ZZ2VQNKTSVJTD /data/compact/0@16913680754407745960/01JGDCJAB9A8TNQV7X1HEBJAC8 /data/compact/0@16913680754407745960/01JGJHBJVR0HHH7YZTF62SG216]: postings offset table size exceeds 4 bytes: 4362102329"

level=error ts=2025-02-25T10:47:24.990556601Z caller=compact.go:487 msg="critical error detected; halting" err="compaction: group 300000@16913680754407745960: compact blocks [/data/compact/300000@16913680754407745960/01JMWGP05QRK52MT31CSGYAMXQ /data/compact/300000@16913680754407745960/01JMWH526GMJRCTPZK805AT0XP /data/compact/300000@16913680754407745960/01JMWHK2NGG4AK7V3GMQ1BHGKY /data/compact/300000@16913680754407745960/01JMWJ29EVWGMRMFKJR32AK72B /data/compact/300000@16913680754407745960/01JMWJHNDKZ45VWN31RY489WRS /data/compact/300000@16913680754407745960/01JMWK0HT63DPMEWRKBRV6ZQP1 /data/compact/300000@16913680754407745960/01JMWKFGT6EK2DSVH18V75SHZF]: 2 errors: populate block: add series: symbol table size exceeds 4294967295 bytes: 4298073971; symbol table size exceeds 4294967295 bytes: 4298073971"

Principle#

When one of the errors above happen, the idea is to mark the blocks to not compact them.

This can be done either with the GUI ("one shot" solution) but it is repetitive. For more than one iteration, this should be done with a script.

"One shot" solution#

Port-forward on the compactor:

kubectl -n <namespace> port-forward thanos-compactor-0 10902:10902

On the console http://localhost:10902:

  • use the search bar to find a block by its ulid,
  • click on the block,
  • on the right panel that appears, at the bottom, click on Mark No Compaction
  • iterate for the other blocks of the list in the log line.

Iterative process with a script#

Tip

This is a long operation that should be done with a CAASCHR. You may take CAASCHR-3201 as an example.

Use this script (/tmp/generate-obsutil-line.sh) - tested on NGOT only:

#! /bin/bash

CLIENT=awesome # example: CLIENT=vis

read -r LINE

printf "\n"
printf "h1. %s\n" "$(date +"%H:%M:%S (%d/%m/%Y)")"
printf "{code}\n%s\n{code}\n" "${LINE}"

BLOCKS=($(echo "${LINE}" | sed -e 's#.*compact blocks \[##' | sed -e 's#\]:.*$##' | sed -e 's#/[^ ]*/##g'))

printf "h2. Marquage des blocs 'no compaction'\n"
printf "{code}\n"
printf "kubectl -n monitoring-stack-client-obs-%s port-forward thanos-compactor-0 10902:10902\n" "${CLIENT}"
printf "{code}\n"
printf "{code}\n"
for b in "${BLOCKS[@]}"; do
  printf "curl 'http://localhost:10902/api/v1/blocks/mark' -X POST --data-raw 'id=%s&action=NO_COMPACTION'\n" "$b"
done
printf "{code}\n"

# printf "h2. Suppression définitive des blocs\n"
# printf "{code}\n"
# for b in "${BLOCKS[@]}"; do
#   printf "obsutil rm -f -r obs://thanos-storage-svc-monitoring-stack-client-${CLIENT}/%s\n" "${CLIENT}" "$b"
# done
# printf "{code}\n"

printf "h2. Redémarrage du pod\n"
printf "{code}\n"
printf "kubectl -n monitoring-stack-client-obs-%s exec -it thanos-compactor-0 -- rm -rf /data/downsample /data/compact /data/meta-syncer\n" "${CLIENT}"
printf "kubectl -n monitoring-stack-client-obs-%s delete pod thanos-compactor-0\n" "${CLIENT}"
printf "{code}\n"

You can use it this way: 

CLIENT=awesome # example: CLIENT=vis

# Check the logs
kubectl -n monitoring-stack-client-obs-${CLIENT} logs thanos-compactor-0 |grep level=error | tail -1

# Generate some Jira stuff for CAASCHR
kubectl -n monitoring-stack-client-obs-${CLIENT} logs thanos-compactor-0 |grep level=error | tail -1 | bash /tmp/generate-obsutil-line.sh

Then copy/paste the output to a comment into your CAASCHR and execute it right after.

Controls in Grafana#

CLIENT=awesome # example: CLIENT=vis
  • thanos_compact_todo_xxx_blocks 
    xdg-open "https://grafana.obs-corp-prd.cloudservicesfactory.com/explore?schemaVersion=1&panes=%7B%22fjl%22:%7B%22datasource%22:%22thanos%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%28thanos_compact_todo_compaction_blocks%7Bnamespace%3D%5C%22monitoring-stack-client-obs-${CLIENT}%5C%22%7D%29%20by%20%28group,%20__name__%29%22,%22range%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22,%22hide%22:false,%22instant%22:true%7D,%7B%22refId%22:%22B%22,%22expr%22:%22sum%28thanos_compact_todo_downsample_blocks%7Bnamespace%3D%5C%22monitoring-stack-client-obs-${CLIENT}%5C%22%7D%29%20by%20%28group,%20__name__%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22,%22hide%22:false%7D,%7B%22refId%22:%22C%22,%22expr%22:%22sum%28thanos_compact_todo_deletion_blocks%7Bnamespace%3D%5C%22monitoring-stack-client-obs-${CLIENT}%5C%22%7D%29%20by%20%28group,%20__name__%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22thanos%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22,%22hide%22:false%7D%5D,%22range%22:%7B%22from%22:%22now-3h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1"
  • thanos_compact_halted 
    xdg-open "https://grafana.obs-corp-prd.cloudservicesfactory.com/explore?schemaVersion=1&panes=%7B%2207g%22%3A%7B%22datasource%22%3A%22thanos%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22thanos_compact_halted%7Bnamespace%3D%5C%22monitoring-stack-client-obs-${CLIENT}%5C%22%7D+%22%2C%22range%22%3Atrue%2C%22instant%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22thanos%22%7D%2C%22editorMode%22%3A%22code%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-1h%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1"
  • Logs 
    xdg-open "https://grafana.obs-corp-prd.cloudservicesfactory.com/explore?schemaVersion=1&panes=%7B%22o9z%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22monitoring-stack-client-obs-${CLIENT}%5C%22,container%3D%5C%22thanos-v2%5C%22,component%3D%5C%22thanos-compactor%5C%22%7D%20%7C%20json%20%7C%3D%20%60level%3Derror%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22,%22direction%22:%22backward%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D,%22panelsState%22:%7B%22logs%22:%7B%22visualisationType%22:%22logs%22%7D%7D%7D%7D&orgId=1"

Case "overlaps found while gathering blocks"#

Warning

In most cases the procedure must be executed multiple times (one iteration is not enough).

Pre-requisites#

LogQL query : {pod="thanos-<namespace>-compactor-0"} |= "overlaps found"

You can find logs like this :

level=error ts=2021-10-01T09:16:42.470565161Z caller=compact.go:423 msg="critical error detected; halting" err="compaction: group 0@13948379997987102035: pre compaction overlap check: overlaps found while gathering blocks. [mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s, blocks: 2]: <ulid: 01EXMV4NWZYVXFV3W7PB1D5PSP, mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s>, <ulid: 01EXMV3QA5JEEVXT057HBGGYZT, mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s>\n[mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s, blocks: 2]: <ulid: 01EXV93GT7NWSWBE4TV0Q3PAV6, mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s>, <ulid: 01EXV94TD179XKQ6VFGV53HNJA, mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s>\n[mint: 1623700800000...

Pre-requisite

Deploy and configure s3cmd to connect to the S3 of the cloud zone.

Retrieve the list of failed objects#

Copy the full log message in a file (named a) :

kubectl logs thanos-compactor-0 -n <monitoring_namespace> | grep "overlaps found" > a

Extract the bucket informations :

$ cat a \
    | sed -e 's/^.*overlaps found while gathering blocks. //' \
    | sed -e 's/\\n/\n/g' \
    > b
$ cat b
[mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s, blocks: 2]: <ulid: 01EXMV4NWZYVXFV3W7PB1D5PSP, mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s>, <ulid: 01EXMV3QA5JEEVXT057HBGGYZT, mint: 1612375200000, maxt: 1612382400000, range: 2h0m0s>
[mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s, blocks: 2]: <ulid: 01EXV93GT7NWSWBE4TV0Q3PAV6, mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s>, <ulid: 01EXV94TD179XKQ6VFGV53HNJA, mint: 1612591200000, maxt: 1612598400000, range: 2h0m0s>
[mint: 1623700800000...

$ export S3PREFIX=s3://thanos-<zone type>-<zone name>
# Examples :
$ export S3PREFIX=s3://thanos-cloud-caascad-ocb-test06
$ export S3PREFIX=s3://thanos-cloud-client-ocb-sbma
$ export S3PREFIX=s3://thanos-cloud-app-ocb-test05

$ export S3PREFIX=s3://thanos-storage-<svc_zone_name> # where <svc_zone_name> is ngot_service label in alert.

$ cat b \
    | awk '{print $10 $18}' \
    | tr ',' ' ' \
    | while read a b; do \
        echo "========================="; \
        echo "$a"; \
        echo "$b"; \
        s3cmd ls -r ${S3PREFIX}/$a; \
        s3cmd ls -r ${S3PREFIX}/$b; \
      done \
    > r1
$ cat r1 # this output is not an example for reality : the dates do not correspond
=========================
01EXMV4NWZYVXFV3W7PB1D5PSP
01EXMV3QA5JEEVXT057HBGGYZT
2021-10-13 09:00  60444137   s3://thanos-cloud-client-ocb-sbma/01EXMV4NWZYVXFV3W7PB1D5PSP/chunks/000001
2021-10-13 09:00  20304616   s3://thanos-cloud-client-ocb-sbma/01EXMV4NWZYVXFV3W7PB1D5PSP/index
2021-10-13 09:00       784   s3://thanos-cloud-client-ocb-sbma/01EXMV4NWZYVXFV3W7PB1D5PSP/meta.json
2021-10-13 09:00  60444137   s3://thanos-cloud-client-ocb-sbma/01EXMV3QA5JEEVXT057HBGGYZT/chunks/000001
2021-10-13 09:00  20304648   s3://thanos-cloud-client-ocb-sbma/01EXMV3QA5JEEVXT057HBGGYZT/index
2021-10-13 09:00       784   s3://thanos-cloud-client-ocb-sbma/01EXMV3QA5JEEVXT057HBGGYZT/meta.json
=========================
01EXV93GT7NWSWBE4TV0Q3PAV6
01EXV94TD179XKQ6VFGV53HNJA
2021-10-13 09:00  60444137   s3://thanos-cloud-client-ocb-sbma/01EXV93GT7NWSWBE4TV0Q3PAV6/chunks/000001
2021-10-13 09:00  20304616   s3://thanos-cloud-client-ocb-sbma/01EXV93GT7NWSWBE4TV0Q3PAV6/index
2021-10-13 09:00       784   s3://thanos-cloud-client-ocb-sbma/01EXV93GT7NWSWBE4TV0Q3PAV6/meta.json
2021-10-13 09:00  60444137   s3://thanos-cloud-client-ocb-sbma/01EXV94TD179XKQ6VFGV53HNJA/chunks/000001
2021-10-13 09:00  20304648   s3://thanos-cloud-client-ocb-sbma/01EXV94TD179XKQ6VFGV53HNJA/index
2021-10-13 09:00       784   s3://thanos-cloud-client-ocb-sbma/01EXV94TD179XKQ6VFGV53HNJA/meta.json
=========================
...

Delete overlap block#

There are several steps to choose which block(s) to delete:

  • old blocks: we check the age of the blocks and if they are older than the retention, we delete the both blocks
  • identical blocks: we compare them, if they are identical, we delete one at random
  • different blocks: if we are not in the first two cases, we delete the smallest
Remove old blocks#

If the blocks are older than the retention period, you can freely remove them with s3cmd commands :

for i in 01EXV93GT7NWSWBE4TV0Q3PAV6; do s3cmd rm -r ${S3PREFIX}/$i ; s3cmd rm ${S3PREFIX}/debug/metas/$i.json; done
Compare blocks (are they the same ?)#

When blocks are recent, you have to compare them.

If the dates are the same, you need to compare their checksums :

$ cat b \
    | awk '{print $10 $18}' \
    | tr ',' ' ' \
    | while read a b; do \
        echo "========================="; \
        echo "$a"; \
        echo "$b"; \
        s3cmd get ${S3PREFIX}/$a/chunks/000001 - | md5sum; \
        s3cmd get ${S3PREFIX}/$b/chunks/000001 - | md5sum; \
      done \
    > r2
$ cat r2
=========================
01EXMV4NWZYVXFV3W7PB1D5PSP
01EXMV3QA5JEEVXT057HBGGYZT
b73c441f36e13b3d01fcc19c251da190  -
b73c441f36e13b3d01fcc19c251da190  -
=========================
01EXV93GT7NWSWBE4TV0Q3PAV6
01EXV94TD179XKQ6VFGV53HNJA
418bfa97ed60ebfd5e5be1bd0db454f3  -
418bfa97ed60ebfd5e5be1bd0db454f3  -

When the checksums of the S3 objects are the same, you can purge one of the two :

$ cat b \
    | awk '{print $10 $18}' \
    | tr ',' ' ' \
    | while read a b; do \
        echo "s3cmd rm -r ${S3PREFIX}/$a"; \
        echo "s3cmd rm ${S3PREFIX}/debug/metas/${a}.json"; \
      done \
    > menage.sh
$ cat menage.sh
s3cmd rm -r s3://thanos-cloud-client-ocb-sbma/01EXMV4NWZYVXFV3W7PB1D5PSP
s3cmd rm s3://thanos-cloud-client-ocb-sbma/debug/metas/01EXMV4NWZYVXFV3W7PB1D5PSP.json
s3cmd rm -r s3://thanos-cloud-client-ocb-sbma/01EXV93GT7NWSWBE4TV0Q3PAV6
s3cmd rm s3://thanos-cloud-client-ocb-sbma/debug/metas/01EXV93GT7NWSWBE4TV0Q3PAV6.json

Warning

Verify the script menage.sh twice ! There should be half less lines than the number of S3 objects : we will remove 1 of each 2 pairs.

bash -x menage.sh
Clean when dates and/or checksums differ#

When the date or checksums of the S3 objects differ, you have to decide what object to purge. Today, the only criteria we have are:

  • to take the date into account (when the dates differ), but today (2021-10-18), we don't know if it is better to remove the older or the newest one
  • to keep the biggest object, and remove the smaller one

In order to obtain the script that remove the smaller block :

cat b \
        | awk '{print $10 $18}' \
        | tr ',' ' ' \
        | while read a b; do \
            c1=$(s3cmd du ${S3PREFIX}/$a |  awk '{print $1}'); \
            c2=$(s3cmd du ${S3PREFIX}/$b |  awk '{print $1}'); \
            x="$a"; \
            [ $c2 -lt $c1 ] && x="$b"; \
            echo "s3cmd rm -r ${S3PREFIX}/$x"; \
            echo "s3cmd rm ${S3PREFIX}/debug/metas/${x}.json"; \
      done > menage.sh

Warning

Verify the script menage.sh twice ! There should be half less lines than the number of S3 objects : we will remove 1 of each 2 pairs.

bash -x menage.sh

Restart the compactor#

kubectl -n <namespace> get pod
kubectl -n <namespace> delete <compactor pod name>

Watch the compactor restart and the alert disappear.

Case with PVC full (NGOT)#

Increase the size of the PVC in envs-ng#

In zones/ngot-zones edit the client zone parameters monitoring-stack-client.

Add or update this parameter:

parameters: {
    thanos: {
        compactor: storage: size: "xxxGi"
    }
}

Set a very higher size in a first step (example: if you have 200Gi, change to 500Gi). After the incident is finished, we know the real needed size and we will downsize the volume.

Redeploy the Compactor with the new PVC#

On your laptop :

CLIENT=xxx # example for obs-pf : CLIENT=pf
trackbone plan -z svc-monitoring-stack-client-${CLIENT} -c thanos-store

You will find all lines automatically marked "deleted" by the system besides the lines you modified, which is normal, don't worry about it. The following operations will add them back.

kubectl -n monitoring-stack-client-obs-${CLIENT} delete sts thanos-compactor
kubectl -n monitoring-stack-client-obs-${CLIENT} delete persistentvolumeclaim/thanos-compactor-data-volume-thanos-compactor-0
trackbone apply -z svc-monitoring-stack-client-${CLIENT} -c thanos-store

Checks :

  • the compactor is redeployed

  • the volume has the new size:

    kubectl -n monitoring-stack-client-obs-${CLIENT} exec -t thanos-compactor-0 -c thanos -- df -h /data

Create a MR#

Create a MR with the changes.

In the CI, ensure with trackbone plan ngot that it is a "no-change"

Merge the MR.

Register the MR in the CAASINC.

Set the CAASINC On Hold with a For Observation reason. Set a date with duration that should be 2 days.

Review the PV size again#

Note

This step should happen after the compaction step is finished.

To check when it is finished, observe the PV usage (Grafana - Persistent Volumes). The usage should have returned to a value near 0Gb.

To determine the optimal size for the PV, the methode is the following:

"Divide the value of the maximum observed volume by 0.72, then round (0.72 = 0.8 for a 20% rule with an additional 0.9 for a 10% margin)."

You can simply follow these steps:

  • Check in Grafana the actual usage of the PV (this represents the maximum observed volume)
  • Divide that value by 0.8
  • Finally, add a 10% margin by dividing the previous result by 0.9

Adapt the Envs-ng configuration.

Re-apply the procedure above with the definitive size of the PV. Update the MR, ask for validation then merge.

Case with PVC full (Caascad)#

Increase the size of the PVC in envs-ng#

Edit envs-ng/thanos.cue. Set a specific configuration like this :

envs: ["ocb-corp"]: configurations: "thanos-cloud-app": {
        helm: values: thanos: compactor: claimSize: "300Gi"
}

(replace the cloud zone name, the thanos-cloud-app and the claimSize).

Set a very higher size in a first step (example: if you have 200Gi, change to 500Gi). After the incident is finished, we know the real needed size and we will downsize the volume.

Redeploy the Compactor with the new PVC#

On your laptop :

  • trackbone plan -z <zone> -c thanos-cloud-app (replace thanos-cloud-app with the wanted configuration).

You will find all lines automatically marked "deleted" by the system besides the lines you modified, which is normal, don't worry about it. The following operations will add them back.

  • kubectl -n monitoring-app delete sts thanos-monitoring-app-compactor (replace monitoring-app in the namespace and in the statefulset name)
  • kubectl -n monitoring-app delete persistentvolumeclaim/thanos-compactor-data-volume-thanos-monitoring-app-compactor-0 (replace monitoring-app in the namespace and in the PVC name)
  • trackbone apply -z <zone> -c thanos-cloud-app (replace thanos-cloud-app with the wanted configuration). You will find all lines automatically marked "added" by the system.

Checks :

  • the compactor is redeployed
  • the volume has the new size kubectl -n monitoring-app exec -t thanos-monitoring-app-compactor-0 -c thanos -- df -h /data (replace monitoring-app in the namespace and in the pod name)

Create a temporary MR#

Create a MR with the changes.

Register the MR in the CAASINC.

Document the CAASINC and say that the PV size should be reviewed and the MR should be applied again and merged with the definitive size.

Set the CAASINC On Hold with a For Observation reason. Set a date with duration that should be 2 days.

Review the PV size again#

Note

This step should happen after the compaction step is finished.

To check when it is finished, observe the PV usage (Grafana - Kubernetes-Persistent Volumes). The usage should have returned to a value near 0Gb.

To determine the optimal size for the PV, the methode is the following:

"Divide the value of the maximum observed volume by 0.72, then round (0.72 = 0.8 for a 20% rule with an additional 0.9 for a 10% margin)."

You can simply follow these steps:

  • Check in Grafana the actual usage of the PV (this represents the maximum observed volume)
  • Divide that value by 0.8
  • Finally, add a 10% margin by dividing the previous result by 0.9

Adapt the Envs-ng configuration.

Re-apply the procedure above with the definitive size of the PV. Update the MR, ask for validation then merge.