Troubleshooting

Visualization of metrics

Case where we only see recent metrics and we have the following warning message :

receive series from ... fetch series from block ... unexpected EOF

This may show a problem for one of the blocks in S3.

We can identify this block in the warning message.

Two possible solutions :

• repair block
• delete block

Repair block

Warning

This solution may not work, if so do the "Delete block" solution.

In Thanos Store-Gateway pod launch repair command.

kubectl exec -it -n monitoring-stack-client-obs-<client> thanos-store-gateway-0 -- sh
/bin/thanos tools bucket verify --log.level=debug --objstore.config-file=/etc/config/thanos-store.yaml  --objstore-backup.config-file=/tmp/thanos-store-backup.yaml --id=<block_id> --issues=index_known_issues

Delete block

Prerequisite : s3cmd tool.

Removing broken block :

s3cmd rm -rf s3://thanos-storage-svc-monitoring-stack-client-<client>/<block_id>/