Aller au contenu

PrometheusTSDBBlocksLoadedLow#

This alert is triggered when the number of blocks is low (Caascad) or not at the maximum size (NGOT).

There are two levels of severity :

  • warning : there is only two blocks remaining on the disk (Caascad) or at least one block out of 13 is missing (NGOT)
  • critical : there are one or no more blocks remaining on the disk (Caascad) or only 6 blocs out of 13 are remaining (NGOT)

This situation can happen when we reach the retention size defined in Prometheus.

Important

On Caascad, this alert can be triggered when there is no client zone attached to the cloud zone. This explains why there are no blocks yet in Thanos.

On Caascad, before going further with this procedure, check zones.json file to be sure there are client zones attached to the cloud zone. If there are none, this alert is normal, and you just need to put a silence on the alert (label : cc_prom_source=<client_zone-name>; comment : ACK! No blocks yet in Thanos because no client zone yet).

Troubleshooting Hints#

Two possible cases :

  • old blocks or/and old files have not been deleted by Prometheus
  • many metrics are managed by Prometheus : this will naturally grow the /prometheus/wal/* files and let less and less space for TSDB blocks.

Note

  • The resolution on NGOT and Caascad is the same, even if the alert expression differ.
  • For NGOT, follow the "xxx by Thanos Receiver" cases.

Case "old blocks or/and old files have not been deleted by Prometheus"#

To verify that this is the case, we can :

  • go to the Prometheus pod
  • list block (in /prometheus) and wal (in /prometheus/wal) files (a block is a directory with upper-case letters and numbers like /prometheus/01FKFY0VMF0YVDX426NMPHXDVK)
  • delete block or wal files if older than the retention period (WAL or TSDB retention : see below).
kubectl exec -it -n <monitoring_namespace> <prometheus_pod> -c prometheus -- sh
ls -lath /prometheus/wal/
ls -lath /prometheus
rm ...

WAL Retention infos

In /prometheus/wal, in most cases there are checkpoint.xxxxxxxxx.tmp files (with .tmp extension`) that are older than the retention. You can remove those files.

On Caascad, the retention is defined in envs-ng/contexts/caascad/kube-prometheus-stack.cue. Search for retention: parameter.

On NGOT, the retention is defined in zones in parameters."monitoring-stack".prometheus.retention.size for Central and Client Prometheus. It is defined in envs-ng/contexts/ngot/kube-prometheus-stack.cue for Cluster Prometheus; search for prometheusRetentionSize.

TSDB Retention infos

TSDB retentions can be found in caascad zones. Search for <zone>.monitoring.metrics.<subzone>.retention.raw.

Case "old blocks or/and old files have not been deleted by Thanos Receiver"#

To verify if the filesystem usage issues are caused by this known case, do the following verifications:

  • connect to the Thanos Receive pod (more information about ephemeral containers connection procedure here):

    # Connect to Thanos pod
IMAGE=alpine:latest
NAMESPACE=monitoring-stack-corp-obs-corp-{prd|stg}
POD=thanos-receive-xx
TARGETCONTAINER=receive
kubectl -n "${NAMESPACE}" debug -it --image="${IMAGE}" --profile=general --target="${TARGETCONTAINER}" "${POD}" -- sh
    - list and delete block/wal files older than Thanos Receiver retention period (1 day):

    # Check timestamp
cd /proc/1/root
cd var/thanos/receive/
ls -lath /default-tenant/wal/
ls -lath /default-tenant
rm ...

    Warning

    Check if data has been sent to S3 in default-tenant/thanos.shipper.json file before removing a block. If yes, you can safely remove the block without data loss. Otherwise you may loose 2h of data contained in each block that you remove.

    Any removal in the wal/ directory results in data loss.

Case "many metrics are managed by Prometheus"#

On NGOT#

In this case it is necessary to increase the retention size as well as the size of the Prometheus volume.

  • Step 1 : extend the PV using the Extend PV documentation
  • Step 2 : increase the retention size parameter. Best practice : the retention size should be 80% of the disk size.
    • Prometheus-Central : edit the zone parameters and add or change parameters."monitoring-stack".prometheus.retention.size and parameters."monitoring-stack".prometheus.storage.size (in ngot-zones)
    • Prometheus-Client : edit the zone parameters and add or change parameters."monitoring-stack".prometheus.retention.size and parameters."monitoring-stack".prometheus.storage.size (in ngot-zones)
    • Prometheus-Cluster : on 15/02/2024, this parameter cannot be easily overriden. A dirty and quick fix is to edit envs-ng/contexts/ngot/kube-prometheus-stack.cue. However it would be better to implement a way to override it with ngot-zones.

Tip

The alert will disappear after 2h or 4h. This is the time necessary for Prometheus to write new TSDB blocks thanks to the new parameters.

Warning

If the alert is critical, it is possible that it created overlaps in Thanos. In this case the alert ThanosCompactionHalted is triggered. See ThanosCompactionHalted help alerts.

On Caascad#

In this case it is necessary to increase the retention size as well as the size of the Prometheus volume.

  • Step 1 : extend the PV using the Extend PV documentation

  • Step 2 : increase the retention size parameter in envs-ng/kube-prometheus-stack.cue. Best practice : the retention size should be 80% of the disk size.

    Example on ocb-test05 with Prometheus volume as 15 Gi and retention size as 12 Gb for both cloud-client and cloud-app stacks : 

    envs: ["ocb-test05"]: configurations: [=~"^kube-prometheus-stack"]: {
    _kubePrometheusStackFlavorMap: [=~"cloud-client|cloud-app"]: {
        prometheusStorageSpec: volumeClaimTemplate: spec: resources: requests: storage: "15Gi"
        prometheusRetentionSize: "12GB"
    }
}

Tip

The alert will disappear after 2h or 4h. This is the time necessary for Prometheus to write new TSDB blocks thanks to the new parameters.

Warning

If the alert is critical, it is possible that it created overlaps in Thanos. In this case the alert ThanosCompactionHalted is triggered. See ThanosCompactionHalted help alerts.

Case "many metrics are managed by Thanos Receiver"#

If the Thanos volume has become insuficient to manage all the metrics, use the following procedure.