Aller au contenu

Update metrics retention#

The component in charge of managing metrics retention is Thanos compactor.

In order to update metrics retention policy, it is necessary to:

  • change Thanos configuration.
  • change storage size (pv, pvc) : to be able to store data for the period defined in the previous step.

To find more information about metrics retention default policies, see retention.md.

Steps#

In order to update metrics retention values:

1/ Update Thanos configuration

Retention is configured by using the following flags. Not setting or setting them to 0s means no retention.

  • --retention.resolution-raw: no compaction performed
  • --retention.resolution-5m: downsampling every 5 minutes
  • --retention.resolution-1h: downsampling every 1 hour

Info

Retention is applied right after Compaction and Downsampling loops. If those are failing, data will be never deleted.

Set compactor.retention.resolution parameter here.

2/ Increase the size of the PVC in envs-ng

The size of persistent volumes has been calculated for the current retention policy.

In case data retention policy need to be extended, you will probably need to also adapt the size of persistent volumes (pv) and persistent volume claims (pvc).

Information refered by parameter claimSize envs-ng/thanos.cue.

3/ Redeploy the Compactor

Danger

After Thanos redeployment, the new retention configuration is applied. The metrics older than the retention policies are deleted.

  • trackbone plan -z <zone> -c thanos-cloud-app (replace thanos-cloud-app with the wanted configuration).
  • kubectl -n <monitoring_namespace> delete sts thanos-monitoring-app-compactor (replace monitoring-app in the namespace and in the statefulset name)
  • kubectl -n <monitoring_namespace> delete persistentvolumeclaim/thanos-compactor-data-volume-thanos-monitoring-app-compactor-0 (replace namespace and PVC name)
  • trackbone apply -z <zone> -c thanos-cloud-app (replace thanos-cloud-app with the wanted configuration).

4/ Verifications

  • Check that the compactor is redeployed
  • Check that the volume has the new size: 
    kubectl -n <monitoring_namespace> exec -t <thanos_compactor_pod_name> -c thanos -- df -h /data`
  • Check that the new retention values have been taken into account by Thanos: 
    kubeclt -n <monitoring_namespace> get pod <thanos_compactor_pod_name< -o json | jq -r '.spec.containers[].args[]
/bin/thanos compact --log.level=info --data-dir=/data --retention.resolution-raw=15d --retention.resolution-5m=92d --retention.resolution-1h=190d --wait --objstore.config-file=/etc/config/thanos-store.yaml

1/ Update Thanos configuration

Retention is configured by using the following flags. Not setting or setting them to 0s means no retention.

  • --raw: no compaction performed
  • --5m: downsampling every 5 minutes
  • --1h: downsampling every 1 hour

Info

Retention is applied right after Compaction and Downsampling loops. If those are failing, data will be never deleted.

Set thanos.retention parameter envs-ng/zones/ngot_zones/client-xxx.

Exemple:

// prometheus clients services
"svc-monitoring-stack-client-XXXX": {
    type:               "service"
    subtype:            "monitoring-stack-client"
    parent_zone_name:   "kub-XXXX"
    contract_zone_name: "obs-XXXX"
    parameters: {
        thanos: {
            retention: {
                raw:             "15d"
                downsampling_5m: "60d"
                downsampling_1h: "60d"
            }
        }

2/ Increase the size of the PVC in envs-ng

The size of persistent volumes has been calculated for the current retention policy.

In case data retention policy need to be extended, you will probably need to also adapt the size of persistent volumes (pv) and persistent volume claims (pvc).

Information refered by parameter storage: size: envs-ng/zones/ngot_zones/client-xxx}.

parameters: {
        thanos: {
            storage: size: "150Gi"

3/ Redeploy the Compactor

Danger

After Thanos redeployment, the new retention configuration is applied. The metrics older than the retention policies are deleted.

  • trackbone plan -z svc-monitoring-stack-client-${CLIENT} -c thanos-store.
  • kubectl -n <monitoring_namespace> delete sts thanos-monitoring-app-compactor (replace monitoring-app in the namespace and in the statefulset name)
  • kubectl -n <monitoring_namespace> delete persistentvolumeclaim/thanos-compactor-data-volume-thanos-monitoring-app-compactor-0 (replace namespace and PVC name)
  • trackbone apply -z svc-monitoring-stack-client-${CLIENT} -c thanos-store

4/ Verifications

  • Check that the compactor is redeployed
  • Check that the volume has the new size: 
    kswitch svc-monitoring-stack-client-${CLIENT}
kubectl -n monitoring-stack-client-obs-${CLIENT} exec -t thanos-compactor-0  -- df -h /data `

  • Check that the new retention values have been taken into account by Thanos: 

    kubectl get statefulset -n monitoring-stack-client-obs-${CLIENT} -o yaml thanos-compactor | grep retention
    Output

    /bin/thanos compact --log.level=info --data-dir=/data --retention.resolution-raw=15d
--retention.resolution-5m=60d --retention.resolution-1h=60d --objstore.config-file=/etc/config/thanos-store.yaml