Extend Prometheus PV on NGOT#

Note

This doc is specific to FE environment. It has not been validated yet.

Warning

When restarting Prometheus on Corp (central) zone, you may experience some over-incidents like:

many "out-of-order samples" that causes a loss of data
Prometheus needing too much time to restart and being killed by the Kubernetes Probes
too many metrics to be sent, causing a need of more Prometheus sockets than it can create (causing the Kubernetes Probes to fail and kill it)

For these known cases, you may want to follow this documentation.

If the service is still available (2 replicas)#

Warning

This procedure applies only if you have 2 or more replicas. E.g. on Prometheus Client and Prometheus Cluster.

Step 1: calculate the retention size and the storage size#

Note

The storage.size parameter controls the size of the volume for the WAL, chunk_head and all blocks.

The retention.size parameter controls the maximum size that Prometheus allows on the volume. It will remove blocks if the actual size is higher.

First, find out the actual capacity used by the WAL, chunk_head and theorical 13*nb_blocks.

In Grafana (Prod or Staging), try this expression.

ClientCluster

Replace XXXX with the client name (e.g. for obs_demo, replace XXXX with demo),
Find the max value over the time (check for 2 or 7 days) then retry with the other pod (replace -0 with -1).

(
    (
        kubelet_volume_stats_used_bytes{persistentvolumeclaim=~".*prometheus-obs-XXXX-prometheus-0"}
        - 
        on() prometheus_tsdb_storage_blocks_bytes{pod="prometheus-obs-XXXX-prometheus-0"}
    )
+ on() (
    13 *
        (
            prometheus_tsdb_storage_blocks_bytes{pod="prometheus-obs-XXXX-prometheus-0"}
            /
            prometheus_tsdb_blocks_loaded{pod="prometheus-obs-XXXX-prometheus-0"}
        )
    )
)/1024/1024

Replace XXXX with the cluster name (examples: kub-1, kub-18...)
Find the max value over the time (check for 2 or 7 days) then retry with the other pod (replace -0 with -1).

(
    (
        kubelet_volume_stats_used_bytes{cluster="kub-XXXX", persistentvolumeclaim="prometheus-cluster-prometheus-db-prometheus-cluster-prometheus-0"}
        - 
        on() prometheus_tsdb_storage_blocks_bytes{cluster="kub-XXXX", pod="prometheus-cluster-prometheus-0"}
    )
+ on() (
    13 *
        (
            prometheus_tsdb_storage_blocks_bytes{cluster="kub-XXXX", pod="prometheus-cluster-prometheus-0"}
            /
            prometheus_tsdb_blocks_loaded{cluster="kub-XXXX", pod="prometheus-cluster-prometheus-0"}
        )
    )
)/1024/1024

Note

This expression can be understood as :

get the pvc used bytes (kubelet_volume_stats_used_bytes)
remove the WAL+chunk_head used bytes (prometheus_tsdb_storage_blocks_bytes)
add 13 times the average size of the blocks (prometheus_tsdb_storage_blocks_bytes/prometheus_tsdb_blocks_loaded)

where 13 is the max number of blocks to guarantee 24 hours with blocks of 2 hours. You can see the blocs number with this expression in Grafana prometheus_tsdb_blocks_loaded{namespace="monitoring-stack-client-obs-xxx"}

When you have the actual capacity for both pods/pvc, calculate the retention size :

get the maximum value of each of the 2 pods
add some margin : add 20% (divide by .8).

The storage size is the retention size divided by .8 (add more 20% margin).

Step 2 : Extend Prometheus volumes#

Apply this procedure for the PVC of all replica. DO NOT RESTART THE PODS YET.

1/ Extend prometheus volumes

kswitch <zone>

Extend the disk by changing .spec.resources.requests.storage parameter:

NAMESPACE=<namespace>
kubectl -n "${NAMESPACE}" get pvc | grep "prometheus"
kubectl -n "${NAMESPACE}" edit pvc <prometheus volume>

Do it for the volumes of all Prometheus replicas.

2/ Verifications

Check the new size of the pvc
```
kubectl -n "${NAMESPACE}" get pvc | grep "prometheus"
```
Note

If the volume does not extend, wait few minutes and retry: it may take some time to extend.

Check size and presence of data on the volume used by the 2 pods prometheus-X

kubectl -n "${NAMESPACE}" exec -it <prometheus pod name>-0 -- df -h /prometheus # check disk extended size
kubectl -n "${NAMESPACE}" exec -it <prometheus pod name>-0 -- ls -l /prometheus # check presence of metrics

kubectl -n "${NAMESPACE}" exec -it <prometheus pod name>-1 -- df -h /prometheus # check disk extended size
kubectl -n "${NAMESPACE}" exec -it <prometheus pod name>-1 -- ls -l /prometheus # check presence of metrics

Step 3 : protect the K8S resources against Prometheus-Operator#

1/ Scale down Prometheus-Operator

kubectl -n monitoring scale --replicas=0 deploy/ngot-prometheus-operator
sleep 2
kubectl -n monitoring get pod |grep ngot-prometheus-operator

Ensure with the last command that the Prometheus-Operator it no more running.

2/ Delete the Statefulset (without the Prometheus Pods)

ClientCluster

CLIENT=xxxx   # example for obs-pf : CLIENT=pf
kubectl -n "monitoring-stack-client-obs-${CLIENT}" delete statefulset --cascade=orphan "prometheus-obs-${CLIENT}-prometheus"

kubectl -n monitoring delete statefulset --cascade=orphan prometheus-cluster-prometheus

This will delete the Statefulset but the pods will continue running.

Step 4 : modify Prometheus configuration#

Create a new branch and edit the Prometheus configuration.

Examples (with storage size = 20Gi and retention size as 80% of the storage size):

ClientCluster

In ngot-zones edit the parameters. Add or update these 2 values :

parameters: {
    "monitoring-stack": {
        prometheus: {
            retention: size: "16GB"
            storage: size:   "20Gi"
        }
    }
}

15/02/2024 : you can do it only in quick and dirty way.

Edit envs-ng/contexts/ngot/kube-prometheus-stack.cue. Add some block like this:

envs: [=~"^(kub-xxxx)$"]: configurations: [=~"^kube-prometheus-stack"]: {
        _kubePrometheusStackFlavorMap: ["cluster"]: {
                prometheusStorageSpec: volumeClaimTemplate: spec: resources: requests: storage: "20Gi"
                prometheusRetentionSize: "16GB"
        }
}

Regenerate the static zones:

generate-static-zones-files

Commit the changes in envs-ng (volumetry) to git repo.

Step 5 : Trackbone apply#

Run trackbone apply on your laptop (CI is discouraged here).

trackbone apply -z <zone> -c kube-prometheus-stack

Warning

To determine the value of <zone> , check the label of the alert: If the label is namespace=monitoring, then the zone is: kub-xx. Otherwise (for example, namespace=monitoring-stack-client-obs-xxx), use: svc-monitoring-stack-client-xxx

The Prometheus object should be modified but nothing else should happen because the Prometheus Operator is still scaled to 0.

Step 6 : Gracefully redeploy each pod#

Scale up the Prometheus Operator.

kubectl -n monitoring scale --replicas=1 deploy/ngot-prometheus-operator

As a consequence:

the statefulset should be automatically redeployed
the pods will automatically be restarted one after the other.

Verifications:

Ensure at least one pod restarted soon.
Check the logs of the prometheus pod (container prometheus) and ensure it restarted without any problem.
```
kubectl -n "${NAMESPACE}" logs <prometheus pod name> -c prometheus
```
When the service is available, ensure the other pod also restarted (also check the logs).

Step 6 : Verifications#

In Grafana (Prod or Staging):

Check the dashboard "Persistent Volumes"
On Prometheus Corp (central), double-check the logs. Select the Loki datasource and check with this expression:
```
{namespace="monitoring-stack-client-obs-XXXX", container="prometheus"}
```
and
```
{namespace="monitoring-stack-corp-obs-corp-<LINE>",container="prometheus"}
```
Where LINE can take one of the values prd or stg.

Ensure you are not in another incident that can be resolved with this documentation.

After 2h and 4h, check that the number of tsdb blocks is increasing:

ClientCluster

prometheus_tsdb_blocks_loaded{namespace="monitoring-stack-obs-XXXX"}

prometheus_tsdb_blocks_loaded{namespace="monitoring", cluster="kub-XXX"}

If the service has not been available for more than 2 hours#

Fully delete Prometheus:

trackbone destroy -t purge=true -z <zone> -c kube-prometheus-stack

Check that the old PV are automatically removed.
Edit the configuration in envs-ng and set the new volumetry:
```
trackbone apply -z <zone> -c kube-prometheus-stack
```