Extend Prometheus PV on Caascad#
Note
This doc was validated on 2022-02-15 on a FE environment.
The old method below doc was validated on 2021-11-02 on a FE environment.
If the service is still available#
Step 1 : Get volumes names#
Write down the names of the prometheus pods and the name of the volumes used by these pods (pv, pvc):
kubectl -n <namespace> get pvc | grep prometheus
Step 2: Update Prometheus configuration#
The size requested by Prometheus and the size of the volumes used by Prometheus need to be coherent.
1/ Find a way to force Prometheus to flush its WAL. After this happens, Thanos should upload the data on S3 soon. FIXME : we do not know how to do it (20/01/2022)
2/ Modify Prometheus configuration by editing the envs-ng/kube-prometheus-stack.cue.
This must be done in a new branch, in order to merge it later.
Examples (with size = 15Gi) :
envs: [=~"^(ocb-xxxxxx)$"]: configurations: [=~"^kube-prometheus-stack"]: {
_kubePrometheusStackFlavorMap: ["cloud-caascad"]: {
prometheusStorageSpec: volumeClaimTemplate: spec: resources: requests: storage: "15Gi"
// prometheusRetentionSize: "5GB"
}
}
envs: [=~"^(ocb-xxxxxx)$"]: configurations: [=~"^kube-prometheus-stack"]: {
_kubePrometheusStackFlavorMap: ["cloud-client"]: {
prometheusStorageSpec: volumeClaimTemplate: spec: resources: requests: storage: "15Gi"
// prometheusRetentionSize: "5GB"
}
}
envs: [=~"^(ocb-xxxxxx)$"]: configurations: [=~"^kube-prometheus-stack"]: {
_kubePrometheusStackFlavorMap: ["cloud-app"]: {
prometheusStorageSpec: volumeClaimTemplate: spec: resources: requests: storage: "15Gi"
// prometheusRetentionSize: "5GB"
}
}
Important
If you want to change the prometheusRetentionSize too, you can do it here (remove the comment in the example above).
The prometheusRetentionSize parameter controls only the files outside the /prometheus/wal directory.
When calculating the size of the disk, set a value larger than the sum of the /prometheus/wal directory size and of the prometheusRetentionSize parameter.
3/ Commit the changes in envs-ng (volumetry) to git repo.
4/ Apply the new configuration with trackbone apply
5/ Both Prometheus pods will restart at the same time (StatefulSet) due to the configuration update.
Wait until they are both up and running.
Note
At this step, /prometheus still has the old size.
Danger
When Prometheus is restarted this way, federated metrics are not scrapped. This may lead to data loss. However there is no better way to do it.
Step 3 : Extend prometheus-1 volume#
1/ Extend prometheus-1 volume
Extend the disk by changing .spec.resources.requests.storage parameter:
NAMESPACE=<namespace>
PROMETHEUS=prometheus-1
kubectl -n "${NAMESPACE}" get pvc | grep "${PROMETHEUS}"
kubectl -n "${NAMESPACE}" edit pvc <prometheus-1 volume>
There is no need to kubectl get pvc to check : at this step you will see the old size. We need to restart the pod to get the new size.
2/ Delete the prometheus-1 pod
kubectl -n "${NAMESPACE}" get pod | grep "${PROMETHEUS}"
kubectl -n "${NAMESPACE}" delete pod "${NAMESPACE}-${PROMETHEUS}"
The pod will automatically restart. Wait until it is up and running.
3/ Verifications
- Check the new size of the pvc
kubectl -n "${NAMESPACE}" get pvc | grep ${PROMETHEUS}" - Check size and presence of data on the volume used by prometheus-1 pod
kubectl -n "${NAMESPACE}" exec -it "${NAMESPACE}-${PROMETHEUS}" -- df -h /prometheus # check disk extended size kubectl -n "${NAMESPACE}" exec -it "${NAMESPACE}-${PROMETHEUS}" -- ls -l /prometheus # check presence of metrics - Check the logs of the prometheus-1 pod (container
prometheus) and ensure it restarted without any problem.kubectl -n "${NAMESPACE}" logs "${NAMESPACE}-${PROMETHEUS}" -c prometheus
4/ Wait 5 more minutes
Wait a few minutes in order to let Prometheus collect metrics before we restart the other pod.
Step 4 : Extend prometheus-0 volume#
Follow the same procedure as step 3 with prometheus-0 volume.
NAMESPACE=<namespace>
PROMETHEUS=prometheus-0
# follow the procedure above
At the end, check on Grafana if some metrics were lost during the operation.
Old method#
Info
The new method above has been tested only once, on FE cluster 1.19.
When this method is known to work everywhere (on AWS and Azure too), the old method may be removed from this documentation.
Step 3 : Extend prometheus-1 volume#
In order to modify the pv size, we need to stop the pod that is using it. Setting Prometheus replica to 1 gives time to do it.
Warning
This step should be performed as fast as possible because prometheus-1 will not scrap.
1/ Get prometheus name
kubect get prometheus -n <namespace>
This step will stop prometheus-1 pod:
kubectl -n <namespace> edit prometheus <prometheus name>
As the volume (pvc) is no longer used by pod prometheus-1 it can be easily extended by changing .spec.resources.requests.storage parameter:
kubectl edit pvc <prometheus-1 volume> -n <namespace>`
This will start prometheus-1 pod:
kubectl -n <namespace> edit prometheus <prometheus name>
- Check size and presence of data on the volume used by prometheus-1 pod
kubectl exec -it <prometheus-1> -n <namespace> -- df -h /prometheus # check disk extended size kubectl exec -it <prometheus-1> -n <namespace> -- ls -l /prometheus # check presence of metrics - Check the logs of the prometheus-1 pod (container
prometheus) and ensure it restarted without any problem.kubectl logs <prometheus-1> -n <namespace> -c prometheus
6/ Wait more than 2 hours.
Important
If the following conditions are all fulfilled, you can wait a few minutes instead of 2 hours.
- step 2 took less than one minute
- prometheus-1 pod started correctly
- we lost no data in
/prometheus
Step 4 : Extend prometheus-0 volume#
Method 1: without data loss#
This method needs dexterity because the time frame to perform it is really short. You can try it a few times, but if you don't succeed, try the second method. Do not restart the pod too many times.
1/ Restart pod prometheus-0
kubectl delete pod <prometheus-0> -n <namespace>
While the pod is restarting, change the .spec.resources.requests.storage parameter:
kubectl edit pvc <prometheus-0 volume> -n <namespace>
kubectl exec -it <prometheus-0> -n <namespace> -- df -h /prometheus # check disk size
kubectl exec -it <prometheus-0> -n <namespace> -- ls -l /prometheus # check presence of metrics
Method 2: with data loss#
This method works all the time. Cons: data loss.
1/ Remove the pvc of pod prometheus-0
kubectl delete pvc <prometheus-0 volume> -n <namespace>
Note
This command will block until the pod is deleted. This is normal.
2/ Restart pod prometheus-0
kubectl delete pod <prometheus-0> -n <namespace>
3/ Verifications
Pod prometheus-0 is re-created.
kubectl exec -it <prometheus-0> -n <namespace> -- df -h /prometheus # check disk size
kubectl exec -it <prometheus-0> -n <namespace> -- ls -l /prometheus # no presence of metrics
Note
No need to remove the pv. It has been removed automatically when the pvc was removed.
If the service has not been available for more than 2 hours#
1/ Delete Prometheus
trackbone destroy -z <zone> -c <kube-prometheus-stack-xxx>
kubectl delete pvc <prometheus-1 volume> -n <namespace>
kubectl delete pvc <prometheus-0 volume> -n <namespace>
trackbone apply -z <zone> -c <kube-prometheus-stack-xxx>