FileSystemSpaceAlmostOutOfSpace#

It's an alert on the space used.

Note

This alert is deactivated for:

NFS fstype for client clusters
PVCs for K8s clusters.

If warning, the filesystem has less than 5% left.

If critical, the filesystem has less than 3% left.

Troubleshooting hints#

Get the device and the mountpoint from the alert description (for example: /dev/sdd and /mnt/paas/kubernetes/kubelet/...), the instance and the cluster info (for Caascad and NGOT zone check the label in the alerts)
Connect (with ssh, through the bastion, or with kubectl node-shell) to the server
Identify the filesystem (PV).
- method 1 based on the mountpoint: the mountpoint has this pattern : <path>/mount. Read the contents of the file <path>/vol_data.json. The specVolID should be the PV name (aka pvc-xxxx).
- method 2 based on the device: the device has this pattern: /dev/sdX. Run sudo mount | grep <device> | grep pvc- : several mountpoints should appear, some of them should show the name of the PV (aka pvc-xxxx)
On Grafana, check the history for that filesystem. You may see a short augmentation. If the augmentation persists, you have a problem. If the augmentation was short and the use of the disk becomes flat again, the alert should disappear soon. Anyway, when the use of the disk is high, check if this is normal.

Known cases#

Concourse worker#

For /opt/concourse/worker or /opt/concourse/worker/volumes, see Concourse operations documentation.

K8s Nodes#

On a client cluster, if the device is the emptyDir path of the node (example: /mnt/paas/kubernetes/kubelet)
- check if Prometheus has garbage in its disk: kubectl -n caascad-monitoring exec -it prometheus-caascad-prometheus-0 -c prometheus -- ls -l /prometheus/wal (files with .tmp extension and older than 24h are considered as garbage).
- If yes, feel free to restart the pod (kubectl -n caascad-monitoring delete pod prometheus-caascad-prometheus-0): there are no useful data in the pod and the restart will clean the emptyDir disk space (including the garbage).
It is also possible that the filesystem of the node is full because of deployment logs:
- Check the use of the "dockersys" disk:
```
df -h -P /var/lib/docker
```
- If this disk is full, you must check its use by each container:
```
du -csh /var/lib/docker/containers/* | sort -h
```
- Based on the directories size, check if a specific container generates a lot more logs than the others (each directory corresponds to a container ID).
```
du -bs /var/lib/docker/containers/* | sort -k1 -n | awk '$1 > 700000000 {print $2"/*.log*"}' | xargs bash -c 'ls -lht $@' -
```
- Find which container the directory corresponds to:
```
docker ps --filter id=<id_du_repertoire_container>
```
- Investigate why the container in question generates so many logs, and empty the logs in order to save space by deleting the oldest log files (except the current log).
Warning

Do not delete files that end with .log because they are the current logs. Rather, delete oldest ones, those that end with a number .log.<number>.
It is also possible that the filesystem of the node is full because of deleted container or orphaned volumes:
- Check the use of the "dockersys" disk:
```
df -h -P /var/lib/docker
```
- Do :
```
docker container prune -f
docker volume prune -f
```
- Check again the use of the "dockersys" disk:
```
df -h -P /var/lib/docker
```

Persistent Volume (PV) Almost Full#

If a storage alert targets a path like /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/..., it means the filesystem mounted on the Persistent Volume is almost full.

allowVolumeExpansion is enabled on our StorageClasses. Increasing the PVC size will automatically expand the PV. see How to extend PVC documentation.
After the expansion is complete, restart the pod of the node-exporter pod mentioned in the alert to force it to see the new capacity (and stop alerting with the old capacity).

Gitlab webservices#

If this alert is for Corp environment, and the mount point is /var/lib/docker, the issue might be caused by Gitlab webservice pods that generate a lot of logs which are causing the filesystem low space availability.

Note

There is a cronjob that restarts automatically (rollout every friday at 9h05) Gitlab's WebService & Sidekiq deployements. See Branch CAASINC-1065 : applications/gitlab/helm/templates/cronjobs-workaround-restart-deployment.yaml

However, you can execute a manual rollout with these commands:

kubectl -n gitlab rollout restart deployment/gitlab-webservice-default
kubectl -n gitlab rollout restart deployment/gitlab-sidekiq-all-in-1-v1