KubePersistentVolumeUsageCritical#
Postgres#
In the case of postgres, this could be caused by a bad synchonization of wal logs between slaves and masters. Due to this desynchronization, pod will cache wal files until overload the PV
Troubleshooting#
-
To fix that list cluster member using patronictl
kubectl -n gitea exec pocwatt-gitea-cluster-0 -it -- bash -
List all cluster member of postgres
root@gitea-postgresql-cluster-0:/home/postgres# patronictl list + Cluster: gitea-postgresql-cluster (6886439782798721093) ----------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +----------------------------+--------------+--------+--------------+----+-----------+ | gitea-postgresql-cluster-0 | 172.16.0.188 | Leader | running | 9 | | | gitea-postgresql-cluster-1 | 172.16.0.130 | | running | 9 | 0 | | gitea-postgresql-cluster-2 | 172.16.0.144 | | start failed | | unknown | +----------------------------+--------------+--------+--------------+----+-----------+In this case we have loose member gitea-postgresql-cluster-2, since the PV is full, it can't start.
-
In some case, all members disk can be full
root@gitea-postgresql-cluster-0:/home/postgres# patronictl list + Cluster: gitea-postgresql-cluster (6886439782798721093) ----------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +----------------------------+--------------+--------+--------------+----+-----------+ | gitea-postgresql-cluster-0 | 172.16.0.188 | Leader | running | 9 | | | gitea-postgresql-cluster-1 | 172.16.0.144 | | start failed | | unknown | +----------------------------+--------------+--------+--------------+----+-----------+ root@gitea-postgresql-cluster-0:/home/postgres# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-252:1-16386-2bf34ce2dbd37f0ab02adf0bc98cb3d9399c413c3fae1471d61a30bf719cec00 9.8G 672M 8.7G 8% / tmpfs 64M 0 64M 0% /dev tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/vgpaas-kubernetes 9.8G 127M 9.2G 2% /etc/hosts tmpfs 7.8G 8.0K 7.8G 1% /dev/shm /dev/mapper/vgpaas-dockersys 18G 6.2G 11G 37% /etc/hostname /dev/sdd 20G 20G 0 100% /home/postgres/pgdata tmpfs 7.8G 12K 7.8G 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 7.8G 0 7.8G 0% /proc/acpi tmpfs 7.8G 0 7.8G 0% /proc/scsi tmpfs 7.8G 0 7.8G 0% /sys/firmware root@gitea-postgresql-cluster-0:/home/postgres# cd pgdata/pgroot/data/pg_wal && du -h 4.0K ./archive_status 18G .You can also check postgres-operator documentation for pg_wal cleanup for master out of sync
Solutions#
-
Try to delete the pod manually and check if the member is in running state
kubectl delete pod -n gitea gitea-postgresql-cluster-2 kubectl get pods -n gitea kubectl -n gitea exec pocwatt-gitea-cluster-0 -it -- bashroot@gitea-postgresql-cluster-0:/home/postgres# patronictl list -
If not working, enter into the responsible pod
kubectl -n gitea exec pocwatt-gitea-cluster-2 -it -- bash- Stop the service pgdq
root@gitea-postgresql-cluster-2:/home/postgres# sv stop /etc/service/pgqd- Delete all core file and open a new tab on your terminal
root@gitea-postgresql-cluster-2:/home/postgres# cd pgdata/pgroot/data/ root@gitea-postgresql-cluster-2:/home/postgres/pgdata/pgroot/data# ll root@gitea-postgresql-cluster-2:/home/postgres/pgdata/pgroot/data# rm core.*- After deleting all core file, reinit the cluster member to solve the incident. It should be done quickly before core files come back again.
root@gitea-postgresql-cluster-0:/home/postgres# patronictl reinit gitea-postgresql-cluster gitea-postgresql-cluster-2 + Cluster: gitea-postgresql-cluster (6886439782798721093) ----------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +----------------------------+--------------+--------+--------------+----+-----------+ | gitea-postgresql-cluster-0 | 172.16.0.188 | Leader | running | 9 | | | gitea-postgresql-cluster-1 | 172.16.0.130 | | running | 9 | 0 | | gitea-postgresql-cluster-2 | 172.16.0.62 | | start failed | | unknown | +----------------------------+--------------+--------+--------------+----+-----------+ Are you sure you want to reinitialize members gitea-postgresql-cluster-2? [y/N]: y Success: reinitialize for member gitea-postgresql-cluster-2 root@gitea-postgresql-cluster-0:/home/postgres# patronictl list + Cluster: gitea-postgresql-cluster (6886439782798721093) -----+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +----------------------------+--------------+--------+---------+----+-----------+ | gitea-postgresql-cluster-0 | 172.16.0.188 | Leader | running | 9 | | | gitea-postgresql-cluster-1 | 172.16.0.130 | | running | 9 | 0 | | gitea-postgresql-cluster-2 | 172.16.0.62 | | running | 9 | 0 | +----------------------------+--------------+--------+---------+----+-----------+- Restart the service pgqd previously stopped
root@gitea-postgresql-cluster-2:/home/postgres/pgdata/pgroot/data# sv start /etc/service/pgqd ok: run: /etc/service/pgqd: (pid 2104) 0s
The pod is the one has a big volume of Lag, in this case pocwatt-gitea-cluster-1
-
Then, reinit manuelly the pod. delete pod may fix that also, by forcing the reinit. It should be done quickly before core files come back again.
-
If you pod fail to restart or sync verify master pod is up, running and functional
root@gitea-postgresql-cluster-0:/home/postgres# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-252:1-16386-2bf34ce2dbd37f0ab02adf0bc98cb3d9399c413c3fae1471d61a30bf719cec00 9.8G 672M 8.7G 8% / tmpfs 64M 0 64M 0% /dev tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/vgpaas-kubernetes 9.8G 127M 9.2G 2% /etc/hosts tmpfs 7.8G 8.0K 7.8G 1% /dev/shm /dev/mapper/vgpaas-dockersys 18G 6.2G 11G 37% /etc/hostname /dev/sdd 20G 20G 0 100% /home/postgres/pgdata tmpfs 7.8G 12K 7.8G 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 7.8G 0 7.8G 0% /proc/acpi tmpfs 7.8G 0 7.8G 0% /proc/scsi tmpfs 7.8G 0 7.8G 0% /sys/firmwareYou can also check postgres-operator documentation for pg_wal cleanup for master out of sync