PostgreSQLMemberDown#
A instance in postgres cluster is down.
Troubleshooting#
- Check the postgres pods status
kubectl get po -n clair
NAME READY STATUS RESTARTS AGE
clair-c987b4987-9rzl8 1/1 Running 0 14d
clair-postgresql-cluster-0 2/2 Running 0 17m
clair-postgresql-cluster-1 2/2 Running 0 18m
clair-postgresql-cluster-2 2/2 Running 0 17m
- Check patroni status
kubectl exec -it -n clair clair-postgresql-cluster-0 -c postgres -- bash -c "patronictl list"
+ Cluster: clair-postgresql-cluster (7025907046366744645) -+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+----------------------------+--------------+--------------+---------+----+-----------+
| clair-postgresql-cluster-0 | 172.16.1.8 | Sync Standby | running | 6 | 0 |
| clair-postgresql-cluster-1 | 172.16.0.247 | | stopped | | unknown |
| clair-postgresql-cluster-2 | 172.16.1.18 | Leader | running | 6 | |
+----------------------------+--------------+--------------+---------+----+-----------+
- Check global filesystem
kubectl exec -it -n clair clair-postgresql-cluster-0 -c postgres -- bash -c "df -h"
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-252:1-786433-ed027b82808642de329323e51fbb4c1c936f6e03eff87f89280c3c062c8222d2 9.8G 672M 8.7G 8% /
tmpfs 64M 0 64M 0% /dev
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
tmpfs 7.8G 8.0K 7.8G 1% /dev/shm
/dev/mapper/vgpaas-kubernetes 9.8G 41M 9.3G 1% /etc/hosts
/dev/mapper/vgpaas-dockersys 18G 3.7G 14G 22% /etc/hostname
/dev/sdj 49G 440M 49G 1% /home/postgres/pgdata
tmpfs 7.8G 12K 7.8G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7.8G 0 7.8G 0% /proc/acpi
tmpfs 7.8G 0 7.8G 0% /proc/scsi
tmpfs 7.8G 0 7.8G 0% /sys/firmware
- Check the logs of the pod
Solution#
Fix member issue, go to :