Aller au contenu

PostgresAlerts#

This means one member of the postgres cluster has a desynchro resulting increase of wal files that will explose the volume attached to pod.

Note

All commands mentioned in this procedure, can be performed on any of the postgres cluster member, as long as the pod is able to accept commands.

  1. Check postgres cluster members and identify the Leader

This is done by connecting to a pod that is part of the postgres cluster (and that is also able to accept commands): 

kubectl exec -it gitea-postgresql-cluster-2  -n gitea sh
cd /home/postgres
patronictl -c postgres.yml list

  - Cluster: gitea-postgresql-cluster ---------------------------+----+-----------+
  |           Member           |     Host     |  Role  |  State  | TL | Lag in MB |
  +----------------------------+--------------+--------+---------+----+-----------+
  | gitea-postgresql-cluster-0 | 172.16.0.72  |        | running |  6 |         0 |
  | gitea-postgresql-cluster-1 | 172.16.0.125 | Leader | running |  6 |     39295 |
  | gitea-postgresql-cluster-2 | 172.16.0.103 |        | running |  6 |         0 |
  -----------------------------+--------------+--------+---------+----+-----------+

From the output, we can see that the member with high Lag is also the Leader.

To fix the issue in this case, it is necessary to move the Leader role to another cluster member.

  1. Move Leader role
 kubectl exec -it -n gitea gitea-postgresql-cluster-0 -- bash -c "patronictl switchover"
  1. Reinit the pod

Manually reinit the pod with high volume of Lag, by using patronictl.

```
kubectl exec -it gitea-postgresql-cluster-2  -n gitea sh #command can be executed on any pod
cd /home/postgres
patronictl -c postgres.yml reinit  gitea-postgresql-cluster gitea-postgresql-cluster-1
```

Where gitea-postgresql-cluster is the CLUSTER_NAME and gitea-postgresql-cluster-1 is the MEMBER_NAME.

Note

Deleting the pod may also fix the issue, by forcing the reinit.

  1. Verification

See if the problem is fixed, by checking the amount of space taken by wal files.

Before fixing the issue:

❯ kubectl exec -it -n gitea gitea-postgresql-cluster-1 -- bash -c "du -sh /home/postgres/pgdata/"
  Defaulted container "postgres" out of: postgres, prometheus-exporter
  4.3G   /home/postgres/pgdata/

After fixing the issue: 

❯ kubectl exec -it -n gitea gitea-postgresql-cluster-0 -- bash -c "du -sh /home/postgres/pgdata/"
Defaulted container "postgres" out of: postgres, prometheus-exporter
264M        /home/postgres/pgdata/