Aller au contenu

PostgresReplicaLag#

Postgresql database has too many replication lag (using patronictl list command).

If an alert is raised, the error could be:

  • error on cluster synchronization status (delay)
  • error to get the leader pod of the postgresql cluster
  • error to connect on the leader pod of the postgresql cluster.

Known cases#

Database problem#

You can find more info about patroni commands here, for example, you may delete a pod to force member resync: PostgresAlerts

If one of more member(s) has a lag, use patronictl reinit [CLUSTER_NAME] [CLUSTER_MEMBER]

Get leader pod#

# get pods name
kubectl -n concourse get pods

NAME                             READY   STATUS    RESTARTS   AGE
concourse-postgresql-0           2/2     Running   0          6d23h
concourse-postgresql-1           2/2     Running   0          6d23h
concourse-web-7d9b849896-dswdj   1/1     Running   0          4d21h


# Get info about the cluster (from any pod)
kubectl -n concourse exec -it -c postgres concourse-postgresql-0 -- patronictl list

+ Cluster: concourse-postgresql (6969851826928627779) ----+----+-----------+
|         Member         |     Host    |  Role  |  State  | TL | Lag in MB |
+------------------------+-------------+--------+---------+----+-----------+
| concourse-postgresql-0 | 10.0.10.141 | Leader | running | 13 |           |
| concourse-postgresql-1 |  10.0.20.61 |        | running | 10 |      6976 |
+------------------------+-------------+--------+---------+----+-----------+

Connect to leader#

# connect to the container
kubectl -n concourse exec -it -c postgres concourse-postgresql-0 -- bash

# then you can reinit the member "concourse-postgresql-1"

patronictl list

+ Cluster: concourse-postgresql (6969851826928627779) ----+----+-----------+
|         Member         |     Host    |  Role  |  State  | TL | Lag in MB |
+------------------------+-------------+--------+---------+----+-----------+
| concourse-postgresql-0 | 10.0.10.141 | Leader | running | 13 |           |
| concourse-postgresql-1 |  10.0.20.61 |        | running | 10 |      6976 |
+------------------------+-------------+--------+---------+----+-----------+

patronictl reinit concourse-postgresql concourse-postgresql-1

+ Cluster: concourse-postgresql (6969851826928627779) ----+----+-----------+
|         Member         |     Host    |  Role  |  State  | TL | Lag in MB |
+------------------------+-------------+--------+---------+----+-----------+
| concourse-postgresql-0 | 10.0.10.141 | Leader | running | 13 |           |
| concourse-postgresql-1 |  10.0.20.61 |        | running | 10 |      6976 |
+------------------------+-------------+--------+---------+----+-----------+
Are you sure you want to reinitialize members concourse-postgresql-1? [y/N]: y
Success: reinitialize for member concourse-postgresql-1

# you should have fix the lag
patronictl list
+ Cluster: concourse-postgresql (6969851826928627779) ----+----+-----------+
|         Member         |     Host    |  Role  |  State  | TL | Lag in MB |
+------------------------+-------------+--------+---------+----+-----------+
| concourse-postgresql-0 | 10.0.10.141 | Leader | running | 13 |           |
| concourse-postgresql-1 |  10.0.20.61 |        | running | 13 |         0 |
+------------------------+-------------+--------+---------+----+-----------+