Vault Failed Reachable Pages

A pod is in BackOffStart with message the failed container exited with ExitCode: 2

In the log we have:

panic: freepages: failed to get all reachable pages (page 56: multiple references)

    goroutine 29 [running]:
    go.etcd.io/bbolt.(*DB).freepages.func2()
    go.etcd.io/bbolt@v1.3.6/db.go:1056 +0x99
    created by go.etcd.io/bbolt.(*DB).freepages
    go.etcd.io/bbolt@v1.3.6/db.go:1054 +0x1f6

Analysis

The database file, for this pod, is corrupt. If only one pod is down, you can try to delete the corresponding volume and restart the pod.

If more thant one pod is down, we have to follow the backup procedure

Solution

We try to delete vault-cloud-ha-0, but it is still down.

kubectl delete -n vault pod vault-cloud-ha-0
kubectl get -n vault pods -o wide                               
NAME                                            READY   STATUS             RESTARTS      AGE     IP             NODE           NOMINATED NODE   READINESS GATES
vault-cloud-ha-0                                0/1     CrashLoopBackOff   5 (18s ago)   3m43s   172.16.0.151   10.0.191.228   <none>           <none>
vault-cloud-ha-1                                1/1     Running            0             15h     172.16.0.87    10.0.171.233   <none>           <none>
vault-cloud-ha-2                                1/1     Running            0             15h     172.16.0.163   10.0.194.64    <none>           <none>
vault-cloud-ha-agent-injector-b5fdcf79f-5wpxk   1/1     Running            0             448d    172.16.0.183   10.0.194.64    <none>           <none>
vault-cloud-ha-agent-injector-b5fdcf79f-p76j4   1/1     Running            0             45h     172.16.1.25    10.0.171.233   <none>           <none>

We will delete his PV to force re-creation of his database file.

kubectl get -n vault pvc
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
data-vault-cloud-ha-0   Bound    pvc-8db0d004-68d0-4402-912f-55178ba21fc5   10Gi       RWO            caascad-storage-standard   2y166d
data-vault-cloud-ha-1   Bound    pvc-3ae5f980-5777-4a00-9fe8-827389e7412f   10Gi       RWO            caascad-storage-standard   2y166d
data-vault-cloud-ha-2   Bound    pvc-ae5816e5-772b-4af0-8bf4-1aba4359acb2   10Gi       RWO            caascad-storage-standard   2y166d

kubectl get -n vault pv pvc-8db0d004-68d0-4402-912f-55178ba21fc5
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS               REASON   AGE
pvc-8db0d004-68d0-4402-912f-55178ba21fc5   10Gi       RWO            Delete           Bound    vault/data-vault-cloud-ha-0   caascad-storage-standard            2y166d

kubectl delete -n vault pvc data-vault-cloud-ha-0                  
persistentvolumeclaim "data-vault-cloud-ha-0" deleted

Theoretically the pod will auto-unseal, but you can have another issue: VaultFailedToUnsealCore