Vault Failed To Unseal Core

We try to restart a vault pod but the pod is Unhealthy.

In the log we have:

[INFO]  core: stored unseal keys supported, attempting fetch
[WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"

Analysis

Theoretically the job vault-cloud-ha-join-raft-node-* call for vault operator raft join automatically.

But the job is note present or the pod that fail is vault-cloud-ha-0

The current configuration assumes vault-cloud-ha-0 is the leader, so if vault-cloud-ha-0 is down (or is not the current leader), no pod can enter the cluster.

Solution

We will add the necessary job. The simpler way is to load trackbone shell and ask helm to generate the job.

But first we need to now who is the current leader.

export ZONE_NAME=ocb-xx
export VAULT_ADDR="https://vault.${ZONE_NAME}.caascad.com"
vault login -method oidc
vault operator raft list-peers

Node                                    Address                                          State       Voter
----                                    -------                                          -----       -----
d31bc8e9-c0f1-f754-20be-5bbf542c0ad6    vault-cloud-ha-0.vault-cloud-ha-internal:8201    follower    true
d837bbbe-f940-447d-75e1-7a73d29736a5    vault-cloud-ha-1.vault-cloud-ha-internal:8201    follower    true
1606399e-f5a5-22fb-f4e3-69f9ea4e11cc    vault-cloud-ha-2.vault-cloud-ha-internal:8201    leader      true

Here vault-cloud-ha-2. is the leader. In the nix-shell, I call helm template (without --no-hooks parameter) to generate vault-cloud-ha-join-raft-node-* jobs.

trackbone shell -z ${ZONE_NAME} -c vault
helm template --namespace "vault"  -f values.yaml "vault-cloud-ha" "./helm"  --output-dir gen
cat gen/vault-caascad/templates/join-raft.yaml

The helm chart generate only two jobs for vault-cloud-ha-1 and vault-cloud-ha-2. I modified the file to create a job for vault-cloud-ha-0 that target vault-cloud-ha-2 as leader.

---
apiVersion: batch/v1
kind: Job
metadata:
  name: "vault-cloud-ha-join-raft-node-0"
  labels:
    app.kubernetes.io/managed-by: "Helm"
    app.kubernetes.io/instance: "vault-cloud-ha"
    app.kubernetes.io/version: 1.10.4
  annotations:
    # This is what defines this resource as a hook. Without this line, the
    # job is considered part of the release.
    "helm.sh/hook": post-install
    "helm.sh/hook-weight": "10"
    "helm.sh/resource-policy": keep
spec:
  ttlSecondsAfterFinished: 300 # 5min
  backoffLimit: 5
  template:
    metadata:
      name: "vault-cloud-ha-join-raft-node-0"
      labels:
        app.kubernetes.io/managed-by: "Helm"
        app.kubernetes.io/instance: "vault-cloud-ha"
    spec:
      restartPolicy: Never
      volumes:
        - name: vault-internal-tls
          secret:
            secretName: vault-internal-tls
      containers:
      - name: post-install-join-raft-node
        image: docker-registry.caascad.com/external/vault:miswl9gam8yw1g1pn5ddlgram87y14n6
        volumeMounts:
          - name: vault-internal-tls
            mountPath: /vault/vault-internal-tls
        env:
          - name: VAULT_CACERT
            value: /vault/vault-internal-tls/ca.crt
          - name: VAULT_ADDR
            value: https://vault-cloud-ha-0.vault-cloud-ha-internal:8200
        command:
          - "/bin/sh"
          - "-ec"
          - |
            `vault operator raft join` -leader-ca-cert="$(cat "$VAULT_CACERT")" \
              "https://vault-cloud-ha-2.vault-cloud-ha-internal:8200"

Apply the job and delete current pod vault-cloud-ha-0.

kubectl apply -n vault -f gen/vault-caascad/templates/join-raft.yaml
kubectl delete -n vault po vault-cloud-ha-0
kubectl logs -n vault -f vault-cloud-ha-0

Wait the pod to become available, then remove the jobs. (This is optional beacause the job will be destroy after 5 minutes ttlSecondsAfterFinished)

kubectl delete -n vault jobs vault-cloud-ha-join-raft-node-0

Modification of a leader

By default, the leader is the node 0.

In envs-ng, the default value and authorized values are set in: contexts/vault.cue:

configurations: ["vault"]: #HelmConfig & {

    helm: values: server: ha: raft: leader: *0 | 1 | 2
}

If you need to change the default value for a specific zone, add following lines into :

• contexts/caascad/vault.cue : if the zone belongs to the caascad context
• contexts/ngot/vault.cue : if the zone belongs to the ngot context

envs: [string]: {
    configurations: ["vault"]: {
        helm: {
            values: {
                vault: server: {
                    if zone.name == "infra-stg" {
                        ha: raft: leader: 1
                    }
                    if zone.name == "ocb-xxx" {
                        ha: raft: leader: 2
                    }
                }
            }
        }
    }
}