Aller au contenu

Usage#

Velero is a Kubernetes operator which aims to ease backups & restores of applications.

Backups vs Snapshots#

  • A snapshot is a copy of a disk at a certain point in time. Its implementation is provider-dependant. The content of a snapshot might be collacated on the same disk as the disk it initiates from.
  • A backup is a copy of a disk stored in another location than the original disk

Caascad deployment#

For Caascad deployment, we use velero as a backup operator, using restic to backup disk data. We are currently not using snapshots for multiple reasons:

  • it wasn't available at the time we started using velero, and we didn't have the time to re-evaluate the solution
  • it is unclear whether it can be used to actually backup data in disaster recovery scenario (location of the snapshot compared to original disk ?)
  • snapshots might have quotas (edit: unlimited quota on FE) ?
  • on FlexibleEngine, everest-csi-controller might also use APIs incompatible with velero. (everest-csi-controller currently works agains v1beta1.snapshot.storage.k8s.io while velero-plugin-for-csi requires v1.snapshot.storage.k8s.io)

Warning

Don't try to use snapshots, it will not work as it requires a velero plugin to handlethe snapshot. We don't deploy such a plugin currently.

Info

All velero Custom Resources have to be created in velero namespace, that's the only namespace velero is watching !

Creating a Backup#

Most of the time we want to create a Schedule rather than a Backup (as we want the backup to run periodically).

The following Schedule manifest backs up vault: 

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: vault-snapshot
  namespace: velero
spec:
  schedule: 0 0 * * *
  template:
    includedNamespaces:
    - vault
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: vault
        component: server
        vault-active: "true"

For more details on backups and schedules see backup spec & restore spec for details).

To help you create your backup or Schedule, you can also use velero CLI (available in the toolbox). See https://git.corp.caascad.com/caascad/demo/velero-demo for basic usage.

Pod Volumes backup#

To back up a volume (with restic), an annotation has to be added on the pod that is attached to it. 2 modes are available:

  • whitelisting volumes to backup using annotation backup.velero.io/backup-volumes: volume1,volume2,...
  • blacklisting volumes to ignore, using annotation backup.velero.io/backup-volumes-excludes: volume1,volume2,...

Or you can instead set defaultVolumesToRestic to true in Backup spec to backup all volumes.

Info

when backing up a volume of type PersistentVolumeClaim, the PersistentVolumeClaim and PersistentVolume manifests must be present in the Backup. This is necessary because velero derives new PVC and PV from those objects during its restore procedure.

Warning

Be mindful with your backups, and try to make them application consistent (i.e. ensure validity of your backup procedure, or you might have surprises during restore).

What happens when I'm backing up a PV manifest ? (Volume backup = Pod Backup)#

Note that the following explanation is linked to our current velero deployment, it might be subject to change if we support CSI plugin for snapshots.

Backing up a PersistentVolume (i.e. the manifest describing a volume) only backs up the manifest. Data contained inside the volume are only backed up when velero plugins are configured to do so. That's not the case for our current velero deployment since we use restic for the same purpose.

In this context, backing up the PV manifest and restoring it will seemingly work only if the volume wasn't removed in the first place (that's the case when spec.persistentVolumeReclaimPolicy: Retain is set in the PV: the volume is only redeclared in Kubernetes during a restore, but was actually never deleted from the Cloud Provider perspective).

It DOES NOT backup the underlying volume data.

Warning

TLDR: to backup data, don't backup PersistentVolume, backup Pod instead.

Info

Backing up PersistentVolume & PersistentVolumeClaim manifests is still required for restic restores (they are used by Velero Restore Actions to reprovision persistent volumes)

Backup Hooks#

Velero is able to execute hook commands before and after backing up pod's attached PV. These hooks run from one of the container where the volume is mounted (required commands need to be available in the container image).

You can configure pre-hooks & post-hooks either from pod annotations:

  1. Pre hooks:

    • pre.hook.backup.velero.io/container: The container where the command should be executed. Defaults to the first container in the pod. Optional.
    • `pre.hook.backup.velero.io/command{ : The command to execute. This command is not executed within a shell by default. If a shell is needed to run your command, include a shell command, like /bin/sh, that is supported by the container at the beginning of your command. If you need multiple arguments, specify the command as a JSON array, such as ["/usr/bin/uname", "-a"]. See examples of using pre hook commands. Optional.
    • pre.hook.backup.velero.io/on-error: What to do if the command returns a non-zero exit code. Defaults is Fail. Valid values are Fail and Continue. Optional.
    • pre.hook.backup.velero.io/timeout: How long to wait for the command to execute. The hook is considered in error if the command exceeds the timeout. Defaults is 30s. Optional.

  2. Post hooks

    • post.hook.backup.velero.io/container: The container where the command should be executed. Default is the first container in the pod. Optional.
    • post.hook.backup.velero.io/command: The command to execute. This command is not executed within a shell by default. If a shell is needed to run your command, include a shell command, like /bin/sh, that is supported by the container at the beginning of your command. If you need multiple arguments, specify the command as a JSON array, such as ["/usr/bin/uname", "-a"]. See examples of using pre hook commands. Optional.
    • post.hook.backup.velero.io/on-error: What to do if the command returns a non-zero exit code. Defaults is Fail. Valid values are Fail and Continue. Optional.
    • post.hook.backup.velero.io/timeout: How long to wait for the command to execute. The hook is considered in error if the command exceeds the timeout. Defaults is 30s. Optional.

Info

Hooks can also be specied inside The Backup or Schedule manifest (see backup spec (cue definitions) for details)

Creating Restore#

Restore manifests are used to recreate resources from a velero Backup or Schedule.

Restore example: 

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: xma-test-20210825181733
  namespace: velero
spec:
  backupName: xma-test
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  includedNamespaces:
  - '*'

Info

By default, velero does not override resources already present in the cluster. It only creates missing ones. This behavior can be modified, see existingResourcePolicy in schedule spec

Restore Hooks#

Velero is able to execute hook commands during and after pods' restore operations. These hooks run from an initContainer when executing during restore, or inside one of the containers when executed after restore (required commands need to be available in the container image).

Restore hooks can be configured either from annotations in the Pod definition or from the Restore resource. It's not possible to provide hooks both from annotations & from the Restore resource.

Restore hooks stored in annotations#

  1. InitContainer Restore Hooks: Use an InitContainer hook to add init containers into a pod before it’s restored. You can use these init containers to run any setup needed for the pod to resume running from its backed-up state. The InitContainer added by the restore hook will be the first init container in the podSpec of the restored pod. In the case where the pod had volumes backed up using restic, then, the restore hook InitContainer will be added after the restic-wait InitContainer.

    • init.hook.restore.velero.io/container-image: The container image for the init container to be added.
    • init.hook.restore.velero.io/container-name: The name for the init container that is being added.
    • init.hook.restore.velero.io/command: This is the ENTRYPOINT for the init container being added. This command is not executed within a shell and the container image’s ENTRYPOINT is used if this is not provided.

  2. Exec Restore Hooks: Use an Exec Restore hook to execute commands in a restored pod’s containers after they start.

    • post.hook.restore.velero.io/container: The container name where the hook will be executed. Defaults to the first container. Optional.
    • post.hook.restore.velero.io/command: The command that will be executed in the container. Required.
    • post.hook.restore.velero.io/on-error: How to handle execution failures. Valid values are Fail and Continue. Defaults to Continue. With Continue mode, execution failures are logged only. With Fail mode, no more restore hooks will be executed in any container in any pod and the status of the Restore will be PartiallyFailed. Optional.
    • post.hook.restore.velero.io/exec-timeout: How long to wait once execution begins. Defaults to 30 seconds. Optional.
    • post.hook.restore.velero.io/wait-timeout: How long to wait for a container to become ready. This should be long enough for the container to start plus any preceding hooks in the same container to complete. The wait timeout begins when the container is restored and may require time for the image to pull and volumes to mount. If not set the restore will wait indefinitely. Optional.

Restore hooks stored Restore resource#

Restore hooks can also be specied inside The Restore manifest (see restore spec (cue definitions) for details)

Using restic CLI with (restic) backups#

Restic CLI is available from caascad/toolbox and can be useful to fetch or list available backups.

  1. Configure the CLI: 

    ZONE="ocb-XXXX"
KUBERNETES_NAMESPACE="YYYY"
kswitch "${ZONE}"
export VAULT_ADDR="https://vault.${ZONE}.caascad.com"; vault login -method=oidc
export RESTIC_PASSWORD="$(kubectl get secret -n velero velero-restic-credentials -o go-template='{{index .data "repository-password" | base64decode}}')"

# FlexibleEngine
export AWS_ACCESS_KEY_ID="$(vault kv get -field=access_key secret/bucket/velero)"
export AWS_SECRET_ACCESS_KEY="$(vault kv get -field=secret_key secret/bucket/velero)"
export RESTIC_REPOSITORY="s3:https://oss.eu-west-0.prod-cloud-ocb.orange-business.com/velero-${ZONE}/velero/restic/${KUBERNETES_NAMESPACE}"

# AWS
eval "$(vault read aws/sts/bucket_velero -format=json | jq -r '.data | "export AWS_ACCESS_KEY_ID=\(.access_key); export AWS_SECRET_ACCESS_KEY=\(.secret_key); export AWS_SESSION_TOKEN=\(.security_token)"')"
export RESTIC_REPOSITORY="s3:s3-eu-west-3.amazonaws.com/velero-${ZONE}/velero/restic/${KUBERNETES_NAMESPACE}"

# Azure: you first need to login using az CLI:
# az login --tenant <tenant_id>
export AZURE_ACCOUNT_NAME="${ZONE/-/}velero"
export AZURE_ACCOUNT_KEY="$(az storage account keys list -g rg-${ZONE} -n ${ZONE/-/}velero -o json | jq -r '.[0].value')"
export RESTIC_REPOSITORY="azure:velero:/velero/restic/${KUBERNETES_NAMESPACE}/"

  2. list snapshots: 

    # list all snapshots
restic snapshots

# show last snapshot
restic snapshots --latest 1 --json | jq -r '. | sort_by(.time) | reverse | .[0]'

  3. List files in a snapshot: 

    restic ls <SNAPSHOT_ID>

  4. Fetch a snapshot: 

    restic restore <SNAPSHOT_ID> --target snapshot.tar.gz