Aller au contenu

CRD management#

State of the art#

Today, some of the tools we deploy (rancher, cert-manager, velero, etc.) use CRDs.

In the context of Caascad, they also generate interdependencies in the deployment of applications on a cluster. This is why the management of these resources is deported to a dedicated Git repository.

This mode of operation is far from ideal since it goes beyond the common Trackbone workflow and these CRDs are very little maintained. It is important that we find a better way to manage these resources, especially when upgrading our tooling.

Lifecycle#

CRDs are global resources for the cluster, which means that once installed, they can be used by any user. Moreover, CRDs are mutable resources, which creates significant impacts when adding/modifying/deleting these resources. Kubernetes is not very careful about the changes made since these changes are not really controlled. If you delete a CRD all instantiated objects disappear. Same thing for the upgrade where you can make a CRD incompatible from one version to another by changing its schema.

Kubernetes defines a workflow to set up and update CustomResourceDefinition. When creating a CRD, it is possible to mention several versions in the specifications. Only one of these is defined as the storage version. This means that Kubernetes only stores the schema of this version in the Etcd.

When an object is created, it is persisted in the storage version defined at that moment. If the storage version changes, existing objects are not migrated. However, all newly created or modified objects are stored with this new version. It is therefore possible to have objects stored in a version that no longer exists.

N.B: When you request an object, you specify the version. If the version is different from the persisted one, Kubernetes returns the object at the requested version but no real change is made (other than changing the apiVersion field).

Add a new version#

When we need to add a new version of CRD, the recommended process is as follows: 1. Knowing that the cluster can support several versions of the same resource, it is possible that the schema of the requested resource differs from that of the storage version. If this is the case, it is necessary to choose a conversion strategy that involves the use of a webhook conversion. Otherwise a simple modification of the apiVersion field is sufficient. 2. If necessary, deploy the conversion webhook. 3. Update the CustomResourceDefinition to include the new version in spec.versions with the served:true attribute. Then set the conversion strategy in spec.conversion.

Updating an existing resource#

When deprecating a version of a CRD, two choices are possible.

**Option 1: Use the Storage Version Migrator

  • Install the [Storage Version Migrator] (https://github.com/kubernetes-sigs/kube-storage-version-migrator) so that it updates the instantiated objects to the storage version of the CRD.
  • Remove the old version from the CRD status.storedVersions field.

Option 2: Manually migrate the objects to the new version

Example for moving from a v1beta1 version to v1

  • Set v1 as the stored version by editing the CRD. The storedVersions field now contains v1beta1 and v1.
  • Update the version of all existing objects so that they reference v1.
  • Remove v1beta1 from the status.storedVersions field of the CRD.

Delete a version#

When you want to deprecate a version of a CRD, a set of tasks is required: 1. Ensure that all clients have migrated to a new version of the resource (no real programmatic way to do this). 2. Set the served attribute to false in the spec.versions field of the CRD. This will have the effect of triggering errors when calling this resource for clients still using the deprecated version. 3. Ensure that the new version is defined as storage version and that the deprecated version is no longer referenced in the status.storedVersions field. 4. Remove the old version from the spec.versions field of the CRD. 5. Edit the webhook conversion so that it no longer supports this version.

*N.B: as of version 1.19 it is no longer possible to use the webhook.