Migrate your storage to another availability zone

This guide walks you through the steps required to migrate an existing Persistent Volume Claim (PVC) and its data to another availability zone (AZ).

Prerequisites

• A node pool in the target availability zone. For further information, read our documentation on node pools.
• Access to the Kubernetes cluster in which you want to migrate the volumes.

Before you begin

Before you start the migration, make sure that you have a tested backup and a rollback plan. A storage migration involves moving and renaming live data, so you should:

• Create a tested backup of the volume data and verify that a restore works.
• Plan a downtime window for the workload that uses the volume.
• Document a rollback procedure (for example, keep the old PVC in a renamed state until the new volume is verified).

Choose a migration approach

This guide uses pv-migrate, which copies the data over the network into a freshly provisioned volume. The network copy works independently of the storage backend and deterministically targets the AZ of the pv-migrate pod.

For other workload types, consider these alternatives:

• Stateless/single RWO PVC: Use pv-migrate as shown in this guide.
• StatefulSet: The rename approach described here does not apply, because StatefulSets use fixed PVC names derived from volumeClaimTemplates. Migrate each replica’s volume individually with pv-migrate, or use the application’s own replication mechanism (for example, database replica streaming) to bring up a new replica in the target AZ.
• Database: Use the database’s own backup/restore mechanism or a managed offering such as STACKIT PostgreSQL Flex.

Migrate your storage

To migrate a volume from one availability zone to another, you can use the tool pv-migrate. Install it by following the installation instructions.

To migrate a volume, you need to create a new volume with the same parameters as the old one. Therefore, fetch the old volume to get its definition with the following command:

kubectl get pvc <pvc-name> -n <namespace> -o yaml > pvc.yaml

Open pvc.yaml in the editor of your choice for the next steps. The result should look similar to the following example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"old-volume","namespace":"docs"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"volumeMode":"Filesystem"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
  creationTimestamp: "2022-02-04T13:21:03Z"
  finalizers:
    - kubernetes.io/pvc-protection
  name: old-volume
  namespace: docs
  resourceVersion: "49199"
  uid: 4ad285b1-3be2-44c3-afd6-098631a6a6ec
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: premium-perf1-stackit
  volumeMode: Filesystem
  volumeName: pv-shoot--garden--mig-test-4ad285b1-3be2-44c3-afd6-098631a6a6ec
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

Remove unnecessary annotations

To turn this definition into a manifest for a new volume, remove the fields that Kubernetes manages automatically. From metadata.annotations, delete the following entries:

• kubectl.kubernetes.io/last-applied-configuration
• pv.kubernetes.io/bind-completed
• pv.kubernetes.io/bound-by-controller
• volume.beta.kubernetes.io/storage-provisioner

Also remove:

• creationTimestamp
• finalizers
• resourceVersion
• uid

Under spec, delete volumeName. Also delete the entire status section.

Lastly, you have to give the PVC a new name. In this guide new-volume is used. Keep the same storageClassName as the source PVC for now; the next step explains how to verify its binding mode.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: new-volume
  namespace: docs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: premium-perf1-stackit
  volumeMode: Filesystem

Verify the volume binding mode

The target PVC must remain Pending until a pod consumes it, so the AZ can be influenced through the pv-migrate pod. This only works with volumeBindingMode: WaitForFirstConsumer. Check the binding mode of the StorageClass used by the source PVC:

kubectl get storageclass <storage-class-name> -o jsonpath='{.volumeBindingMode}{"\n"}'

If the result is WaitForFirstConsumer, you can continue with the same StorageClass. If it is Immediate, the volume would be provisioned right away in the source AZ, before pv-migrate can influence scheduling. In that case, you need to create a custom StorageClass with late binding before continuing:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium-perf1-stackit-wait
provisioner: cinder.csi.openstack.org
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Then set storageClassName in the new PVC manifest to premium-perf1-stackit-wait (or the name you chose) and apply it again.

Read our documentation on storage classes and topologies for background on the available classes and binding modes.

After editing the file, save it and create the new PVC using the following command:

kubectl apply -f pvc.yaml

Check if the PVC has been created successfully. The result should look similar to the following example:

kubectl get pvc -n <namespace>

NAME         STATUS    VOLUME                                                            CAPACITY   ACCESS MODES   STORAGECLASS                 AGE
new-volume   Pending                                                                                               premium-perf1-stackit-wait   15m
old-volume   Bound     pv-shoot--garden--mig-test-4ad285b1-3be2-44c3-afd6-098631a6a6ec   1Gi        RWO            premium-perf1-stackit        15m

Be aware that the new volume should be in Pending status because of the WaitForFirstConsumer binding mode. If you created a custom StorageClass, make sure to set it in the PVC manifest.

Next, identify the availability zone of the target node pool. List the zones available to your nodes:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}'

Choose your target AZ and create a file called values.yaml with the following content, replacing eu01-3 with your target zone:

rsync:
  nodeSelector:
    topology.kubernetes.io/zone: "eu01-3"

The pv-migrate pod inherits this node selector, so it is scheduled in the target AZ. Because the new PVC uses WaitForFirstConsumer, the volume is then provisioned in the same AZ as the pv-migrate pod.

Now you should shut down the application mounting the volume you want to migrate. After the pod that mounted the volume has been shut down successfully, run the following command, replacing <old-pvc-name>, <new-pvc-name>, and <namespace> with your values.

The command below uses the current pv-migrate CLI syntax. For the full set of options, see the pv-migrate migration guide and the CLI reference. To keep the behaviour reproducible, pin the pv-migrate version you use.

pv-migrate --source <old-pvc-name> --dest <new-pvc-name> --source-namespace <namespace> --dest-namespace <namespace> -f values.yaml

🚀 Starting migration
💭 Will attempt strategies: mount, clusterip
🚁 Attempting strategy: clusterip
🔑 Generating SSH key pair
📂 Copying data... 100% |████████████████████████████████████████████████████████████| ()
🧹 Cleaning up
✨ Cleanup done
✅ Migration succeeded

If the migration was successful, the data is now available in the new availability zone. To confirm this, inspect the nodeAffinity of the newly created persistent volume.

kubectl get pv <new-volume-name> -o yaml | grep -A5 nodeAffinity

nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
    - key: topology.cinder.csi.openstack.org/zone
      operator: In
      values:
      - eu01-3

The values entry shows the availability zone the volume was provisioned in.

As the data is now in the correct availability zone, you need to point your application at the new volume. You can do this in two ways: bind the new PVC directly in your deployment, or rename the new PVC to the old PVC’s name so the deployment stays unchanged.

Changing the PVC reference in a deployment is usually straightforward, but for StatefulSets this does not work. StatefulSets derive their PVC names from volumeClaimTemplates and keep stable identities per replica. If you are using a StatefulSet, prefer application-native replication or a per-replica migration, and review the StatefulSet documentation before proceeding.

This guide uses the renaming approach: instead of changing your deployment to reference the new PVC, the old PVC is renamed and the new PVC takes the original name. This keeps the old PVC available in case a rollback is required.

Before you proceed, make sure the backup from the before you begin section is in place and tested, as renaming PVCs affects live workloads.

1. Install rename-pvc by following its installation instructions.

2. Verify the installation:

   rename-pvc --help

3. Move the old PVC aside by renaming it to a temporary backup name (<old-pvc-name>-backup):

   rename-pvc -n <namespace> <old-pvc-name> <old-pvc-name>-backup

4. Give the new PVC the original name:

   rename-pvc -n <namespace> <new-pvc-name> <old-pvc-name>

Now you can safely restart your application. It will use the volume in the correct availability zone without any changes to the deployment’s specification.

After verifying that the application works correctly with the new volume, clean up the old PVC that was renamed to <old-pvc-name>-backup:

kubectl delete pvc <old-pvc-name>-backup -n <namespace>

Your storage and its data are now migrated to the new availability zone.