Migrating ArgoCD PVCs from One Cluster to Another
Most of the credit for this article comes from Zach’s great post about migrating Persistent Volume Claims (PVCs).
A notable difference since that post was written is the widespread adoption of ArgoCD for running applications. This includes letting ArgoCD handle PVC creation instead of manual kubectl
commands.
The Problem
If you followed the steps outlined in Zach’s post, you will encounter sync issues for ArgoCD applications with PVC manifests:
The PersistentVolumeClaim "postgres" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims
This is because the PVC was also migrated using kubectl
. Once a PVC is created, certain key-value pairs are automatically generated and become immutable thereafter.
One example is volumeName
on which determines the specific Persistent Volume it is bound to.
When you attempt to sync ArgoCD using the same manifest, it will trigger an attempt to remove the volumeName
, which is Forbidden!
The Solution
Since modifying the PVC after its creation is not possible, you can consider the following alternative approach:
Let ArgoCD handle application deployment, including PVC creation.
Modify the bound PV.
Steps
This will incur downtime. You should have extracted the PV manifest from your old cluster as well as stated in Zach’s blog.
After ArgoCD deployment, scale your application down to 0 replicas.
Note: By having an initial replica, the newly created PVC transitions from aPending
state to aBound
state. We can now scale down the replicas once the state is set.Grab the
uid
andvolumeName
of the PVC.
# uid
kubectl get pvc <name> -n <namespace> -o jsonpath='{.metadata.uid}'
# volume_name
kubectl get pvc <name> -n <namespace> -o jsonpath='{.spec.volumeName}'
Delete the PV bound to it.
The PVC’s status will update toLost
.
kubectl delete pv <volume_name>
Update your PV manifest from your old cluster with
name
anduid
with PVC values from Step 2.Apply the PV manifest.
Verify that the PVC updates back to a
Bound
status.
This should take less than a minute.Scale up your application and verify it is working properly.
I hope this demystifies the ownership issue that arises during PVC migrations when ArgoCD is involved. I would appreciate hearing your thoughts on this approach for handling migrations. Feel free to post a comment or share your input. Thanks in advance!