Persistent volumes on Kubernetes: ReadWriteOnce, Recreate and the snapshot before deletion
A rolling update with a volume is a deadlock waiting to happen. And deleting a volume from the manifest is data loss waiting to happen. Five rules that prevent both.
Stateless is the ideal, but most applications have files somewhere: CMS uploads, a search index, a cache, a small SQLite database. On Kubernetes, that means a PersistentVolumeClaim on Block Storage. This works well as long as you respect five properties of Block Storage that Kubernetes doesn’t take care of for you.
Rule 1: ReadWriteOnce means exactly one node
Scaleway Block Storage is a disk attached to a virtual machine. Kubernetes calls this ReadWriteOnce: a volume can be mounted on one node at a time. Two pods on two nodes cannot share it. Start three replicas with the same volume and you get one running and two with Multi-Attach error.
The consequence: a service with a volume has exactly one instance. No autoscaler, no fixed count greater than one. If you need several instances with shared files, you need Object Storage or a network file system, not Block Storage.
Rule 2: Recreate instead of RollingUpdate
A Deployment’s default strategy is RollingUpdate: the new pod starts while the old one is still running. With a ReadWriteOnce volume, that’s a deadlock. The new pod waits for the volume the old one still holds, and the old one is only terminated once the new one is ready. The Deployment stays at "1 of 1 updated, 0 ready" forever.
The solution is strategy: Recreate: the old pod is terminated, the volume is released, the new pod starts. That costs a few seconds of downtime per rollout. For an application with a single instance, that is the case anyway.
Rule 3: Undeploy doesn’t delete a volume
A claim is an object of its own. Deleting a Deployment doesn’t delete the claim, and that is how it should be. Deleting the claim hands the disk over to the storage class’s reclaim policy, which is usually set to Delete. The disk is then gone – immediately, without asking.
That’s why a volume never belongs on the same deletion path as the workload. Undeploy removes the Deployment, Service, Ingress and Secrets. The claim stays until someone explicitly deletes the data, with confirmation – not because they removed a line from a manifest.
Rule 4: Before deleting, a snapshot that has finished
Block Storage supports snapshots: a copy of the disk at a point in time, independent of the node, restorable as a new disk. A snapshot before every deletion is the difference between "the data is gone" and "the data can be recovered". Two details matter:
- The snapshot must have reached the
availablestate before the claim is deleted. A snapshot that has merely been triggered is not a snapshot. - If the snapshot fails, nothing is deleted. Abort instead of skipping.
Add a schedule on top: a snapshot daily or weekly, with retention in days, and automation that only deletes the snapshots it created itself. Snapshots created manually in the console remain untouched.
Rule 5: Size only grows, class stays
A disk can be enlarged, not shrunk. And a claim’s storage class is fixed once it is created. A tool should enforce both instead of leaving it to the user: a claim with a smaller size in the manifest is rejected, and so is a different class. If you really want to shrink, create a new volume, copy the data, and delete the old one via the protected path.
What does fsGroup have to do with it?
A disk is mounted with root permissions. A container running as non-root then cannot write to it. fsGroup in the pod’s SecurityContext sets the group of the mount, and Kubernetes adjusts the permissions when mounting. That is the usual reason for "Permission denied" on the first start of an application with a volume, and it has nothing to do with the disk.
How do you restore a volume?
A snapshot produces a new disk that replaces the old one behind the claim. By hand, that means stopping the pod, creating a disk from the snapshot in the Scaleway console, swapping the PersistentVolume and rebinding the claim. Three rules go with it. First, stop the service beforehand so nobody writes to the old disk. Second, save the current state as a snapshot yourself first, in case the chosen snapshot was the wrong one. Third, only start again afterwards.
In Clusterward, that’s one click: "Restore…" on the snapshot in the Volumes card stops the service, saves the current state as an additional snapshot, replaces the disk and starts the service again, with the progress shown on the card. Restoring still remains a deliberate step, not an automatic fallback, as described under Volumes & snapshots.
Which applications really need a volume?
Not every file needs Block Storage. The rule of thumb: a volume for data that a single instance needs locally and fast. Object Storage for data shared by several instances or several applications. A managed database for anything that needs queries.
Data | Right place | Why |
|---|---|---|
CMS uploads | Object Storage | Multiple instances, direct delivery, backup as objects |
Search index | Volume | Local, fast, one instance, rebuildable |
SQLite of a small app | Volume | One instance, snapshot as backup |
Sessions and caches | No volume | Ephemeral, in memory or in Redis |
Customer data with queries | Managed database | Roles, backups, import, no multi-attach problem |
Many applications that “need a volume” actually need an S3 adapter for uploads. That is usually a single line of configuration and makes the application horizontally scalable.
Snapshots aren’t backups until you’ve tested them
A snapshot is a copy of the disk, consistent at the block level. For an application that is in the middle of writing, that can mean a half-written file. For a search index, that doesn’t matter – it gets rebuilt. For a SQLite database, it can mean a corrupted state. The application should be quiet before the snapshot, or the file system and the database must be able to cope with half-finished writes; SQLite in WAL mode can.
And as with offboarding: a snapshot nobody has ever restored from is a hope. Once a quarter, create a disk from the latest snapshot, attach it to a test pod and check the files.
How Clusterward does it
A service with volumes automatically gets Recreate and a single instance; the claim is created before the deployment and survives every undeploy. Deleting is only possible via a dedicated path that requires typing the name, and only after a snapshot that is ready. Snapshots run on a schedule with retention; only snapshots from Clusterward’s own list are deleted. Size can only go up, the class is fixed, and removing a volume from the list is rejected. Details under Volumes & snapshots.
Conclusion
Volumes on Kubernetes are safe when five rules apply: one instance, Recreate, the claim survives undeploy, a snapshot before deletion, grow only. Each rule stands for a data loss someone has already suffered. Build them into a tool before you suffer it too.
Storing data safely on Kubernetes? Tell us which application needs files on disk. We’ll show you what volume, snapshots and restore look like for it. Ask about volumes →
Sources and further reading
Frequently asked questions
- Because a ReadWriteOnce volume can only be mounted on one node at a time. The new pod waits for the volume the old one still holds, and the old one is only terminated once the new one is ready. The solution is strategy: Recreate, which costs a few seconds of downtime per rollout.