Kubernetes upgrades on Kapsule without the fear

Kubernetes upgrades have a bad reputation, a holdover from the days of self-built clusters. On a managed offering like Kapsule, an upgrade is one click and twenty minutes of waiting. The fear doesn’t come from the upgrade itself, but from what you didn’t check beforehand.

What does Scaleway handle during an upgrade, and what doesn’t it?

Kapsule can update the control plane automatically if you want, but only patch versions, and only within a maintenance window. Going from 1.31.2 to 1.31.4 happens overnight on its own. Going from 1.31 to 1.32 only happens when someone triggers it. And it’s one minor version per step: if you’re on 1.30 and want 1.33, that’s three upgrades.

Node pools are not upgraded automatically along with the control plane unless you choose that option. A cluster with a 1.32 control plane and nodes on 1.31 runs fine, because Kubernetes allows kubelets to lag a few minor versions behind. But it runs with a maintenance backlog that becomes a blocker at the next upgrade.

What do you check before you click “Upgrade”?

1. Look at the target version. Only the next minor version and newer patches of the current one are reachable. The Kapsule API lists them; if you don’t see them, you may be on a version Scaleway no longer supports and have to get there first.
2. Check add-ons. Every Helm chart has a kubeVersion range. An ingress-nginx whose chart excludes the target version may keep running after the upgrade, but can no longer be updated. Same for cert-manager and the Reflector. First bring add-ons to a version that covers both Kubernetes versions, then upgrade Kubernetes.
3. Look for removed APIs. Every minor version removes deprecated API groups. A manifest that still uses policy/v1beta1 is no longer valid after the upgrade. The Kubernetes deprecation guide lists them per version.
4. PodDisruptionBudgets and replicas. When a pool is upgraded, nodes get drained. A deployment with a single replica briefly disappears. Run two replicas, or pick a maintenance window where it doesn’t matter.
5. Volumes. A pod with a ReadWriteOnce volume can only run on one node. When the node is drained, the pod moves, and that takes as long as it takes to reattach the disk. Plan for it, don’t be surprised by it.

The sequence

Control plane first. Kapsule swaps it out in the background; the API is unreachable for a few seconds, running pods notice nothing. Then the pools, one after the other: new nodes with the new version come up, old ones are drained and removed. For a pool with three nodes that takes about ten minutes, for the whole cluster twenty to thirty.

What you see during that time: nodes with different kubelet versions, pods moving around, briefly one more node than usual. What you shouldn’t do: deploy or update an add-on in the meantime. One upgrade at a time.

After the upgrade

• Are all nodes on the target version? A node that got stuck while draining will otherwise stay behind.
• Are all pods running that were running before? A pod that depended on a deprecated API object won’t come back.
• Are the add-ons still current for the new version, or is an add-on upgrade due now?

After that, things are quiet until next quarter. That’s the real trick: one upgrade per quarter, small and boring, instead of one every two years that changes everything at once. Regular upgrades are also part of the evidence an auditor wants to see under NIS2; see Audit & NIS2 and NIS2 in Kubernetes operations.

Is there a way back?

Kubernetes upgrades are not reversible. There is no control plane downgrade. The way back is a new cluster on the old version and a migration of the workloads, which on Kapsule is doable with a second cluster in the same private network, but it’s not a push of a button. Hence the checklist, hence staging first.

An upgrade, step by step

Here’s what a quarterly upgrade of a small production cluster with two pools looks like, the way we run it regularly:

1. Monday, staging. Check add-ons, choose the target version, trigger the upgrade including pools. Wait twenty minutes, test the applications. Note anything unusual.
2. Wednesday, production, outside core hours. Any add-ons that had to be updated on staging get updated on production first. Then the Kubernetes upgrade including pools.
3. While it runs. Deploy nothing. The cluster page shows nodes on two versions, pods moving and the progress per pool.
4. Afterwards. All nodes on the target version, all pods running, add-ons still compatible with the new version. A note in the operations log, done.

The whole thing takes an hour of attention per quarter. An upgrade across four minor versions after two years takes a day, with surprises.

The three surprises we’ve run into

• A chart that excludes the new version. The ingress controller kept running after the upgrade but could no longer be updated via Helm, because its chart didn’t know the Kubernetes version. Since then: add-ons first.
• A pod with a volume that hung for minutes while draining. The disk had to be detached from the old node and attached to the new one. Not an error, just patience. It’s been on the checklist ever since.
• A node that wouldn’t drain. A PodDisruptionBudget with minAvailable: 1 on a deployment with a single replica. The budget can never be satisfied, so the node never empties. Either two replicas or no budget.

Automatic patches: yes, but

The maintenance window for automatic patch upgrades should be on. Patches fix security vulnerabilities and don’t change behavior. But put the window at a time when someone will look at the cluster the next morning, not the night before a public holiday.

How Clusterward supports this

The version card only shows target versions that Kapsule can reach from here. The upgrade requires you to type the cluster name, upgrades the pools along with it if you want, is monitored in the background until it finishes and reported as a notification. Add-ons show their compatibility with the target version beforehand, and add-on upgrades are locked while a Kubernetes upgrade is running. Details under Cluster updates.

Conclusion

A Kapsule upgrade is a small operation if add-ons and APIs are in order beforehand. Do it every quarter, on staging first, with pools upgraded along with it. The upgrade isn’t the scary part. The scary part is the upgrade that waited two years.

Preparing your next Kapsule upgrade? Send us your Kubernetes version and your add-ons. We’ll tell you what to check before the upgrade. Ask about your upgrade →

Sources and further reading

• Kubernetes: Release cycle
• Scaleway: How to upgrade the Kubernetes version of a Kapsule cluster
• Kubernetes: Deprecated API migration guide
• Helm documentation: Charts (kubeVersion field)