Clusterward
← Back to the blog
OperationsFlorian Apel

Zero-downtime deployments on Kubernetes: readiness, rolling updates and rollback

The rollout was green, yet users saw 502 errors for 20 seconds. What a rolling update alone does not give you: readiness, a clean shutdown, disruption budgets, migrations in two steps and a rollback to a pinned version.

Cover image: Zero-downtime deployments on Kubernetes: readiness, rolling updates and rollback

Friday, 2 pm, a release of the order API with two replicas. Kubernetes reports success, yet monitoring shows 20 seconds of 502 errors at the ingress. Two causes almost every team hits once: the new pod got traffic before the application had loaded its configuration, and the old one was stopped with requests still in flight.

A zero-downtime deployment is a rollout in which every request during the whole switch is answered by a version that is ready – no dropped connections, no error pages. Kubernetes provides the building blocks; not all come set up right.

What does a rolling update do – and what not?

RollingUpdate is a Deployment’s default strategy. maxSurge sets how many extra pods may be created, maxUnavailable how many may be missing; both default to 25%, the first rounded up, the second down. With one replica that means one extra pod, none missing – the old one goes once the new one is ready. With four, one may be missing; to prevent that, set maxUnavailable: 0.

“Ready” means the readiness probe succeeded. Without one, a container counts as ready once it runs – exactly when the first 502 arrives.

Readiness, liveness, startup: which probe for what?

  • Readiness decides about traffic. If it fails, Kubernetes removes the pod from its Services’ EndpointSlices; it keeps running but gets no requests.
  • Liveness decides about restarts, for instance after a deadlock. It does not wait for readiness, so without a delay it restarts a slow starter in a loop.
  • Startup protects slow starters: until it succeeds, the other two do not run.

A liveness probe that checks the database restarts every pod at once when the database stalls – the Kubernetes docs explicitly warn about such cascades. And a TCP check only shows that the port is open; an HTTP check on /ready knows whether caches and connections are in place.

How do I stop a pod without losing requests?

On shutdown the kubelet sends SIGTERM to the main process, waits for terminationGracePeriodSeconds – 30 seconds by default – and then sends SIGKILL. At the same time, not before, the control plane removes the pod from the EndpointSlices; ingress controllers learn of it late and briefly keep sending requests to a process already shutting down.

  1. A preStop hook pausing a few seconds bridges that, since Kubernetes 1.30 without a shell in the image: preStop: {sleep: {seconds: 5}}.
  2. After SIGTERM the application accepts no new connections, finishes requests in flight and exits within the grace period.
  3. The process must receive the signal: the shell form of ENTRYPOINT starts it under /bin/sh -c, which does not pass signals on. The exec form avoids that.

Node drains: what is a PodDisruptionBudget for?

Node upgrades and shrinking a pool empty nodes; tools such as kubectl drain use the Eviction API for that. A PodDisruptionBudget defines how many of an application’s pods may be missing at once – it does not limit rolling updates.

  • From two replicas: maxUnavailable: 1 – the drain moves one pod at a time.
  • Never a budget that allows no disruption: the drain then never completes.
  • With one replica no budget helps; if you want no gap, run two.

Database migrations: expand first, clean up later

During a rolling update, old and new version share one database. If the schema only fits the new one, no probe helps. The remedy is expand/contract, or parallel change:

  1. Expand: add the new column or table, nullable or with a default – the old version notices nothing.
  2. Migrate: roll out code that writes both and reads the new one; backfill old rows.
  3. Contract: remove the old part only in the next release.

A rename thus becomes three releases, each reversible on its own.

When a short pause is unavoidable

A ReadWriteOnce volume is attached to exactly one node. If the new pod lands on another one, it cannot mount the volume while the old pod holds it – the rollout stalls. Such services use Recreate: old pods first, then the new one. The pause lasts until the volume has moved and the application has started; plan for it. More in Persistent volumes on Kubernetes: ReadWriteOnce, Recreate and the snapshot before deletion.

Rollback – and when canary or blue-green pay off

kubectl rollout undo goes back to the previous ReplicaSet. If the template points at a tag that now means another image, the old version does not return. A rollback only deserves the name when it points at a digest – more in Rollbacks on Kubernetes: why a tag is not a version.

Canary and blue-green go further: a small share of traffic goes to the new version, or a second environment waits for the switch – for example with Argo Rollouts, Flagger or weighted routes in the Gateway API. That pays off when 5% of your traffic produces meaningful error rates within minutes and metrics decide automatically. For most teams without a platform team, clean probes, shutdown and migrations bring more.

What Clusterward takes care of – and what not

Clusterward rolls out git and container services as Deployments with Kubernetes’ default rolling update. Every container gets a readiness probe as a TCP check on its port (first check after 5 seconds, then every 10, not ready after three failures). A deployment is healthy only once every instance of the new version accepts connections and the old ones are gone; if the new version does not get there within the readiness timeout – 120 seconds by default, adjustable per service up to 3,600 – it counts as failed, and the message says the rollout continues in the cluster. Services with a volume run one instance with Recreate (Volumes & snapshots). Rollouts from the Scaleway registry are pinned to the digest; “Restore this version…” in the deployment history brings exactly that image back (Deployments).

What it does not offer: canary or blue-green, configurable liveness or startup probes, an HTTP path for readiness, a preStop hook or a longer grace period, and PodDisruptionBudgets for git and container services. Your application must handle SIGTERM itself. A Helm chart service renders its own objects and can bring its own probes and budget.

Conclusion

Rolling out without an outage is mostly a matter of details: a readiness probe that really means ready, a shutdown that finishes requests, a budget for drains and migrations that tolerate two versions. Where a volume is attached to one node, a short pause remains – the honest answer is to plan for it.

Releases without 502s? Tell us how your application starts and shuts down today. We’ll tell you where the gap in your rollout is. Ask a question →

Sources and further reading

Frequently asked questions

  • A rollout in which every request during the whole switch is answered by a version that is ready. It needs a rolling update, a readiness probe that checks real readiness, a clean shutdown of old pods with SIGTERM and a short pause, and database migrations that tolerate the old and the new version at once.