Network policies on Kubernetes: truly separating environments and customers

A penetration test at an agency running three customers’ apps on one Kapsule cluster, each with staging and production in its own namespaces. The tester opens a shell in customer A’s staging pod and types curl http://api.kunde-b-prod.svc.cluster.local:8080/internal/stats. It answers at once: an internal endpoint without a login, built on the assumption that only the inside could reach it. Nobody had misconfigured anything – this is Kubernetes out of the box.

A Kubernetes network policy is an object that defines which network traffic may reach and leave the pods of a namespace – and it only works if the cluster’s network plugin enforces it. Below: a two-policy baseline that separates environments and customers, and where it breaks in practice.

Why don’t namespaces separate network traffic?

Namespaces separate names, permissions and quotas. To the network they mean nothing: according to the Kubernetes documentation, a pod without a policy is isolated in neither direction – all connections are allowed. Cluster DNS resolves every service in every namespace, and every pod reaches every other pod by its IP.

Convenient for one team with one app. For an agency with several customers, or a SaaS with one namespace per tenant, a hole in one app opens the way to all the others. Separating tenants by namespace, as in the article Multi-tenant: namespace or row?, needs network policies for the separation to hold on the network too.

How does a Kubernetes network policy work?

Four documented rules explain almost every surprise:

• Selection isolates. A pod becomes isolated for a direction as soon as any policy selects it and lists that direction in policyTypes. An empty podSelector: {} selects every pod in the namespace.
• Policies add up. There are no deny rules and no order: what is allowed is the union of what all applicable policies allow.
• Replies are implicitly allowed. Rules only describe who may start a connection.
• No enforcement, no effect. Without a network plugin that implements it, a policy is just an object in the database. Kapsule clusters run Cilium by default (the Scaleway CLI’s default for cni), and Cilium enforces policies.

One exception: traffic to and from the pod’s own node is always allowed, so the kubelet’s probes still reach pods in an isolated namespace.

What does a sensible baseline per environment look like?

Per environment – one namespace with an app’s services – two policies selecting every pod in it are enough:

1. Ingress: only ingress controllers and neighbours. Allowed: the installed ingress controllers’ namespaces, selected via the automatic label kubernetes.io/metadata.name, and pods in the same namespace, so the app still reaches its worker or cache. Everything else is refused.
2. Egress: everything except the metadata. First an explicit rule for cluster DNS (pods labelled k8s-app: kube-dns in kube-system, UDP and TCP 53), then 0.0.0.0/0 except 169.254.0.0/16. The node’s metadata service lives in this link-local range, on Scaleway at 169.254.42.42.

Why not block all egress? Almost every app talks to the outside: the managed database on the private network, object storage, payment and mail services. A default deny suits an environment whose destinations you know completely; as a baseline for existing apps it breaks more than it protects. Blocking the metadata range costs no app anything and takes a favourite target away from server-side request forgery (SSRF).

Why does Cilium need its own DNS rule?

Because an ipBlock with 0.0.0.0/0 is not what it looks like in Cilium. Cilium’s documentation is clear: CIDR rules do not apply to traffic where both sides are managed by Cilium, and ipBlock rules do not match pod or node IPs inside the cluster by default. CoreDNS is a pod inside the cluster – so it falls through.

The symptom is treacherous: connections to IP addresses work, names no longer resolve. Node.js reports EAI_AGAIN, and the pod status stays green. The DNS rule therefore belongs at the top of every egress policy.

Which pitfalls should you expect?

• A hard-coded controller namespace. Run Envoy Gateway next to ingress-nginx to move hosts gradually and you have two controllers in two namespaces. If the policy only names ingress-nginx, the Envoy proxies are refused and every moved host returns an error instead of the app.
• Monitoring that scrapes from outside. A Prometheus in the monitoring namespace scraping /metrics is refused after isolation; its namespace has to be admitted explicitly.
• Calls across namespaces. A frontend calling another namespace’s API through cluster DNS no longer reaches it. Put both in one environment, or route the call through the public host.
• Jobs with needs of their own. Migrations, backups to S3 and webhooks need egress – the baseline keeps it open. Tighten egress later, and check every CronJob.
• Policies that go stale quietly. A controller is added or removed, the policy stays. Nobody notices until a host moves – the list of admitted namespaces has to follow the cluster, not a wiki page.

How does Clusterward isolate environments?

In Clusterward, isolation is a switch per environment: exactly the two policies above (bk-baseline-ingress and bk-baseline-egress), written idempotently into the namespace. Admitted are the namespaces of the cluster’s healthy ingress controllers – nginx, Envoy Gateway or Traefik. Before isolating, Clusterward re-reads which controllers are really installed; when one is added or removed later, it realigns the ingress policy of every isolated environment on the cluster and records that in the audit log. A realignment never puts back an egress policy someone removed.

Isolation is only offered where it cannot cut an app off: on a Kapsule cluster with stored access and at least one installed ingress controller – otherwise the switch does not appear. The state is stored and checked daily: each environment carries the chip “Isolated” or “Open to the cluster”, and Settings → Security lists open environments first. New environments can be isolated at creation, effective after their first successful rollout; in tenant pipelines every new environment block preselects it, so each tenant of that pipeline starts isolated. More under Security & access and Network & ingress.

Conclusion

Namespaces organise, network policies separate. A baseline of two policies – ingress only from ingress controllers and neighbours, egress to everything but the metadata range, DNS first – keeps environments and customers apart without breaking running apps. The hard part is not the YAML but keeping the controller list current.

Separating your environments properly? Tell us how many customers and environments share your cluster. We’ll show you what isolation would look like. Ask a question →

Sources and further reading

• Kubernetes docs: Network Policies
• Cilium docs: Layer 3 policies
• Scaleway CLI: scw k8s, default CNI
• Scaleway: Metadata API and user data