Why your app sees the wrong IP behind the load balancer
Every visitor comes from the same IP, the login limit locks out all customers at once, and the allowlist lets nobody in. That’s not an attack, that’s the load balancer.
The problem always shows up only in production. The application runs, the domain responds, the certificate is green. Then the login limit locks out everyone after the third customer, or the IP allowlist that should admit only the office doesn’t even admit the office. One look at the logs: every request comes from the same private address.
What happens
A Scaleway Load Balancer operates at layer 4. It accepts the visitor’s TCP connection and opens a new connection to the node running ingress-nginx. From nginx’s point of view, the sender is the load balancer. The visitor’s address is gone before the first HTTP header is read.
X-Forwarded-For doesn’t help here: that header is set by an HTTP proxy, and the load balancer isn’t one. It just passes bytes through.
The solution: PROXY protocol
The PROXY protocol is a small header the load balancer puts in front of the TCP connection: source address, source port, destination address, destination port. nginx reads it when use-proxy-protocol is enabled, and from then on knows the real address. Both sides have to play along:
- The load balancer has to send the protocol. On Scaleway, this is controlled by the annotation
service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2on the ingress controller’s Service; the Cloud Controller Manager then configures the load balancer accordingly. - The controller has to read it. For ingress-nginx, that’s
use-proxy-protocol: "true"in the ConfigMap; for Envoy Gateway, a ClientTrafficPolicy.
If one side is missing, the connection breaks: if the load balancer sends the header and nginx doesn’t expect it, nginx sees garbage instead of HTTP. If nginx expects the header and the load balancer doesn’t send it, nginx waits in vain. That’s why the switch is a reinstall of the controller with both values at once, and for a few seconds no host on the cluster responds.
Why this is more than a logging problem
Without real addresses, three things break that you’d normally take for granted:
- Rate limits. A per-address login limit sees one address for everyone. After the first failed attempts by any visitor, everyone is locked out.
- Allowlists. An access list for office addresses never sees the office, because everyone arrives as the load balancer.
- Audit and forensics. Who signed in when, and from where? The answer is always the same address. The evidence an auditor wants to see here is described in Audit & NIS2.
We ran into this live: a cluster with several customer applications and a control plane, all behind a load balancer without PROXY protocol. The control plane’s rate limits lumped all customers together. Since then, PROXY protocol is enabled by default in Clusterward for new clusters, and for existing ones the switch is a typed confirmation on the setup card; see Cluster provisioning.
And behind Cloudflare?
If you route hosts through the Cloudflare proxy, you have a second layer. The load balancer now sees Cloudflare’s edge address, and the visitor’s real address is in the CF-Connecting-IP header. So the PROXY protocol delivers the Cloudflare address, and the application has to read the header, but only if the connection really comes from a Cloudflare address. Otherwise anyone can forge the header and pose as any address they like.
The clean rule is: take the address from the PROXY protocol. If it’s one of Cloudflare’s published addresses, take CF-Connecting-IP instead. Otherwise ignore the header. The list of Cloudflare addresses rarely changes, but it does change; hard-code it and sooner or later you have wrong addresses. Clusterward reloads it daily and applies this one rule to login limits, allowlists and audit entries, as described under Security & access.
Checklist
- Do your application’s logs show a single address for all visitors? Then PROXY protocol is missing.
- Is it enabled on the load balancer but not in the controller, or the other way around? Then nothing responds at all.
- Do hosts run behind Cloudflare? Then the application needs the rule for
CF-Connecting-IP, including a check of the sender address. - Do you use per-host IP restrictions? They only work with the real address, and only for hosts that don’t go through Cloudflare.
How to check in five minutes
- Open one of your applications from outside and look at its access log. Does it show your public address, or a private one from the cluster network?
- Check the ingress controller’s Service: does it carry the PROXY protocol annotation for the Scaleway Load Balancer?
- Check the controller’s ConfigMap for
use-proxy-protocol. - If only one of the two is set, the host doesn’t respond at all. If neither is set, it responds, but with the wrong address.
The switch itself is a reinstall of the controller with both values, in one go. Plan for a few seconds of interruption for every host on the cluster, and don’t do it on a Friday afternoon.
Why X-Forwarded-For still gets set
Once nginx knows the real address, it sets X-Forwarded-For and X-Real-IP itself for the application behind it. So the application keeps reading the header, as it’s used to with any reverse proxy. The difference lies only in front of it: without PROXY protocol, the header contains the load balancer’s address; with PROXY protocol, the visitor’s. Frameworks that trust proxies need a setting like TRUST_PROXY or trusted proxies for this, otherwise they ignore the header out of caution.
A special case: health checks
The load balancer checks its backends regularly. These checks arrive without a PROXY protocol header because they don’t come from a visitor. But ingress-nginx now expects the header. On Scaleway this is solved because the health check goes to a separate port that the controller serves without PROXY protocol. If you put the health check on the traffic port, you’ll see backends marked as dead even though they’re running.
One more reason: lockouts after failed attempts
Recently, even more depends on the real address. Clusterward locks an address after too many failed sign-ins and reports a sign-in from a new address by email. Behind a load balancer without PROXY protocol, the lockout would hit all users at once, and every sign-in would appear to come from the same address. More under Security & access.
Conclusion
PROXY protocol is one line of configuration that you either set on day one or retrofit on your worst day. New clusters should have it from the start. Everything based on addresses depends on it.
Real client IP in your cluster? Tell us which load balancer and which ingress controller you run. We’ll tell you what it takes to get the real address. Ask about your setup →
Sources and further reading
Frequently asked questions
- Because a Scaleway Load Balancer operates at layer 4. It accepts the visitor’s TCP connection and opens a new connection to the node running ingress-nginx. From nginx’s point of view, the sender is therefore the load balancer, and the visitor’s address is gone before the first HTTP header is read.