Question
Design a layer-4 (TCP/UDP) load balancer that fronts a stateful service with 1M concurrent long-lived TCP connections at 2M packets/sec, where a given connection must keep hitting the same backend for its whole lifetime even as the LB fleet and backend pool change. Targets: p99 added latency under 200us, near-even backend spread, and when an LB node is added/removed, existing connections must not be reset just because a different LB node now handles their packets. Walk through the packet path, the connection-affinity model, and the central trade-off.
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts — data model, bottlenecks, consistency, failure modes — and name the trade-offs you are making.