Layer-4 load balancer

DesignHard45 minGoogle style

Your task

Design a layer-4 (TCP/UDP) load balancer that fronts a stateful service with 1M concurrent long-lived TCP connections at 2M packets/sec, where a given connection must keep hitting the same backend for its whole lifetime even as the LB fleet and backend pool change. Targets: p99 added latency under 200us, near-even backend spread, and when an LB node is added/removed, existing connections must not be reset just because a different LB node now handles their packets. Walk through the packet path, the connection-affinity model, and the central trade-off.

It covers

  • Maglev hashing
  • Connection stickiness
  • Direct server return