Question
Design a reverse proxy that sits in front of a fleet of slow upstream services (some upstreams have p99 latency of 3s). You're terminating 100k client connections but the upstreams can only handle a few thousand concurrent in-flight requests. Naively mapping each client connection to an upstream connection exhausts the upstreams. Design the connection model: how you pool/multiplex upstream connections, how you apply backpressure when upstreams are slow, how you avoid head-of-line blocking, and how a slow upstream is prevented from consuming all proxy resources.
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts — data model, bottlenecks, consistency, failure modes — and name the trade-offs you are making.