Retries hammer degraded node
Your task
Your clients use a client-side least-connections / least-loaded load balancer across a fleet of backend replicas, with automatic retries on failure. One backend replica develops a problem (a slow disk) and gets slow but not down, its requests pile up. Clients retry the slow requests; the least-loaded LB, seeing the slow node's in-flight requests as 'busy', is supposed to route AWAY from it, yet you observe the OPPOSITE in one failure mode: retries keep landing on the SAME degraded node and it gets hammered into total failure while healthy nodes sit underused. How do you triage, and what load-balancing subtlety is biting you?
It covers
- Retry amplification
- Load balancing
- Hot node