Code RoomOutlier detection cascades pod ejection
HardPrep Room Coding #2659

Outlier detection cascades pod ejection

On-callReliability & on-callSenior–Staff~45 min

Your mesh has Envoy outlier detection enabled: a backend host that returns enough 5xx is ejected from the load-balancing pool for a cooldown. At 18:00 one backend pod (of 12) for the catalog service gets briefly slow due to a GC pause and emits some 5xx. Envoy ejects it. But then a SECOND pod, now carrying extra load, also crosses the 5xx threshold and gets ejected; then a third. Within 4 minutes the catalog service's pool has collapsed from 12 healthy hosts to 3, the surviving 3 are overwhelmed, and catalog is effectively down. Dashboards: each individual pod looked momentarily unhealthy, but the underlying cause was transient. No deploy. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 45 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.