On-callMediumoc-g415

Subject Resource exhaustionLevel Senior–Staff~35 minCommon in Concurrency · Algorithms & data structures interviewsIndustries Technology, Software development

Question

An async image-processing service uses a bounded in-memory work queue (capacity 1000) feeding a fixed pool of workers. At 17:00 the queue saturates at 1000/1000 and producers start getting rejected (HTTP 429 'queue full'); end-to-end latency for accepted jobs balloons. Worker CPU is only ~30% and there are free CPU cores. A thread/goroutine dump shows most workers blocked waiting on a downstream thumbnail-storage write that got slow after a storage-tier change at 16:50. Ingest rate is flat — no spike. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.