Work queue saturated slow downstream

On-call incidentMedium35 min

Your task

An async image-processing service uses a bounded in-memory work queue (capacity 1000) feeding a fixed pool of workers. At 17:00 the queue saturates at 1000/1000 and producers start getting rejected (HTTP 429 'queue full'); end-to-end latency for accepted jobs balloons. Worker CPU is only ~30% and there are free CPU cores. A thread/goroutine dump shows most workers blocked waiting on a downstream thumbnail-storage write that got slow after a storage-tier change at 16:50. Ingest rate is flat, no spike. Triage and mitigate.

It covers

  • Bounded queue
  • Backpressure
  • Semaphore