Worker threads blocked on decode
Your task
An image-processing pipeline (a pool of 64 worker threads pulling jobs off a queue) grinds to a halt: the queue backlog grows by tens of thousands, but completed-jobs-per-minute drops to near zero. Dashboards: all 64 workers show 'busy' but throughput is ~0; CPU is low. A thread sample shows most workers stuck inside the same library call decoding a particular video container with no timeout. A user started uploading a batch of malformed/huge files an hour ago. Triage and stabilize the pipeline.
It covers
- Poison message
- Stuck workers
- Queue backlog