Vendor calls block and timeout
Your task
A service guards calls to a flaky vendor with a Semaphore(50) to cap concurrency. Over hours it slowly degrades: throughput to the vendor falls steadily, and eventually every call blocks on semaphore.acquire() and times out, even though the vendor itself is responding fine and your traffic hasn't increased. A graph of available permits trends downward over time and never recovers, like a slow leak, flattening at 0. Restarting the pod fixes it instantly, then it slowly leaks again. A retry/timeout change shipped two days ago. Triage and find the leak.
It covers
- Resource leak
- Rate limiter
- Permit not released