Vendor calls block and timeout

On-call incidentMedium35 min

Your task

A service guards calls to a flaky vendor with a Semaphore(50) to cap concurrency. Over hours it slowly degrades: throughput to the vendor falls steadily, and eventually every call blocks on semaphore.acquire() and times out, even though the vendor itself is responding fine and your traffic hasn't increased. A graph of available permits trends downward over time and never recovers, like a slow leak, flattening at 0. Restarting the pod fixes it instantly, then it slowly leaks again. A retry/timeout change shipped two days ago. Triage and find the leak.

It covers

  • Resource leak
  • Rate limiter
  • Permit not released