Sync-over-async thread starvation

On-call incidentHard40 minMicrosoft style

Your task

A C#/.NET async web service (ASP.NET Core) that normally handles 30k RPS on a small thread pool starts collapsing under a moderate load increase: throughput craters, latency climbs to seconds, and the runtime keeps slowly adding threads (you see the thread count creeping up over minutes and brief recoveries each time a thread is injected). CPU is only ~40%. A recent change added a call into an older library method that is synchronous and does network I/O, wrapped as SomeLegacyClient.GetData() and called with .Result / .GetAwaiter().GetResult() from inside an async handler. How do you triage and mitigate?

It covers

  • Thread pool exhaustion
  • Async
  • Blocking call