Async queue overflows to OOM

On-call incidentHard40 min

Your task

A streaming ingestion service consumes from Kafka and writes to a slower downstream store via an async, in-memory bounded-but-large work queue. During a traffic surge it slowly heads toward OOM: RSS climbs steadily over ~20 minutes, GC pauses lengthen, the in-flight async task queue grows into the hundreds of thousands, consumer lag rises, and eventually pods get OOM-killed and restart, losing in-flight work, then the cycle repeats. The downstream store is healthy but its write latency is ~3x the consume rate. No deploy; just sustained higher input. Explain the failure and how you stabilize it.

It covers

  • Unbounded queue
  • Memory growth
  • No backpressure