Async queue overflows to OOM
Your task
A streaming ingestion service consumes from Kafka and writes to a slower downstream store via an async, in-memory bounded-but-large work queue. During a traffic surge it slowly heads toward OOM: RSS climbs steadily over ~20 minutes, GC pauses lengthen, the in-flight async task queue grows into the hundreds of thousands, consumer lag rises, and eventually pods get OOM-killed and restart, losing in-flight work, then the cycle repeats. The downstream store is healthy but its write latency is ~3x the consume rate. No deploy; just sustained higher input. Explain the failure and how you stabilize it.
It covers
- Unbounded queue
- Memory growth
- No backpressure