Stream consumer OOMKilled on restart

On-call incidentHard40 minDatabricks and Snowflake style

Your task

A stream-processing consumer (Kafka to transform to write to a slow downstream warehouse) starts growing RSS and eventually gets OOMKilled, then on restart immediately reprocesses a backlog and OOMs again, a crash loop. Dashboards show consumer lag climbing for hours, an in-memory work queue between the fetch stage and the write stage growing without bound, and downstream warehouse write latency up 5x after the warehouse began a maintenance/compaction window. Consumer fetch rate is unchanged. Triage and fix.

It covers

  • Queue backpressure
  • Unbounded buffer
  • Memory pressure