Code RoomConsumer lag on few partitions
HardPrep Room Coding #2620

Consumer lag on few partitions

On-callDistributed systemsSenior–Staff~35 min

A Kafka consumer group that processes user events with at-least-once, in-order-per-partition semantics develops a strange consumer-lag pattern: total lag climbs steadily but only on a handful of partitions, while most partitions stay near zero. End-to-end event-processing latency for affected users spikes to minutes. Broker, network, and consumer CPU are all fine, and throughput on the healthy partitions is normal. A new event type was recently added whose handler occasionally makes a slow (~5s) external call. Events for a few high-volume 'whale' tenants all key to the same partitions. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.