Question

A Kafka consumer group that processes user events with at-least-once, in-order-per-partition semantics develops a strange consumer-lag pattern: total lag climbs steadily but only on a handful of partitions, while most partitions stay near zero. End-to-end event-processing latency for affected users spikes to minutes. Broker, network, and consumer CPU are all fine, and throughput on the healthy partitions is normal. A new event type was recently added whose handler occasionally makes a slow (~5s) external call. Events for a few high-volume 'whale' tenants all key to the same partitions. How do you triage and mitigate?