Kafka partition skew from hot customer

On-call incidentHard35 min

Your task

The clickstream Kafka topic has 24 partitions, keyed by customer_id, consumed by a 24-instance group (one partition each). Alert: end-to-end lag on the topic is 4M and growing, but only on one partition, partition 7's lag is 3.9M while the other 23 are near zero. The consumer instance owning partition 7 is at 95% CPU; the rest are at ~10%. Produce rate is normal in aggregate. Recent context: a single large enterprise customer (customer_id 90021) onboarded last week and now generates ~40% of all clickstream volume. Triage and mitigate this partition skew.

It covers

  • Kafka
  • Hot partition
  • Keying