Skewed partition key starves consumers
Your task
Your event pipeline consumes a Kafka topic with 12 partitions via a consumer group. Volume tripled after onboarding a big customer, and consumer lag is now climbing without bound even though you scaled the consumer group from 12 to 40 instances. CPU on the consumers is low. You notice that only 12 consumers are ever doing work and the other 28 sit idle. The producers are keyed by customerId, and the new big customer is ~70% of all traffic. How do you triage and get the pipeline to keep up?
It covers
- Partition limit
- Kafka
- Throughput ceiling