Partition count limits consumer scaling
Your task
Your event-processing service consumes a Kafka topic with 8 partitions via a consumer group. After a customer onboarding tripled event volume, consumer lag grows steadily and processing falls behind SLA. You scaled the consumer deployment from 8 to 24 pods to catch up, but lag kept growing and 16 of the 24 pods sit idle consuming nothing. CPU on the 8 active pods is high. There was no code change. How do you triage and what's the durable fix?
It covers
- Consumer parallelism
- Partition cap
- Kafka