Stream partition rebalancing
Your task
Design the partition-assignment and rebalancing protocol for a large consumer group on a partitioned event stream. 2,000 partitions, 200 consumer instances, autoscaling so consumers join/leave every few minutes. Problem: every join/leave currently triggers a 'stop the world' rebalance where all consumers pause and reshuffle, causing latency spikes and reprocessing. Make rebalances cheap and minimize partition movement while keeping assignment balanced.
It covers
- Consumer rebalancing
- Stop the world
- Incremental cooperative