Stream partition rebalancing

DesignHard45 minDatabricks and Uber style

Your task

Design the partition-assignment and rebalancing protocol for a large consumer group on a partitioned event stream. 2,000 partitions, 200 consumer instances, autoscaling so consumers join/leave every few minutes. Problem: every join/leave currently triggers a 'stop the world' rebalance where all consumers pause and reshuffle, causing latency spikes and reprocessing. Make rebalances cheap and minimize partition movement while keeping assignment balanced.

It covers

  • Consumer rebalancing
  • Stop the world
  • Incremental cooperative