Consumer group partition rebalancing

DesignHard45 min

Your task

Design the partition-assignment + rebalancing coordination for a consumer-group system processing a 2,000-partition event stream across an autoscaling pool of consumers (5–100 nodes). Each partition must be owned by exactly one consumer at a time, ownership must rebalance when nodes join/leave, and rebalancing must not cause a long global stop-the-world pause where no partitions are being processed. How do you assign partitions and minimize disruption during rebalances?

It covers

  • Partition assignment
  • Rebalancing
  • Consumer groups