Leader election for stream partitions
Your task
Design leader election for a sharded stream-processing system where each of 4,000 partitions must have exactly one active processor writing to a downstream sink. Processors run on autoscaled, preemptible VMs that can pause (GC, network partition) for tens of seconds. A second processor must never write the same partition's output concurrently, even if the 'old' leader is merely slow and not dead. How do you elect leaders and guarantee single-writer at the sink?
It covers
- Lease
- Fencing token
- Split brain