Flapping partition leadership
Your task
Your Kafka-style streaming cluster (and its coordination layer) sits across three AZs. Starting 02:15 you get intermittent connectivity loss between one AZ and the other two, packets drop for 10–30 seconds, recover, then drop again (a flapping partition, not a clean cut). Dashboards: leader elections are firing repeatedly; partition leadership keeps moving; consumer lag oscillates wildly; producer error rate spikes during each flap. CPU/disk are fine. Recent context: a cloud-provider networking incident is in progress for that AZ. How do you triage and mitigate?
It covers
- Az isolation
- Leader election
- Flapping