Flapping partition leadership

On-call incidentHard45 min

Your task

Your Kafka-style streaming cluster (and its coordination layer) sits across three AZs. Starting 02:15 you get intermittent connectivity loss between one AZ and the other two, packets drop for 10–30 seconds, recover, then drop again (a flapping partition, not a clean cut). Dashboards: leader elections are firing repeatedly; partition leadership keeps moving; consumer lag oscillates wildly; producer error rate spikes during each flap. CPU/disk are fine. Recent context: a cloud-provider networking incident is in progress for that AZ. How do you triage and mitigate?

It covers

  • Az isolation
  • Leader election
  • Flapping