Asymmetric network partition in cluster

On-call incidentHard40 minAmazon style

Your task

Your stateful cluster spans AZ-a, AZ-b, AZ-c. At 02:20 you see a confusing 'gray' failure: AZ-b can still reach AZ-a and AZ-c (its outbound is fine and health checks it initiates pass), but AZ-a and AZ-c can no longer reach AZ-b, packets from them to AZ-b time out. Dashboards: AZ-b nodes look healthy from their own vantage and keep reporting themselves up; the cluster keeps a leader in AZ-b but followers in a/c can't replicate from it; clients pinned to AZ-b succeed while others see write stalls and timeouts. No deploy; cloud provider reports a network event in one AZ. How do you triage and mitigate?

It covers

  • Partial partition
  • Asymmetric
  • Gray failure