Etcd cluster writes failing

On-call incidentHard45 min

Your task

Your config service is backed by a 5-node etcd cluster across 3 AZs (2/2/1). At 04:00 all writes start failing with 'no leader' / context-deadline-exceeded, though reads still work. Dashboards: an AZ hosting 2 etcd nodes lost network at 03:58; the remaining cluster has 3 of 5 nodes; leader elections are flapping; raft proposals time out; clients that need to WRITE config are blocked. The 2-node AZ isn't coming back soon. How do you triage and mitigate?

It covers

  • Partition tolerance
  • Consensus
  • Write availability