Etcd cluster writes failing
Your task
Your config service is backed by a 5-node etcd cluster across 3 AZs (2/2/1). At 04:00 all writes start failing with 'no leader' / context-deadline-exceeded, though reads still work. Dashboards: an AZ hosting 2 etcd nodes lost network at 03:58; the remaining cluster has 3 of 5 nodes; leader elections are flapping; raft proposals time out; clients that need to WRITE config are blocked. The 2-node AZ isn't coming back soon. How do you triage and mitigate?
It covers
- Partition tolerance
- Consensus
- Write availability