On-callHardoc-g218

Subject Distributed failuresLevel Senior–Staff~45 minCommon in Distributed systems interviewsIndustries Technology, Software development

Question

Your config service is backed by a 5-node etcd cluster across 3 AZs (2/2/1). At 04:00 all writes start failing with 'no leader' / context-deadline-exceeded, though reads still work. Dashboards: an AZ hosting 2 etcd nodes lost network at 03:58; the remaining cluster has 3 of 5 nodes; leader elections are flapping; raft proposals time out; clients that need to WRITE config are blocked. The 2-node AZ isn't coming back soon. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.