On-callHardoc-g204

Subject Stale leaderLevel Senior–Staff~45 minCommon in Reliability & on-call interviewsIndustries Technology, Software development

Question

Your job scheduler uses a lock in ZooKeeper to elect a single 'leader' worker that processes a queue. At 11:50 you see duplicate side effects: some jobs run twice (double-charged invoices). Dashboards: a worker (worker-7) had a 12s JVM GC pause at 11:48; during that pause ZooKeeper expired its session and a new leader (worker-3) was elected and started processing; then worker-7 resumed and kept processing jobs as if it were still leader for ~9s before noticing. The DB has no fencing on writes. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.