Code RoomStale leader processing after GC pause
HardPrep Room Coding #2573

Stale leader processing after GC pause

On-callReliability & on-callSenior–Staff~45 min

Your job scheduler uses a lock in ZooKeeper to elect a single 'leader' worker that processes a queue. At 11:50 you see duplicate side effects: some jobs run twice (double-charged invoices). Dashboards: a worker (worker-7) had a 12s JVM GC pause at 11:48; during that pause ZooKeeper expired its session and a new leader (worker-3) was elected and started processing; then worker-7 resumed and kept processing jobs as if it were still leader for ~9s before noticing. The DB has no fencing on writes. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 45 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.