Stale leader processing after GC pause
Your task
Your job scheduler uses a lock in ZooKeeper to elect a single 'leader' worker that processes a queue. At 11:50 you see duplicate side effects: some jobs run twice (double-charged invoices). Dashboards: a worker (worker-7) had a 12s JVM GC pause at 11:48; during that pause ZooKeeper expired its session and a new leader (worker-3) was elected and started processing; then worker-7 resumed and kept processing jobs as if it were still leader for ~9s before noticing. The DB has no fencing on writes. How do you triage and mitigate?
It covers
- Fencing token
- Lease
- Zombie write