Stale leader processing after GC pause

On-call incidentHard45 minStripe style

Your task

Your job scheduler uses a lock in ZooKeeper to elect a single 'leader' worker that processes a queue. At 11:50 you see duplicate side effects: some jobs run twice (double-charged invoices). Dashboards: a worker (worker-7) had a 12s JVM GC pause at 11:48; during that pause ZooKeeper expired its session and a new leader (worker-3) was elected and started processing; then worker-7 resumed and kept processing jobs as if it were still leader for ~9s before noticing. The DB has no fencing on writes. How do you triage and mitigate?

It covers

  • Fencing token
  • Lease
  • Zombie write