Distributed cron single-execution guarantee
Your task
Design a distributed cron/scheduler that runs ~5,000 recurring jobs (send daily digest emails, generate invoices, expire sessions) across a fleet of 30 worker nodes. Each scheduled job must run exactly once per scheduled tick, never zero times (missed) and never twice (duplicate invoice). Nodes can crash mid-run or be slow. How do you coordinate which node runs which job and guarantee single execution per tick?
It covers
- Job scheduling
- Exactly once
- Idempotency