Cron interval duplicate job runs
Your task
Customers complain they received the same marketing email 3–4 times this morning. Dashboards: the nightly 'send digest' job's run count for today shows 4 overlapping executions instead of 1; each run took longer than the cron interval (the job's p95 runtime crept past the schedule period over the last weeks as the audience grew); the email provider's send volume is ~4x baseline; the job has no locking. A scheduler config change last week shortened the interval from hourly to every 15 minutes for an unrelated job, and the change accidentally applied to this one too. How do you triage, stop the over-sending, and prevent it recurring?
It covers
- Cron overlap
- Distributed lock
- Non idempotent
- Emails