Midnight cron stampede

On-call incidentMedium30 min

Your task

Every day at exactly midnight UTC, your shared Postgres and internal config service get hammered for ~90 seconds: CPU spikes, latency for unrelated online traffic degrades, and you see a burst of identical queries. It turns out hundreds of services each run a 0 0 * * * cron that refreshes a cached config blob and recomputes daily aggregates at exactly 00:00:00. The fleet has grown, so the midnight spike now causes real customer-facing latency. How do you triage and flatten the spike?

It covers

  • Cron
  • Synchronized jobs
  • Batch