Design a job orchestration system

DesignMedium40 minDatabricks style

Your task

A company runs ~2000 interdependent data jobs as a DAG: a late or failed upstream job cascades into hundreds of downstream tables missing their morning SLA, and on-call wastes hours figuring out which root failure caused the storm of downstream alerts. Jobs have different priorities (a few are board-report-critical, most are not). Design an orchestration system that models dependencies, handles failures and retries sensibly, prioritizes critical paths, and gives clear SLA/root-cause visibility instead of alert storms.

It covers

  • Dag orchestration
  • Dependencies
  • Sla misses