Multi-source order consolidation

DesignMedium40 minDatabricks and Snowflake style

Your task

Design a nightly ETL that consolidates orders, refunds, and shipping data from 6 source systems (3 Postgres OLTP DBs, a Stripe export, a CSV drop from a 3PL logistics partner, and a Salesforce API) into a unified fact_orders table powering finance dashboards. Volumes: ~5M orders/day total. Finance needs the table fresh by 6am and trusts it for board reporting, so a silently-wrong number is worse than a late table. Design the extraction, the transform/conform layer, and the quality gates that decide whether to publish.

It covers

  • Transformation
  • Data quality gates
  • Scheduling