Data quality gating for warehouses

DesignHard40 minDatabricks and Snowflake style

Your task

Design a data-quality gating system for a daily partitioned warehouse pipeline where bad upstream data (a source bug dumping nulls, a 10x row-count drop, a timezone shift, a duplicate load) currently silently corrupts a finance fact table that ~40 downstream dashboards and an ML feature pipeline depend on. The goal: a bad partition must be detected and BLOCKED from being published/consumed before it poisons downstream, while a healthy partition flows through automatically with no human in the loop. False positives that block good partitions are nearly as costly as missed bad ones.

It covers

  • Quality gates
  • Partition blocking
  • Circuit breaker