Data quality gating for warehouses
Your task
Design a data-quality gating system for a daily partitioned warehouse pipeline where bad upstream data (a source bug dumping nulls, a 10x row-count drop, a timezone shift, a duplicate load) currently silently corrupts a finance fact table that ~40 downstream dashboards and an ML feature pipeline depend on. The goal: a bad partition must be detected and BLOCKED from being published/consumed before it poisons downstream, while a healthy partition flows through automatically with no human in the loop. False positives that block good partitions are nearly as costly as missed bad ones.
It covers
- Quality gates
- Partition blocking
- Circuit breaker