Feature pipeline quality gates

DesignHard45 minDatabricks style

Your task

An ML feature pipeline computes ~600 features daily that feed recommendation and ranking models in production. Twice this year a silent upstream change (a field went 100% null, a currency unit flipped) corrupted features, the model degraded for days, and revenue dropped before anyone noticed because nothing 'errored'. Design a data-quality and observability system that catches these silent corruptions automatically, distinguishes real anomalies from legitimate shifts (e.g. Black Friday traffic), and decides whether to block a bad feature from reaching the model.

It covers

  • Anomaly detection
  • Circuit breaker
  • Observability