Data lineage impact analysis

DesignMedium35 minDatabricks style

Your task

A data platform has ~10k tables, ~4k dbt/Spark transformation jobs, and hundreds of BI dashboards and ML features, all interdependent. An engineer who changes or drops a column has no idea what breaks downstream, so changes ship and silently corrupt dashboards discovered days later, and on-call can't tell which upstream failure caused a given dashboard to go stale. Design a lineage system that gives pre-merge impact analysis ('what does this change break') and post-incident root-cause ('what upstream broke this dashboard').

It covers

  • Impact analysis
  • Blast radius
  • Column level