What is Data Lineage?

How it works

Lineage is a directed graph: nodes are datasets (tables, files, topics, dashboards, ML features) and edges say “this was built from that”. It can be recorded at two levels:

• Table-level: fct_orders reads stg_orders and stg_customers.
• Column-level: fct_orders.segment comes from stg_customers.segment, which is useful for tracing PII and for exact impact analysis.

Lineage is collected in three main ways. Parsing code: dbt builds its graph from ref() and source(), and SQL parsers read FROM and JOIN clauses. Capturing at run time: Unity Catalog records lineage from the queries and jobs that actually run on Databricks, and OpenLineage events from Airflow, Spark or dbt are sent to a server such as Marquez. Declaring it: asset-based orchestrators such as Dagster model each table’s inputs explicitly.

The two everyday uses run in opposite directions. Walking downstream answers “what breaks if I change or drop this column?” Walking upstream answers “this dashboard number is wrong; which source or step caused it?”

Example

Table-level lineage parsed from each model’s SQL, then a downstream walk to find everything a change to raw.customers would affect (Python 3.11).

import re
from collections import defaultdict

# Each model and the SQL that builds it (like dbt models or warehouse views).
models = {
    "stg_orders":    "SELECT * FROM raw.orders",
    "stg_customers": "SELECT * FROM raw.customers",
    "fct_orders":    "SELECT o.*, c.segment FROM stg_orders o JOIN stg_customers c USING (customer_id)",
    "rpt_revenue":   "SELECT segment, SUM(amount) FROM fct_orders GROUP BY segment",
    "rpt_churn":     "SELECT * FROM stg_customers WHERE churned",
}

downstream = defaultdict(set)
for model, sql in models.items():
    for source in re.findall(r"(?:FROM|JOIN)\s+([\w.]+)", sql, re.I):
        print(f"{source:<14} -> {model}")
        downstream[source].add(model)

def impact(node):
    """Everything that would break or go stale if `node` changes."""
    hit, todo = set(), [node]
    while todo:
        for child in downstream[todo.pop()]:
            if child not in hit:
                hit.add(child); todo.append(child)
    return sorted(hit)

print("change raw.customers affects:", impact("raw.customers"))

raw.orders     -> stg_orders
raw.customers  -> stg_customers
stg_orders     -> fct_orders
stg_customers  -> fct_orders
fct_orders     -> rpt_revenue
stg_customers  -> rpt_churn
change raw.customers affects: ['fct_orders', 'rpt_churn', 'rpt_revenue', 'stg_customers']

Common mistakes

• Drawing lineage by hand in a wiki, where it is out of date within weeks.
• Capturing only table-level lineage when the question is about one sensitive or broken column.
• Losing lineage at tool boundaries, such as Python scripts or BI extracts that no parser sees.
• Collecting lineage but never using it before schema changes, when it is most valuable.