Data Lineage Explained

Data lineage is the documented path a piece of data takes from the system where it was created, through every copy, join, filter and calculation, to the report, model or file where someone uses it. It answers three questions about any number on a dashboard: where did it come from, what was done to it, and what else depends on it. It is how you prove a figure, find the step that broke it, judge the blast radius of a change, answer an auditor or a privacy request, and show what data an AI model was trained on. You can capture it by hand for a few reports, from metadata your tools already emit, or automatically with a catalog.

What data lineage is, with an example

Take a single number: Net Revenue on a monthly dashboard. Its lineage, written out, looks like this.

1. Source. Posted sales invoice lines in Dynamics 365 Business Central.
2. Extract. An Azure Data Factory pipeline copies new and changed rows every night into a staging schema in Azure SQL Database.
3. Transform. A dbt model renames columns and casts types; a second model removes credit memos and intercompany lines, converts amounts to USD at the month-end rate, and joins customer and date dimensions.
4. Semantic layer. A Power BI semantic model defines the measure Net Revenue as the sum of the converted amount, filtered by posting date.
5. Consumption. The finance dashboard, a scheduled PDF to the board, and a CSV exported for the bank.

When the dashboard shows a drop that finance does not recognize, lineage lets you walk backward one step at a time: did the export run, did the currency table update, did someone change the credit memo filter. Without it, the investigation starts with a meeting.

Three neighboring terms get mixed up with lineage. A data catalog is an inventory of data assets with descriptions and owners; lineage is one feature a catalog can hold. Provenance is the origin and custody of a dataset. A data flow diagram shows systems and interfaces, not individual fields.

Lineage can be recorded at system, table, column or row level; table and column level is where most of the value sits.

Business lineage vs technical lineage

The same path can be described for two audiences, and you need both. Business lineage is the sentence a controller or an auditor can read. Technical lineage is the trace an engineer needs to fix something.

Aspect	Business lineage	Technical lineage
Audience	Finance, operations, compliance, auditors	Data engineers, analysts, DBAs
Question answered	Which system and business rule does this metric come from?	Which tables, columns, jobs and transformations produce it?
Granularity	Metric and system level	Table and column level, sometimes job and run level
Typical form	A sentence in the metric dictionary, a simple diagram	A graph in a catalog, a dbt DAG, pipeline metadata
Example	Net Revenue is posted invoice value from Business Central, less credit memos and intercompany, converted at the month-end rate.	SalesInvoiceLine.Amount, pipeline pl_bc_daily, model fct_revenue, column amount_usd, measure Net Revenue

The practical move is to link them: every metric definition cites its source system and rule in plain language, and every technical trace is tagged with the metric it feeds. A catalog does this linking for you; a spreadsheet can do it for a dozen metrics.

Data lineage use cases

Lineage earns its keep in six situations. If none of them applies to you yet, a light version is enough.

• Trust and dispute resolution. Two reports disagree. Lineage shows they start from different tables or apply different filters, and the argument ends in minutes.
• Impact analysis before a change. The ERP team wants to rename a field or retire a table. Lineage lists every pipeline, model and report that will break, so the change is planned instead of discovered.
• Root cause for bad data. A data quality check fails on a fact table. Lineage tells you which upstream step introduced the problem, and whether the fix belongs in the source system, the pipeline or the model.
• Audits and compliance. Controls over financial reporting, HIPAA questions about where protected health information flows, privacy requests that require deleting a person from every copy, and SOC 2 evidence all depend on knowing which systems hold which data and how it moves.
• AI and machine learning. Models are only as defensible as their training data. Lineage records which datasets, versions and filters went into a model, which is what frameworks like the NIST AI RMF expect you to be able to document.
• Migrations and decommissioning. Before retiring a legacy database or moving to Snowflake, Databricks or Microsoft Fabric, lineage tells you what feeds what, so nothing silently stops updating after cutover.

Lineage is one of the first capabilities a data governance program delivers, because owners, definitions and lineage reinforce each other: the owner approves the definition, and the definition cites the lineage.

Data lineage tools

Most organizations already own a tool that captures part of their lineage. The gaps are between tools, which is what catalogs and open standards fill.

Tool	Lineage it captures	Best fit
Microsoft Purview	Automatic lineage from Azure Data Factory, Fabric pipelines and Power BI, scans of SQL Server and Azure SQL, manual lineage for everything else	Microsoft-centric estates
Power BI and Fabric lineage view	Workspace-level graph of sources, dataflows, semantic models and reports, with impact analysis	Anyone using Power BI; included with the service
dbt (dbt Core docs, dbt Cloud Explorer)	Model-level graph from ref() calls; column-level lineage in the cloud edition	SQL transformation teams on any warehouse
Databricks Unity Catalog	Automatic table- and column-level lineage for queries, notebooks and jobs run on Databricks	Databricks customers
Snowflake	Object dependencies and column lineage from access history and the catalog interface (edition-dependent; check current docs)	Snowflake customers
Informatica Cloud Data Governance and Catalog	Scanner-based lineage across ERPs, databases and ETL, including PowerCenter mappings	Large, mixed estates with legacy ETL
Collibra, Alation, Atlan	Connector-based automated lineage plus glossary, ownership and policy workflows	Mid-sized and larger governance programs
OpenLineage and Marquez	Open standard for lineage events, with emitters for Airflow, Spark, dbt and Flink; Marquez is the reference server	Open-source and multi-cloud stacks

The governance platforms differ most in connectors, workflow and price. Data Governance Tools Compared: Purview, Collibra, Alation, Atlan goes through the trade-offs if you are choosing one.

A low-cost way to start

You do not need a platform to get the first and largest share of the value. Start with the ten reports or metrics people argue about most and build a lineage register for them.

1. Write the business lineage sentence for each metric: source system, business rule, exclusions, currency or unit conventions.
2. Write the technical trace: source table and field, each pipeline or job, each model or stored procedure, the final table and column, the measure name. One row per hop.
3. Harvest what already exists. Power BI’s lineage view, dbt docs, the Azure Data Factory pipeline list, and SQL Server dependency views give you most hops for free.
4. Add two rules. No new pipeline or report goes live without a register entry, and every metric in the dictionary cites its source.
5. Graduate when the register hurts. When the register passes a few hundred rows or changes weekly, move to automatic capture: Purview on a Microsoft stack, Unity Catalog on Databricks, OpenLineage on open-source orchestration, or a governance catalog.

Metric: Net Revenue (finance dashboard)
Business lineage: Posted invoice value from Business Central, less credit memos
  and intercompany, converted to USD at the month-end rate.
Hop 1: Business Central -> Sales Invoice Line.Amount, Posting Date
Hop 2: ADF pipeline pl_bc_daily -> azsql.stg.sales_invoice_line (nightly)
Hop 3: dbt model fct_revenue -> amount_usd (filters: credit memo, intercompany)
Hop 4: Power BI semantic model Finance -> measure [Net Revenue]
Owner: Controller   Last verified: quarterly

A founder with a billing system and spreadsheets keeps a register of five metrics on one page. A 200-person company on Business Central, Salesforce and Power BI relies on the Power BI lineage view and dbt docs, with the register covering only the hops between them. A division inside a larger group usually inherits Purview or Collibra and contributes its assets and owners to it.

Where PS3G fits

PS3G sets up lineage as part of data governance services for organizations of every size: the metric dictionary, the lineage register, ownership, and the move to automatic capture in Purview, Unity Catalog or a governance catalog when it is worth it. Our data engineering team builds pipelines on Azure Data Factory, Fabric, dbt, Snowflake and Databricks so that lineage is emitted as a by-product of the build rather than documented afterward. A founder usually needs the register and a few rules; a larger company usually needs the catalog configured and connected to its sources. Either way, we start with the reports people argue about.

Related reading

• Data Governance Tools Compared: Purview, Collibra, Alation, Atlan
• Data Quality Management: Dimensions, Rules, Tools and Process
• What Is Data Governance? A Plain-English Guide for Growing Companies
• Data governance services

Frequently asked questions

What is data lineage in simple terms?

Data lineage is the record of where a piece of data came from, every step that changed it, and where it ends up. For a number on a report, it names the source system, the pipelines and transformations in between, and the final measure. It lets you trace a figure backward to prove it or to find the step that broke it.

What is the difference between data lineage and a data catalog?

A data catalog is an inventory of your data assets: tables, reports, files, their descriptions and owners. Data lineage is the map of how those assets connect and feed each other. Most catalogs include lineage as a feature, and lineage is far easier to maintain inside a catalog than in a separate document, but you can keep a simple lineage register without one.

Which tools capture data lineage automatically?

Microsoft Purview captures lineage from Azure Data Factory, Fabric and Power BI. Databricks Unity Catalog records table and column lineage for work run on Databricks. dbt derives model lineage from its own code, and Snowflake exposes dependencies and column lineage on higher editions. Collibra, Alation, Atlan and Informatica add connector-based lineage across many systems, and OpenLineage is an open standard for emitting it.

Do small companies need data lineage?

Small companies need lineage for the handful of numbers that drive decisions and for any data that is regulated. A one-page register covering five to ten metrics is usually enough, and it pays for itself the first time two reports disagree. You add automatic capture when the number of pipelines and reports makes the register hard to keep current.

What does data lineage have to do with AI?

An AI model inherits every flaw in its training data, so you need to know which datasets, versions and filters went into it. Lineage provides that record. It also supports impact analysis when a source changes, and it is the evidence that AI governance frameworks such as the NIST AI RMF expect when they ask you to document data provenance and quality.