What is Data Lineage? Request a Demo
Created with Fabric.js 5.2.4

Your warehouse knows what happened inside it

Where Databricks Unity Catalog lineage stops, and the two questions it leaves open

A model validator picks one exposure figure from your first-quarter capital report and asks for its data lineage as it stood on March 31. The figure’s path runs through an Oracle stored procedure, an Informatica mapping, and a customer master on the mainframe, and your migration plan moves all three onto Databricks over the next three phases. What happens when that request arrives before the second phase of the migration plan?

Whoever runs the review will ask two things of that lineage. Show me the path from this figure back to the systems it came from, and then show me that path as it stood on the date the report was filed. Unity Catalog captures data lineage for every workload that runs on Databricks and answers the first well for everything there. However, outside the platform, it answers neither, and you can confirm that on your own data estate before the review.

What Unity Catalog lineage does well

Any query that runs through the Spark DataFrame or Databricks SQL interfaces leaves table-level lineage behind it, and column-level lineage wherever the platform can derive it, with no agent to deploy and no scanner to schedule.1 Your team does nothing to make that happen, and it works. Lineage shown in Catalog Explorer “is retained indefinitely,” with everything captured after September 1, 2024, available, and only the lineage system tables keep a rolling one-year window.2 A vendor who told you last year that Databricks forgets lineage after twelve months was describing an older product.

Two other things you may have heard about Unity Catalog are also out of date: that its lineage cannot leave the lakehouse, and that you can’t author it manually. The external lineage feature lets you register upstream sources such as Salesforce or MySQL and downstream tools such as Tableau or Power BI as external assets, so they appear “alongside your Unity Catalog tables in a single graph.”3 For an architect planning a migration, two narrower questions remain: what the warehouse knows about the systems outside it, and what it can tell you about its own past.

Question one: show me the path back to the source

The documentation sets the condition: queries must use the Spark DataFrame or Databricks SQL interfaces for lineage to be captured.4 Unity Catalog records only what executes on its own compute, so a transformation that runs anywhere else, in an Oracle stored procedure, an Informatica workflow, or a COBOL batch job, produces no lineage event on Databricks, because nothing on Databricks ran it.

For everything beyond Databricks, external lineage lets you describe the connection yourself. You create an external metadata object for the Tableau dashboard or the MySQL source, then configure a lineage relationship between it and a Unity Catalog table, manually through the Catalog Explorer interface, the External Lineage APIs, or the Python SDK.5 The one automatic path is a Lakeflow Connect ingestion pipeline, which records lineage from the source tables it copies to the destination tables it writes.6 Every other external connection is hand-authored.

The cost of that description depends on whether a platform derives lineage for you or only records what you tell it. Unity Catalog can hold the column-level lineage of an Informatica mapping once someone has worked out what the mapping does and told it. A ticket that reads “add the Oracle stored-procedure lineage to Unity Catalog” asks an engineer to open the procedure and trace each output column to its inputs, record what they find, and repeat the work whenever the procedure changes. A handful of feeds takes an afternoon, and the 45 source systems behind a global bank’s lending book take a program of work.

Furthest out sit the mainframe systems, for which Databricks documents no lineage capture at all. COBOL, JCL, and PL/I jobs leave no event, so the customer master on z/OS enters the lineage graph, if at all, as an external asset with hand-authored connections. The same limit applies to any lineage tool that captures only what runs on its own compute or only what it can scan. The trace stops at the first system the tool cannot observe, and in a bank those systems are usually the oldest and the most regulated.

Question two: show me that path as it stood on the reporting date

Between March 31 and November, when the model validator asked for the exposure figure’s lineage, the second migration phase moved the customer turnover feed from Oracle onto Databricks, so the figure’s upstream path now runs through a different system. A platform engineer also renamed the staging schema during the cutover. The validator opens Catalog Explorer and sees a lineage graph that is accurate for today. For March, the documentation states the consequence: “Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns.”7 Part of the March path is gone, and the rest has been replaced by the current one.

Indefinite retention does not restore the March path, because retention and history are different things. The documentation describes how long lineage is kept and a time-range filter for viewing it, and says nothing about holding two states of the lineage graph or comparing them.8 A validator can narrow the current graph to a window but cannot put March 31 beside today and see which connections changed, which columns moved, and who approved the change. Delta Lake time travel does not help either. It versions the rows in a table so that you can query last quarter’s data, but nothing versions the lineage graph that explains where those rows came from. The same question reaches the AI context layer above the data, which cannot tell you what it said in March.

Governed tag propagation, in beta since October 12, turns the lineage question into an access-control question as well. Unity Catalog now uses lineage to apply a governed tag to derived tables and views whenever a supported operation runs, so that classification and attribute-based access control follow data downstream.9 A control that follows lineage reaches only as far as the lineage does, and a personal-data tag on a column that entered the platform from a mainframe extract has no upstream connection to inherit from.

Lineage that answers both questions

The European Central Bank’s guide on risk data aggregation and risk reporting already expects banks to maintain “complete and up-to-date data lineages on data attribute level.”10 Neither “complete” nor “up-to-date” describes a lineage graph that stops where Databricks stops or overwrites its own past. An examiner who tests a report’s lineage at a past reporting date against BCBS 239, the Basel Committee on Banking Supervision’s principles for risk data aggregation, will notice both. Any lineage layer that meets the expectation across a mixed data estate has to do four things:

  1. Derive lineage from the source artifacts: Parse the stored procedure, the ETL mapping, and the COBOL copybook directly, with no one describing them first.
  2. Hold the Databricks lineage in the same model as everything else: The end state and the legacy environment should appear to a validator as one path.
  3. Keep every version of the lineage model: Any two dates should be comparable at the column level, in one view, without exporting system tables.
  4. Model a change before it runs: A team should be able to stage a migration phase in a sandboxed copy, see the downstream reports it touches, and have an accountable owner approve it before cutover.

Solidatus is a data lineage platform built to do those four things alongside Unity Catalog. Its connectors parse transformation logic in Oracle, SAP, and Informatica, among dozens of other technologies, and a mainframe connector reads the COBOL, JCL, and PL/I that never reach Databricks compute. A Databricks Unity Catalog connector pulls the catalog structure down to individual columns, along with column-level dataflow and jobs, into the same lineage model. That model keeps both the date a change became true and the date it was recorded so that a validator can compare March 31 with today in a single view. Every change, including one drafted by the AI Lineage Assistant, goes through forks, diffs, and pull requests before it reaches production, the workflow that software teams already use. Unity Catalog continues to govern what runs on Databricks, and the lineage platform answers the two questions for systems outside it.

Run the two questions on your own data estate

You can test your own data environment against the two questions, path to source and path as of the reporting date, before a reviewer does:

  1. Trace one critical data element until the lineage graph stops: Pick a figure from a regulatory report and follow its lineage upstream from the Databricks table that feeds it. Note the first system where the trace ends or turns into a hand-authored external connection, and ask who derived that connection and when it was last checked against the code. That system is the first place an independent lineage layer is worth adding, because the reviewer will ask about it.
  2. Ask for two dates in one view: Request the lineage of that element as it stood on your last reporting date, beside today’s, at the column level, in a single comparison rather than an export of system tables and change tickets. An answer that involves reconstruction tells you what the second question will cost.
  3. Model the next migration phase before it runs: Stage the next set of data flows scheduled to move onto Databricks in a sandboxed copy of the lineage model, renamed schemas and swapped sources included, so the downstream reports the phase touches are listed and approved before anything changes. The change ticket’s risk rating will not list them, as your riskiest change ticket says low risk.

The quickest way to see both questions answered in a real environment is a technical demo run against your own Databricks workspace and the systems around it.

[1]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage

[2]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage

[3]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage

[4]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage

[5]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage

[6]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage

7Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage

[8]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage

[9]Databricks. “October 2026: Governed tag propagation is in Beta.” Databricks Documentation, October 12, 2026.
https://docs.databricks.com/aws/en/release-notes/product/2026/october

[10]European Central Bank. “Guide on Effective Risk Data Aggregation and Risk Reporting.” ECB Banking Supervision, May 2024. https://www.bankingsupervision.europa.eu/ecb/pub/pdf/ssm.supervisory_guides240503_riskreporting.en.pdf

Frequently asked questions

01.

Does Databricks Unity Catalog capture data lineage outside Databricks?

Only when you describe it. Unity Catalog captures data lineage for queries that run through the Spark DataFrame or Databricks SQL interfaces. For systems outside the platform, its external lineage feature lets you register sources such as Salesforce or MySQL and tools such as Tableau or Power BI as external assets, then author the lineage relationships yourself through the Catalog Explorer interface, the APIs, or the Python SDK. Lakeflow Connect ingestion pipelines are the one case where that lineage is recorded for you. Databricks does not parse transformation logic inside external systems, and it does not document lineage capture for mainframe languages such as COBOL.

02.

How long does Unity Catalog keep data lineage?

Lineage shown in Catalog Explorer is retained indefinitely for everything captured after September 1, 2024, according to the Databricks documentation. The lineage system tables are different and keep a rolling one-year window. The older claim that Databricks forgets lineage after twelve months no longer applies to the Catalog Explorer view. Retention describes how long the current lineage graph is kept, and the documentation says nothing about holding earlier states of the graph or comparing two of them.

03.

What is the difference between lineage retention and lineage history?

Retention is how long a lineage platform keeps the lineage it has captured. History is whether the platform keeps every earlier state of the lineage graph so you can compare two dates. A graph that is retained indefinitely but updated in place shows only the current path, and the Databricks documentation states that lineage is not preserved for renamed catalogs, schemas, tables, views, or columns. A lineage model with history records both when a change became true and when it was recorded so that a validator can compare March 31 with today at the column level.

04.

Does Delta Lake time travel give you historical data lineage?

No. Delta Lake time travel versions the rows in a table, so you can query a table as it was at an earlier point. It does not version the lineage graph that explains where those rows came from, which systems fed them, or which transformations ran. To show a figure’s lineage as it stood on a past reporting date, a model validator needs a lineage model with its own version history. A snapshot of the data does not supply it.

05.

Can Solidatus and Databricks Unity Catalog work together?

Yes. Solidatus is a data lineage platform that runs alongside Unity Catalog. A Databricks Unity Catalog connector pulls the catalog structure, column-level dataflow, and jobs into the Solidatus lineage model, where they sit beside lineage that Solidatus derives from Oracle, SAP, Informatica, and mainframe sources that never run on Databricks compute. Solidatus keeps every version of that model so that you can compare any two dates in one view, and changes go through forks, diffs, and pull requests before reaching production. Unity Catalog keeps governing what runs on Databricks.

06.

Where should a data platform lead start?

Pick one figure from a regulatory report and trace its data lineage upstream from the Databricks table that feeds it until the trace ends or becomes a hand-authored external connection. That system is the first place to add an independent lineage layer. Then ask for the lineage of the same figure as it stood on your last reporting date, beside today’s, at the column level in a single view. An answer that requires reconstruction from system tables and change tickets shows what a supervisory review will cost.

Published on: November 17, 2026

Contents

Related articles

How Banks Can Meet Regulatory Compliance with Data Lineage
Blog

Basel IV runs on data that BCBS 239 already governs

Where risk data aggregation and capital calculation rules overlap

Solidatus Named in 2024 FT 1000 Ranking of Europe’s Fastest-growing Companies
Blog

Your AI context layer cannot tell you what it said in March

What your data lineage has to prove before an examiner asks about a past decision

Bringing Attribute Lineage to IBM z/OS Mainframes
Blog

Your mainframe is not going to emit OpenLineage events

What runtime capture reaches, and what you have to model yourself.

Data dashboards
Blog

A sovereign cloud does not make your data sovereign

Why a residency commitment is not evidence of custody.

From Complexity to Confidence -A Modern Data Strategy in Action webinar
Blog

Your BCBS 239 evidence expires the day you file it

Three criteria that make regulatory data lineage reusable

The EU AI Act deadline has moved, but data lineage can’t wait
Blog

Your risk reports run on an unassessed supply chain

How data lineage brings assessment discipline to financial services data flows

What to look for in a data lineage platform
Blog

What to look for in a data lineage platform

A buyer's guide to choosing the foundation of your AI stack.

Data Lineage is Finally Getting the Attention
Blog

Your riskiest change ticket says low risk

Pre-change impact analysis for data estates that feed AI

Solidatus Chosen as Microsoft Purview’s Data Lineage Integration Partner
Blog

Why banks can’t meet modern regulations without data lineage

What BCBS 239 and SR 26-2 now demand at the column level

Blog

Solidatus 2026.3 puts data lineage where AI agents can reach it

MCP support, Bring Your Own LLM, and an assistant that keeps its context

Solidatus Named in 2024 FT 1000 Ranking of Europe’s Fastest-growing Companies
Blog

Lineage tools forget what your AI model saw

Bi-temporal data lineage is the foundation of forensic AI investigation

blogImgShadowAI
Blog

Shadow AI is a sovereignty problem

Why an AI use policy cannot enforce itself.

Ai lineage company
Blog

Your Data Governance Team Already Built Your AI Governance Foundation

How LSEG turned data lineage into a strategic asset for AI trust

Proving data lineage to regulators
Blog

Proving data lineage to regulators

What regulators expect when they ask you to prove data lineage

Blog

Data lineage vs metadata management: The architecture behind AI governance

Why catalog-first tools fall short when regulators ask how your training data moved

Ai lineage platform
Blog

Model Risk Starts in the Data Supply Chain

When an AI model starts producing unexplainable results, the first instinct is to blame the model. Teams often rush to...

Solidatus Named in 2024 FT 1000
Blog

The Data Fairy is Dead

Five data lineage myths that the masterclass got right

Four AI governance questions your data catalog cannot answer
Blog

Four AI governance questions your data catalog cannot answer

What regulators are already asking about your AI, and what it takes to respond

Ai lineage company
Blog

The Engineering of Trust: Why Metadata isn’t enough for AI

Three questions your AI governance approach must answer

FT 1000 Ranking of Europe’s Fastest-growing Companies
Blog

The 48-Hour Test: Does Your AI Have Complete Data Lineage?

Three institutions receive the same question during Model Risk Management reviews: “Walk us through the complete data lineage for your...

Data lineage and AI
Blog

How data lineage prevents AI failures in financial services

Solidatus’ Tina Chace and fellow experts reveal why 90% of AI model failures trace back to upstream data changes

Blog

Why Data Lineage is Essential for AI: 7 Governance Challenges Solved by AI-Ready Lineage

AI-ready data lineage is a comprehensive, auditable record of how data flows through your organization, designed to support AI governance...

ai data lineage
Blog

Detailed Expectations Around End-to-End Data Lineage and BCBS 239 From the European Central Bank RDARR Guide

In May 2024, the ECB released its ‘Guide on effective risk data aggregation and risk reporting (RDARR)’...

Blog

Why is Advanced Data Lineage Fundamental for Financial Services Organizations?

Read why advanced data lineage is crucial for business success

Blog

Continuing Innovation in Advanced Data Lineage to Help Answer Business Questions

An update on some recent developments in our latest product releases

Blog

Unveiling the Path: Why Data Lineage is Crucial for Building Effective AI Products

Read more about data lineage and its business impact, including on AI, BCBS 239 and more

Blog

Solidatus & Microsoft Purview: Elevating Data Governance in the AI Era

Solidatus data lineage partners with Microsoft Purview to help enterprises trust their data

Solidatus Recognized in 2022 Gartner® Market Guide
Blog

Blasting Off: Why Proactive Data Governance is Propelling Innovation

Read our key takeaways from Gartner D&A Summit 2024

Solidatus Named in 2024 FT 1000 Ranking of Europe’s Fastest-growing Companies
Blog

Live Demo: Explore our All-new Interface

Video introducing our new interface and core features like Connected Catalog and Data Map

Blog

Visualize Snowflake Horizon and Enhance its Impact

Read about Solidatus and Snowflake Horizon's governance solution

What to look for in a data lineage platform
Blog

Advanced Data Lineage: The Cornerstone of Modern Data Governance

Explore the various aspects of data lineage and its crucial role in your organization.

Achieving Basel III Compliance: A 3-Step Action Plan
Blog

Achieving Basel III Compliance: A 3-Step Action Plan

Basel III is changing – are you prepared? Read 3 easy steps with Solidatus

Solidatus Named in 2024 FT 1000 Ranking of Europe’s Fastest-growing Companies
Blog

Building a Data Community

Read how we helped successfully launched the Houston Women in Data Chapter

Blog

Data Lineage for Better Planning

Exploring the parallels between urban planning and data planning projects

Blog

Data Distress: Data Leaders on Brink of Quitting Jobs

71% of senior data leaders in financial services polled are close to quitting their jobs

Blog

Supercharging your Snowflake Governance with Solidatus

Take a look at what's new in our partnership with Snowflake

Blog

The Value of Data: Reflections from Attending Gartner

VP Product, Tina Chace, reflects on the Gartner conference, covering data governance and AI

Blog

Douze Points for New Way to Visualize Eurovision Data

We’ve linked the Eurovision Song Contest to the realm of data governance and data lineage

Blog

Quick Answer: What is Active Metadata?

In the latest Gartner® research note, find out what active metadata is

Blog

The Amazing World of Active Metadata

The role of metadata, dynamic visualization and inference across metadata

Blog

5 Ways Great Metadata Connectors are Game-Changers

Automatic connectors are essential for efficiently mapping metadata but not all are created equal. We look at the most important...

Blog

5 Things we Can’t Wait to Do at the Gartner Summit, Orlando

We discuss injecting active metadata into your governance and 4 other things we’re looking forward to at the Gartner® Data...