Your warehouse knows what happened inside it
Where Databricks Unity Catalog lineage stops, and the two questions it leaves open
A model validator picks one exposure figure from your first-quarter capital report and asks for its data lineage as it stood on March 31. The figure’s path runs through an Oracle stored procedure, an Informatica mapping, and a customer master on the mainframe, and your migration plan moves all three onto Databricks over the next three phases. What happens when that request arrives before the second phase of the migration plan?
Whoever runs the review will ask two things of that lineage. Show me the path from this figure back to the systems it came from, and then show me that path as it stood on the date the report was filed. Unity Catalog captures data lineage for every workload that runs on Databricks and answers the first well for everything there. However, outside the platform, it answers neither, and you can confirm that on your own data estate before the review.
Any query that runs through the Spark DataFrame or Databricks SQL interfaces leaves table-level lineage behind it, and column-level lineage wherever the platform can derive it, with no agent to deploy and no scanner to schedule.1 Your team does nothing to make that happen, and it works. Lineage shown in Catalog Explorer “is retained indefinitely,” with everything captured after September 1, 2024, available, and only the lineage system tables keep a rolling one-year window.2 A vendor who told you last year that Databricks forgets lineage after twelve months was describing an older product.
Two other things you may have heard about Unity Catalog are also out of date: that its lineage cannot leave the lakehouse, and that you can’t author it manually. The external lineage feature lets you register upstream sources such as Salesforce or MySQL and downstream tools such as Tableau or Power BI as external assets, so they appear “alongside your Unity Catalog tables in a single graph.”3 For an architect planning a migration, two narrower questions remain: what the warehouse knows about the systems outside it, and what it can tell you about its own past.
The documentation sets the condition: queries must use the Spark DataFrame or Databricks SQL interfaces for lineage to be captured.4 Unity Catalog records only what executes on its own compute, so a transformation that runs anywhere else, in an Oracle stored procedure, an Informatica workflow, or a COBOL batch job, produces no lineage event on Databricks, because nothing on Databricks ran it.
For everything beyond Databricks, external lineage lets you describe the connection yourself. You create an external metadata object for the Tableau dashboard or the MySQL source, then configure a lineage relationship between it and a Unity Catalog table, manually through the Catalog Explorer interface, the External Lineage APIs, or the Python SDK.5 The one automatic path is a Lakeflow Connect ingestion pipeline, which records lineage from the source tables it copies to the destination tables it writes.6 Every other external connection is hand-authored.
The cost of that description depends on whether a platform derives lineage for you or only records what you tell it. Unity Catalog can hold the column-level lineage of an Informatica mapping once someone has worked out what the mapping does and told it. A ticket that reads “add the Oracle stored-procedure lineage to Unity Catalog” asks an engineer to open the procedure and trace each output column to its inputs, record what they find, and repeat the work whenever the procedure changes. A handful of feeds takes an afternoon, and the 45 source systems behind a global bank’s lending book take a program of work.
Furthest out sit the mainframe systems, for which Databricks documents no lineage capture at all. COBOL, JCL, and PL/I jobs leave no event, so the customer master on z/OS enters the lineage graph, if at all, as an external asset with hand-authored connections. The same limit applies to any lineage tool that captures only what runs on its own compute or only what it can scan. The trace stops at the first system the tool cannot observe, and in a bank those systems are usually the oldest and the most regulated.
Between March 31 and November, when the model validator asked for the exposure figure’s lineage, the second migration phase moved the customer turnover feed from Oracle onto Databricks, so the figure’s upstream path now runs through a different system. A platform engineer also renamed the staging schema during the cutover. The validator opens Catalog Explorer and sees a lineage graph that is accurate for today. For March, the documentation states the consequence: “Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns.”7 Part of the March path is gone, and the rest has been replaced by the current one.
Indefinite retention does not restore the March path, because retention and history are different things. The documentation describes how long lineage is kept and a time-range filter for viewing it, and says nothing about holding two states of the lineage graph or comparing them.8 A validator can narrow the current graph to a window but cannot put March 31 beside today and see which connections changed, which columns moved, and who approved the change. Delta Lake time travel does not help either. It versions the rows in a table so that you can query last quarter’s data, but nothing versions the lineage graph that explains where those rows came from. The same question reaches the AI context layer above the data, which cannot tell you what it said in March.
Governed tag propagation, in beta since October 12, turns the lineage question into an access-control question as well. Unity Catalog now uses lineage to apply a governed tag to derived tables and views whenever a supported operation runs, so that classification and attribute-based access control follow data downstream.9 A control that follows lineage reaches only as far as the lineage does, and a personal-data tag on a column that entered the platform from a mainframe extract has no upstream connection to inherit from.
The European Central Bank’s guide on risk data aggregation and risk reporting already expects banks to maintain “complete and up-to-date data lineages on data attribute level.”10 Neither “complete” nor “up-to-date” describes a lineage graph that stops where Databricks stops or overwrites its own past. An examiner who tests a report’s lineage at a past reporting date against BCBS 239, the Basel Committee on Banking Supervision’s principles for risk data aggregation, will notice both. Any lineage layer that meets the expectation across a mixed data estate has to do four things:
Solidatus is a data lineage platform built to do those four things alongside Unity Catalog. Its connectors parse transformation logic in Oracle, SAP, and Informatica, among dozens of other technologies, and a mainframe connector reads the COBOL, JCL, and PL/I that never reach Databricks compute. A Databricks Unity Catalog connector pulls the catalog structure down to individual columns, along with column-level dataflow and jobs, into the same lineage model. That model keeps both the date a change became true and the date it was recorded so that a validator can compare March 31 with today in a single view. Every change, including one drafted by the AI Lineage Assistant, goes through forks, diffs, and pull requests before it reaches production, the workflow that software teams already use. Unity Catalog continues to govern what runs on Databricks, and the lineage platform answers the two questions for systems outside it.
You can test your own data environment against the two questions, path to source and path as of the reporting date, before a reviewer does:
The quickest way to see both questions answered in a real environment is a technical demo run against your own Databricks workspace and the systems around it.
[1]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
[2]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
[3]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage
[4]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
[5]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage
[6]Databricks. “External lineage.” Databricks Documentation, October 7, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/external-lineage
7Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
[8]Databricks. “Lineage in Unity Catalog.” Databricks Documentation, September 29, 2026.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
[9]Databricks. “October 2026: Governed tag propagation is in Beta.” Databricks Documentation, October 12, 2026.
https://docs.databricks.com/aws/en/release-notes/product/2026/october
[10]European Central Bank. “Guide on Effective Risk Data Aggregation and Risk Reporting.” ECB Banking Supervision, May 2024. https://www.bankingsupervision.europa.eu/ecb/pub/pdf/ssm.supervisory_guides240503_riskreporting.en.pdf
01.
Only when you describe it. Unity Catalog captures data lineage for queries that run through the Spark DataFrame or Databricks SQL interfaces. For systems outside the platform, its external lineage feature lets you register sources such as Salesforce or MySQL and tools such as Tableau or Power BI as external assets, then author the lineage relationships yourself through the Catalog Explorer interface, the APIs, or the Python SDK. Lakeflow Connect ingestion pipelines are the one case where that lineage is recorded for you. Databricks does not parse transformation logic inside external systems, and it does not document lineage capture for mainframe languages such as COBOL.
02.
Lineage shown in Catalog Explorer is retained indefinitely for everything captured after September 1, 2024, according to the Databricks documentation. The lineage system tables are different and keep a rolling one-year window. The older claim that Databricks forgets lineage after twelve months no longer applies to the Catalog Explorer view. Retention describes how long the current lineage graph is kept, and the documentation says nothing about holding earlier states of the graph or comparing two of them.
03.
Retention is how long a lineage platform keeps the lineage it has captured. History is whether the platform keeps every earlier state of the lineage graph so you can compare two dates. A graph that is retained indefinitely but updated in place shows only the current path, and the Databricks documentation states that lineage is not preserved for renamed catalogs, schemas, tables, views, or columns. A lineage model with history records both when a change became true and when it was recorded so that a validator can compare March 31 with today at the column level.
04.
No. Delta Lake time travel versions the rows in a table, so you can query a table as it was at an earlier point. It does not version the lineage graph that explains where those rows came from, which systems fed them, or which transformations ran. To show a figure’s lineage as it stood on a past reporting date, a model validator needs a lineage model with its own version history. A snapshot of the data does not supply it.
05.
Yes. Solidatus is a data lineage platform that runs alongside Unity Catalog. A Databricks Unity Catalog connector pulls the catalog structure, column-level dataflow, and jobs into the Solidatus lineage model, where they sit beside lineage that Solidatus derives from Oracle, SAP, Informatica, and mainframe sources that never run on Databricks compute. Solidatus keeps every version of that model so that you can compare any two dates in one view, and changes go through forks, diffs, and pull requests before reaching production. Unity Catalog keeps governing what runs on Databricks.
06.
Pick one figure from a regulatory report and trace its data lineage upstream from the Databricks table that feeds it until the trace ends or becomes a hand-authored external connection. That system is the first place to add an independent lineage layer. Then ask for the lineage of the same figure as it stood on your last reporting date, beside today’s, at the column level in a single view. An answer that requires reconstruction from system tables and change tickets shows what a supervisory review will cost.
Published on: November 17, 2026