Cloud platforms and lakehouse
Databricks Unity Catalog
Unity Catalog knows what happened inside Databricks. Solidatus takes that knowledge and puts it in context with everything upstream and downstream, so your lakehouse is part of the enterprise picture rather than a world of its own.
What it is
Metadata, lineage and job extraction from Databricks workspaces, driven by Unity Catalog.
What Solidatus captures
Catalogs as layers, schemas as objects, tables as groups and columns as attributes; column-level dataflow as transitions; and, optionally, Databricks jobs as objects in their own layer.
What you need
- Tables registered in a Unity Catalog metastore. External assets need to be registered as external metadata objects in Unity Catalog before their lineage can be captured.
- Workloads that use Spark DataFrame or Databricks SQL interfaces, since that is what Unity Catalog records lineage for.
- At least the BROWSE privilege on the parent catalog, and that catalog accessible from the workspace. Notebook, job, dashboard and pipeline lineage additionally needs the relevant object permissions, including CAN VIEW on Unity Catalog-enabled pipelines.
- Databricks Runtime 11.3 LTS or above for streaming lineage between Delta tables, and 13.3 LTS or above for column lineage on declarative pipeline workloads.
- A workspace host URL, plus either a personal access token (user or service principal) or OAuth client credentials for a service principal. OAuth is recommended for system-to-system use.
Good to know
Databricks runs as a Standalone job today, which keeps first setup simple. Unity Catalog is what supplies the lineage, so we run a short readiness check with you and then get your lakehouse into the enterprise model.
