Types of Data Lineage

What does column level lineage mean?

Column-level lineage is a form of lineage that goes to the level of detail of tracing the flow of data through your organization at the column level of a system – as opposed to only the table level. When teams look at an overview map and see an issue for example, they can drill right down to the column level, to find the root cause of issues including their transformations – such as a faulty calculation formula in a specific column.

Solidatus is built to provide column level, detailed lineage. As it’s our expertise, we’ve ensured that the most information possible can be extracted from different systems and visualised by the business to perform proper root cause analysis.

What is the difference between column level lineage and other lineage?

Column-level lineage is far more detailed than table level lineage, which only shows how data flows between different tables in an organization’s systems – not which specific columns in the tables are part of complex data flows. Without column-level lineage, crucial detail is missed, which can make the difference between a quick find and resolution of the issue versus a slow one – or not at all.

What is manual lineage?

Manual lineage is a form of mapping data that is done as it sounds, manually. Some organizations have attempted to map data lineage manually themselves in absence of a proper data lineage solution. This may have been because when they started, they thought their needs were simple – and it may have served initial requirements for a high-level overview of systems and how data flows between them. But as their needs and understanding matures, the business grows, acquisitions are done etc, this format becomes more challenging and limiting. In a manual form of lineage, all features have to be coded manually and developers have to add in ingestion abilities, visual impact analysis, business context and more – adding many hours of work and development time and costs. This kind of lineage is very expensive to maintain and manage, is hard to encourage collaboration in a growing organization, is not sustainable, and is subject to human error.

What is basic data lineage versus advanced data lineage? 

Basic lineage refers to the type of lineage that is often included in data catalog solutions and offers simple table-to-table level lineage and usually only enables you to see other systems near a system, as opposed to all systems and connections. This type of lineage supports some uses, but does not support detailed root cause analysis, strict compliance or provide insight into transformations across all systems.

Advanced data lineage ensures a full end-to-end view of how data flows across ALL systems, enables you to do proper root cause analysis and to drill down to see detailed transformations in fine detail at the column level. It also enables you to see detailed snapshots of the past and predict future outcomes. This helps companies reduce risk in their business and react quickly to issues in data, as well as to stay fully compliant. It also links the technical data lineage with how it is used in business use cases, providing full value to the business.