Compliance and Business Insight – Delivered by Data Lineage

From 29 – 30 September, A-Team Insight brought together the US data management community to explore the latest challenges, opportunities and data innovations facing sell side and buy side financial institutions.

On day two of the event, Solidatus Co-CEO, Philip Dutton, joined data experts Dennis Slaterry from EDMworks, Andrew Foster from Deutsche Bank, Olga Maydanchik from Voya Financial and Harpreet Singh from Luxoft on a panel titled: “The power of data lineage to deliver compliance and business insight”. A wide range of key challenges were covered from managing ever increasing data volumes to the mainstream adoption of cloud.

The panel started with each member stating their definitions of data lineage, with Philip adding an important point: the power of lineage transcends technical flows.

“As a panel, we’re trying to educate the community that lineage enables a lot more than just the technical pipes where the data flows – it brings in huge amounts of impact analysis and transformational change to an organisation. There is an additional layer of value which can be utilised when we start to think about lineage in a different way than just the purely technical sense.”

The term automation means many things to many people – Philip believed that its definitions in the market are misguided.

“Unfortunately, automation takes time, it takes understanding, and it takes a lot of effort. It really comes down to the ‘why’ – what’s the value that we’re generating? The state of automation that most people are looking to achieve is if a system changes, tell me about it. The problem in global investment banks and complex organisations is that if you’re hearing about a change after it’s happened in production, you’ve got a bigger problem in the organisation. We need to move away from this reactive approach to a more proactive lineage approach – we should be thinking about planning and execution, with the automation as the validation that what we’ve done is what we plan to do.”

The creation and retention of data has increased exponentially in recent years, with a growing remote workforce catalysing the production and addition of more cloud applications to companies’ data landscapes. Philip made the point that the journey to the cloud is like any other transformational change companies enact.

“You need to know where you’re starting from and where you want to end up – an organisation looks at their current state on-premise and plans to move these elements into the cloud, and typically, people don’t pick up their whole organisation and start moving all the data into the cloud at once; it’s this journey that takes time and part of that is designing a cloud migration strategy to ensure you don’t breach regulations like cross-border data sharing, or that you sufficiently structure the migration so you’re not moving huge volumes of data up to the cloud and then back down into on-prem, generating huge costs. Lineage allows us to simulate what that future state of an organisation looks like while remaining in the safety of a confined space.”

Before the session ended, the panel discussed the relevance on AI and Machine learning within metadata management – Philip observed that much of what the vendors are pushing in the market is overhyped mainly because we work in metadata, and not data. The record sets for metadata management are so shallow that you can never train an AI to really deal with it. When you’re working with the actual underlying data and trying to infer lineage based on that data, there’s a much better case for using an AI or an ML – it’s important to understand the distinction between the two.

Solidatus for the cloud – Migrate, Optimise and Transform

Businesses are turning to the cloud for reliability, scalability, flexibility and consistent processing – but, with great transformation comes complexity and risk. With Solidatus, prepare your migration strategy by identifying mission–critical data and eliminating redundant information – enrich this data with current regulations, service agreements and business intelligence allowing for in-depth impact analysis before each asset is moved into the cloud.