Visualize Snowflake Horizon and enhance its impact

Blog banners snowflake

What is Snowflake Horizon?

Snowflake Horizon is Snowflake’s built-in governance solution that unifies compliance, security, privacy, interoperability, and access capabilities ​enabling​ customers to easily govern data, apps, and more across clouds, teams, partners, and customers. 

Find Solidatus on Snowflake Horizon here.

Solidatus and Snowflake Horizon

Solidatus supercharges Snowflake’s governance capabilities with active metadata management and advanced visualization. With an end-to-end blueprint of your data landscape, you can understand its usage, define a control strategy, and make major time and cost savings. 

  • Enhance data observability 
  • Boost governance and compliance 
  • Highlight use of restricted data 
  • Reduce risk and improve operating efficiency 
  • Fast-track Snowflake migrations
Untitled design 6

With Solidatus you can:

Bring all the components of your data fabric into one, unified view

Solidatus’ powerful automation creates data blueprints covering your entire data landscape, including the sources upstream of, within and downstream of your Snowflake instance. By doing this, you can gain a clear vantage point from which to view your data ecosystem, answer the questions that matter most, and carry out strategic change.

Simplify and streamline data asset discovery, mapping, and monitoring in both hybrid multi-cloud and on-premise environments.

  • Create dynamic blueprints of a Snowflake instance, illustrating databases, schemas, tables, and views, displaying column-level dependencies and lineage, including between Snowflake databases. 
  • Boost data governance by integrating Snowflake governance rules such as tags, row access and data masking policies into your data blueprint. 
  • Derive unique insights by analyzing Snowflake governance rule application to tables, views and columns. 

Enhance data security

  • Highlight and investigate active Snowflake row access policies on relevant tables, accessing ownership and logic details, and easily visualize dynamic data masking policies across Snowflake databases. 
  • Rapidly determine if the table you’re using has access restrictions and ensure proper protection for dependent views. 
  • Identify and govern sensitive data movement across your data landscape, both upstream and downstream from Snowflake. 
  • Ensure consistent application of security policies across your data supply chain and enhance your data security insight and capability. 

Leverage tailored business insights

  • Snowflake tags enhance tables and columns with predefined allowable code sets. Solidatus incorporates these allowable values into your data blueprint. 
  • Easily view all Snowflake tags and their application across Snowflake databases and downstream targets. 
  • Apply Solidatus rules and filters to your data blueprint to capture additional business insights and identify gaps, issues and exceptions. 

Collaborate across teams and automate your audit trail

Solidatus provides a complete audit trail of all changes, automatically maintained and always accessible, with powerful visualization to display all differences along a timeline. Coordinate, control and plan changes throughout your organization regardless of the type of system, the data in use, where it is or who owns it. 

Looking to the future

We’ll be enhancing the depth and breadth of Snowflake Horizon by​ enabling users to​: 

  • Ingest Snowflake roles and visualize access permissions to database objects to improve audit control and governance. 
  • Automatically categorize semantic and privacy information for most accessed tables – such as PII as Snowflake tags – and visualize ​them atop the​ir​ organization’s data blueprint through Snowflake Classification. 
  • Broaden the scope of our end-to-end data blueprint by directly ingesting Snowpipe data flow information. 
Joining forces orange and red HIGH RES 1 scaled

Solidatus partners with Artefact to enhance the value of data management practices and provide data-driven solutions to customers. 

London and Houston, December 7 2023. Artefact, a leading data analytics and AI services firm specializing in helping organizations deliver measurable value from data, is excited to join Solidatus. This new partnership will help simplify data lineage and data management processes for customers in the Financial Services sector.

Artefact is a vertically integrated consultancy with expertise across data management, AI solutions, data marketing, AI/ML, and data science and analytics, dedicated to Financial Services. Artefact are a connected independent global network, with a footprint in 20 offices across 16 countries, and they partner with 1000+ ambitious clients around the globe.

The new addition to the Solidatus consultancy partner portfolio is set to empower organizations to govern their data more effectively, working closely to ensure we reach new financial services customers in need of building faster, automated data infrastructure.Over a quarter of US Artefact consultants are certified in Solidatus, adding to their wealth of data expertise to assist clients through all stages of their Solidatus journey.

Mike Pierce, VP of Global Alliances at Solidatus,acknowledges the power of this partnership in delivering results to customers: “We’re delighted to join forces with data & AI consulting leaders Artefact, whose expertise in enabling enterprises to maximize data potential perfectly complements Solidatus’ strengths. We look forward to jointly delivering Solidatus’ unique approach to building data transparency and establishing trust across complex environments, empowering clients to meet governance and regulatory requirements.”

Akhilesh Kale, Senior Director and Financial Services & Insurance lead of Artefact US, shares this sentiment, stating, “We are thrilled to announce our partnership with Solidatus enabling our financial services clients to break free from the challenges of data flow transparency while also improving trust in AI by having a clear sense of quality and origins of data. As we join forces, we are looking forward to accelerating value creation through data discovery for our clients’ front office, risk and regulatory use cases delivering tangible bottom and top-line value.”

Download the Artefact e-book for data in finance now: https://report.artefact.com/l/597421/2022-09-20/hzyrj2

About Solidatus: 

Solidatus is a leading data management solution that empowers
organizations to connect, visualize, and govern their data relationships
through next-generation data lineage. For more information, please visit www.solidatus.com

About Artefact: 

Artefact is a global data-driven services company specialising in consulting for data transformation and data-driven digital marketing. We help companies transform data into business impact by delivering tangible results across the entire value chain. www.artefact.com.  

red black and white ribbons HIGH RES 2 scaled

2023 shows no signs of slowing down for us and our partners, marked by a significant win at the Banking Tech Awards from FinTech Futures for Best Use of Tech in Business Lending. We take pride in this recognition, which reflects our successful and ongoing efforts to deliver effective services to our customers.

1701428373096

Our joint submission showed how HSBC supercharged their Wholesale Lending business with our next-gen version-controlled graph tech.

Within six months of launching, a small business team documented and modelled HSBC’s entire wholesale lending book, demonstrating traceability from source to consumption. The team has now successfully modelled 2,000 source tables with 80,000+ fields, and 20,000+ data linkages across 45 source systems used globally.

With this Winter win of 2023, we are in high anticipation of 2024 and what it will bring. If you want to learn more about our data sharing work with HSBC, you can access it all in our case study 

amazon web services solidatus partnership scaled

Solidatus joins the AWS Marketplace to bring connected governance to AWS customers

Unlocking the true power of AWS with the world’s leading data lineage solution 

Solidatus, the leading enterprise data lineage and metadata management provider, today announced its availability in the Amazon Web Services (AWS) Marketplace, allowing AWS customers to seamlessly discover, purchase, and deploy Solidatus. With integrations to AWS services like S3, Redshift, and Glue, plus connectivity to other AWS data sources, Solidatus delivers an integrated data landscape view across AWS and non-AWS environments. This unified approach allows for effective governance and risk management by bridging cloud, on-premises, and hybrid systems on one pane of glass. 

Simon Bustamante-Dick, Partner Analytics Leader, EMEA AWS, said: “AWS Marketplace transforms how enterprises worldwide find, subscribe to, deploy, and govern third-party software, data, containers, machine learning models, and professional services. It’s also become the most strategic channel for ISVs, data providers, and resellers to acquire new customers, migrate existing customers to the cloud, and grow revenue. I’m excited to welcome Solidatus to the AWS Marketplace. Their solution lets customers create integrated data maps spanning AWS and non-AWS environments, and this consolidated visibility allows organizations to simplify governance across AWS cloud, multicloud, hybrid, and on premises systems. Solidatus fills a key need for our customers seeking end-to-end data insight.”

With an extensive list of trusted partnerships and out-of-the-box integrations with leading data management vendors like Collibra, Snowflake, Alteryx, Google BigQuery, BigID, Denodo, Data.World, Informatica, Oracle, Microsoft, and IBM Redhat, Solidatus gives AWS customers confidence in leveraging their data from end to end.

Philip Dutton, Solidatus CEO, added: “Through AWS Marketplace, it’s now simpler than ever for customers to leverage Solidatus to visualize their data journeys and unlock deeper insights across their business. Our technology’s flexibility is a key asset, and we look forward to supporting AWS users with diverse data intelligence needs, from governance to analytics.”  

Key benefits of Solidatus for AWS users:

It doesn’t matter how data enters your business, how it’s transported, or where it’s stored, Solidatus consolidates your enterprise metadata into a single, manageable view. This gives users the power to create dynamic blueprints that visualize real-time data flow, providing an operational control center for comprehensive data management and analysis at scale.   

  • Automate end-to-end data lineage across AWS services  
  • Connect data sources for integrated analytics and improved decisions 
  • Map data to critical business processes to streamline operations 
  • Streamline data governance workflows  
  • Identify security gaps for enhanced data protection
  • Rapidly embed risk controls into data and processes  
  • Identify data quality issues affecting reporting 

View us on the Marketplace

For more information contact hello@solidatus.com

For AWS customers contact aws-marketplace@solidatus.com 

Fiber optics blue HIGH RES scaled

After a remarkable year filled with success and expanding our reach, we are happy to announce our win in the ‘Best Data Governance Solution’ category at the A-Team Data Management Insight Awards USA 2023.

This marks our third win for us in the ‘Best Data Governance Solution’ category – a testament to our transformative data governance capabilities in data management.

Our data governance solution goes far beyond basic data management, adding different dimensions through powerful dynamic visualizations and this award solidifies our position as leaders in this domain. In today’s complex digital landscape, having a visual representation of the connections that define and drive an organization can be the key differentiator between success and failure.

DataManagmentInsightsAwards 023

Solidatus CEO & Founder Philip Dutton

With high hopes for the year ahead, we anticipate 2024 will be just as rewarding, and we eagerly anticipate sharing these accomplishments with our colleagues, partners, clients, and industry peers.

For more information on this great awards ceremony, you can check out the A-Team’s dedicated showcase page: https://a-teaminsight.com/awards/data-management-insight-awards-usa/?section=winners

red black and white ribbons HIGH RES 2 scaled

Solidatus and HSBC shortlisted for prestigious banking technology award

We are thrilled to announce that Solidatus and our customer, HSBC, have been selected as finalists for the Banking Tech Awards by Fintech Future. Our nomination falls under the category of “Best Use of Tech in Business Lending”. 

Our joint submission detailed how HSBC leveraged the power of Solidatus’ next-generation version-controlled graph technology to revolutionize their Wholesale Lending business under a multimillion-dollar transformation program that helped them:  

  • Slash credit decisions from months to minutes 
  • Reduce fund distribution to customers from months to hours 
  • Comply with evolving regulations and internal risk management controls 

Sid Mubashar, Head of Credit Insights & Intelligence, Wholesale Credit & Lending, HSBC, said: “At HSBC, we serve millions of customers across 63 countries and territories, with over one million customers utilising $3.6 trillion in approved credit limits – resulting in a highly complex data infrastructure. With Solidatus, I can visualize the data model for each business outcome, manage the requirements of key stakeholders and provide greater clarity over the intricacies involved in addressing their requests. This new data-led framework plays a major role in HSBC’s global strategy.” 

How HSBC Transformed their Data Capabilities with Solidatus

Within six months of launch, a small business team documented and modeled HSBC’s entire wholesale lending book, demonstrating traceability from source to consumption. The team has now successfully modeled 2,000 source tables with 80,000+ fields, and 20,000+ data linkages across 45 source systems used globally.

By seamlessly ingesting metadata via Solidatus’ connectors, HSBC gained complete lending book transparency with dynamic end-to-end lineage visualization. They can efficiently evaluate change impacts by referencing current, historical, and future data views, substantially reducing manual workload. Additionally, HSBC established a shared data language, eliminating redundant efforts for the bank and its customers, with significant financial benefits.

This self-service, single source of truth is available to hundreds of internal users, visualizing all data requirements across the lending book. In contrast, with the same sized team, given the complexity and scale of the organization, traditional solutions would require an estimated 18-24 months and would not provide the same breadth of transparency and capability.

Philip Dutton, CEO, Solidatus, added: “HSBC understands the complexities of modern banking better than most – and seeing such a small team deliver such a big impact using Solidatus is extremely powerful. Being shortlisted for this prestigious award is a testament to their innovative work and the quality of our technology.”

Read the full case study by following the link below.

transforming HSBCs lending business

Transforming HSBC's Lending Business with Solidatus

Within six months of launch, a small business team documented and modeled HSBC’s entire wholesale lending book, demonstrating traceability from source to consumption. They now have a highly scalable and automated solution that is being applied to several applications from ESG to liquidity calculations and other regulatory uses.

23.07 000011 BROC Data distress are banks on the brink of a mental health crisis top banner scaled

Senior global data leaders within banking and financial services firms are currently experiencing alarmingly high levels of data-related stress in the workplace, with 64% reporting that their data-related stress levels are sometimes or always high.

87% of those who reported any data-related stress, regardless of how often they feel it, say that it has affected their mental health and well-being, with 74% having taken sick days as a result, and 61% enduring an average of two to six nights of disrupted sleep per week.

And this anxiety has prompted 71% to consider quitting their jobs.

We call this phenomenon ‘data distress’, and these statistics are just a handful of the headline findings from new research recently commissioned by Solidatus.

The first major study of its kind, it involved 300 senior data leaders across the US and the UK in the financial services sector answering a series of questions on their levels of data-related stress and their views on the contributory factors.

Whitepaper

The assessment is bleak, and you can read about it in our whitepaper: Data Distress: Is the Data Office on the Brink of Breakdown? How US and UK Data Leaders in Banking and Financial Services are Facing Data Burnout. And the conversations we have day in, day out with practitioners – the ones that prompted us to commission this work in the first place – bear out our findings, so these will be very familiar pain points to people in our space.

But there’s hope; by quantifying it, as we do in the report, remedies have emerged.

Mistrust and the burden of regulations

So, what’s getting practitioners down?

A broad area we’ve defined as ‘data ambiguity and uncertainty’ appears to be the most significant cause of data-related stress, with 82% of respondents choosing at least one option from a range of answers that fall within this umbrella category.

A particular source of frustration is how time-consuming and stressful managing data for financial regulations is. The most significant category of reason cited for why is one we designate ‘tech deficiencies’. 93% of respondents’ answers fell into this bucket, and this doesn’t just cause data distress; it’s a huge commercial distraction.

“With tactical firefighting and fine avoidance being the default, productivity and opportunity discovery will be stifled,” said Philip Dutton, CEO and founder of Solidatus, adding that with “global banking estimated to be worth around $20 trillion per year, even as little as a 5% drop in strategic activity due to data distress represents a $1 trillion reduction in value”.

Better tech and governance

In our report, we identify how the right tech stack and a better approach to data governance are both key to unlocking the cure for data distress, increasing trust in your data and generally increasing practitioners’ levels of happiness. By heeding the advice in the report, you can:

  • Deliver data that you and your colleagues can trust and on which you can base decisions confidently;
  • Reveal business opportunities that might otherwise have been obscured in the attempt to demonstrate compliance with suboptimal systems; and
  • Reduce data distress.

Download the report to dig deeper into our quantitative research.

The Happy CDO Project

You’ll notice that we mention happiness above. We use this word more than in passing, as this research represents the first piece of activity in our The Happy CDO Project, a new initiative from Solidatus.

We’re shining a light on the issues that matter to data leaders to help them to be successful, fulfilled and happy in their work.​

By focusing on their challenges, we can highlight solutions and strategies to help CDOs, data leaders and data practitioners to be their best.

We look forward to sharing more on this project with you in the months ahead.

Data Distress: Is the Data Office on the Brink of Breakdown?

23.06 000010 GRIC Alteryx partnership SG top banner scaled

Solidatus extends technology partnership with Alteryx to demonstrate ‘connected governance’

Enhanced alliance will help customers visualize data flow across their Alteryx workflows through Solidatus interface, demonstrating governance and building greater confidence to grow their Alteryx footprint

London and Houston, 4th July 2023. Solidatus, the leading data management solution that empowers organizations to connect, visualize and govern their data relationships through data lineage, has extended its strategic technology partnership with Alteryx, the preeminent analytics automation platform.

Leveraging a partnership with a leading Global Systems Integrator (GSI), the two companies have been able to successfully facilitate two large-scale transformation projects: one for a global systemically important bank (G-SIB); the other, a multinational financial services company.

Both shared customers derived value from Alteryx-enabled workflows, which brought significant improvements to their business-critical analysis capabilities by streamlining the production of regulatory reports and reducing the frustrations and challenges felt by their internal IT groups. In order to grow, they faced a business requirement to address data governance concerns over audit, processing and best practices. Each wanted to demonstrate end-to-end lineage to fulfil regulatory obligations specifically around MIFiD and Solvency 2 (capital adequacy), as well as ESG disclosures.

This is where Solidatus came to the fore: by mapping the Alteryx workflows into Solidatus and then through their companies’ systems, processes, controls and their governance platform, Solidatus could easily demonstrate governance processes, leading the senior business leaders of both customers to extend their Alteryx footprint and bring further value to their organizations.

Following the success of these projects, Solidatus has extended its support for more Alteryx tool sets and fully automated the integration. Customers can not only get fine-grained, field-level lineage but can show that their data is under control by allowing them to complete an end-to-end picture of the whole process and resulting data estate.

Håkan Söderbom, Sr. Director, Technology Alliances, at Alteryx, said: “Working with Solidatus enables shared customers to benefit from lineage to quickly demonstrate that their data is properly governed. Solidatus lineage makes it easy to connect technical and non-technical applications and platforms that maintain policies, controls, CDEs and regulations to visually prove compliance much faster than would typically be possible.”

Howard Travers, Head of Technology Alliances at Solidatus, said: “Alteryx workflows are a critical part of the management of organizational data flows for regulatory reporting at many of the largest banks in the world, especially in the USA. The Solidatus-Alteryx integration creates a field-level lineage-focused view of an Alteryx workflow, which can be automatically linked to upstream and downstream lineage information to provide a clear end-to-end view. We believe the Solidatus-Alteryx connector will provide confidence to any financial or regulated organization wishing to expand their use of Alteryx.

“Our vision is to ‘make the unknown known’ by enabling organizations to rapidly build an enterprise data blueprint. Solidatus is a powerful data management solution in its own right. However, we are being increasingly used as a lineage bridge to other best-of-breed applications like our partners Alteryx, Snowflake, Collibra and BigID, to name but a few. Our relationship with Alteryx proves the value of ‘connected governance’ to demonstrate governance, removing concerns, building confidence and driving expansion.”

– Ends –

For more information please contact:

Solidatus press office

About Solidatus

Solidatus is an innovative data management lineage solution that empowers organizations to connect and visualize their data relationships, simplifying how they identify, access, and understand them. With a sustainable data foundation in place, data-rich enterprises can meet regulatory requirements, drive digital transformation, capture business insights, and make better, less risky and more informed data-driven decisions. We provide solutions to several key areas of endeavor, including: governance and regulatory compliance; data risk and controls; business integration; environment, social, governance (ESG); and data sharing. Our clients and investors include top-tier global financial services brands such as Citi and HSBC, healthcare, and retail organizations as well as government institutions.

www.solidatus.com

snowflakes scaled

This week, six members of the Solidatus team – including our CEO and co-founder, Philip Dutton – are at the Snowflake Summit in Las Vegas.

Stationed at Booth 2122-C, they’re reporting growing interest from the gathered masses in how we can supercharge Snowflake’s governance capabilities (PDF) with active metadata management and advanced visualization.

It’s all about increased certainty, which is a little ironic, given that the Summit’s venue is famous for high-risk stakes and gambling.

With such a concentration of interested parties in our immediate vicinity, this is a great time to announce that we’ve enhanced our partnership with the tech giant.

Key new benefits include:

  • Deeper levels of granularity, meaning you can now create dynamic blueprints of Snowflake schemas, whether tables or views, and see column-level dependencies and lineage
  • Capture business insights in real-time by analyzing Snowflake governance rule application to tables, views and columns
  • The facility to automatically map multiple Snowflake instances

Speaking from the Summit, Howard Travers, Head of Technology Alliances at Solidatus, said: “The buzz here is exhilarating. This is because when it comes to data and the cloud, the possibilities with Snowflake are endless. We feel an affinity with Snowflake because we apply the same philosophy to metadata, and I’m so excited to see this important partnership grow in strength.”

For details of the complete package, view our concise brochure on Solidatus and Snowflake (PDF), which also offers free access to a simplified sample model.

On the subjects of technology partnerships and connectors more generally, check out our recent news about our enhanced partnership with Alteryx.

If you’re at the Summit, do pop by our booth to ask how our partnership with Snowflake can bring new levels of visualization, better understanding and greater control over your data. We’ll be there until Thursday.

If you can’t make it along, here’s a preview video of the Snowflake model we’re showcasing:

Play Video
reflections scaled

The value of data and other reflections from attending last month’s Gartner® Data & Analytics Summit 2023 in London, U.K.

By Tina Chace, VP Product

In April, the latest chapter in my 10-year career in fintech product management began: I was appointed VP Product at Solidatus.
Though I’m new to this side of data and analytics (DA) governance, I had been that person who felt the pain when data flows, systems, lineage and quality aren’t mapped; it leads to inefficiency or failure in your operations, and it’s my mission at Solidatus to help put this right for people who subscribe to our sophisticated software.
From previous roles, I also came equipped with experience of AML products, model risk management, and artificial intelligence (AI) and machine learning (ML).
 
So I was delighted when asked to fly in from New York to join the team at last month’s Gartner® Data & Analytics Summit in London.
 
 
tina et al

That’s me on the far left of the picture above, accompanied by some of my colleagues, including our founder and CEO, Philip Dutton, second from the right.

But my main objective in attending was to listen to people from beyond the Solidatus stand. What was the mood music in this space? And what might I be able to do with it in my new role?

We’ll start with an overview of what I felt were the major themes, these views and observations of mine being an amalgam of the various sessions I attended from a wide variety of speakers and organizations.

Major themes

As a first-time attendee to a data and analytics conference, I observed that:

  • Articulating the value of data and governance projects and teams is still challenging. Putting real numbers or quantifying the impact of a governance project can be challenging, and recommendations of talking about financial influencers (such as enabling faster decision-making) should be touted as highly as direct tactical impact, such as headcount reduction. The best examples of getting buy-in from stakeholders outside the data office are real-world use cases which tell the story of value.
  • Culture and people are key to a successful data governance organization. There were many examples of success – where an inspirational data leader was able to align technologies, processes and people to achieve their outcomes, and potential pitfalls where there were warnings that purchasing a particular piece of technology isn’t enough if there are no trained data engineers to work in the product and end-users don’t adopt those tools.
  • As you’d expect, artificial intelligence was talked about everywhere. It seemed like there was a proliferation of AI/ML/data science (DS) vendors or more traditional vendors that touted how AI/ML was powering their platforms. A significant proportion of sessions had AI as a topic as well. In my view, there are two ways to approach AI through a data and analytics lens: how data governance is a key part of a successful enterprise application of AI and ML, and how AI/ML can assist data and analytics governance. More on this further in the blog.

Let’s expand on these points by rolling the first two into one discussion and finishing with the third on its own.

Data governance programs: the vision

The ideal state for a successful data and governance program is for self-service data products and data governance to be owned by their various domains within a centralized framework – essentially, this dovetails with descriptions I saw of how data fabric architecture and data mesh operating models can be used in combination.

For the more practical data leader, a valuable steppingstone is just breaking the silos between domains to have an accurate map of their own data ecosystem. Our customer, Lewis Reeder from the Bank of New York Mellon, presented with Philip Dutton on the bank’s success in leveraging Solidatus’ metadata layered with business context to clearly visualize their data ecosystem.

One general impression I got from this topic as a whole is of the merits of ensuring that getting to the vision is taken in reasonable and planned increments that each deliver value themselves. The value of understanding your data enables tactical things – such as making the data office operate more efficiently – but also underpins all other functions in the organization. Understanding where your data comes from enables functions such as KYC and AML programs, financial and ESG reporting, algorithmic trading functions, and of course feeding the ML models that help drive operational efficiency and automation in other domains.

The verdict on AI

As discussed in a session entitled The Enterprise Implications of ChatGPT and Generative AI on Monday 22nd May, Gartner has rated techniques such as LLMs (large language models) “not ready for prime time” but something to monitor and research.

I agree wholeheartedly with this judgement based on my past experience of trying to implement them for large and highly regulated institutions. The main limitations are the inability of enterprise customers to safely use and potentially contribute to the open-source datasets that drive models such as OpenAI’s ChatGPT, the effort and responsibility for creating and curating a private dataset while utilizing GPT3/4 technologies to build your own LLMs, and the lack of a big enough use case to make these efforts worth the time and money.

The main use case demonstrated is to enable what I saw called “natural language questioning”. This allows business users to easily ask questions to assist them with self-service data governance like “which systems contain information on customer addresses?” or “what is the impact of adding a new field, tax ID number, to a customer entity?”. I’d describe this as search on steroids.

More traditional machine learning methods, such as classification, matching, data extraction, anomaly detection, and topic modeling, can be used to great effect to support data and analytics governance. Plus, the subject matter expertise will be easier to come by, and implementation is more straightforward.

Classification, for example, can be used to determine whether a system should be controlled by certain procedures based on its properties and relationships to other systems. Matching on both name and semantics can be used to suggest mappings between systems for both technical and business terms to assist in creating and enforcing a centralized data dictionary.

On the flip side, understanding your data ecosystem is key to a successful AI/ML project. From my experience in implementing ML products for regulated enterprise customers, we ran into many roadblocks that a well-mapped understanding of the available data and impacted systems would have solved. Understanding what datasets are available and the quality of the datasets to train your ML model is helpful; there were times when it took days or weeks to approve data to just test one of the ML models we implemented – even on prem. And there were many times when data that could have improved the automation rate of the ML model was excluded because the business stakeholders didn’t know or trust the quality of the data.

Finally, there were many scenarios where changes to upstream systems negatively impacted the performance of ML models and the customer didn’t find out until it had already flowed through.

Putting it into practice

It was great to see so many data leaders talking about their real use cases and success stories. But it was even more interesting to hear about their struggles and what they have learned while implementing their data governance programs.

There was a lot to learn from the panels and sessions but also from talking directly to practising data leaders about their own specific scenarios and pain points. The willingness to share knowledge with peers will surely drive the D&A industry forward as a whole.

I can’t wait to put these insights into practice in my new role.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.