Enhancing Digital Transformation with Data Mesh

The distribution of important business information across a myriad of different systems and in a range of different formats is a major obstacle to the success of digital transformation projects. But taking a new approach to information management, by employing a data mesh, can make an immense difference to the success of digital transformation endeavours.

Every activity in your organisation generates data and that data is used to make decisions. Some of that comes from or is captured in internal systems. Other information is less structured and comes from external sources such as social media. And every customer interaction, whether it comes into a contact centre, from your website, a chatbot or over the phone is information you want to capture and use.

The data mesh is a set of principles, first developed by Zhamak Dehghani, that sees data as a product that is packaged and managed by domain experts and accessed using a self-service data infrastructure with a federated governance model. That data can be accessed by users through a self-service platform that enables business users to rapidly access data to extract insights that allow them to make decisions. Instead of putting technology teams in the role of data custodians, the data mesh sees data as a product with domain-oriented, decentralised data ownership and architecture.

For example, the credit card business unit at a bank knows and understands its data. Its domain expertise allows it to determine the best way to share that data, as a product, to others who may need it. Those experts can set data quality rules at the source and create ways to share that data that make sense to them. This can be using a data fabric that leverages APIs (application programming interfaces) to give access to the data without the need to copy it or traditionally extract, transform and load (ETL) processes that are complex, costly to develop and maintain, and result in the need for more computing and storage capability.

One of the key benefits of successful digital transformation is vastly improved business agility. That means being able to react more quickly to rising challenges and being able to identify and take advantage of new opportunities. The ETL approach was developed at a time when business moved more slowly and reaction times could be measured in weeks, months or even years. It is no longer fit for purpose in today’s world.

But if they choose a data fabric approach the data can stay in place. The data fabric does not rely on the centralisation or copying of data for analysis. This can reduce the time to insight from hours to seconds. And it changes data from something used to look back on into a tool that can be used for forward-looking analytics. Rather than waiting for an ETL process to complete, a data fabric approach to the data mesh makes data accessible from where it is created in near real-time.

One of the biggest obstacles for successful digital transformation is siloed data. The data warehouses and data lakes of the past tried to become the one source of information truth for businesses but failed to live up to that expectation. The data mesh, brought to life through data fabric technologies, renders those technologies impotent as it gives decision-makers the data they need, when they need it, to enable vastly improved processes.

Unlike other data access and management systems, the data mesh is technology-agnostic. Vertically integrated data management systems used to create vendor lock-in by forcing you to use one vendor to ensure data lake and business intelligence tools worked together. The data mesh allows data from almost any source to be queried and analysed by whatever business intelligence tool you prefer.

True digital transformation is data-driven. Whether that’s for decision-making or linking systems to create streamlined processes, fast access to data makes it possible to truly transform. A data mesh makes data accessible by putting domain experts in control of how the data is managed and shared.

Without a data mesh, digital transformation efforts are doomed to mediocrity or complete failure.