Scalability, agility, and governance issues are limiting the use of traditional centralised data architectures in the rapidly changing field of data engineering and analytics. A new paradigm known as “data mesh” has arisen to address these issues and enables organisations to adopt a decentralised approach to data architecture. This blog post describes the idea of data mesh and explains how Dataplex, a BigQuery suite data fabric feature, may be utilised to achieve the advantages of this decentralised data architecture.
What is Data mesh
An architectural framework called “data mesh” encourages treating data like a product and decentralises infrastructure and ownership of data. More autonomy, scalability, and data democratisation are made possible by empowering teams throughout an organisation to take ownership of their respective data domains. Individual teams or data products assume control of their data, including its quality, schema, and governance, as opposed to depending on a centralised data team. Faster insights, simpler data integration, and enhanced data discovery are all facilitated by this dispersed responsibility paradigm.
An overview of the essential components of data mesh is provided in Figure 1.
Data mesh architecture
Let’s talk about the fundamentals of data mesh architecture and see how they affect how we use and manage data.
Domain-oriented ownership:
Data mesh places a strong emphasis on assigning accountability to specific domains or business units inside an organisation as well as decentralising data ownership. Every domain is in charge of overseeing its own data, including governance, access controls, and data quality. Domain specialists gain authority in this way, which promotes a sense of accountability and ownership. Better data quality and decision-making are ensured by this approach, which links data management with the particular requirements and domain expertise of each domain.
Self-serve data infrastructure:
Within a data mesh architecture, domain teams can access data infrastructure as a product that offers self-serve features. Domain teams can select and oversee their own data processing, storage, and analysis tools without depending on a centralised data team or platform. With this method, teams may customise their data architecture to meet their unique needs, which speeds up operations and lessens reliance on centralised resources.
Federated computational governance:
In a data mesh, a federated model governs data governance instead of being imposed by a central authority. Data governance procedures are jointly defined and implemented by each domain team in accordance with the demands of their particular domain. This methodology guarantees that the people closest to the data make governance decisions, and it permits adaptation to domain-specific requirements with flexibility. Federated computational governance encourages responsibility, trust, and adaptability in the administration of digital assets.
Data as a product:
Data platforms are developed and maintained with a product mentality, and data within a data mesh is handled as such. This entails concentrating on adding value for the domain teams, or end users, and iteratively and continuously enhancing the data infrastructure in response to input. Teams who employ a product thinking methodology make data platforms scalable, dependable, and easy to use. They provide observable value to the company and adapt to changing requirements.
Google Dataplex
Dataplex is a cloud-native intelligent data fabric platform that simplifies, integrates, and analyses large, complex data sets. It standardises data lineage, governance, and discovery to help enterprises maximise data value.
Dataplex’s multi-cloud support allows you to leverage data from different cloud providers. Its scalability and flexibility allow you to handle large volumes of data in real-time. Its robust data governance capabilities help ensure security and compliance. Finally, its efficient metadata management improves data organisation and accessibility. Dataplex integrates data from various sources into a unified data fabric.
How to apply Dataplex on a data mesh
Step 1: Establish the data domain and create a data lake.
We specify the data domain, or data boundaries, when building a Google Cloud data lake. Data lakes are adaptable and scalable big data storage and analytics systems that store structured, semi-structured, and unstructured data in its original format.
Domains are represented in the following diagram as Dataplex lakes, each controlled by a different data provider. Data producers keep creation, curation, and access under control within their respective domains. In contrast, data consumers are able to make requests for access to these subdomains or lakes in order to perform analysis.
Step 2: Define the data zones and create zones in your data lake.
We create zones within the data lake in this stage. Every zone has distinct qualities and fulfils a certain function. Zones facilitate the organisation of data according to criteria such as processing demands, data type, and access needs. In the context of a data lake, creating data zones improves data governance, security, and efficiency.
Typical data zones consist of the following:
- The raw zone is intended for the consumption and storage of unfiltered, raw data. It serves as the point of arrival for fresh data that enters the data lake. Because the data in this zone is usually preserved in its original format, it is perfect for data lineage and archiving.
- Data preparation and cleaning occurs in the curated zone prior to data transfer to other zones. To guarantee data quality, this zone might include data transformation, normalisation, or deduplication.
- Zone of transformation:This area contains high-quality, structured, and converted data that is prepared for use by data analysts and other users. This zone’s data is arranged and enhanced for analytical uses
Step 3: Fill the data lake zones with assets
We concentrate on adding assets to the various data lake zones in this step. The resources, data files, and data sets that are ingested into the data lake and kept in their designated zones are referred to as assets. You can fill the data lake with useful information for analysis, reporting, and other data-driven procedures by adding assets to the zones.
Step 4: Protect your data lake
We put strong security measures in place in this stage to protect your data lake and the sensitive information it contains. Protecting sensitive data, assisting in ensuring compliance with data regulations, and upholding the confidence of your users and stakeholders all depend on having a safe data lake.
With Dataplex’s security approach, you can manage access to carry out the following actions:
Establishing zones, building up more data lakes, and developing and linking assets are all part of managing a data lake.obtaining data connected to a data lake through the mapped asset (storage buckets and BigQuery data sets, for example)obtaining metadata related to the information connected to a data lake
By designating the appropriate fundamental and preset roles, the administrator of a data lake controls access to Dataplex resources (such as the lake, zones, and assets).Table schemas are among the metadata that metadata roles can access and inspect.The ability to read and write data in the underlying resources that the assets in the data lake reference is granted to those who are assigned data responsibilities.
Benefits of creating a data mesh
Enhanced accountability and ownership of data:
The transfer of data ownership and accountability to individual domain teams is one of the fundamental benefits of a data mesh. Every team now has accountability for the security, integrity, and quality of their data products thanks to the decentralisation of data governance.
Flexibility and agility:
Data meshes provide domain teams the freedom to make decisions on their own, enabling them to react quickly to changing business requirements. Iterative upgrades to existing data products and faster time to market for new ones are made possible by this agility.
Scalability and decreased bottlenecks:
By dividing up data processing and analysis among domain teams, a data mesh removes bottlenecks related to scalability. To effectively handle growing data volumes, each team can extend its data infrastructure on its own terms according to its own requirements.
Improved data discoverability and accessibility:
By placing a strong emphasis on metadata management, data meshes improve both of these metrics. Teams can find and comprehend available data assets with ease when they have access to thorough metadata.
Collaboration and empowerment:
Domain experts are enabled to make data-driven decisions that are in line with their business goals by sharing decision-making authority and data knowledge.
Cloud technologies enable scalable cloud-native infrastructure for data meshes.Serverless computing and elastic storage let companies scale their data infrastructure on demand for maximum performance and cost-efficiency.
Strong and comprehensive data governance: Dataplex provides a wide range of data governance solutions to assure data security, compliance, and transparency. Dataplex secures data and simplifies regulatory compliance via policy-driven data management, encryption, and fine-grained access restrictions. Through lineage tracing, the platform offers visibility into the complete data lifecycle, encouraging accountability and transparency. By enforcing uniform governance principles, organisations may guarantee consistency and dependability throughout their data landscape.
Effective data governance procedures are further enhanced by Dataplex’s centralised data catalogue governance and data quality monitoring capabilities.Businesses can gain a number of advantages by adopting the concepts of decentralisation, data ownership, and autonomy.Better data quality, accountability, agility, scalability, and decision-making are benefits. This innovative strategy may put firms at the forefront of the data revolution, boosting growth, creativity, and competitiveness.
