Explore a whole new catalog experience with Dataplex, which is currently widely accessible.
Organisations are finding that they require a central catalog for their data assets as a result of the ever-growing volumes and varieties of data. Whether your resources are on-premises or in Google Cloud, Dataplex Catalog, Google Cloud’s next-generation data asset inventory platform, offers a uniform inventory for all of your metadata. It is currently broadly available.
Dataplex Data catalog
A comprehensive inventory of both on-premises and Google Cloud resources, including BigQuery, is offered by Dataplex Data catalog. You add metadata for third-party resources into Dataplex Catalog, and metadata for Google Cloud resources is automatically collected.
You can add more commercial and technical metadata to your inventory using Dataplex Catalog in order to fully capture the context and knowledge about your resources. You can enable data governance over your data assets and search and find your data throughout the organisation with Dataplex Data catalog.
The following tasks can be completed with Dataplex Catalog:
Find out about and comprehend your data. Throughout the company, Dataplex Data catalog gives you visibility over your data resources. It facilitates finding pertinent resources for your needs when consuming data. It gives data resources context, which enables you to judge whether or not they are appropriate for the requirements of your data consumers.
Make data management and governance possible. Your data governance and management capabilities can be strengthened and informed by the metadata provided by Dataplex Catalog.
Keep your metadata in a complete, expandable library. You may access and save metadata that is automatically gathered from your Google Cloud resources using Dataplex Data catalog. Your own metadata from non-Google Cloud systems can be integrated. Technical and commercial metadata annotations can be added to enhance any metadata.
The operation of Dataplex Catalog
The following ideas form the foundation of Dataplex Catalog:
- Entry A data asset is represented by an entry. Aspects inside an entry describe the majority of the metadata. This is comparable to Data Catalog entries. See Entries for further details.
- Aspect: Within an entry, an aspect is a collection of connected metadata fields. An aspect might be thought of as extra metadata attached to an entry, or as one of its building blocks. This is comparable to Data Catalog tags, except the aspects are contained in the entries rather than existing as separate resources. See Aspects for additional details.
- Aspect type: An aspect type is an aspect template that can be used again. Each aspect is an example of a certain aspect type. This is comparable to Data Catalog’s tag templates. Go to Aspect types for further details.
- Entry group: An entry group is a unit of management for entries, acting as a container for them. You can set up IAM access control, project attribution, or location for the entries in an entry group, for instance, using an entry group. This reminds me of Data Catalog entry groupings. Refer to Entry groups for further details.
- An entry type is a template that can be used to create entries. It lays forth the necessary metadata components, which are described as a set of conditions for this kind of entry. See Entry types for additional information.
What is it that Dataplex Catalog can do for you?
You can search and find your data throughout the company with Dataplex Data catalog. You can also enable data governance over your data assets, gain a better understanding of the context of your data, and capture context and knowledge about your data domain by adding more business and technical metadata to your data.
How Dataplex Data catalog may assist you with daily data discovery and governance inquiries is as follows:
- You can go through related metadata and seek data resources all around the company as a business analyst or data analyst.
- You can annotate your data resources as a data producer or governor by adding more technical, semantic, and business metadata.
- Establishing the guidelines for annotation and custom resources will help you, as the data owner, steward, or governor, maintain consistency in your metadata.
You have a consolidated inventory of all the resources you have as a data engineer, including resources from Google Cloud (harvested by Dataplex Catalog automatically) and resources from other systems (harvested by you and ingested into Dataplex Data catalog).
A solitary, user-friendly API and a strong metamodel are provided by the Dataplex Catalog.
The following are some advantages of utilising Dataplex Catalog:
- You can interact with and store a variety of metadata types, including complicated structures like lists, maps, and arrays, with an expressive metadata structure.
- For consistent and efficient ingestion, you can self-configure the metadata schema for your unique resources.
- One atomic CRUD operation can be used to interact with all of the metadata associated with an entry, and you can retrieve various metadata annotations linked to search or list responses.
Basic API functions (create, read, update, and delete) and searches conducted against specific Dataplex Data catalog resources are free of charge. The storage of metadata is paid for, nevertheless. Dataplex Data catalog offers complete support for Terraform providers and can be accessed through the google cloud CLI, the console, and an API.
At Google Cloud, their goal is to simplify their integration process for partners so that google cloud can increase combined value. To expand their data management capabilities into hybrid and multi-cloud systems, google cloud collaborate closely with a wide range of partners. For clients who use Dataplex and Collibra together, Dataplex Data catalog is now linked with Collibra to simplify governance across cloud, on-premises, self-managed, and edge locations. Keep checking back for further details regarding new alliances that will improve their data management skills and benefit their clients even more.
