BigQuery Unified Governance: Centralized Data Management

BigQuery Unified Governance

Data is the essential building block of artificial intelligence, most of its promise is still unrealized. Why? Data quality is still a major obstacle. Businesses must be able to locate, comprehend, and have faith in their data assets in order to use enterprise data to inform analytics-driven decisions and create unique AI. This calls for efficient data governance that includes access control, sharing, quality assurance, cataloguing, metadata management, and discovery.

The stakes are really high. Over 60% of AI initiatives will fail to meet business SLAs and be abandoned by 2026, according to Gartner, if organizations do not enable and support their AI use cases through an AI-ready data approach.

Google Cloud is introducing BigQuery unified governance at Google Cloud Next 25, which offers strong data governance features to help businesses stay up to date with governance complexity. Significant risks are created as innovation is hampered by data silos, fragmented information, and unclear ownership. BigQuery unified governance gives businesses the tools and services they need to streamline data management and produce insights that can be put to use.

By assisting organizations in identifying, comprehending, and utilising their resources, BigQuery’s integrated intelligent governance turns governance from a hassle into a potent instrument for data activation. Business, technical, and runtime metadata are brought together by BigQuery Universal Catalogue, a uniform, AI-powered data catalogue that natively integrates Dataplex, BigQuery sharing, security, and metastore capabilities. It is a key component of BigQuery unified governance.

The unified governance features of BigQuery include:

• Unified: BigQuery enables the discovery, comprehension, governance, and use of your data assets and AI models by putting governance right at the Centre of your data-to-AI lifecycle. This provides end-to-end data-to-AI lineage, data profiling, insights, and safe sharing, giving data administrators, stewards, and custodians strong capabilities for managing metadata and enforcing policies. Additionally, the new universal semantic search makes it as easy as posing a natural language query to retrieve the appropriate facts.

• Intelligent: Gen AI-powered new governance features have the potential to completely transform data management. BigQuery universal catalogue can help you discover hidden relationships between BigQuery data assets, automate governance, enable intelligent query recommendations and automated metadata curation at scale, and democratize data-driven insights throughout the company by leveraging the power of large language models (LLMs).

• Open: With support for open storage standards like Apache Iceberg and a unified runtime metastore spanning SQL, open-source engines, and AI/ML, BigQuery universal catalogue protects you from change. A multi-engine, multi-vendor architecture for the governance and utilization of fully managed Iceberg data is made possible by the BigQuery metastore, which is part of the BigQuery universal catalogue and is Iceberg-compliant.

The BigQuery universal catalogue is used by ANZ Bank, a global provider of banking and financial services, for thorough data governance, discovery, and observability.

Noteworthy features

BigQuery’s new unified governance experience offers a centralized platform for managing, safeguarding, and sharing data and AI assets inside the BigQuery user interface. There are also launching a plethora of significant new features and capabilities in the areas of security, sharing, and governance.

Governance

• Full-catalog search with semantic understanding (preview): Full-catalog semantic search now allows users to find data and AI resources in BigQuery across projects and data silos. With the introduction of natural-language search capabilities, this feature facilitates catalogue searches for both technical and non-technical users.

• Automated metadata curation (preview): To enhance data discovery and support future AI applications, BigQuery universal catalogue can now automatically create metadata for BigQuery tables, including table and column descriptions.

• AI-powered knowledge engine (preview): With automatic entity-relationship visualization, users may quickly uncover hidden relationships in a collection. BigQuery global catalogue helps new data teams quickly become familiar with new data assets by using inferred associations to provide suggestions for cross-table searches and natural language questions.

• Data products (preview): Data owners can build, share, and manage sets of data assets by use case using BigQuery data products. They can package and distribute these data assets both inside and outside of organizations in a way that is consistent, controlled, and compliant with security best practices.

• Business lexicon (GA): BigQuery’s business lexicon gives businesses a common understanding of their data. Customers may improve context, collaboration, and search by defining and managing company terms, designating data stewards for these terms, and attaching them to data asset fields.

• Automated large-scale BigLake and object table (GA) cataloguing: BigQuery Universal Catalogue instantly generates query-ready BigLake tables at scale by extracting current metadata for both structured and unstructured data from Cloud Storage.

• Automated anomaly detection (preview): BigQuery Universal Catalogue helps you find data flaws, inconsistencies, and outliers in your data by automating data anomaly detection. This saves you time while figuring out and fixing data problems.

Sharing

• Google Cloud Marketplace integration with BigQuery sharing (preview): Data owners can use Google Cloud Marketplace to make money off of datasets in BigQuery sharing (previously Analytics Hub).

• BigQuery (GA) stream sharing: Use Pub/Sub topics to choose and distribute worthwhile real-time streams.

• BigQuery stored procedure sharing (preview): Without disclosing the actual code, share SQL stored procedures and allow execution in the subscriber’s project.

• BigQuery (preview) query template sharing: Use publisher-defined query templates to personalize, re-use, and limit SQL queries in a clean environment.

Read more on GCP Cloud SQL Enterprise Plus Query Insights for Bottlenecks

Security

• Column data policies (preview): Make data-masking and raw access policies that are directly related to a column and that are applicable to other columns and tables.

• Row-level security (GA) combined with subquery support: Row filtering can now be enabled without altering current data models to BigQuery Universal Catalog’s support for SQL subqueries in security access policy definitions.

• The BigQuery platform’s integrated governance innovations enable businesses to fully realise the potential of their data and AI projects.

There continue to collaborate with outside catalogue suppliers to enhance their governance capabilities in addition to the innovation in BigQuery. Collibra’s enterprise-wide governance for data and AI provides end-to-end visibility, quality, and stewardship across hybrid and multicloud environments, extending BigQuery’s universal catalogue. Each use case is accelerated and improved by guaranteeing that more teams can locate and trust the data they need to execute AI, regardless of location.

BigQuery universal catalog’s integration of governance and automation of metadata management are assisting companies in overcoming operational inefficiencies and data silos, which will ultimately spur innovation and increase business impact.