Amazon Athena federated queries now support Amazon SageMaker Lakehouse integrated access controls.
It unveiled Amazon SageMaker‘s next iteration today, a single platform for data, analytics, and AI that combines popular AWS analytics and machine learning features. Its central component is SageMaker Unified Studio, a unified platform for developing AI and data for large data processing, rapid SQL analytics, model construction and training, generative AI application development, and data exploration, preparation, and integration. Among the features included in this release is Amazon SageMaker Lakehouse, which allows you to create robust analytics and artificial intelligence and machine learning (AI/ML) applications using a single copy of data by integrating data from data lakes and data warehouses.
Along with these releases, it’s great to share that Amazon SageMaker Lakehouse now has data catalog and permissions features that let you connect, find, and centrally manage permissions to data sources.
In order to optimize for particular use cases and scaling needs, organizations nowadays store data across several platforms. Data silos spanning databases, streaming services, data lakes, and data warehouses are frequently the outcome of this. Connecting to and analyzing data from these many sources presents difficulties for analysts and data scientists. They have to maintain several access permissions, set up specific connections for every data source, and often copy data, which raises expenses and may result in inconsistent data.
By making it easier to connect to well-known data sources, categorize them, provide rights, and make the data accessible for analysis using SageMaker Lakehouse and Amazon Athena, the new feature tackles these issues. No matter where your data sources are located, you may utilize the AWS Glue Data Catalog as a single metadata repository. A consolidated view of all the data that is available is therefore provided.
You don’t need to establish connections again since data source connections are made once and may be utilized again. Databases and tables are automatically cataloged and registered with AWS Lake Formation as you connect to the data sources. After cataloging, you provide data analysts access to those databases and tables, saving them from having to connect to each data source separately and from needing to know the built-in secrets of those sources. When querying with Athena, Lake Formation permissions may be used to establish fine-grained access control (FGAC) policies that are consistently enforced across data lakes, data warehouses, and online transaction processing (OLTP) data sources.
There is no need for expensive and time-consuming data transfers or duplications because the data stays in its original location. In Data Catalog, you may setup built-in connectors to a variety of data sources, such as Google BigQuery, Amazon Redshift, Amazon Aurora, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), and more. You can also create or reuse existing data source connections.
Getting started with the integration between Athena and Lake Formation
Lets utilize a preset setup using Amazon DynamoDB as a data source to demonstrate this functionality. To properly illustrate the functionality, the environment is configured with the relevant tables and data. For this example, it utilize the SageMaker Unified Studio interface.
Projects may be created and managed here, and they function as communal workplaces. Team members may cooperate, interact with data, and create machine learning models together through these projects. When a project is created, AWS Glue Data Catalog databases are immediately set up, a Redshift Managed Storage (RMS) data catalog is created, and the required permissions are granted.
Project management options include creating a new project by selecting Create project or seeing an extensive list of current projects by selecting Browse all projects. In make use of two current projects: marketing-project, where analysts work with limited data access rights, and sales-group, where administrators have all access privileges to all data. The difference between administrative and restricted user access levels is well-represented by this configuration.
In this step, was configured a federated catalog for Amazon DynamoDB, the target data source. Just select the + (plus) symbol to add data after going to Data in the left navigation pane. After selecting Add connection, Simply select Next.
After selecting Amazon DynamoDB, Simply choose Next.
And select Add data after entering the information. kindly now used SageMaker Lakehouse to generate the Amazon DynamoDB federated catalog. This is where resource policies are used by your administrator to provide you access. The resource policies in this environment are already set up by me. It will now demonstrate how SageMaker Unified Studio’s fine-grained access restrictions operate.
Lets start by choosing the sales-group project, where administrators are in charge of maintaining and having complete access to client data. Phone numbers, client IDs, and zip codes are among the variables in this collection. Anyone can use Query with Athena to run queries in order to evaluate this data.
The Query Editor opens instantly upon clicking choose Query with Athena, giving me a workspace to create and run SQL queries on the lakehouse. Data exploration and analysis are made easy with this integrated query environment.
To demonstrate what an analyst sees when they run their queries and see that the fine-grained access control permissions are there and functional, And move to the marketing-project in the second section.
In the second section, it turn to the marketing-project context to illustrate the analyst’s point of view. This enables us to confirm that the fine-grained access control restrictions are applied correctly and are successfully limiting access to data as intended. Users may see how analysts work with the data while adhering to the defined security restrictions by looking at sample queries.
To confirm the access controls, Then run a SELECT statement on the table using the Query with Athena option. As anticipated, the results show that it can only read the zipcode and cust_id columns; the phone column is still blocked due to the established permissions.
You can now improve security governance, expedite AI/ML development, and streamline data operations while preserving data integrity and compliance throughout your whole data ecosystem with Amazon SageMaker Lakehouse‘s new data catalog and permissions features.
Now available
Data catalog and permissions in Amazon SageMaker Lakehouse simplify interactive analytics through federated query when connecting to a unified catalog and permissions with Data Catalog across multiple data sources to define and enforce fine-grained security policies for a high-performing query experience.
US East (N. Virginia), US West (Oregon), US East (Ohio), Europe (Ireland), and Asia Pacific (Tokyo) AWS Regions have this capability.
