AWS S3 Tables
Use S3 to store tabular data at scale
Optimize query cost and performance as your data lake grows.
Amazon S3 Tables simplify the process of storing tabular data at scale and offer the first cloud object store with integrated Apache Iceberg support. Specifically designed for analytics applications, S3 Tables offer up to 10x more transactions per second and up to 3x quicker query performance than self-managed Iceberg tables housed in general-purpose S3 buckets.
The Apache Iceberg standard is supported by S3 Tables, making it simple to query your tabular data using well-known AWS and third-party query engines like Amazon Athena, Redshift, EMR, and Apache Spark. Utilize S3 Tables to store tabular data, such as ad impressions, streaming sensor data, or daily purchase transactions, as an iceberg table in S3. Automatic table maintenance will help you optimize cost and performance as your data changes.
AWS S3 table Advantages
Scalability
Streamline data lakes at any size, whether you’re managing hundreds of tables in your Iceberg environment or you’re just starting out.
Improved efficiency
Compared to storing Iceberg tables in general-purpose S3 buckets, you can achieve up to 10 times more transactions per second and up to 3 times quicker query performance.
Completely controlled
To automatically optimise query efficiency and costs over time, carry out ongoing table maintenance actions like compaction, snapshot management, and unreferenced file removal.
Smooth integration
Through the S3 Tables preview interface with AWS Glue Data Catalogue, you can use sophisticated Iceberg analytics features and query data using well-known AWS services like Amazon Athena, Redshift, and EMR. Popular open source tools can be used with S3 Tables.
Streamlined security
To easily control access to tables, create them as first-class AWS resources and apply rights.
How it operates
S3 Tables offer dedicated S3 storage for Apache Parquet-formatted structured data storage. Tables can be created directly in S3 as first-class resources within a table bucket. These tables are accessible by programs or tools that support the Apache Iceberg standard and can be secured with table-level permissions specified in identity- or resource-based rules. The underlying data in S3 is saved as Parquet data when you build a table in your table bucket. The metadata required for your applications to query that Parquet data is then maintained by S3.
Query engines use the client library included in table buckets to access and modify the Iceberg metadata of the tables within the bucket. Multiple clients can safely read and write data to your tables when this library is used in conjunction with new S3 APIs for table operations. S3 automatically rewrites, or “compacts,” your objects over time to optimize the underlying Parquet data. Compaction optimizes your data on S3 to reduce expenses and enhance query performance.
For simple queries utilizing well-known query engines like Amazon Athena, Amazon EMR, and Apache Spark, Amazon S3 Tables offer storage that is optimized for tabular data, including daily purchase transactions, Ad impressions and streaming sensor data in Apache Iceberg format. You can anticipate up to three times quicker query performance and up to ten times more transactions per second when using a fully managed service as opposed to self-managed table storage. You can also anticipate part-and-parcel operational efficiency.
With thousands of AWS customers utilizing Iceberg to query through frequently billions of files comprising petabytes or even exabytes of data, Iceberg has emerged as the most widely used method for managing Parquet files.
Tables, Namespaces, and Table Buckets
Alongside the current directory and general purpose buckets, table buckets are the third kind of S3 bucket. A table bucket can be compared to an analytics warehouse that holds Iceberg tables with different schemas. S3 Tables also offer the same performance, scalability, availability, and durability as S3 itself. They also automatically optimize your storage to reduce costs and optimize query performance.
Every table bucket has a name that needs to be distinct within the AWS account in relation to the region in which it sits. In addition to having a resource policy, buckets are referred by ARN. Lastly, each bucket groups its tables logically using namespaces.
Structured datasets kept in a table bucket are called tables. They reside in one of the bucket’s namespaces and share ARNs and resource policies with table buckets. Tables provide complete management capabilities, including compaction, dated snapshot management, and unreferenced file removal, along with automated, adjustable continuous maintenance. An S3 API endpoint is available for storing operations for every table.
Access management can be made simpler by referencing namespaces from access policies.
Table Maintenance
Some crucial maintenance tasks that you would be in charge of if you were building and maintaining your own Iceberg tables are handled by table buckets. The following maintenance tasks are carried out automatically to free you from these responsibilities and allow you to spend more time at your table:
Compaction: To increase query performance, this technique merges several small table objects into a single, larger object. The goal is to achieve a target file size that can be set to range from 64 to 512 megabytes. A fresh snapshot is created using the updated item.
Snapshot Management: With configuration parameters for the minimum number of snapshots to retain and the maximum age of a snapshot to retain, this process expires and eventually deletes table snapshots. After a predetermined number of days, expired photos are designated as non-current and then removed.
Removal of Unreferenced Files: This procedure eliminates and expunges items that are not mentioned in any table snapshots.
Important Information
You should be aware of the following two crucial facts regarding tables and table buckets:
AWS connection: The connection of S3 Tables with AWS Glue Data Catalogue is in preview. This integration enables you to use AWS Analytics services like Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight to query and visualise data.
S3 API Support: GetObject, HeadObject, PutObject, and multi-part upload operations are among the pertinent S3 API services that table buckets support.
Security: Table buckets automatically encrypt all items stored within. Block Public Access is enforced by the configuration of table buckets.
Pricing: You pay for requests, storage, compaction costs, and an object monitoring fee. To learn more, see the S3 Pricing page.
Regions: This new functionality is available in the US West (Oregon) and US East (Ohio, N. Virginia) AWS Regions.
