What is Amazon FSx for Lustre?
Amazon FSx for Lustre is a fully managed solution offering high-performance, scalable, and reasonably priced storage computational workloads. The most widely used high-performance file system in the world serves as the foundation for fully controlled shared storage.
Amazon FSx for Lustre advantages
Quicken computation workloads
Shared storage with sub-millisecond latencies, hundreds of gigabytes/second throughput, and millions of IOPS can speed up computing workloads. In only a few minutes, deploy a fully managed Lustre file system.
Use Amazon S3 to access and handle data sets
By connecting your file systems to S3 buckets, you may access and handle Amazon S3 data from a high-performance file system.
Optimize the cost and functionality of storage for your workload
Use a variety of deployment choices, such as replication level, performance tier, and storage type, to balance cost and performance.
How it operates
With the scalability and performance of the well-liked Lustre file system, Amazon FSx for Lustre offers fully managed shared storage.
Use cases
Increase the speed of machine learning (ML)
Training durations can be shortened with optimized throughput to your computational resources and convenient access to training data stored in Amazon S3.
High performance computing (HPC) should be enabled
Fast, highly scalable storage that is directly integrated with AWS computation and orchestration services can power even the most demanding HPC applications.
Open up big data analytics
Support petabytes of data and thousands of computing instances executing sophisticated analytics workloads.
Boost the agility of the media workload
With storage that grows with your computer, you can adapt to ever-tinier visual effects (VFX), rendering, and transcoding timelines.
Amazon Lustre FSx Features
Overview
Amazon FSx for Lustre provides controlled, cost-effective, high-performance, scalable compute storage. Based on Lustre, the world’s most popular high-performance file system, FSx for Lustre provides shared storage with sub-ms latency, terabytes per second throughput, and millions of IOPS. When coupled to Amazon Simple Storage Service (S3) buckets, FSx for Lustre file systems can access and process data simultaneously.
Improve workload performance
Overview
AWS FSx for Lustre file systems can handle terabytes per second and millions of IOPS. FSx for Lustre handles concurrent access to files and directories from thousands of compute instances. Low file operation latencies are guaranteed by FSx for Lustre.
Most common high-performance file system
The Lustre open source file system is the most popular file system for the 500 fastest computers in the world because it efficiently and cheaply processes the world’s expanding data collections. It is battle-tested in energy, life sciences, media production, and financial services for genome sequencing, video transcoding, machine learning, and fraud detection.
Use for any compute workload
Overview
Popular Linux-based AMIs like Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux are compatible with FSx for Lustre.
Simple import/export Amazon S3 info
Amazon FSx for Lustre allows native S3 data access for data-processing tasks.
You can join one or more S3 buckets to a file system in Amazon FSx with a few clicks. After connecting your S3 bucket to your file system, FSx for Lustre transparently displays S3 objects as files and lets you post results to S3. Objects added, altered, or removed from your S3 bucket update your connected file system automatically. As files are added, edited, or removed, FSx for Lustre updates your S3 bucket automatically. Data is exported back to S3 quickly using parallel data-transfer techniques by FSx for Lustre.
Utilize computing services easily
AWS FSx for Lustre can be used on Amazon EC2 instances or on-premises machines. Your file system’s files and directories can be accessed like a local file system once mounted. Amazon EKS containers can access FSx for Lustre file systems.
Increase Amazon SageMaker instructor positions
Amazon Sagemaker supports Amazon FSx for Lustre input data. Amazon SageMaker and Amazon FSx for Lustre expedite machine learning training jobs by skipping the initial S3 download phase and reducing TCO by avoiding repeated downloads of common items (saving S3 request costs) for iterative jobs on the same data set.
Compute management services simplify deployment
Amazon FSx for Lustre interfaces with AWS Batch via EC2 Launch Templates. Our cloud-native batch scheduler supports HPC, ML, and other asynchronous workloads. AWS Batch launches instances and runs jobs using existing FSx for Lustre file systems and dynamically sizes instances to job resource requirements.
Lustre FSx works with AWS ParallelCluster. Deploy and manage HPC clusters with AWS ParallelCluster, an open-source cluster management tool. During cluster creation, it can automatically create Lustre FSx or use existing file systems.
Access data quickly
File data access has sub-millisecond first-byte latency on SSDs and single-digit millisecond on HDDs.
Metadata servers with low-latency SSD storage support all Amazon FSx for Lustre file systems, regardless of deployment type, storage type, or throughput performance. The SSD-based metadata server delivers metadata operations, which make up most file system activities, with sub-millisecond latencies.
Save money
Reduce paperwork and scale capacity and performance as needed
You can construct and scale a high-performance Lustre file system with a few clicks via Amazon FSx console, CLI, or API. The time-consuming administration responsibilities of managing file servers and storage volumes, updating hardware, setting software, running out of space, and tweaking performance are automated by Amazon FSx file systems.
Various deployments
For short-term and long-term data processing, Amazon FSx for Lustre supports scratch and persistent file systems. Scratch files are suited for short-term data storage and processing. A failed file server does not replicate or save data. For long-term storage and workloads, persistent file systems are best. A persistent file system replicates data and replaces failed servers.
For additional data protection and business and regulatory compliance, Amazon FSx may automatically take incremental backups of persistent file systems. Amazon S3 backups are 99.999999999% durable.
Many storage choices
To optimise cost and performance for your workload, Amazon FSx for Lustre offers SSD and HDD storage solutions. SSD storage can be used for low-latency, IOPS-intensive workloads with tiny, random file operations. HDD storage can handle throughput-intensive workloads with huge, sequential file operations.
To provide sub-millisecond latencies and better IOPS for frequently visited files in an HDD-based file system, provision an SSD cache.
Storage quotas can monitor and limit user- and group-level storage consumption on file systems to prevent unnecessary capacity use. Storage quotas are for file system administrators who service various users, teams, or projects.
Data compression lowers storage costs
File system backups and storage can be reduced by data compression. The data compression feature uses the LZ4 algorithm, which optimizes compression without affecting file system speed. Data compression allows FSx for Lustre to compress and uncompress newly written files before writing them to disk and reading them.
Get rid of old files
After exporting files to Amazon S3, release inactive data to maximize storage capacity. After a file is released, its data is removed from the file system and stored on S3, but its metadata remains. A released file is automatically and transparently loaded from your S3 bucket onto your file system when accessed.
Ensure security and compliance
Overview
Amazon FSx for Lustre file systems are secured at-rest and in-transit in specific areas.
AWS helps customers manage their requirements with the longest-running cloud compliance program. The security of Amazon FSx meets global and industry requirements. In addition to HIPAA, it is PCI DSS, ISO 9001, 27001, 27017, and 27018 compliant and SOC 1, 2, and 3. Visit our compliance site for resources. To see all services and certifications, visit the Services in Scope by Compliance Program page.
Isolated networks
Amazon VPC endpoints allow you to isolate your Amazon FSx file system in your virtual network. Configure security group rules and network access to Amazon FSx file systems.
Resources-level permissions
Amazon FSx integrates with AWS IAM. You can govern how AWS IAM users and groups create and delete file systems using this connection. Amazon FSx resources can be tagged to restrict IAM user and group actions.
One-stop backup and compliance with AWS Backup
Integration with AWS Backup allows fully managed, policy-based backup and recovery for Amazon FSx file systems. Integration with AWS Backup protects customer data and ensures AWS service compliance for business continuity.
Regional and account backup compliance
Copying Amazon FSx file system backups across AWS Regions, accounts, or both can improve data protection and meet business continuity, disaster recovery, and compliance requirements.
