Microsoft has announced the general availability of Azure Managed Lustre, a managed high-performance parallel file system designed for high-performance computing (HPC) and AI workloads. Azure Managed Lustre offers the Lustre file system as a first-party managed service on Azure, allowing users to leverage the benefits of a complete HPC solution, including compute and high-performance storage.
Need for Azure Managed Lustre
Lustre is a widely used distributed parallel file system in the HPC world, primarily deployed on-premises to meet the scalable and high-throughput storage needs of HPC workloads. With the increasing demand for high-throughput storage in generative AI applications, Lustre has experienced renewed growth in adoption. Customers in industries that rely on high-performance computing are seeking a seamless end-to-end infrastructure that includes compute, storage, and networking, which can be deployed quickly and easily.
Azure Managed Lustre benefits
- Managed PaaS experience: Users can deploy and consume Lustre file systems as a worry-free managed service.
- Reduced TCO: Pay-as-you-go pricing helps reduce total cost of ownership.
- Throughput options: Different pricing levels offer varying throughput capabilities to accommodate different workloads.
- Easy deployment of independent clusters: Users can deploy independent Lustre clusters with predictable performance, eliminating the “noisy neighbor” problem commonly encountered in on-premises deployments.
- Quick deployment: Azure Managed Lustre can be deployed within minutes, eliminating the need for extensive planning and meeting compliance and data residency requirements.
- Regional deployment options: Azure Managed Lustre supports deployment in multiple Azure regions, allowing users to meet compliance and data residency requirements.
- Scalability: Azure Managed Lustre can scale to hundreds of petabytes (PB) and offers benefits such as durability, availability, and cost savings through tiering to Azure Blob storage.
- Seamless integration with Azure compute services: Azure Managed Lustre integrates seamlessly with existing Azure compute services, including Azure HPC services, Azure Machine Learning, and Azure Kubernetes Service.
- Investment protection: Users migrating from on-premises Lustre deployments can retain their existing automation and platform development.
Azure Managed Lustre currently offers two persistent durable instances based on solid-state drives (SSDs), differentiated by their performance options:
- AMLFS Standard: 125 MB/s
- AMLFS Premium: 250 MB/s
Users can download the Lustre client packages for their preferred Linux distribution and kernel version from packages.microsoft.com. Additionally, prebuilt HPC images with Lustre client packages are available for Ubuntu 18.04, 20.04, 22.04, and Alma 8.7.
Several customers from various industries have already benefited from Azure Managed Lustre
- Petronas, a world leader in energy, leverages Azure Managed Lustre to accelerate their upstream energy research, enabling highly complex quantitative interpretation of seismic processing workloads with high throughput.
- Meteomatics, a weather intelligence provider, utilizes Azure Managed Lustre for their weather modeling and simulations, benefiting from Lustre’s high IOPS and throughput to handle massive data volumes.
- Duke Clinical Research Institute employs Azure Managed Lustre to address their high I/O needs, supporting their analytics tools for clinical research with a scalable and simple deployment model.
Azure Managed Lustre
Azure Managed Lustre is delivered as a hosted-on-behalf-of subscription within the customer’s virtual network. Users can easily configure and deploy Azure Managed Lustre through the Azure portal, and the service provides performance metrics for monitoring cluster performance.
The hierarchical storage management functionality in Azure Managed Lustre allows seamless data transfer between Azure Managed Lustre and Azure Blob. Users can establish data archival, retention, and protection strategies by importing active data into Lustre clusters while archiving the remaining data in Azure Blob. This approach minimizes runtime costs. Integration with Azure Blob enables data accessibility through multiple protocols, including NFS, HDFS, and REST.
Microsoft also announces the general availability of the Container Storage Interface (CSI) driver for Azure Managed Lustre, allowing containerized applications running on Azure Kubernetes Service to easily utilize Azure Managed Lustre.
Azure Managed Lustre is now available in multiple regions through the Azure portal. For the most up-to-date information on available regions, users can refer to the products available by region page. Pricing details are available on the Azure Managed Lustre pricing page.
[…] called Microsoft Dev Box are now available for general use. This announcement was made at the Microsoft Build event, where several new features and enhancements were introduced. Microsoft Dev Box is […]