#intelligenttiering

Amazon FSx Lustre Intelligent Tiering

The newest Amazon FSx for Lustre storage class delivers the first completely elastic Lustre flannel storage at the lowest pricing.

AWS launched Amazon FSx for Lustre Intelligent Tiering today, a new storage class that offers the lowest-cost cloud Lustre file storage, the only fully elastic Lustre file storage, and almost limitless scalability. FSx for Lustre Intelligent-Tiering offers the lowest high-performance cloud file storage for less than $0.005 per GB-month and can save up to 96% on infrequently visited data compared to other managed Lustre options. Elastic file systems allow you to pay for the data you keep without upfront storage capacity provisioning since they expand and contract as you add or remove data.

FSx for Lustre Intelligent-Tiering optimises expenses by assigning cold data to a lower-cost storage tier based on use patterns. An optional SSD read cache boosts performance for latency-sensitive tasks. Intelligent Tiering performs well with petabyte-scale datasets or gigabytes of experimental data for your most demanding HPC and AI/ML applications.

Intelligent Tiering lets you adjust file system performance regardless of storage and offers up to 34% better price performance than on-premise HDD file systems. Intelligent-Tiering storage is appropriate for HDD-based or mixed HDD/SSD workloads with hot and cold data. FSx for Lustre Intelligent-Tiering lets you relocate and execute these workloads without modifying applications, eliminating the need to plan and manage storage capacity and only paying for what you need.

Before this launch, clients used FSx for Lustre SSD storage class to speed up HPC and machine learning applications that need low-latency data access and all-SSD performance. Many applications contain both hot and cold data, therefore all-SSD storage is not needed for colder data parts. FSx for Lustre is increasingly used in AI/ML applications to enhance GPU utilisation. Being one of these burden solutions is now cheaper.

Lustre Intelligent-Tiering FSx

Your data automatically switches between Frequent Access, Infrequent Access, and Archive storage tiers, saving you money without any upfront costs. Tiering works as follows:

Frequent Access: This tier's data was accessed within 30 days.

Infrequent Access: This tier keeps data not retrieved in 30–90 days and saves 44% over Frequent Access.

Archive: Data not accessed in 90 days or more is kept in this category at 65% less cost than Infrequent Access.

For redundancy and availability, your data is stored across multiple AWS Availability Zones regardless of storage tier, unlike on-premises implementations. Your data can be retrieved in milliseconds.

Making a filesystem

Use the AWS Management Console, CLI, API, or CloudFormation to create a file system. When building a file system, choose “Persistent, Intelligent Tiering” for deployment and storage. Selection of Metadata IOPS and necessary throughput. The SSD read cache is configured automatically after the throughput capacity.

After creation, throughput, metadata IOPS, security groups, and SSD read cache can be modified. Users can increase throughput capacity to meet workload needs by paying only for resources consumed.

Data in any storage tier is dispersed across various AWS Availability Zones for redundancy and availability.

Important points

FSx for Lustre Intelligent Tiering can handle many terabytes per second.FSx for Lustre per client with GPU Direct Storage (GDS) and Elastic Fabric Adapter (EFA) support has 12 times (up to 1200 Gbps) the throughput of previous systems.

Write and cached read IOPS max out at tens of millions. SSD read cache has submillisecond time-to-first-byte latencies, but other data takes tens.

Availability

Remember these two points:

AWS Asia Pacific (Hong Kong, Mumbai, Seoul, Singapore, Sydney, Tokyo), Europe (Frankfurt, Ireland, London, Stockholm), Canada (Central), and the US East and West (N. Virginia, Ohio) now provide the new FSx for Lustre storage class.