#amazonfsx

Amazon FSx Lustre Intelligent Tiering

The newest Amazon FSx for Lustre storage class delivers the first completely elastic Lustre flannel storage at the lowest pricing.

AWS launched Amazon FSx for Lustre Intelligent Tiering today, a new storage class that offers the lowest-cost cloud Lustre file storage, the only fully elastic Lustre file storage, and almost limitless scalability. FSx for Lustre Intelligent-Tiering offers the lowest high-performance cloud file storage for less than $0.005 per GB-month and can save up to 96% on infrequently visited data compared to other managed Lustre options. Elastic file systems allow you to pay for the data you keep without upfront storage capacity provisioning since they expand and contract as you add or remove data.

FSx for Lustre Intelligent-Tiering optimises expenses by assigning cold data to a lower-cost storage tier based on use patterns. An optional SSD read cache boosts performance for latency-sensitive tasks. Intelligent Tiering performs well with petabyte-scale datasets or gigabytes of experimental data for your most demanding HPC and AI/ML applications.

Intelligent Tiering lets you adjust file system performance regardless of storage and offers up to 34% better price performance than on-premise HDD file systems. Intelligent-Tiering storage is appropriate for HDD-based or mixed HDD/SSD workloads with hot and cold data. FSx for Lustre Intelligent-Tiering lets you relocate and execute these workloads without modifying applications, eliminating the need to plan and manage storage capacity and only paying for what you need.

Before this launch, clients used FSx for Lustre SSD storage class to speed up HPC and machine learning applications that need low-latency data access and all-SSD performance. Many applications contain both hot and cold data, therefore all-SSD storage is not needed for colder data parts. FSx for Lustre is increasingly used in AI/ML applications to enhance GPU utilisation. Being one of these burden solutions is now cheaper.

Lustre Intelligent-Tiering FSx

Your data automatically switches between Frequent Access, Infrequent Access, and Archive storage tiers, saving you money without any upfront costs. Tiering works as follows:

Frequent Access: This tier's data was accessed within 30 days.

Infrequent Access: This tier keeps data not retrieved in 30–90 days and saves 44% over Frequent Access.

Archive: Data not accessed in 90 days or more is kept in this category at 65% less cost than Infrequent Access.

For redundancy and availability, your data is stored across multiple AWS Availability Zones regardless of storage tier, unlike on-premises implementations. Your data can be retrieved in milliseconds.

Making a filesystem

Use the AWS Management Console, CLI, API, or CloudFormation to create a file system. When building a file system, choose “Persistent, Intelligent Tiering” for deployment and storage. Selection of Metadata IOPS and necessary throughput. The SSD read cache is configured automatically after the throughput capacity.

After creation, throughput, metadata IOPS, security groups, and SSD read cache can be modified. Users can increase throughput capacity to meet workload needs by paying only for resources consumed.

Data in any storage tier is dispersed across various AWS Availability Zones for redundancy and availability.

Important points

FSx for Lustre Intelligent Tiering can handle many terabytes per second.FSx for Lustre per client with GPU Direct Storage (GDS) and Elastic Fabric Adapter (EFA) support has 12 times (up to 1200 Gbps) the throughput of previous systems.

Write and cached read IOPS max out at tens of millions. SSD read cache has submillisecond time-to-first-byte latencies, but other data takes tens.

Availability

Remember these two points:

AWS Asia Pacific (Hong Kong, Mumbai, Seoul, Singapore, Sydney, Tokyo), Europe (Frankfurt, Ireland, London, Stockholm), Canada (Central), and the US East and West (N. Virginia, Ohio) now provide the new FSx for Lustre storage class.

What is Amazon FSx for Lustre?

Amazon FSx for Lustre is a fully managed solution offering high-performance, scalable, and reasonably priced storage computational workloads. The most widely used high-performance file system in the world serves as the foundation for fully controlled shared storage.

Amazon FSx for Lustre advantages

Quicken computation workloads

Shared storage with sub-millisecond latencies, hundreds of gigabytes/second throughput, and millions of IOPS can speed up computing workloads. In only a few minutes, deploy a fully managed Lustre file system.

Use Amazon S3 to access and handle data sets

By connecting your file systems to S3 buckets, you may access and handle Amazon S3 data from a high-performance file system.

Optimize the cost and functionality of storage for your workload

Use a variety of deployment choices, such as replication level, performance tier, and storage type, to balance cost and performance.

How it operates

With the scalability and performance of the well-liked Lustre file system, Amazon FSx for Lustre offers fully managed shared storage.

Use cases

Increase the speed of machine learning (ML)

Training durations can be shortened with optimized throughput to your computational resources and convenient access to training data stored in Amazon S3.

High performance computing (HPC) should be enabled

Fast, highly scalable storage that is directly integrated with AWS computation and orchestration services can power even the most demanding HPC applications.

Open up big data analytics

Support petabytes of data and thousands of computing instances executing sophisticated analytics workloads.

Boost the agility of the media workload

With storage that grows with your computer, you can adapt to ever-tinier visual effects (VFX), rendering, and transcoding timelines.

Amazon Lustre FSx Features

Overview

Amazon FSx for Lustre provides controlled, cost-effective, high-performance, scalable compute storage. Based on Lustre, the world’s most popular high-performance file system, FSx for Lustre provides shared storage with sub-ms latency, terabytes per second throughput, and millions of IOPS. When coupled to Amazon Simple Storage Service (S3) buckets, FSx for Lustre file systems can access and process data simultaneously.

Improve workload performance

Overview

AWS FSx for Lustre file systems can handle terabytes per second and millions of IOPS. FSx for Lustre handles concurrent access to files and directories from thousands of compute instances. Low file operation latencies are guaranteed by FSx for Lustre.

Most common high-performance file system

The Lustre open source file system is the most popular file system for the 500 fastest computers in the world because it efficiently and cheaply processes the world’s expanding data collections. It is battle-tested in energy, life sciences, media production, and financial services for genome sequencing, video transcoding, machine learning, and fraud detection.

Use for any compute workload

Overview

Popular Linux-based AMIs like Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux are compatible with FSx for Lustre.

Simple import/export Amazon S3 info

Amazon FSx for Lustre allows native S3 data access for data-processing tasks.

You can join one or more S3 buckets to a file system in Amazon FSx with a few clicks. After connecting your S3 bucket to your file system, FSx for Lustre transparently displays S3 objects as files and lets you post results to S3. Objects added, altered, or removed from your S3 bucket update your connected file system automatically. As files are added, edited, or removed, FSx for Lustre updates your S3 bucket automatically. Data is exported back to S3 quickly using parallel data-transfer techniques by FSx for Lustre.

Utilize computing services easily

AWS FSx for Lustre can be used on Amazon EC2 instances or on-premises machines. Your file system’s files and directories can be accessed like a local file system once mounted. Amazon EKS containers can access FSx for Lustre file systems.

Increase Amazon SageMaker instructor positions

Amazon Sagemaker supports Amazon FSx for Lustre input data. Amazon SageMaker and Amazon FSx for Lustre expedite machine learning training jobs by skipping the initial S3 download phase and reducing TCO by avoiding repeated downloads of common items (saving S3 request costs) for iterative jobs on the same data set.

Compute management services simplify deployment

Amazon FSx for Lustre interfaces with AWS Batch via EC2 Launch Templates. Our cloud-native batch scheduler supports HPC, ML, and other asynchronous workloads. AWS Batch launches instances and runs jobs using existing FSx for Lustre file systems and dynamically sizes instances to job resource requirements.

Lustre FSx works with AWS ParallelCluster. Deploy and manage HPC clusters with AWS ParallelCluster, an open-source cluster management tool. During cluster creation, it can automatically create Lustre FSx or use existing file systems.

Access data quickly

File data access has sub-millisecond first-byte latency on SSDs and single-digit millisecond on HDDs.

Metadata servers with low-latency SSD storage support all Amazon FSx for Lustre file systems, regardless of deployment type, storage type, or throughput performance. The SSD-based metadata server delivers metadata operations, which make up most file system activities, with sub-millisecond latencies.

Save money

Reduce paperwork and scale capacity and performance as needed

You can construct and scale a high-performance Lustre file system with a few clicks via Amazon FSx console, CLI, or API. The time-consuming administration responsibilities of managing file servers and storage volumes, updating hardware, setting software, running out of space, and tweaking performance are automated by Amazon FSx file systems.

Various deployments

For short-term and long-term data processing, Amazon FSx for Lustre supports scratch and persistent file systems. Scratch files are suited for short-term data storage and processing. A failed file server does not replicate or save data. For long-term storage and workloads, persistent file systems are best. A persistent file system replicates data and replaces failed servers.

For additional data protection and business and regulatory compliance, Amazon FSx may automatically take incremental backups of persistent file systems. Amazon S3 backups are 99.999999999% durable.

Many storage choices

To optimise cost and performance for your workload, Amazon FSx for Lustre offers SSD and HDD storage solutions. SSD storage can be used for low-latency, IOPS-intensive workloads with tiny, random file operations. HDD storage can handle throughput-intensive workloads with huge, sequential file operations.

To provide sub-millisecond latencies and better IOPS for frequently visited files in an HDD-based file system, provision an SSD cache.

Storage quotas can monitor and limit user- and group-level storage consumption on file systems to prevent unnecessary capacity use. Storage quotas are for file system administrators who service various users, teams, or projects.

Data compression lowers storage costs

File system backups and storage can be reduced by data compression. The data compression feature uses the LZ4 algorithm, which optimizes compression without affecting file system speed. Data compression allows FSx for Lustre to compress and uncompress newly written files before writing them to disk and reading them.

Get rid of old files

After exporting files to Amazon S3, release inactive data to maximize storage capacity. After a file is released, its data is removed from the file system and stored on S3, but its metadata remains. A released file is automatically and transparently loaded from your S3 bucket onto your file system when accessed.

Ensure security and compliance

Overview

Amazon FSx for Lustre file systems are secured at-rest and in-transit in specific areas.

AWS helps customers manage their requirements with the longest-running cloud compliance program. The security of Amazon FSx meets global and industry requirements. In addition to HIPAA, it is PCI DSS, ISO 9001, 27001, 27017, and 27018 compliant and SOC 1, 2, and 3. Visit our compliance site for resources. To see all services and certifications, visit the Services in Scope by Compliance Program page.

Isolated networks

Amazon VPC endpoints allow you to isolate your Amazon FSx file system in your virtual network. Configure security group rules and network access to Amazon FSx file systems.

Resources-level permissions

Amazon FSx integrates with AWS IAM. You can govern how AWS IAM users and groups create and delete file systems using this connection. Amazon FSx resources can be tagged to restrict IAM user and group actions.

One-stop backup and compliance with AWS Backup

Integration with AWS Backup allows fully managed, policy-based backup and recovery for Amazon FSx file systems. Integration with AWS Backup protects customer data and ensures AWS service compliance for business continuity.

Regional and account backup compliance

Copying Amazon FSx file system backups across AWS Regions, accounts, or both can improve data protection and meet business continuity, disaster recovery, and compliance requirements.

Amazon FSx for NetApp ONTAP file systems can now be created up to 9 times faster than previously. The file systems are completely managed, as is already the case with this service, with latency to primary storage being in the sub-millisecond range and latency to the capacity pool being in the tens of milliseconds. You’ll be able to move even more of your most taxing workloads in statistical computing, visual effects, and electronic design automation (EDA) to the cloud thanks to this improved performance.

Scaling Success with Amazon FSx Strategies

Current FSx for ONTAP scale-up file systems have a maximum throughput of 4 GBps and 192 TiB of SSD storage. They are powered by a single pair of servers in an active/passive high availability (HA) arrangement. To provide continuous availability even in the event that an Availability Zone is unavailable, the server pair can be placed in two different Availability Zones or across different fault domains within a single Availability Zone.

With today’s release, ONTAP file systems powered by two to six HA pairs can now have scale-out FSx created for them.

Building a File System for Scaling Out

Using the AWS Management Console, AWS Command Line Interface (AWS CLI), or by creating code that invokes the Amazon FSx CreateFileSystem function, You can construct a scale-out file system. Using the console, you will first select Amazon FSx

You select a Single-AZ deployment, type in a name, select Standard create, and enter the amount of SSD storage you want. You have three alternatives (more on that in a moment): accept the suggested throughput capacity, select a value from the menu, or enter a value to see my possibilities.

There’s a handy magical feature in the dropdown. It will display one or more options based on your input of the required throughput capacity:

Occasionally, the console may present you with multiple choices for the same required throughput capacity. The following suggestions will assist you in selecting a solution that will perform well for your workload.

Low Throughput: You will only be able to use one HA pair if you select a plan that offers 4 GBps or less. Choosing this option is the easiest if you don’t want a high throughput.

High Storage and/or Throughput: The maximum throughput varies depending on how many HA pairs you supply. Additionally, selecting a solution with more pairs will increase your headroom for potential future increases in allocated storage.

You make your choices, input the typical values for the remaining options, and then click Next to see my settings again. After making sure that my selections for any attributes that cannot be changed after creation are correct, you click Create file system.

You have the ability to alter the supplied IOPS up to once every six hours and to expand the storage capacity.

Currently, if the file system uses multiple HA pairs, the provisioned throughput capacity cannot be altered after formation.

Accessible Right Now

You can begin developing and utilizing scale-out file systems right now in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Sydney), and Europe (Ireland) Regions.