Efficient cloud workloads with Amazon ECS and EBS
It is now possible to run a greater variety of data processing workloads with Amazon Elastic Container Service (Amazon ECS) thanks to the integration with Amazon Elastic Block Store (Amazon EBS), which AWS are announcing currently. You can provision Amazon EBS storage for your ECS tasks running on AWS Fargate and Amazon Elastic Compute Cloud (Amazon EC2) without needing to manage storage or compute.
With the advent of Amazon ECS integration with Amazon EBS, organizations can now run a wider range of workloads than before. Many organizations opt to deploy their applications as containerized packages.
Data workloads that need high transaction volumes and throughput support storage, like extract, transform, and load (ETL) jobs for big data, can be executed. These jobs retrieve pre-existing data, process it, and store the processed data for later use. Your data processing workloads are now more resilient and handle less work because Amazon ECS fully manages the storage lifecycle, saving you from having to construct extra scaffolding to handle infrastructure updates.
Now, when running containerized applications on Amazon ECS, you have a range of storage options to select from:
- The default ephemeral storage for your Fargate tasks is 20 GiB. For your Fargate tasks, you can configure up to 200 GiB of ephemeral storage for applications that require extra storage to download large container images or for scratch work.
- Amazon Elastic File System (Amazon EFS) can be mounted to your ECS tasks running on both EC2 and Fargate, if your application spans multiple tasks and requires concurrent access to a shared dataset. Common examples of such workloads include web applications such as content management systems, internal DevOps tools, and machine learning (ML) frameworks. The architecture of Amazon EFS allows it to be attached to multiple tasks at once and be accessible throughout a Region.
- You can set up Amazon ECS to provision and attach Amazon EBS storage to your tasks that are running on both Amazon EC2 and Fargate, for applications that require high-performance, low-cost storage that does not need to be shared among tasks. Within an availability zone, Amazon EBS is intended to offer block storage with minimal latency and optimal performance.
Refer to the Amazon ECS documentation’s “Using data volumes in tasks” and “Persistent storage best practices” for further details.
Starting to integrate your ECS tasks with EBS volume
For your Amazon ECS task, you can pass the Amazon EBS storage requirements at runtime and set the volume mount point for your container in the task definition. Simply specifying the volume size required for the task will suffice to get things started in the majority of use cases. The file system you wish to use to format the volume and all other EBS volume attributes are configurable at your option.
1.Make a task description first
Select Create new task definition after opening the Amazon ECS console and going to Task definitions.
EBS volume can be configured as a new configuration type by selecting Configure at deployment in the Storage section. Linux file systems allow you to provision and attach a single volume per task.
When you choose Configure at task definition creation, you can configure existing storage options such as bind mounts, Docker volumes, EFS volumes, Amazon FSx for Windows File Server volumes, or Fargate ephemeral storage.
Now that the source EBS volume is selected, you can specify a mount path for the volume to be mounted in the task as well as a container in the task definition.
You can also use $aws ecs register-task-definition –cli-input-json file://example.json command line to register a task definition to add an EBS volume.
2. Install the EBS volume and launch your task
Select Run new task after navigating to your ECS cluster. You can choose the launch type, your task definition, and the compute options.
A new or existing ECS service can be configured to use EBS volumes with the desired configuration, even though this example focuses on deploying a standalone task with an attached EBS volume.
Configuring the additional storage is done in a new section called Volume. As stated in your task definition, the mount points, type, and volume name are those. Determine the IOPS, desired throughput, sizes (GiB), and EBS volume types.
Existing EBS volumes cannot be attached to ECS tasks. You can select your snapshot ID, though, if you wish to build a volume from an already-taken snapshot instead. Leave this field empty if you would like to create a new volume. In Linux, the file system type that you can select is between xfs, ext3, and ext4.
Amazon ECS automatically removes the attached volume upon task termination. Uncheck “Delete on termination” if you need the data in the EBS volume to be kept after the task finishes. Furthermore, in order to authorize Amazon ECS to make API calls on your behalf, you must create an AWS Identity and Access Management (IAM) role for volume management that has the necessary permissions. Visit the AWS documentation’s infrastructure role for additional details on this policy.
Using either customer-managed keys or Amazon-managed keys, you can also set up encryption by default on your EBS volumes. Consult Amazon EBS encryption in the AWS documentation to gain further insight into the available choices.
Click Create to begin your task after you’ve adjusted every task setting.
3. Open the EBS volume and deploy your task
The volume information is displayed on the task details page once your task has begun. To find your created EBS volume details, choose a task and then click the Volumes tab.
The organization and management of EBS volumes can be better coordinated by your team. DevOps engineers can configure the actual EBS volume attributes at runtime when the application is deployed, and application developers can specify the path where your application expects storage to be available in the task definition.
With different EBS volume configurations in each environment (e.g., gp3 volumes in development environments and io2 volumes in production), DevOps engineers can now deploy the same task definition to multiple environments.
Currently accessible
The nine AWS Regions that now offer Amazon ECS integration with Amazon EBS are US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm). Like EBS volumes and snapshots, you only pay for what you use.
