Amazon MSK Express Brokers For Fast Scaling And Throughput

Express brokers for Amazon MSK are now available to provide your Kafka clusters with high throughput and quicker scaling.

Express brokers, a new broker type for Amazon Managed Streaming for Apache Kafka (Amazon MSK), are now generally available. Compared to standard brokers running Apache Kafka, it is intended to provide up to three times the throughput per broker, scale up to 20 times faster, and cut recovery time by 90%. Express brokers support Kafka APIs, are preconfigured with Kafka best practices by default, and offer the same low latency performance that Amazon MSK clients are accustomed to, allowing them to keep using their current client applications without any modifications.

When utilizing Amazon MSK provided clusters, express brokers offer enhanced compute and storage elasticity for Kafka applications. Scalable and highly available Apache Kafka-based systems may be developed and run more easily with the help of Amazon MSK, a fully managed AWS service.

Let’s examine some of the main characteristics and advantages of Express brokers in more detail:

• Easy operations and hands-free storage management: By offering limitless storage without preprovisioning, express brokers eliminate disk-related bottlenecks. With just ingress and egress throughput divided by the suggested per-broker throughput, cluster sizing is easier. By removing a possible source of failure, this improves resilience and streamlines cluster administration by doing away with the requirement for proactive disk capacity monitoring and scaling.
• Reduced brokers with throughputs of up to three times each Smaller clusters can handle the same demand thanks to higher throughput per broker. While m7g.16xl standard brokers may comfortably handle 154 MBps ingress, their throughput must account for client traffic and background processes. During cluster events, m7g.16xl size instances may securely handle up to 500 MBps ingress without sacrificing availability or performance because to express brokers’ opinionated settings and resource isolation.
• 20 times faster scaling and higher utilization: By reducing data transfer, Express brokers may scale up to 20 times faster than Standard brokers. This makes cluster resizing faster and more dependable. Over-provisioning in anticipation of traffic surges is no longer necessary because you can add brokers in a matter of minutes and keep an eye on each broker’s ingress throughput capacity.
• Improved resilience and 90% quicker recovery: Express brokers are made for mission-critical applications that need to be highly resilient. Best-practice defaults, such as 3-way replication (RF=3), are preconfigured in them to minimize misconfiguration-related failures. In addition, compared to regular Apache Kafka brokers, express brokers recover fromtemporary failures 90% faster. Capacity planning is made easier by express brokers’ recovery and rebalancing, which consume relatively little cluster resources. When right-sizing clusters, this removes the possibility of higher resource consumption and the requirement for ongoing monitoring.

Depending on your preferences and workload, Amazon MSK offers you a variety of options:

Express brokers are the ideal option for all Kafka workloads because they reduce expenses, offer greater resilience, and have fewer operational overheads. Amazon MSK Serverless is an option if you would rather use Kafka without having any control over its configuration, capacity, or scalability. This gives you a completely abstracted Apache Kafka experience that scales automatically, does not require infrastructure administration, and charges you on a pay-per-use basis without requiring you to minimize resource usage.

How to begin using Amazon MSK Express brokers

You can use Amazon MSK’s Sizing and Pricing worksheet to get started with Express brokers. This worksheet provides you with an approximate monthly cost estimate and assists you in determining the cluster size required to support your workload.

The main determinant of your cluster’s size is the throughput demands of your workload. To determine the size and number of brokers your cluster will require, you should also take into account other variables like the amount of connections and the partition. For instance, you can use three express.m7g.large brokers to meet your throughput requirements if your streaming application requires 30 MBps of data ingress (write) and 80 MBps of data egress (read) capacity (assuming the partition count for your workload is within the maximum number of partitions that Amazon MSK recommends for a m7g.large instance).

The following table shows the maximum entrance, egress, and partition counts per instance size that are recommended for secure and long-term operations. The Amazon MSK Developer Guide’s Best Practices section has more information on these suggestions.

Instance size      Ingress (MBps) Egress (MBps)

express.m7g.large           15.6        31.2

express.m7g.4xlarge      124.9     249.8

express.m7g.16xlarge    500.0     1000.0

Create an Amazon MSK provisioned cluster using the CreateCluster API or the AWS Management Console after determining the quantity and size of Express brokers required for your workload.

Choose Express brokers under Broker type when creating a new cluster in the Amazon MSK panel. Next, choose the amount of computational capacity you wish to allocate to the broker. As shown in the screenshot, Graviton-based systems running Apache Kafka 3.6.0 can be used for Express brokers. Express brokers do not require storage to be pre-provisioned.

Some of these configurations can also be altered to further optimize your clusters’ performance to your own tastes.

The AWS CLI create-cluster command creates an MSK cluster.

brokernodegroupinfo.JSON. You specify three subnets for Amazon MSK to distribute broker nodes in JSON.

Once the cluster has been created, you may use the bootstrap connection string to connect your clients to the cluster endpoints.

With Express brokers, you may scale vertically (by changing the instance size) or horizontally (by adding brokers). Vertical scaling doubles throughput without requiring partition redistribution. Horizontal scalability lets you add brokers in groups of three and establish extra partitions, but new brokers handling traffic require partition reassignment.

One of Express Brokers’ key advantages is its ability to add brokers and rebalance partitions in a matter of minutes. Rebalancing partitions after adding Standard brokers may take hours. The graph below shows how long it takes to rebalance partitions when three Express brokers are added to a cluster and 2000 partitions are redistributed to each of the extra brokers.

As shown, reassigning partitions and using the new brokers’ capacity takes about 10 minutes. It performed the same experiment on a similar cluster composed of Standard brokers, and partition reassignment took place over a 24-hour period.

Things to consider

There are a few things you should know about Express brokers:

• Data migration: Amazon MSK Replicator moves data from your Kafka or MSK cluster to an Express broker cluster. and metadata from your cluster to a new cluster.
• Monitoring: Use Amazon CloudWatch metrics to monitor your Express broker cluster at the broker level and within. Additionally, Prometheus may be used to disclose metrics and enable open monitoring with the JMX Exporter and Node Exporter.
• Security: By integrating with AWS Key Management Service (AWS KMS), Amazon MSK provides transparent server-side encryption for the storage in Express brokers, just like other broker types. You can give Amazon MSK the AWS KMS key to encrypt your data when you set up an MSK cluster with Express brokers. it’s at rest. If you do not provide a KMS key, Amazon MSK creates an AWS managed key and uses it on your behalf.

Now available

The Express broker type is currently offered by the following regions: Europe (Frankfurt), Europe (Ireland), Europe (Stockholm), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), US East (Ohio), US East (N. Virginia), and US West (Oregon).

The size and resolution of Apache Kafka broker instances (invoiced at one-second resolution) for Express brokers determine the hourly charge. The number of active brokers in your MSK clusters also determines the hourly charge. Additionally, data written to an Express broker incurs a fee per gigabyte (GB).