APIs for Amazon SQS

The newly launched set of APIs for Amazon SQS allows you to manage dead-letter queue (DLQ) redrive programmatically. These APIs enable you to move messages from the DLQ back to their original queue or a custom queue destination for reprocessing. DLQs are used by Amazon SQS to automatically move messages that were not processed correctly by consumer applications.

History of Queues

Message queues play a crucial role in modern application architectures, enabling asynchronous and message-based communication between services. They help decouple services, ensure resilience to temporary service failures, prioritize message processing, and scale worker nodes. Message queues are widely used in event-driven architectures.

The concept of asynchronous message exchange has been around for decades, with early implementations like TCAM for OS/360 and later IBM MQ Series and Java Messaging Service (JMS). Amazon SQS was launched by AWS on July 12, 2006, as a highly scalable and reliable queuing service with a simple concurrency model.

In 2014, dead-letter queues (DLQs) were introduced in Amazon SQS to prevent messages that failed to be processed from blocking the queue indefinitely. When a message exceeds the maximum receive count specified for the queue, it is moved to the associated DLQ for analysis and debugging. DLQs are typically monitored using alarms to notify system administrators. Messages are commonly moved to the DLQ due to formatting errors, bugs in consumer applications, or prolonged processing times.

At AWS re:Invent 2021, AWS introduced dead-letter queue redrive in the Amazon SQS console, allowing users to easily reinject messages from the DLQ back to the original queue with a few clicks. Now, with the new APIs, you can programmatically handle the redrive process without relying on manual actions. This improves scalability, reduces the risk of human error, and enables automation.

To demonstrate the new API, the provided example showcases a command-line demo. It begins by creating two queues: a dead-letter queue (DLQ) and an application queue. The redrive policy is set on the application queue to send messages to the DLQ after three delivery attempts. A message is then posted to the application queue, followed by three receive attempts without deleting the message, simulating a crash in the consumer application. After three attempts, the message is moved to the DLQ.

To programmatically redrive the message back to the original queue, the API command “start-message-move-task” is used on the DLQ. The status and details of move tasks can be checked using the “list-message-move-tasks” command. Once the redrive task is completed, the application can consume the message again from the original queue.

These DLQ redrive APIs are available in all commercial Regions where Amazon SQS is available.

See Code