The Importance of Data Snapshots

Understanding Amazon Data Lifecycle Manager

The use of pre- and post-snapshot scripts integrated in AWS Systems Manager documents is now supported by Amazon Data Lifecycle Manager. These scripts can be used to guarantee that Data Lifecycle Manager-created Amazon Elastic Block Store (Amazon EBS) snapshots are application-consistent. Scripts provide the ability to flush buffered data to EBS volumes, pause and resume I/O operations, and more. AWS are also releasing a series of in-depth blog entries to accompany this launch that walk you through the process of utilizing this functionality with Windows Volume Shadow Copy Service (VSS) and self-managed relational databases.

Overview of Data Lifecycle Manager (DLM)

To summarize, Amazon EBS volume snapshot creation, retention, and deletion can all be automated with the aid of Data Lifecycle Manager. After completing the necessary steps, like tagging your SSM documents, setting up an IAM role for DLM, and onboarding your EC2 instance to AWS Systems Manager, all you need to do is create a lifecycle policy, tag the relevant Amazon Elastic Compute Cloud (EC2) instances, choose a retention model, and let DLM handle the rest. The policies outline what has to be backed up, when it should be done, and how long the pictures must be kept. See my blog post from 2018 titled “New – Lifecycle Management for Amazon EBS Snapshots” for a comprehensive explanation of DLM.

Consistent with Applications Snapshots

Because EBS snapshots are crash-consistent, they show the state of the corresponding EBS volume as of the snapshot creation date. For many kinds of applications, even those that don’t employ snapshots to record the current state of an active relational database, this is adequate. Application consistency requires accounting for pending transactions (waiting for them to complete or causing them to fail), pausing further write operations for a brief period of time, taking the snapshot, and then returning to regular operations.

And that’s the purpose of today’s launch. The instance can now be instructed by DLM to get ready for an application-consistent backup. Pre-snapshot scripts can handle pending transactions, stop the program or database, freeze the filesystem, and flush in-memory data to persistent storage. After that, the post-snapshot script can thaw the disk, restart the application or database, and reload in-memory caches from persistent storage.

You can use this functionality to automate the creation of VSS Backup snapshots in addition to the basic support for custom scripts:

Both pre- and post-scripts

The updated scripts are applicable to DLM instance policies. Assume for the moment that author have drafted a policy that pertains to a single instance and references SSM documents that have pre- and post-snapshot scripts. The following occurs when the policy is applied according to its schedule:

• The SSM document is where the pre-snapshot script gets started.
• Every command in the script is executed, and the success or failure of the script is recorded. If the policy allows it, DLM will try unsuccessful scripts again.
• For EBS volumes connected to the instance, multi-volume EBS snapshots are started; more management is available through the policy.
• Starting with the SSM document, the post-snapshot script executes each command and records the script’s level status (success or failure).

You have control over what happens (retry, continue, or skip) when one of the scripts times out or fails thanks to the options in the policy. In addition to being recorded and published in Amazon CloudWatch metrics and Amazon EventBridge events, the status is also encoded in tags that are automatically applied to every snapshot.

The pre-snapshot and post-snapshot scripts can execute shell scripts, PowerShell scripts, and other operations that are permitted in a command document. The policy’s timeout, which might last anywhere from 10 to 120 seconds, must be completed by the actions.

Starting Out

To create a strong pair of scripts, you will need to have a thorough understanding of your application or database. Your scripts should not just account for the “happy path” in the event of a successful outcome, but also for multiple failure scenarios. A pre-snapshot script ought to, for instance, fork a background task that functions as a backup in the event that the post-snapshot script malfunctions.