Soft delete, a crucial new data protection feature for Cloud Storage, works with all features and workloads. It lets you keep and restore recently erased data at enterprise scale, protecting against inadvertent and malicious erasure. Soft deletion allows your organisation to proceed faster while “pruning” old data because it gives an undo mechanism for mistakes.
This blog gives you the tools and insights to optimize your soft delete settings, even at scale, to protect your business-critical data.
Soft delete works and is billed how?
Soft deletion keeps removed things in a hidden state for the bucket’s soft deletion retention term instead of permanently deleting them. Run a restoration to copy soft-deleted objects back to live state.
Google made soft deletion with a seven-day retention length the default for new and existing buckets. Soft delete is enabled by default since inadvertent deletions are prevalent and Cloud Storage stores business-critical data. Besides the seven-day default, you can choose any amount between 7 and 90 days or disable the feature.
The storage class of recently deleted objects determines soft delete billing. Soft deletion usually only raises bills by a few percentage points, which should be a decent bargain for the security it gives. Enabling soft deletion on buckets with a lot of short-lived (often erased) data can increase pricing because an item deleted after an hour would be invoiced for the hour it was live plus seven days of soft delete consumption.
Soft delete How valuable is your data?
To get to a position where soft deletion protects you against data deletion threats with the lowest economic impact, ask yourself these three questions:
How vital is my company’s data?
Do they save temporary objects or media transcodes that can be recovered? These situations may not warrant soft delete protection. Do companies store data that could compromise my business and customer connections if lost? Soft deletion could protect this vitally.
They have what level of data protection?
If Cloud Storage is the only copy of your business-critical data, soft delete protection is more necessary than if you store long-term backups in another Google Cloud region, on-perm, or with another cloud provider.
They can afford how much data protection?
Soft delete is cheaper than business backups, but your deletion rates can still affect pricing. Soft erase protects business data used by your workloads, thus companies recommend comparing it to your Google Cloud bill rather than just storage. Left soft delete on all your buckets may just increase your cloud price by a single digit percentage, which may be worth it to protect against inadvertent and malicious deletions.
After deciding where and how much to use it, the next stages rely on your architectural decisions and your organization’s cloud complexity. This blog will discuss how to measure soft delete’s impact and act on that information, starting with bucket-level metrics, then bucket-level settings within a project, Terraform management, and organizational-level management.
Assessing bucket-level impacts
Cloud Monitoring metrics and the Metrics Explorer estimate bucket-level soft delete charges. To determine which buckets are more and less expensive to soft delete, examine a few buckets from different datasets.
Action in a project
A storage administrator implementing it’s settings for a project may wish to carefully review your bucket list and decide what should be protected and what can go without soft delete based on business information. If you have more buckets, utilize the aforementioned analytics to identify those that exceed a billing effect threshold (e.g. 20%) and disable soft deletion on them.
To help, they created a soft delete billing impact Python script on Github that provides a list of buckets in a project that exceed the proportion of billing effect you define, taking into account object storage classes. The script can adjust soft delete policies depending on comparable cost thresholds.
Using Terraform
With an orchestration layer like Terraform, soft delete should be as easy as changing templates and setting retention durations for each task. This may involve establishing new templates for short-lived data to prevent soft delete for buckets built from them. Terraform may update existing buckets to match templates and build new buckets with your parameters after you define them.
Acting across a broad company
For a huge company with thousands of projects and millions of buckets that doesn’t use an orchestration layer, a manual approach is impractical and requires scaled decision-making. If so, bucket-level metrics and how to operate inside a project. They’ll apply these methods to organizations in this part. This section requires an updated cloud CLI, which google presume you have installed.
Assess
If you can act on bucket-level configurations across your organisation, you can use the project-level analysis above to find any buckets that exceed your impact threshold. You might also set all buckets in your organisation to 0d or 14d.
Act
Iterate through all your projects and alter the buckets to update the soft delete policy for all your buckets.
By following the best practices in this blog and utilizing existing tools and controls, you may secure business-critical data with soft delete while minimizing billing impact.
