S3 GFS Retention

Home Assistant makes daily backups. Mine include metrics, which means about 1.8 GB per day. After six months, that was roughly 350 GB sitting in S3. I wanted to keep the safety of a long history without paying for every single day forever.

So I wrote a small script that applies a Grandfather-father-son (GFS) retention policy to any S3 bucket with timestamped objects. It keeps a rolling set of daily, weekly, and monthly backups and prunes the rest.

The Script: GFS Retention for Any S3 Backup

The script lives here: MBW.Tool.S3GFSRetainer
The Readme has the full setup, examples, and rationale: Readme

The core idea is simple:

• Execute the script regularly, possibly even from a serverless system like AWS Lambda
• Extract a timestamp from each object key using a regex.
• Group objects that belong to the same backup run.
• Keep the newest N daily, weekly, and monthly groups.
• Delete old groups once the minimum retention is satisfied.

Because it is just a regex and a timestamp format, it works for more than Home Assistant. Any tool that writes timestamped files to S3 can use the same pattern.

These are the two required environment variables:

• S3_BUCKET: S3 bucket name to scan.
• S3_GFS_REGEX: Regex that captures the timestamp in a single group.

And a couple of optional ones I use a lot:

• S3_GFS_KEEP_DAILY: How many daily backup groups to keep.
• S3_GFS_KEEP_MONTHLY: How many monthly backup groups to keep.
• S3_GFS_DRY_RUN: Start with this set to true to test without deleting.

There are more knobs in the Readme, including the timestamp format and the minimum remaining group count.

My Setup: S3 -> SQS -> Lambda

I run the script as a Lambda, triggered by S3 events that are fan-out to an SQS queue. That keeps the Lambda decoupled from the S3 event stream and lets me retry cleanly.

The flow is:

• S3 bucket receives a new backup object.
• S3 sends a notification to SQS.
  • I could send directly from S3 -> Lambda, but I use SQS to introduce an artificial 2 minute delay. This is to ensure the metadata file from HA has arrived.
• Lambda processes the queue and runs the retention script.

The Lambda is idempotent and only removes older groups, so it is safe to run on every new backup. The Readme has the exact IAM permissions and handler settings I used.

Cost Notes and Storage Classes

I ran three rough scenarios to sanity-check costs. These are back-of-napkin numbers for eu-central-1, 1 object per day at 1.8 GB, and they ignore request, retrieval, and transfer charges. Your mileage will vary, and this is just for comparison.

Constants and constraints I used:

• Storage prices (approx): Standard $0.0245/GB-month, Glacier Flexible $0.0036/GB-month, Deep Archive $0.0010/GB-month.
• Minimum storage durations: some classes require having the object for at least N days, such as: Glacier Flexible at 90 days and Deep Archive at 180 days.
• Objects land in Standard by default unless you set a storage class.
• Transition costs exist (and minimum-storage penalties apply on early deletes), but they are negligible at this scale, as we only make one object each day.

Scenario 1: All Standard, keep 180 days - baseline without any effort

• Policy: Standard only, keep every daily for 180 days, then expire.
• Effective storage: 180 objects * 1.8 GB = ~324 GB.
• Rough cost: 324 * $0.0245 = ~$7.9/month (about $95/year).

Scenario 2: Standard then Glacier Flexible at day 30, keep 180 days - baseline with very simple lifecycle policy

• Policy: Standard for 30 days, then transition to Glacier Flexible, expire at 180 days.
• Effective storage: ~54 GB in Standard + ~270 GB in Glacier Flexible.
• Rough cost: (54 * $0.0245) + (270 * $0.0036) = ~$2.3/month (about $28/year).
• Notes: keeps all dailies, lower cost, and the 90-day minimum is naturally satisfied.

Scenario 3: GFS + Deep Archive after ~16 days, keep monthlies for 12 months

• Policy: GFS prunes most dailies, keeps up to two weeklies (expire around day 15), then only monthlies survive. Those survivors transition to Deep Archive at ~16 days and expire at 12 months.
  • As keeping 1 object each month is practically free, we can expand to a full 12 months of monthlies at (almost) no added cost.
• Effective storage: ~15 recent objects in Standard (~27 GB) + ~11.5 monthlies in Deep Archive (~21 GB).
• Rough cost: (27 * $0.0245) + (21 * $0.0010) = ~$0.68-0.70/month (about $8-9/year).
• Notes: Deep Archive minimum storage is 180 days.

Quick comparison:

Diagnostics

The script outputs logs, which can be seen in the console. If you’re in Lambda like me, logs are written to CloudWatch.

Things to watch out for

There is an issue on the HA issue tracker on having the S3 backup be able to set a storage class. If this is not possible, all objects arrive as Standard storage, even though IA might make more sense.

Wrap-up

I now keep a meaningful backup history without storing hundreds of daily copies forever. The script is tiny, the retention logic is explicit, and the pipeline is easy to reuse for any S3-based backup workflow.

If you want to try it, start with the Readme and a dry run. Once the regex matches your backup naming scheme, the rest is just policy.