Insight: From Snapshots to S3—A Smarter Way to Offload Redshift Data
Insight: From Snapshots to S3—A Smarter Way to Offload Redshift Data
When Compliance Meets Cost Pressure
Imagine you’re managing a mature Redshift deployment that’s been running for years. Some data is critical, some is rarely touched, but it all needs to stick around for compliance, audit, or regulatory reasons. The default solution? Snapshots. They’re easy, they’re automatic, and they live in S3.
But there’s a catch: Redshift snapshots are managed by AWS and stored in the S3 Standard tier. You can’t move them to cheaper storage like Glacier or Deep Archive. And while snapshots are great for point-in-time restore, they’re expensive if all you really need is a cold, readable copy.
Imagine you’re managing a mature Redshift deployment that’s been running for years. Some data is critical, some is rarely touched, but it all needs to stick around for compliance, audit, or regulatory reasons. The default solution? Snapshots. They’re easy, they’re automatic, and they live in S3.
But there’s a catch: Redshift snapshots are managed by AWS and stored in the S3 Standard tier. You can’t move them to cheaper storage like Glacier or Deep Archive. And while snapshots are great for point-in-time restore, they’re expensive if all you really need is a cold, readable copy.
Exporting Tables—Not Snapshots
If your primary goal is to retain data affordably rather than restore clusters quickly, exporting your tables might be the better path. Once exported to S3, that data can live under your control—with lifecycle policies that drop storage costs automatically over time.
Of course, manually writing UNLOAD commands for thousands of tables isn’t scalable. So we built a tool to help.
If your primary goal is to retain data affordably rather than restore clusters quickly, exporting your tables might be the better path. Once exported to S3, that data can live under your control—with lifecycle policies that drop storage costs automatically over time.
Of course, manually writing UNLOAD commands for thousands of tables isn’t scalable. So we built a tool to help.
Meet the Redshift Table Export Script
This Python script connects to your Redshift cluster, automatically discovers all user tables in a schema, and generates a matching UNLOAD statement for each. Each table is exported to its own folder in S3, and everything is driven by environment variables for flexibility.
The tradeoff? You lose snapshot functionality. These exports are not restorable as-is. They’re static copies—perfect for audit, long-term safekeeping, and S3 tiering, but not for point-in-time rollback.
Here’s the public Gist:
This Python script connects to your Redshift cluster, automatically discovers all user tables in a schema, and generates a matching UNLOAD statement for each. Each table is exported to its own folder in S3, and everything is driven by environment variables for flexibility.
The tradeoff? You lose snapshot functionality. These exports are not restorable as-is. They’re static copies—perfect for audit, long-term safekeeping, and S3 tiering, but not for point-in-time rollback.
Here’s the public Gist:
Use Cases
Final Thoughts
Snapshots have their place. But if you're looking to cut costs without cutting corners, sometimes it makes more sense to export and control the storage yourself. This script helps make that transition easier, safer, and much more scalable.
If you've got 2500 tables and 200TB to manage—we hope it saves you a weekend or two. Let us know how it goes.
- Archiving historical data before downsizing a cluster
- Meeting data retention policies without burning money on snapshot storage
- Creating a lightweight offload of infrequently accessed tables
Final Thoughts
Snapshots have their place. But if you're looking to cut costs without cutting corners, sometimes it makes more sense to export and control the storage yourself. This script helps make that transition easier, safer, and much more scalable.
If you've got 2500 tables and 200TB to manage—we hope it saves you a weekend or two. Let us know how it goes.
Need AWS Expertise?
We'd love to help you with your AWS projects. Feel free to reach out to us at info@pacificw.com.
Written by Aaron Rose, software engineer and technology writer at Tech-Reader.blog.
Comments
Post a Comment