Efficiently Archiving Scaleway Block Volume Snapshots to Object Storage via the CLI
Data storage costs are a critical component of managing cloud resources. To optimize these costs, it is wise to archive old snapshots from costlier block storage to more economical object storage solutions. In this post, we explore how you can perform this archival using the Scaleway command-line interface (CLI), reaping savings and promoting a streamlined and automatable process.
I host an couple of small servers on Scaleway, and typically run this type of sequence before weekly updates, as part of an ansible role.
Why Shift Snapshots to Object Storage?
- Cost Savings: Object storage is more economical than block storage, especially for data that requires infrequent access.
- Further Savings: Transitioning archived data to glacier class storage can reduce costs even more.
- Automation: Repetitive tasks like snapshot archival can be automated, ensuring consistency and reliability, particularly before scheduled updates or maintenance.
Setting Up the Scaleway CLI Container
Scaleway conveniently provides a docker container for their CLI, which of course can also be run using Podman - my preference. This nicely avoids cluttering your system.
Check out the doc: https://github.com/scaleway/scaleway-cli/blob/master/docs/docker.md
Keeping the CLI Up-to-Date
Ensure that you are using the most up-to-date version of the Scaleway CLI. Run the following command to pull the latest image:
> $ podman pull docker.io/scaleway/cli:latest
Trying to pull docker.io/scaleway/cli:latest...
Getting image source signatures
Copying blob 573149b0d1be done |
Copying blob 659d66d51139 skipped: already exists
Copying blob 7d848d073608 done |
Copying config e02e6ff9b7 done |
Writing manifest to image destination
e02e6ff9b7e20c735d95c3e6440217ae01dd4add0f306d800f3e177652bf8805
The above command fetches the latest version of the Scaleway CLI Docker image, ensuring you have all the recent features and bug fixes.
Snapshot Archival Process
Here’s how we carry out the snapshot archival:
Listing Volumes
Firstly, list all the volumes to determine which ones will have snapshots created or archived:
> $ scw instance volume list
You will receive a list of volumes with IDs, names, sizes, and properties - crucial data for snapshot management.
Creating New Snapshots
To create a snapshot of a volume, use the command:
> $ scw instance snapshot create name=generic-snapshot-name volume-id=your-volume-id
Replace generic-snapshot-name and your-volume-id with your specific snapshot name and the ID of the volume you want to snapshot.
Listing Snapshots
Review your current snapshots to manage them effectively:
> $ scw instance snapshot list
This command displays an informative list with details like snapshot IDs, names, sizes, and the current state.
If successful, the ouput should be similar to this:
> $ scw instance snapshot list
ID NAME ORGANIZATION PROJECT TAGS VOLUME TYPE SIZE STATE CREATION DATE MODIFICATION DATE ZONE ERROR REASON
f3a6247c-8200-4555-85ba-90e8d77aa434 testvolume-202310231240 9aa34dfa-b999-999a-b0ec-8d5dc81e3820 9aa34dfa-b999-999a-b0ec-8d5dc81e3820 [] b_ssd 20 GB available 1 day ago 23 hours ago fr-par-1 -
e653d02f-e4ef-44f5-bf36-669d44ab99a1 System volume-202310231240 9aa34dfa-b999-999a-b0ec-8d5dc81e3820 9aa34dfa-b999-999a-b0ec-8d5dc81e3820 [] b_ssd 15 GB available 1 day ago 23 hours ago fr-par-1 -
Exporting Snapshots to Object Storage
Move older snapshots to an object storage bucket with the following command:
> $ scw instance snapshot export zone=your-zone snapshot-id=your-snapshot-id bucket=your-bucket-name key=name-of-snapshot-in-object-storage.qcow2
Replace placeholders with the appropriate snapshot ID, zone, bucket name, and the desired object key.
The ouput status will show pending until the snapshot has successfully been moved. For instance:
> $ scw instance snapshot export zone=fr-par-1 snapshot-id=f3a6247c-9999-999-85ba-90e8d77aa434 bucket=testbucket key=testvolume-202310231240.qcow2
Task.ID 4b739f95-e232-4d70-9e81-213070f32cae
Task.Description export_snapshot
Task.Progress 0
Task.StartedAt now
Task.Status pending
Task.HrefFrom /snapshots/f3a6247c-9999-999-85ba-90e8d77aa434/export
Task.HrefResult https://s3.fr-par.scw.cloud/testbucket/testvolume-202310231240.qcow2
Task.Zone
Deleting Redundant Snapshots
After successful export, delete the old snapshots from block storage to start saving costs:
> $ scw instance snapshot delete snapshot-id=your-snapshot-id zone=your-zone
This command will remove the snapshot from your list, eliminating the costs associated with it.
Conclusion
By transitioning snapshots from block to object storage, and potentially further into glacier storage, you can optimize your cloud storage costs effectively. Automation through the CLI not only saves money but also time and adds reliability to your data management operations.