At rest encryption for the cloud

Here’s a example with an email server I manage. I recently encrpted the block volume using LUKS. All valuable data (docker volumes, app db, etc.) is stored on this block volume. The volume itself is formated with ZFS. There a lot of reason to use ZFS but a key aspect here is also that it fits well into my backup workflow. Here a quic

Current setup

ubuntu@mailgafr-sgp2 ~> sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 73.9M 1 loop /snap/core22/864 loop1 7:1 0 9.9M 1 loop /snap/canonical-livepatch/248 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 40.8M 1 loop /snap/snapd/20092 loop4 7:4 0 111.9M 1 loop /snap/lxd/24322 loop5 7:5 0 40.9M 1 loop /snap/snapd/20290 loop6 7:6 0 105.8M 1 loop /snap/core/16202 loop7 7:7 0 9.6M 1 loop /snap/canonical-livepatch/246 loop8 7:8 0 74.1M 1 loop /snap/core22/1033 loop9 7:9 0 63.9M 1 loop /snap/core20/2105 sda 8:0 0 50G 0 disk ├─sda1 8:1 0 49.9G 0 part / ├─sda14 8:14 0 4M 0 part └─sda15 8:15 0 106M 0 part /boot/efi sdb 8:16 0 15G 0 disk ├─sdb1 8:17 0 15G 0 part └─sdb9 8:25 0 8M 0 part

ubuntu@mailgafr-sgp2 ~> zfs list NAME USED AVAIL REFER MOUNTPOINT mailgafr-zfsblock 13.5G 601M 25K /mnt/mailgafr-zfsblock mailgafr-zfsblock/docker-c-projects 1.80G 601M 1.07G /mnt/mailgafr-zfsblock/docker-c-projects mailgafr-zfsblock/docker-root 11.5G 601M 1.29G /mnt/mailgafr-zfsblock/docker-root mailgafr-zfsblock/docker-root/0a898399494abc30dcf79b17086e583be9b318c21c68b890775c5a9fbbfe7594 7.28M 601M 7.28M legacy […] mailgafr-zfsblock/docker-root/volumes 6.30G 601M 3.17G /mnt/mailgafr-zfsblock/docker-root/volumes

Why LUKS

  • flexibility (multiple keys, Tang & Clevis)
  • Easily deal with existing data

In practice

Shut down docker

sudo systemctl stop docker
sudo systemctl stop docker.socket
sudo systemctl stop containerd

Detach the old pool

zfs umount -a

If you get an error message saying the pool is still busy, there are a few extra steps to take:

  • disable the docker systemctl services
  • reboot
  • try to umount again

Create the new volume

Use lsblk to identify the new block volume. Then format it and open it.

cryptsetup luksFormat -s 512 -h sha512 -i 10000 /dev/sdc
sudo cryptsetup open /dev/sdb mailgafr-luks-zfs

Replicate and cleanup

zfs snapshot -r mailgafr-zfsblock@transfer
zfs send -R mailgafr-zfsblock@transfer | sudo zfs receive -F mailgafr-luks-zfs
zfs destroy mailgafr-luks-zfs@transfer

Change the mountpoints (if necessary) and mount the pool

zfs set mountpoint=/mnt/mailgafr-luks-zfs mailgafr-luks-zfs
zfs set mountpoint=/mnt/mailgafr-luks-zfs/docker-root mailgafr-luks-zfs/docker-root
zfs set mountpoint=/mnt/mailgafr-luks-zfs/docker-root/volumes mailgafr-luks-zfs/docker-root/volumes
zfs set mountpoint=/mnt/mailgafr-luks-zfs/docker-projects mailgafr-luks-zfs/docker-projects
zfs mount -a

Edit the docker daemon root to the new dictory (if required).

See /etc/docker/daemon.json.

Start docker once again

systemctl start docker.service

And now?

Your block volume is now encrypted at rest. This is not a perfect solution: keys could easily be extracted from RAM by your cloud providers. However, at-rest encryption already protects you from a lot of threats, like disks getting lost with customer data on them (Scaleway), backups exfiltration following a breach, etc.

Next step? Your services won’t start unless you SSH, manually open the LUKS volume and start the docker daemon. Best practices would be to implement tang and clevis to allow remote unlocking. This will be a future post.