ZFS Zpool Encryption with Geli on FreeBSD

I recently set up a server at my buddies house for remote, off-site, backups using ZFS send/recv. Since SmartOS is lacking any sort of encryption for ZFS I instead used FreeBSD with geli. This way, I get the benefits of ZFS incremental send and receive for doing backups, with the security of knowing my data is encrypted on disk.

To ensure maximum security, I made sure the passphrase for unlocking the drives was not stored anywhere on the server itself, and instead would require manual intervention on my part to decrypt the drives anytime the server is rebooted. To do this, I created 2 pools:

zroot - the main pool, mounted at /, unencrypted
paper - the pool used for backups, mounted at /paper, encrypted

The server will boot into the operating system automatically, and from there I must run a script to unlock the drives and import the paper pool.

PS: The zpool is called paper because the server is called paperdyne. That name was cleverly derived from dataDyne (my storage server) and @papertigerss (my buddy whose house this server now lives).

Setup

Install FreeBSD

The first step, of course, is to install FreeBSD. Go to the website, make a bootable USB (or DVD, or CD, or whatever) and install it.

To make my life easier I disconnected all drives but the 2 I wanted for the operating system (250GB drives mirrored) when installing the OS. After it was installed, I rebooted with all of the drives connected to get prepped for the next step.

Encrypt Each Drive Individually

Geli works by encrypting at the hard drive level. The idea is to encrypt each drive, and then create a zpool with the encrypted drives as the devices.

First, get a list of all drives attached to the server and pick out the ones you want use.

# camcontrol devlist
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 0 lun 0 (pass0,da0)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 1 lun 0 (pass1,da1)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 2 lun 0 (pass2,da2)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 3 lun 0 (pass3,da3)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 4 lun 0 (pass4,da4)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 5 lun 0 (pass5,da5)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 6 lun 0 (pass6,da6)
<ATA SAMSUNG HD154UI 1118>         at scbus0 target 7 lun 0 (pass7,da7)
<WDC WD2500AAKX-753CA1 19.01H19>   at scbus4 target 0 lun 0 (ada0,pass8)
<WDC WD2500AAKX-001CA0 15.01H15>   at scbus6 target 0 lun 0 (ada1,pass9)

The ATA SAMSUNG HD154UI 1118 are the drives I am using. To encrypt them, we first have to make some random keyfiles that will be used for each device.

All of the key files will be stored in /root/geli (can be anywhere though of course).

chmod 700 /root
cd /root
mkdir geli
cd geli

To create the keys, you have to run this command for each drive you want to encrypt:

dd if=/dev/random of=/root/geli/da0.key
dd if=/dev/random of=/root/geli/da1.key
...

Or, use this loop (these commands assume you are using bash as your shell).

for f in da{0..7}; do
    dd if=/dev/random "of=./$f.key" bs=64 count=1
done

Now, each da* device will have a corresponding key in /root/geli. The next step is to initialize each device with the newly generated keys.

for f in da{0..7}; do
    geli init -s 4096 -K "/root/geli/$f.key" "/dev/$f"
done

Each invocation of this will require a passphrase - 8 devices with dual passphrase prompts is 16 password prompts! Luckily this is the only time you have to do this (well, except for the next step). Use a strong password and store it in a password manager (or in your head) - I recommend using the same password for each device to make unlocking them easier, but that isn’t strictly necessary.

Attach the encrypted drives

Each drive is located at /dev/<drive>. When decrypted there will be a new device for use at /dev/<drive>.eli. To attach the drives, use the following commands:

for f in da{0..7}; do
    geli attach -k "/root/geli/$f.key" "/dev/$f"
done

This will prompt you for the newly created password 8 times… this will be fixed below in a script to make it so you are only prompted once.

To verify this process worked, you can list the newly attached devices

# ls /dev/*.eli
/dev/da0.eli    /dev/da1.eli    /dev/da2.eli    /dev/da3.eli    /dev/da4.eli    /dev/da5.eli    /dev/da6.eli    /dev/da7.eli

Create a zpool

Finally, now that all devices are decrypted and attached, create a new zpool! Feel free to use any scheme you’d like - I chose raidz for 8 devices.

zpool create paper raidz1 /dev/da{0..7}.eli

Mounting on Reboot

The server will boot automatically after a reboot, but the drives will still be encrypted. You can adapt and use the script below to make it easy to decrypt and attach the drives, as well as import the zpool.

#!/usr/bin/env bash
zpool='paper'
devices=(da0 da1 da2 da3 da4 da5 da6 da7)

read -s -p 'password: ' pass
echo
for name in "${devices[@]}"; do
    echo "mounting: $name"
    echo -n "$pass" | geli attach -j - -k "/root/geli/$name.key" "/dev/$name" || exit 1
done
echo "import: $zpool"
zpool import -R "$zpool"

I have this script in my home directory and call it whenever the server is rebooted

# ./mount-paper
password:
mounting: da0
mounting: da1
mounting: da2
mounting: da3
mounting: da4
mounting: da5
mounting: da6
mounting: da7
import: paper

Done

Anything stored on /paper will be encrypted. If the server is rebooted and the password is not given, that data should be safe for a long time.