Remotely Unlocking ZFS Pools with ZFSBootMenu

The Problem & The Solution

ZFS (Zettabyte File System) is renowned for its strong data integrity, supported by comprehensive checksumming and advanced snapshotting. However, adding encryption introduces a critical operational dependency: the requirement of physical access. When ZFS pools are protected by passwords, maintenance requires physically connecting a keyboard and monitor, causing significant logistical inconvenience often resolved only via expensive KVM switches.

This guide presents a stable, validated solution derived from my own testing. By leveraging ZFSBootMenu, we can enable remote unlocking of encrypted ZFS datasets during the boot process. This removes reliance on physical presence for routine management, allowing you to manage your system reliably whether you are at your desk or across town. Specifically, this article addresses how to enable remote unlocking of ZFS datasets using ZBM, but it does not cover the required encryption key caching within second-stage initramfs.

Configuration Context

Recent changes to the official ZBM build container invalidated previous approaches. This guide details a robust solution based on practical experience for making remote access a part of your root-on-ZFS boot sequence. If you are comfortable managing such an advanced setup, proceed with these steps.

System Prerequisites

This setup is advanced and assumes familiarity with Linux kernel booting processes and ZFS fundamentals. Please ensure these requirements are met before proceeding:

• Root-on-ZFS System: Your operating system must be installed on ZFS, not traditional partitions.
• Custom Boot Structure Knowledge: You need to know how to replace your existing bootloader with ZFSBootMenu (ZBM) if you aren't already using it.
• Target Layout: Your system must utilize a layout where the kernel and initramfs reside within /efi inside an encrypted dataset while ZBM resides on an unencrypted FAT32 /boot partition

Once implemented, the boot sequence will flow as follows:

1. ZBM initiates
2. System attempts to configure network interfaces via DHCP
3. Dropbear service starts
4. User decrypts datasets via SSH or physical console
5. User selects which dataset/kernel+initramfs to boot

Implementation Steps

Step 1: Install necessary tools

This section requires distribution-specific commands, which depend on your operating system's package manager. Please adjust these commands based on your specific Linux distribution.

sudo pacman -S mkinitcpio dropbear dhclient efibootmgr
sudo yay -S zfsbootmenu

Step 2: Configure ZFSBootMenu

This file is the central brain, controlling how ZFSBootMenu is built with generate-zbm

Components:
  Enabled: false
EFI:
  Enabled: true
  ImageDir: /efi/EFI/ZBM
  Versions: false
Global:
  BootMountPoint: /efi
  InitCPIO: true
  InitCPIOConfig: /etc/zfsbootmenu/mkinitcpio.conf
  InitCPIOHookDirs:
  - /etc/zfsbootmenu/initcpio
  - /usr/lib/initcpio
  ManageImages: true
Kernel:
  Prefix: VMLINUZ
  CommandLine: ro quiet loglevel=0 zbm.timeout=-1
  Path: /boot/vmlinuz-linux-lts

Step 3: Define Modules and Hooks

This file directs the build process of your initramfs, ensuring all necessary tools are bundled and, crucially, that our custom scripts run in the correct order.

# vim:set ft=sh
MODULES=()
BINARIES=(ip dhclient dhclient-script timeout efibootmgr)
FILES=()
HOOKS=(base udev modconf block filesystems keyboard rclocal dropbear zfsbootmenu)
rclocal_hook=/etc/zfsbootmenu/initcpio/rc.local

Step 4: Initialize Networking

This is my custom script, designed for simplicity and resilience in an early boot environment. It brings up all active network interfaces (excluding the loopback interface) and attempts DHCP on them with a conservative timeout. The 10-second timeout ensures that if the system fails to establish network connectivity, it won't hang indefinitely, allowing for potential physical recovery via console access.

#!/bin/sh

IFACES=$(ip -o link show | awk -F': ' '$2 != "lo" {print $2}')

for IFACE in $IFACES; do
    ip link set dev "$IFACE" up || exit 1
done

timeout 10 dhclient -1 $IFACES

Step 5: SSH Hooks

Dropbear was chosen as the SSH daemon to facilitate remote access. Rather than creating custom hooks, I utilized those sourced from ahesford's mkinitcpio-rclocal and mkinitcpio-dropbear forks on GitHub. They are responsible for populating your initramfs with the necessary cryptographic keys. It intelligently checks if existing OpenSSH keys can be converted to Dropbear format; otherwise, it generates new ones.

#!/usr/bin/ash

run_hook ()
{
  [ -d /dev/pts ] || mkdir -p /dev/pts
  mount -t devpts devpts /dev/pts

  echo "Starting dropbear"
  # Configuration processing and dropbear execution is done in a subshell
  # to prevent dropbear.conf from inadvertently clobbering important variables
  # or terminating the shell that is running /init.
  (
    # Load configuration options (currently only default listen address)
    [ -r /etc/dropbear/dropbear.conf ] && . /etc/dropbear/dropbear.conf
    /usr/sbin/dropbear -E -s -j -k -p "${dropbear_listen:-22}"
  )

}

run_cleanuphook ()
{
    umount /dev/pts
    rm -R /dev/pts
    if [ -f /var/run/dropbear.pid ]; then
        kill `cat /var/run/dropbear.pid`
    fi
}

Security Caveat: I used a virtual machine during my setup process primarily for convenience, separating the ZBM deployment environment from the live production OS. The resulting ZBM host and authorized keys are stored unencrypted in the initramfs. If physical access to the machine is compromised, an attacker might exploit this vector for future password extraction through passive or active Man-in-the-Middle (MITM) attacks.

Step 6: Generate ZBM and create boot entry

Once all configurations and hooks are in place, you must generate the final ZFSBootMenu image and instruct UEFI to load it.

generate-zbm -d
efibootmgr -c -d "$DISK" -p "$BOOT_PARTITION" \
  -L "ZFSBootMenu" \
  -l '\EFI\ZBM\VMLINUZ.EFI'

Contact

Looking to explore my public projects, or simply discuss the topic of this article? Choose how you'd like to connect below.