Migrating a ZFS RAIDZ2 pool to multiple mirror pools

2021-05-01

I've had a NAS with ~43.5TB of storage for a while now. However, in the 2-3 years I've had it I only used 1.77TB of it. Since I don't intend to use the space for anything else soon it is best to split the 6 drive pool in 3 pools of 2 drives each. These pools can then be stored at different locations, which is always a good idea in case a disaster happens (e.g. a building catches fire).

Making a backup

Before messing with the current pool, mirror all of it to a spare drive so that if something goes wrong in the process, the data can still be quickly recovered.

# Creates a new pool. Generally you should use /dev/disk/by-id/*, but since
# this is temporary it doesn't matter much.
zpool create -O compression=lz4 tank_backup /dev/sdX

# If you have another pool you can use instead of a spare drive, it is also
# possible to create a new datatset
zfs create tank/tank_backup

Then, send the data from the main pool to the current pool

# Run this on the receiving machine. (only do this on LAN! the data is not
# encrypted).
nc -l -p 47424 | zfs recv -v tank/tank_backup

# Run this on the sending machine
zfs snapshot -r tank@migrate
zfs send -Rvc tank@migrate | nc 192.168.2.20 47424

# If you already sent a previous snapshot, you can use this instead
zfs send -Rvc -i tank@previous tank@migrate | nc 192.168.2.20 47424

Create a new 2 disk mirror pool

If you are certain the data has been correctly mirrored, you can bring the old pool offline.

zpool export tank

Then, create a new pool:

zpool create -f -O compression=lz4 tank_a mirror /dev/diskid/DISK-a /dev/diskid/DISK-b

Bring the old pool back online

zpool import tank

If you run zpool status now, you should get something like this:

  pool: tank_a
 state: ONLINE
  scan: none requested
config:

    NAME                      STATE     READ WRITE CKSUM
    tank_a                    ONLINE       0     0     0
      mirror-0                ONLINE       0     0     0
        diskid/DISK-aaaaaaaa  ONLINE       0     0     0
        diskid/DISK-bbbbbbbb  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-4J
  scan: resilvered 476K in 0 days 00:00:01 with 0 errors on Sat May  1 00:04:20 2021
config:

    NAME                      STATE     READ WRITE CKSUM
    tank                      DEGRADED     0     0     0
      raidz2-0                DEGRADED     0     0     0
        7098730037166419494   FAULTED      0     0     0  was /dev/diskid/DISK-aaaaaaaa
        9705479084576219043   FAULTED      0     0     0  was /dev/diskid/DISK-bbbbbbbb
        diskid/DISK-cccccccc  ONLINE       0     0     0
        diskid/DISK-dddddddd  ONLINE       0     0     0
        diskid/DISK-eeeeeeee  ONLINE       0     0     0
        diskid/DISK-ffffffff  ONLINE       0     0     0

errors: No known data errors

Mirror the data

You can send the data from the old pool to the new pool now.

zfs send -Rcv tank@migrate | zfs recv -F tank_a

When that is done, check if the data has been correctly mirrored. Finally, destroy the old pool, create the new pools and mirror the data to both.

zpool destroy tank
zpool create -O compression=lz4 tank_b mirror /dev/diskid/DISK-c /dev/diskid/DISK-d
zpool create -O compression=lz4 tank_c mirror /dev/diskid/DISK-e /dev/diskid/DISK-f
tmux new-session -ds mirror -n 1 'zfs send -Rcv tank_a | zfs recv -F tank_b'
tmux new-window -dt mirror -n 2 'zfs send -Rcv tank_a | zfs recv -F tank_c'

# May be useful if some of the mountpoints are shared (e.g. /home)
zfs set canmount=off tank_b
zfs set canmount=off tank_c