Multi-device and RAID1 with btrfs

I have been using btrfs, a modern modern copy on write filesystem for Linux, since many years now. But only recently I realized how amateurish my usage has been. Over the last day I switched to multiple devices and threw in a RAID1 level at the same time.

For the last years, I have been using btrfs in a completely naive way, simply creating new filesystems, mounting them, moving data over, linking the directories into my home dir, etc etc. It all became a huge mess over time. I have heard of “multi-device support“, but always thought that this is for the big players in the data centers – not realizing that it works trivially on your home system, too. Thanks to an article by Mark McBride I learned how to better use it!

Btrfs has an impressive list of features, and is often compared to (Open)ZFS (btw, they domain openzfs.org has a botched SSL certificate .. umpf, my trust disappears even more) due to the high level of data security. I have been playing around with the idea to use ZFS for quite some time, but first of all it requires compiling extra modules all the time, because ZFS cannot be included in the kernel source. And much more, I realized that ZFS is simply too inflexible with respect to disks of different sizes in a storage pool.

Btrfs on the other hand allows adding and removing devices to the “filesystem” on a running system. I just added a 2TB disk to my rig, and called:

btrfs device add /dev/sdh1 /

and with that alone, my root filesystem grew immediately. At the end I have consolidated data from 4 extra SSDs into this new “filesystem” spanning multiple disks, and got rid of all the links and loops.

For good measure, and since I had enough space left, I also switched to RAID1 for this filesystem. This again, surprisingly, works on a running system!

btrfs balance start -dconvert=raid1 -mconvert=raid1 /

Here, both data and metadata are mirrored on the devices. With 6TB of total disk space, the balancing operation took quite some time, about 6h in my case, but finished without a hiccup.

After all that, the filesystem now looks like this:

$ sudo btrfs fi show /
Label: none  uuid: XXXXXX
	Total devices 5 FS bytes used 2.19TiB
	devid    1 size 899.01GiB used 490.03GiB path /dev/sdb3
	devid    2 size 489.05GiB used 207.00GiB path /dev/sdd1
	devid    3 size 1.82TiB used 1.54TiB path /dev/sde1
	devid    4 size 931.51GiB used 649.00GiB path /dev/sdf1
	devid    5 size 1.82TiB used 1.54TiB path /dev/sdc1

and using btrfs fi usage / I can get detailed information about the device usage and status.

Stumbling blocks

You wouldn’t expect such a deep rebuilding of the intestines of a system to go without a few bumps, and indeed, there are a few:

First of all, update-grub is broken when device names are used. If you have GRUB_DISABLE_LINUX_UUID=true, so that actual device nodes are used in grub.cfg, the generated entries are broken because they list all the devices. This comes from the fact that grub-mkconfig uses grub-probe --target=device / to determine the root device, and this returns in our case:

# grub-probe --target=device /
/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1

and thus the grub config file contains entries like:

...
menuentry 'Debian GNU/Linux ...' ... {
  ...
  linux	/boot/vmlinuz-5.7.0-rc7 root=/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1 ro 
}

This is of course an invalid entry, but fortunately grub still boots, but ignores the rest of the command line options.

So I decided to turn back to using UUID for the root entry, which should be better supported. But alas, what happened, I couldn’t even boot anymore. Grub gave me very cryptic messages like cannot find UUID device and dropping you into the grub rescue shell, then having the grub rescue shell being unable to read any filesystem at all (not even FAT or ext2!). The most cryptic one was grub header bytenr is not equal node addr, where even Google gave up on it.

At the end I booted into a rescue image (you always have something like SystemRescueCD on an USB stick next to you during these operations, right?), mounted the filesystem manually, and reinstalled grub, which fixed the problem, and now the grub config file contains only the UUID for root.

I don’t blame btrfs for that, this is more like we are, after sooo many years, we still don’t have a good boot system 🙁

All in all, a very smooth transition, and at least for some time I don’t have to worry about which partition has still some space left.

Thanks btrfs and Open Source!

8 Responses

  1. On my laptop machines, I don’t have the luxury of multiple devices. But I do use btrfs on my root partition and backup drives. I mainly use btrfs for its snapshot and de-dupe features.

    On the laptop side of things, the only thing I realized was to organize {sub}volumes in a proper manner. For example, I have separate volumes defined for ~/.cache ~/.local etc. This works out well for backups

    • I never used subvolumes, I really need to look into that, too. Why do you use separate volumes for .cache and .local? And do you have any good pointer on btrfs sub/volumes,?

      Anyway, thanks for the hint, I will start reading up on that!

  2. MrCalvin says:

    Ireally don’t understand why smart people use grub? Are you not aware of systemd-boot?? It will give you back 100% control of the boot process

    • Well, it doesn’t make such a huge difference, right? Both need to find the kernels and initrds, and must be activated from efi. What are the advantages you see of gummyboot over grub? And why not refind or any other efi boot loader?

      • MrCalvin says:

        That you have full control of which device you boot from, as you simple specific specify by UUID, VERY simple.
        You might be one of the very few who actually know how to set this specific in grub bloated config-files, but must don’t
        The rest of us can only have the choice of running a wizard-tool and pray to good it hopefully pick the right disk/partition, if it doesn’t, you’re fucked and have to reinstall all over. Just do a google search “grub” and “unbootable”, yaaak!
        One would think this abomination was coded by MS!

        • Ok, on your recommendation I took the challenge: According to the documentation I should call bootctl install, which I did, the output was promising, all fine, all working, all ok.

          Well, reboot showed a different image: suddenly all Linux boot options are gone, and I can only boot into either Windows or firmware setup.

          GREAT! Well done SystemD again!

          Anyway, I booted into Windows, used EasyUEFI to fix the selected boot loader back to Grub, and now I am again in “normal working state”.

          So, very simply put, systemd-boot is as much hosed and broken – if not much more – than Grub. Grub at least tries to find Linux boot images, systemd-boot didn’t do anything like this and left me with a broken system. Congratulations.

Leave a Reply to Norbert Preining Cancel reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>