Managing your NixOS filesystem (without Disko)

Contents

Introduction ⚓

So, I have a confession to make.

I’m too dumb to use Disko 🔗.

Yes, yes, I know. You can revoke my “NixOS user” card. You can pelt me with snowflakes. My setup isn’t 100% declarative. The horror. But before you make your judgement, just give me 10 minutes to explain what I did instead. And maybe, just maybe, someone else struggling with Disko will see this post and realize there’s another way.

What is Disko? ⚓

Disko is an open source project that extends NixOS (a fully declarative Linux distribution that I’ve posted about several times 🔗) by managing storage devices and partitions.

Ok…what are storage devices and partitions? ⚓

When you install any operating system, not just Linux, you need to install it on some kind of persistent storage device. This is a dedicated bit of hardware that specializes in storing data long-term (i.e. you can retrieve the data even after completely powering down your PC). Unless you’re using a teeny single-board system like a Raspberry Pi, or some specialty hardware, you’re guaranteed to have persistent storage somewhere in your system.

Storage devices have a certain capacity that they support, like 256GB, 512GB, 2TB, etc. You can’t just dump data onto these devices though—you need to prepare them. The first step is to partition them, which means splitting this total capacity into separate “blocks”. This is done for a few reasons:

• It creates logical separators for different functions or data pools. One partition might store data for booting the PC, another might store user files, etc.
• It lets other systems know where to find certain kinds of data, like how UEFI expects to find a partition with bootable operating system images 🔗.

The simplest setup only requires two partitions: an EFI (also called an ESP) partition 🔗 containing instructions on how to boot the system, and a partition that stores all other data: operating system files, your user files, your programs, etc. Most modern operating systems—including most Linux distributions—handle this for you, automatically, without you even knowing. Off-the-shelf PCs like Mac and Windows systems come pre-partitioned, so most people don’t even know it’s a thing.

Here’s the partition layout of the laptop I’m drafting this blog post on. The two partitions are listed at the bottom: 1 GB is allocated to an EFI partition, and the remainder is used up by a single Linux filesystem partition.

aires@Khanda ~ $ sudo fdisk -l /dev/nvme0n1  
Disk /dev/nvme0n1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: MZ9L4256HCJQ-00BMV-SAMSUNG              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: E68222CE-6DBC-4FC2-99AA-9BEBEB4AFB7F

Device           Start       End   Sectors   Size Type
/dev/nvme0n1p1    2048   2050047   2048000  1000M EFI System
/dev/nvme0n1p2 2050048 500117503 498067456 237.5G Linux filesystem

This isn’t the full story, of course. A partition is just a container, a box. In order to put data into the box, you need some way of organizing it first so it doesn’t just end up as one big jumbled mess. This organization system is called a filesystem.

I won’t dive into filesystems, because that’s a whole different topic, but here are the basics: after you create a partition, you format it with a filesystem. This filesystem is what you interact with whenever you create files, install programs, etc. I mean, you don’t interact with it directly, the operating system does, but it’s the top-most layer of this complex storage sandwich. Think of it as the bread that keeps all the inner toppings in place.

What do partitions and filesystems have to do with Disko? ⚓

Simply put, Disko is a tool that creates and manages partitions and filesystems for NixOS systems.

Normally for DIY Linux distros like NixOS and Arch, partitioning is manual. You’re expected to open the right tools, know what kind of partition layout you want, and know how to mount them in the right places, so the installer can work its magic. The installation keeps a record of the partitions you set up (it needs to in order to properly utilize them), but it doesn’t know how to recreate those partitions in case you ever need to reinstall the system. Linux geeks have historically gotten around this problem by documenting their partition layouts themselves, creating shell scripts to automate partition creation, or just YOLOing it and starting over from scratch each time.

But NixOS is different. NixOS is declarative. Ideally, you should be able to rebuild your PC entirely from scratch with your NixOS configuration, storage partitions and all. NixOS doesn’t natively support partition creation, so that’s where Disko comes in.

Here’s an example of a Disko configuration. Don’t worry too much about the details. This creates the same partition layout shown above, only now I can check this configuration into my Git repository and have it be a permanent part of my system configuration.

{
  disko.devices = {
    disk = {
      vdb = {
        device = "/dev/nvme0n1";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              type = "EF00";
              size = "1000M";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}

Great, so why don’t I use Disko? ⚓

If you’re able to use Disko and have it properly create and mount your partitions for you, that’s awesome! By all means, use it! Unfortunately, I wasn’t so lucky.

Like most normal, healthy, well-adjusted adults, I have a favorite filesystem and partition layout (yes, that was sarcasm). Unfortunately, no matter how I tried, I couldn’t get Disko to accept it as a valid layout. This is despite there being an example configuration for what I’m trying to accomplish 🔗. Every time I tried to run Disko, it complained that I didn’t have a root filesystem defined, despite it being right there 🔗. I probably missed something simple that an experienced NixOS user could identify immediately, but after a few days, I gave up. Instead, I made my own reproducible filesystem definition , with blackjack and hookers 🔗.

What I’m using instead of Disko ⚓

Instead of using Disko, I went old school and wrote myself a good old-fashioned Bash script 🔗. The only manual step is creating the two partitions on my storage device before. Then, I let the script work its magic. It does a few things:

1. Formats the first partition as a valid EFI partition.
2. Turns the second partition into a LUKS container so the filesystem is encrypted.
3. Opens the LUKS container and formats it as a BTRFS filesystem 🔗.
4. Creates a bunch of BTRFS subvolumes 🔗. A BTRFS subvolume is kind of a partition-lite. It’s a logical container that can be mounted like a partition, but doesn’t have a fixed size.
5. Re-mounts the BTRFS subvolumes as partitions, so the NixOS installer can write to them.
6. Generates a hardware-configuration.nix file for the current system, but without a filesystem definition. This is because I already have the filesystem defined in my NixOS config (I’ll get to that shortly).

This script doesn’t actually install NixOS, but it does spit out a nixos-install command for convenience.

So what about the filesystems? Where did I set those? Let’s check that out next.

The filesystems module ⚓

I’ve gotten into a bit of a modularization kick with NixOS. In my config, pretty much everything is a module. You just enable or disable it for the hosts you want to use it on, and maybe add some parameters to configure it properly. aux.system.filesystem is a new module I added that defines my preferred partition layout, subvolumes in all.

Note: Although this module is under the aux.system namespace, it’s not actually part of the Auxolotl system template 🔗. It’s just easier for me to add my own modules to this namespace than to maintain two separate ones.

Here’s the module in its entirety (as of this writing):

{ lib, config, ... }:
let
  cfg = config.aux.system.filesystem;

  # LUKS partition will decrypt to /dev/mapper/nixos-root
  decryptPart = "nixos-root";
  decryptPath = "/dev/mapper/${decryptPart}";
in
{
  options = {
    aux.system.filesystem = {
      enable = lib.mkEnableOption (lib.mdDoc "Enables standard BTRFS subvolumes and parameters.");
      partitions = {
        boot = lib.mkOption {
          type = lib.types.str;
          description = "The ID of your boot partition. Use /dev/disk/by-uuid for best results.";
          default = "";
        };
        luks = lib.mkOption {
          type = lib.types.str;
          description = "The ID of your LUKS partition. Use /dev/disk/by-uuid for best results.";
          default = "";
        };
      };
      swapFile = {
        enable = lib.mkEnableOption (lib.mdDoc "Enables the creation of a swap file.");
        size = lib.mkOption {
          type = lib.types.int;
          description = "The size of the swap file to create in MB (defaults to 8192, or ~8 gigabytes).";
          default = 8192;
        };
      };
    };
  };

  config = lib.mkIf cfg.enable {

    # Check for blank parameters
    assertions = [
      {
        assertion = cfg.partitions.luks != "";
        message = "Please specify a LUKS partition to use as the root filesystem.";
      }
      {
        assertion = cfg.partitions.boot != "";
        message = "Please specify your boot partition.";
      }
    ];
    boot.initrd.luks.devices.${decryptPart} = {
      device = cfg.partitions.luks;
      # Enable TPM auto-unlocking if configured
      crypttabExtraOpts = lib.mkIf config.aux.system.bootloader.tpm2.enable [ "tpm2-device=auto" ];
    };
    fileSystems =
      {
        "/" = {
          device = decryptPath;
          fsType = "btrfs";
          options = [
            "subvol=@"
            "compress=zstd"
          ];
        };
        "/boot" = {
          device = cfg.partitions.boot;
          fsType = "vfat";
        };
        "/home" = {
          device = decryptPath;
          fsType = "btrfs";
          options = [
            "subvol=@home"
            "compress=zstd"
          ];
        };
        "/var/log" = {
          device = decryptPath;
          fsType = "btrfs";
          options = [
            "subvol=@log"
            "compress=zstd"
          ];
        };
        "/nix" = {
          device = decryptPath;
          fsType = "btrfs";
          options = [
            "subvol=@nix"
            "compress=zstd"
            "noatime"
          ];
        };
      }
      // lib.optionalAttrs cfg.swapFile.enable {
        "/swap" = {
          device = decryptPath;
          fsType = "btrfs";
          options = [
            "subvol=@swap"
            "noatime"
          ];
        };
      };

    swapDevices = lib.mkIf cfg.swapFile.enable [
      {
        device = "/swap/swapfile";
        size = cfg.swapFile.size;
      }
    ];
  };
}

So, let’s break this down.

The module arguments ⚓

This module takes three arguments:

1. The path to the boot partition (aux.system.filesystem.partitions.boot).
2. The path to the encrypted LUKS volume (aux.system.filesystem.partitions.luks).
3. Optionally, whether to enable a swap file, and what the size of the swap file should be.

The script does some other stuff too, like enabling disk unlocking via TPM if you have it enabled, but that’s not relevant to this post.

For the partitions, I HIGHLY recommend using UUIDs, since they don’t change even if you move your storage device to a different system. With something like /dev/sda or /dev/sdb, the drive’s path could change if you add a second drive, or even if you just plug the drive into a different port. A great way to get your partition’s UUID is by running blkid and looking for the UUID field (not the PARTUUID field):

aires@Khanda ~ $ sudo blkid                
/dev/nvme0n1p1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="B2D7-96C3" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="4234893a-2d10-48f4-89bb-f953e3f51d3f"
/dev/nvme0n1p2: UUID="f5ff391a-f2ef-4ac3-9ce8-9f5ed950b212" LABEL="nixos-crypt" TYPE="crypto_LUKS" PARTUUID="2ec0a1e6-bc0c-4d10-aeb5-951df3ff7853"

I prefer using swap files to swap partitions, so I enable them by default. The file size defaults to 8GB, but I usually bump this up to 16GB to be safe. All my PCs use fast storage and are fairly high capacity, so squirreling away some extra storage space for swap doesn’t hurt. I can always reduce the file size later if needed, which is something you can’t do easily with swap partitions.

Using the module ⚓

Putting this all together, whenever I need to define my filesystems for a system, including subvolumes and swap files, all I need to do is add these lines to its config:

aux.system.filesystem = {
  enable = true;
  partitions = {
    boot = "/dev/disk/by-uuid/${bootUUID}";
    luks = "/dev/disk/by-uuid/${luksUUID}";
  };
  swapFile = {
    enable = true;
    size = 16384;
  };
};

Just swap out ${bootUUID} and ${luksUUID} with your boot and LUKS partition UUIDs respectively, and rebuild. Much better than what I was doing before 🔗, and unlike Disko, it doesn’t complain that I don’t have a root filesystem defined.

Disko is great, it’s just not for me ⚓

Before I end this article, I want to note that I really do like Disko. If this article sounded harsh or critical of it, it’s because I’m frustrated that I couldn’t get it working, not because Disko itself is bad. Plenty of people are actively using Disko just fine, and if Disko works for you, that’s awesome! You should keep using it! This post is for the rest of us who either can’t use it, couldn’t get it working, or just want an alternative that better fits their configuration structure.

If you want to use this module and make it your own, feel free to snag it from GitHub 🔗. Add your own subvolumes, change the mount options, change the name of the unlocked LUKS partition from nixos-root to something else, do whatever you’d like. One of the best ways to learn a system like NixOS is by messing around, because even if you do break things horribly, you can always just reboot back into a working system (unless you break your bootloader, in which case may God have mercy on your soul).

As always, if you have any thoughts or comments on this blog, hit me up on Mastodon 🔗!