Why is enabling automatic updates in NixOS so hard?

Contents

Note: this was originally published on June 1, 2024

Pop quiz: how do you enable automatic updates on your computer?

If you’re using Windows or macOS, chances are automatic updates are already enabled for you. Even certain Linux distros enable them by default. If not, enabling and disabling them is often as simple as opening system settings and unchecking a box, or in the worst case, editing a text file.

And then there’s NixOS. Now, imagine you’re a brand-new NixOS user. You’ve managed to learn the basics: how to configure your system, how to apply that configuration, and how to make changes. Now you want to make sure your system gets regular, automatic updates. You search “NixOS automatic updates” and end up on the NixOS wiki 🔗. Looks easy enough, right? Just copy this block into your configuration, run nixos-rebuild, and go!

system.autoUpgrade = {
  enable = true;
  flake = inputs.self.outPath;
  flags = [
    "--update-input"
    "nixpkgs"
    "-L" # print build logs
  ];
  dates = "02:00";
  randomizedDelaySec = "45min";
};

Well, not quite. For a single-user, single-system configuration, this works perfectly fine. But if you’re doing anything more advanced, like using Flakes or sharing your config across multiple systems, it gets even harder.

So I started looking into solutions. While there are a lot of great ideas and projects floating around, none fit my requirements:

• Automatically apply system updates on multiple systems: two laptops and an always-on home server
• Update and distribute the latest version of flake.lock to all systems without conflicts
• Support passing arguments to nixos-rebuild, specifically --build-host

Instead, I wrote a bash script that solves these problems without requiring any tools beyond Git and nixos-rebuild. But before I get into that, I want to start with the problems. Why is something as simple as automatic system updates so hard in NixOS?

Also, if something in this article is incorrect or you know of an easier way to do this, please let me know 🔗!

Problem #1: Git ⚓

NixOS’ Flakes feature requires you to use Git 🔗 to manage and track changes to your configuration files. New files must be git added before nix will even recognize them. This is because the goal of Flakes is to make NixOS perfectly reproducible. To do that, it has to know exactly how to recreate your system, and to do that, it has to have a copy of your configuration at a specific time. Git makes this effortless.

Why is this a problem? One of the files Flakes uses is flake.lock, which is automatically generated from the inputs in your flake.nix file. For instance, my flake.nix uses Nixpkgs version 24.05 🔗 for its packages. You’ll notice that this URL points to a GitHub repository. When I run nix flake update, Nix goes to that repository, finds the latest commit in the nixos-24.05 branch, and stores the commit hash 🔗 in flake.lock. This essentially pins the version of nixos-24.05 and all of the packages it tracks to that specific commit.

The great thing about this is that it enables perfect reproducibility. If I run nixos-rebuild again without changing the lock file, I’ll get the exact same build as when I ran it the first time. I can even copy this folder to a different computer, run nixos-rebuild, and get the exact same build as on the first computer. I’ve done this multiple times when reinstalling NixOS after replacing hard drives and entire systems.

HOWEVER👆, remember how I mentioned Flakes requires git? Well, git can only track different versions of a file when it’s committed to the repository using git add and git commit. That slick block of code that we copied from the NixOS wiki doesn’t do that. So while we may get a fresh flake.lock each time, it’ll get overwritten as soon as the update service kicks off again, effectively killing reproducibility.

Problem #2: Multiple systems ⚓

Another great benefit of Flakes is the ability to manage multiple computers using a single flake.nix file. I manage a Git repo 🔗 that I use to configure my gaming PC, travel laptop, home server, and Raspberry Pi. All I have to do is clone the repo to the target system, run nixos-rebuild switch --flake .#<hostname>, and I’m good to go!

Oh but wait, what’s this? I had the system.autoUpgrade block configured for both my home server and travel laptop, but they ran at slightly different times. Within that short time span, a bunch of new commits were made to nixpkgs. Now I have two different and divergent versions of flake.lock. Which one is the right one? Which one should I commit to the repo? Which one should I overwrite? How do I computer??? 😵‍💫

One solution would be to only allow one system to run nix flake update, commit the new lock file, push it to my GitHub repo, then have the other systems pull the latest version using git pull. I think this is the method the Nix developers intended, because there’s a convenient flag we can use to update the lock file and commit it in one go: nix flake update --commit-lock-file.

This could work! Wait, but now we need a central repository to host our configuration file, ideally one that’s always online and that every system can access. Unless you’re comfortable running your own code repo, you’re likely going to use GitHub. And that’s fine - lots of NixOS users upload their system declarations on GitHub. But still…why should users have to slingshot their systems around an external service just to enable automatic updates? It adds a whole new dimension of complexity for something that should be as easy as checking a box or editing a text file.

Problem #3: systems.autoUpgrade is incomplete ⚓

Ok, so we’ve figured out how to update our lock file and commit it an external, always-on repository. Our automatic updates are still limited to just one computer though, and if we enable it for every system, we’ll have a bunch of divergent Git repositories all over the place. What’s missing?

Even if we fix system.autoUpgrade to update and commit the lock file, we still need to git push our changes back up to the central repo. This is really where the NixOS wiki’s system.autoUpgrade solution falls short: it only updates the local config, and there’s no option to enable a push.

Basically, we need to reinvent the autoUpgrade wheel by telling NixOS how to pull the latest version of the repo, update and commit the lock file, then push it back up, so other systems can build from it.

One small saving grace is that Nix makes it fairly easy to create and configure automated services via systemd. And by creating our own Nix options 🔗, we can specify which computer should update the lock file, and which ones should just pull and apply the latest version. And that’s what I ended up doing.

The solution ⚓

The fix consists of a single bash script and a systemd service. I’ll start with the script, then end with the service.

The NixOS Upgrade Script ⚓

This bash script performs four key functions:

1. Pull the latest version of the config using git pull.
2. Update your flake.lock file by running nix flake update --commit-lock-file.
3. Push any updates back up to the repository using git push.
4. Apply the updates using nixos-rebuild.

You can find the full script on GitHub 🔗, but here are the most important bits:

if [ -z "${flakeDir}" ]; then
	echo "Flake directory not specified. Use '--flake <path>' or set \$FLAKE_DIR."
	exit 1
fi

cd $flakeDir

echo "Pulling the latest version of the repository..."
/run/wrappers/bin/sudo -u $user git pull

if [ $update = true ]; then
	echo "Updating flake.lock..."
	/run/wrappers/bin/sudo -u $user nix flake update --commit-lock-file && /run/wrappers/bin/sudo -u $user git push
else
	echo "Skipping 'nix flake update'..."
fi

options="--flake $flakeDir $remainingArgs --use-remote-sudo --log-format multiline-with-logs"

echo "Running this operation: nixos-rebuild $operation $options"
/run/wrappers/bin/sudo -u root /run/current-system/sw/bin/nixos-rebuild $operation $options

This script is meant to be run as root. It takes four main arguments:

1. --user: the user account to run git commands as. This is so the script doesn’t accidentally change permissions in your .git folder. This defaults to the user who ran the script.
2. --flake: the path to your flake.nix file (and optionally, the name of the configuration to build). You can also set the FLAKE_DIR environment variable.
3. --operation: which operation to perform (build, boot, switch, etc.). This defaults to switch.
4. --update: tells the script to check for and commit updates to flake.lock. By default, the script will not do this.

Any arguments provided after these will be passed directly to nixos-rebuild.

Creating a systemd service to run automatic NixOS updates ⚓

Now, let’s put everything we’ve learned together into a complete solution.

Essentially, we want to make a sytemd service that calls this script on a timer. We also want to add options to set the user, flake path, operation, and whether or not to update in our NixOS host configuration.

I won’t post the entire thing here, but you can find the full module in my Nix configuration on GitHub 🔗. And to see how to enable & configure this for a host, check out my home server configuration 🔗.

There’s gotta be a better way ⚓

Managing NixOS updates feels more like managing cloud infrastructure than a desktop, which is both a blessing and a curse. Having to set up systemd rules and use external services like GitHub to slingshot files between devices is a burden most regular users won’t be willing to put up with. And granted, you could argue that not using Flakes would make everything easier, but Flakes are such a common part of the NixOS ecosystem now that not using them would be like saying “don’t use apt repositories.” It just cuts you off from too many other options.

That said, if you know of a better way to do this, please let me know 🔗! I think this is one of those gotchas that’s holding NixOS back from being a killer distro. I’m hoping if enough of us rub our brain cells together, we can figure out a good solution that preserves reproducibility without sacrificing accessibility.