I went looking for ideas to borrow for my own NixOS fleet config and ended up reading most of MeccanicOS. It is a desktop distro aimed at non-technical users, BSD-3, about 7,700 lines of Nix plus a Python fleet layer. The desktop half is not what interested me. The fleet layer and the test suite are, and there are a handful of correctness details in there that I think are worth lifting into other people’s tooling.

Two caveats before anyone takes this as a recommendation. The fleet layer is marked experimental in the repo and the README says it has been tested end to end in VMs and not yet in a real organisation. And most of the primitives below are not new. The value is in how they are assembled and in about six small details that hand-rolled deploy scripts usually get wrong. Everything here is from commit 4d6679b; it is a moving target.

I have no connection to the project.

The headline design decision

From docs/fleet/DESIGN.md:

An earlier version used Kanidm, Forgejo, Harmonia, FleetDM, MySQL, Redis and nginx. Each did its job well, but together they were a lot to run, secure, upgrade and back up for what the fleet needs. This version replaces them.

What replaced all of it: one git repository, SSH for every connection, Nix to build each host, SQLite for what the hosts report. No web services, no directory server. Managed hosts keep one SSH connection to a server, the server hands down the store path of the system that host should be running, the host fetches the paths it lacks and switches.

Whether or not you want their architecture, that is a serious argument made by someone who had the full stack working and chose to remove it. Most homelab and small-fleet setups, mine included, accumulate in the opposite direction.

Details worth stealing

1. Run activation in a transient unit, not in your deploy session.

systemd-run --unit mos-fleet-switch-<ts> --wait --collect --pipe --quiet \
  --service-type=exec <system>/bin/switch-to-configuration switch

Their comment: “Outside this service’s cgroup: switching may restart services, and must finish even if the agent itself goes away meanwhile.” If you deploy by SSHing in and running a rebuild, activation lives in a process tree that activation itself can tear down, sshd included.

2. A deploy succeeded only if the exit code is zero AND the new system is actually current.

running = current_system() == system
# Both: switch-to-configuration can fail after the new system became the current one.
ok = code == 0 and running

The failure mode they handle explicitly is “the new system is running, but switching failed”: some units never started, readlink /run/current-system points at the new closure, and any monitoring that checks only the symlink reports success. They report that state separately and tell the operator the previous system is still in the boot menu.

3. Per-host branch, declared in the host’s own data.

Each host directory has a host.toml naming the branch it follows, main or testing. The build server builds each host from its own branch. That gives me real canaries without a separate pipeline: one machine you do not mind breaking follows testing.

4. Two-tier config validation, with the cheap tier in a git hook.

scripts/mos_fleet/repo.py validates the repository without evaluating Nix: unknown keys, duplicate uids and gids, duplicate SSH keys, references to groups or profiles that do not exist, no admin holding both an SSH key and an age key. A pre-receive hook runs that and refuses the push, and also refuses deleting or rewriting main. Full Nix evaluation happens on the operator’s machine and again at build time.

NixOS assertions catch this class of error at eval. A pre-receive hook catches it before it lands on the branch everything else builds from, which is cheaper and keeps the branch always-buildable. post-receive then refreshes derived state and touches a file that a systemd path unit watches, so the build starts on push; a timer also runs it every 10 minutes.

5. Authorisation by forced command, not by trusting the client.

sshd uses an AuthorizedKeysCommand for a single unprivileged account. It emits an authorized_keys line with restrict plus a forced command naming the role and the host, for example mos-fleet-serve host ws-001. The client cannot claim to be a different host, and gets no shell, no PTY and no forwarding. Standard SSH, used properly, and a better fit for a small fleet than anything with a web UI.

6. Fetching a closure where the server decides what you may have.

The transport is nothing special, it is nix-store --export piped over the existing SSH channel, which is what nix copy does. The part worth copying is the authorisation: the server will serve only the closure of that host’s current desired system and nothing else. The agent enumerates what it is missing first with

nix-store --check-validity --print-invalid <paths>

and asks for only those, then verifies the whole system path is valid after importing.

the agent imports as root with no signature checking, on the grounds that the SSH channel already authenticates the server. That is a deliberate trade of signature verification for channel authentication, and it means whoever controls the server controls every managed host. The design doc says so.

7. Log shipping acknowledged by journald cursor, and jobs acknowledged by id.

The server writes a batch and its cursor in one transaction, then acks. On reconnect it tells the agent where its copy ends, so a lost ack re-sends rather than duplicating. Jobs are acked on arrival by id, the agent remembers ids it has seen, and an unacked job is re-sent with the same id. At-least-once with dedup on both sides, written down explicitly. Nothing novel if you have built a log shipper, and easy to get wrong if you have not.

8. Network-bound disk unlock with a fallback that does not hang.

boot.initrd.clevis = {
  enable = true;
  useTang = true;
  devices.cryptroot.secretFile = "/etc/secrets/meccanicos/cryptroot.jwe";
};
boot.initrd.systemd.network.wait-online = {
  anyInterface = true;
  timeout = 30;
};

boot.initrd.clevis is already in nixpkgs, so the module is not the contribution. The 30 second wait-online timeout is: a laptop that unlocks from a Tang server on the office LAN has to give up quickly and ask for a passphrase when it is somewhere else, or it sits in initrd forever. They also generate a systemd-cryptenroll recovery key on the host, age-encrypt it to the admins’ keys, and store only the ciphertext centrally.

9. Weekly store deduplication instead of per-build.

nix.optimise = { automatic = true; dates = [ "weekly" ]; };

With the comment that auto-optimise-store slowed every update down. Small, and I had never bothered to change it.