# Guix system container daemon When developing system containers it is useful to be able to test them on the underlying host. Particularly when using AI agents. This little tool allows starting system containers through a network call. It validates the originating IP and whether this is a true container. Still, make sure this only runs on systems that have limited functionality. I use it on my reduced AI box where the agents run in containers. A daemon that starts Guix system containers (`guix system container ...` launchers from `/gnu/store`) on request. You can run it by hand, but it is meant to run **as root from shepherd (Guix System) or systemd (Debian)**. It refuses to start otherwise: check [Privileges required](#privileges-required). ## Network protocol Line based, one request per connection: ``` START []\n → OK \n container started (detached) → ERR \n rejected KILL \n → OK killed \n SIGKILL sent to a container started by this daemon → ERR \n unknown pid, not ours, etc. LIST\n → OK\n + one " " line per running container LOG \n → OK\n + raw log contents (boot output of the container) FINGERPRINT\n → OK \n the daemon's fingerprint ``` The daemon runs **at most one container at a time**: `START` is refused with `ERR already running container ` while a previously started container is still alive — `KILL` it (or `--list` to find it) first. The client `run-guix-container.rb` speaks this protocol: ```sh ./run-guix-container.rb --daemon 127.0.0.1:8989 HASH GUIX_CONTAINER_DAEMON=127.0.0.1:8989 ./run-guix-container.rb HASH ./run-guix-container.rb --daemon 127.0.0.1:8989 --list ./run-guix-container.rb --daemon 127.0.0.1:8989 --kill PID ./run-guix-container.rb --daemon 127.0.0.1:8989 --log HASH ``` Inspect a running container (as root): ```sh nsenter -at $PID /run/current-system/profile/bin/bash --login ``` ## Daemon fingerprint Start the daemon with `--fingerprint WORD`, a short word or version like `v0.1`: ```sh run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1 ``` If no fingerprint is given, the daemon generates a **random hash** at startup. The fingerprint makes sure everyone talks to the daemon (and thus the containers) they expect: * clients can query it (`FINGERPRINT` command) or pass the expected fingerprint with `START ` — a mismatch is refused with `ERR fingerprint mismatch` * the client has `--fingerprint WORD`, which verifies the daemon via `FINGERPRINT` before doing anything else * every started container receives the fingerprint in its `GUIX_CONTAINER_FINGERPRINT` environment variable; a container OS can show it at login (e.g. write it to `/etc/issue`), so an ssh session immediately reveals which daemon generation started it ## Shutdown behaviour On SIGINT or SIGTERM the daemon terminates the containers it started: SIGTERM to all of them, wait up to **10 seconds** for them to exit, then SIGKILL any survivors, and exit. Containers started by a *previous* daemon instance are not tracked and not touched. ## Access control The daemon listens on **one address only**, given by `--listen IP` (default `127.0.0.1`, i.e. loopback — only local clients can reach it). Pass an explicit address to expose it further, e.g. `--listen 0.0.0.0` for all IPv4 interfaces, or a specific address such as `--listen 192.168.1.10`. IPv6 addresses work too (`--listen ::1`). There is no per-client ACL: whoever can reach the listening address may start containers. Restrict access at the network level (bind to a dedicated interface, firewall the port, or use a private network between the host and its containers). Every request is logged to stderr with a timestamp and client address. ## Validation Before a container script is exec'd, the daemon verifies: * the name resolves to exactly `HASH-run-container` (32-char nix-base32 hash, unique in the store) * the file is a direct child of `/gnu/store`, executable, and immutable (no write bits — Guix store items are `0555` root-owned) * the file is structurally a generated Guix system container launcher: * first line is a guile shebang (`#!/gnu/store/...-guile-.../bin/guile ...`) — this also rules out ELF binaries, which start with `\x7fELF` * it calls `call-with-container`, the container entry point * it sets `GUIX_NEW_SYSTEM`, as generated system containers do * it references `linux-container` A regular binary, an ELF file, or a script that merely mentions these strings but is not a generated launcher will not pass. ## Build & install ```sh cd bin gcc -O2 -Wall -o run-guix-system-container run-guix-system-container.c # Debian: sudo install -o root -g root -m 0755 run-guix-system-container /usr/local/bin/ # Guix System: add it to a channel/package or keep it in a root-owned # directory, e.g. /run/current-system/profile/bin/ # Run the unit tests: cd ../test && ./run.sh ``` ## Privileges required Container creation needs **real root on the host**: `CAP_SETUID` (to write the container's 65536-entry uid map) and `CAP_SYS_ADMIN` (to mount its file systems, `/proc` and `/dev`). Without them every `START` succeeds in forking a launcher that dies within milliseconds and leaves nothing behind: the client sees `OK ` and then a `LIST` that comes back empty. So the daemon checks at startup and **exits with status 1** when `geteuid() != 0` or when `CapEff` from `/proc/self/status` lacks `CAP_SETUID` or `CAP_SYS_ADMIN` (root in a unit with a narrowed `CapabilityBoundingSet=`), e.g.: ``` error: run-guix-system-container must run as root (uid 1000): container creation needs CAP_SETUID and CAP_SYS_ADMIN, without them every START fails ``` Container boot output goes to `/var/log/guix-containers`, or wherever `--log-dir DIR` (or `$GUIX_CONTAINER_LOG_DIR`) points -- the flag wins over the variable, and the default above applies when neither is given. If that directory cannot be created or written (read-only `/var`, hardened unit, a daemon that is not system-wide) the daemon only warns -- containers still run -- but a boot failure then leaves no log to read. With `ProtectSystem=strict` either point `--log-dir` at a writable path or set `LogsDirectory=guix-containers` (equivalently `ReadWritePaths=/var/log/guix-containers`). ## Running it ### systemd (Debian) Basic unit, `/etc/systemd/system/guix-container-daemon.service`: ```ini [Unit] Description=Guix system container launcher daemon After=network.target [Service] ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1 Restart=on-failure [Install] WantedBy=multi-user.target ``` A hardened variant — the daemon only needs to read the store, listen on a socket, fork and exec, so most systemd sandboxing can be enabled (note: `ProtectSystem=strict` would hide `/gnu/store` unless made read-only-visible, which is the default anyway — keep it `read-only`): ```ini [Unit] Description=Guix system container launcher daemon After=network.target [Service] ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1 Restart=on-failure # Logging goes to stderr; let journald capture it. StandardOutput=journal StandardError=journal # The daemon forks detached containers (setsid), which must survive a # daemon restart — do not use KillMode=control-group here. KillMode=process # Sandbox: the daemon needs /gnu/store (read-only), /dev/null, the # socket, and nothing else writable. ProtectSystem=strict ReadOnlyPaths=/gnu/store # Container boot logs; without this /var/log is read-only and the # daemon can only warn that boot output will be lost. LogsDirectory=guix-containers ProtectHome=true PrivateTmp=true NoNewPrivileges=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET AF_INET6 CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_SYS_ADMIN RestrictSUIDSGID=true # Network filtering (cgroup BPF): only allow loopback traffic, deny # everything else — a second layer on top of --listen 127.0.0.1. IPAddressDeny=any IPAddressAllow=localhost # Optionally log traffic from/to the unit: IPAccounting=yes [Install] WantedBy=multi-user.target ``` Enable and watch the log: ```sh sudo systemctl daemon-reload sudo systemctl enable --now guix-container-daemon sudo journalctl -u guix-container-daemon -f # watch the log ``` ### systemd network filtering Beyond `--listen`, systemd can restrict the unit's network at the cgroup level with `IPAddressAllow=` / `IPAddressDeny=` (enforced with BPF, applies to both directions, and to anything the unit's processes spawn — including the started containers, since they are children of the unit's cgroup): ```ini # allow only loopback, deny everything else IPAddressDeny=any IPAddressAllow=localhost ``` Rules accept addresses, prefixes (`192.168.1.0/24`), and the special values `any` and `localhost`. Order does not matter; the most specific match wins. Combined with `RestrictAddressFamilies=` this gives belt-and-braces isolation: even a bug that made the daemon bind `0.0.0.0` could not be reached from outside. Two related options: * `PrivateNetwork=yes` — gives the unit its own network namespace; then *nothing*, not even the host's loopback, can reach it except via a veth/VRF setup. Usually too strong for this daemon. * `IPAccounting=yes` — counts bytes in/out per unit, useful for auditing. Note: containers started by the daemon inherit the unit's cgroup and thus its IPAddress rules — if containers need real network access, keep the rules on the daemon minimal or move started containers to a scope with `Delegate=` semantics. ### shepherd (Guix System) Add a service to the OS configuration: ```scheme (simple-service 'guix-container-da shepherd-root-service-type (list (shepherd-service (provision '(guix-container-daemon)) (requirement '(user-processes)) (start #~(make-forkexec-constructor (list #$(file-append my-tools "/bin/run-guix-system-container") "--listen" "127.0.0.1" "--port" "8989" "--fingerprint" "v0.1") #:log-file "/var/log/guix-container-daemon.log")) (stop #~(make-kill-destructor))))) ``` ## Worked example: a service container other programs talk to The pattern that matters in practice is a container whose services the *host* then uses over loopback. `--network` shares the host's network namespace, so a service listening on a port inside the container is reachable at `127.0.0.1:PORT` on the host -- no port forwarding and no NAT. (The daemon's own `--port` is a different thing: it is the socket you send `START` to.) Build first, start second. `guix system container ...` builds *and* runs the container in the foreground, which is not what you want when a daemon is going to own it: ```sh # 1. get the launcher's derivation, then build the launcher guix system container -d --network --share=/srv/share config.scm # -> /gnu/store/HASH-run-container.drv guix build /gnu/store/HASH-run-container.drv # -> /gnu/store/HASH-run-container # 2. let the daemon start it (it validates the launcher, detaches it, and # enforces one container at a time) ./run-guix-container.rb --daemon 127.0.0.1:8989 CONTAINER_HASH # 3. it is now running; talk to its services on the host curl http://127.0.0.1:PORT/... ``` `--share=HOSTDIR` mounts a host directory inside the container at the same path. A `--share` whose host path does **not** exist at container start is dropped silently -- Guix does not warn -- so create the directory first, and if a container cannot see a shared path, check it exists before looking at the container. Two timings worth knowing when testing (measured on a Slurm cluster container, 2026-11-12): a container reaches its services in about **2 s** from `START` when the store paths are already built, and about **45 s** on its first boot, where activation still has to generate keys and lay out its spool and log directories. If a test client gives up after a few seconds, it is measuring the first case and losing against the second. Boot output is the way to see what a container is doing: `LOG HASH` over the protocol, `--log HASH` from the client, or `/var/log/guix-containers/.log` on the host (which the daemon warns it cannot write without `LogsDirectory=`/`ReadWritePaths=`, see the unit files below). ## Containers and VMs Container definitions live in `services/`, VM definitions in `vms/`. All of them provide ssh and are minimal (hello + ssh server). Containers share the host's whole /gnu/store read-only; the VM image contains only the closure of the system, so it exposes no host file systems at all. Containers are built and started like this: ```sh guix system container --network services/hello-container.scm ./run-guix-container.rb --daemon 127.0.0.1:8900 ``` The VM is built as a disk image and run under qemu (user-mode networking forwards host port 2223 to guest ssh port 22): ```sh guix system image -t qcow2 vms/hello-vm.scm cp /gnu/store/...-image.qcow2 /tmp/vm.qcow2 && chmod u+w /tmp/vm.qcow2 qemu-system-x86_64 -m 1024 -drive file=/tmp/vm.qcow2,format=qcow2 \ -netdev user,id=n0,hostfwd=tcp:127.0.0.1:2223-:22 \ -device virtio-net-pci,netdev=n0 -nographic ``` Definition Description ------------------- ------------------------------------------------------------ hello-container.scm Bare-bones test container running 'hello' with an ssh server on port 2222. Root login with etc/hello-container-key (host key pinned from etc/ssh-container-key, so the identification never changes). Smoke test: ssh -p 2222 -i etc/hello-container-key root@127.0.0.1 then run /run/current-system/profile/bin/hello. debug-container.scm Same OS as hello-container, plus the container fingerprint service (shows the starting daemon generation in /etc/issue) and a DBG2 monitor that dumps ps, herd status and syslog into the container log every 20s. Use this to debug boot problems; its host-name is "debug-container". hello-vm.scm The same minimal OS as a SELF-CONTAINED qcow2 VM. Unlike the containers, the VM's /gnu/store holds only this system's closure (~260 items, not the whole host store) and nothing from the host is mounted. SSH on guest port 22, forwarded to host port 2223. Serial console on ttyS0 for debugging; root password "hello". Host SSH keys are generated at first boot and persist in the (writable) image. The packages field is slim on purpose: just bash, coreutils, findutils, grep, sed, gawk, tar, gzip, shadow, procps, guile and hello (%base-packages would add ~400 binaries: nano, man-db, sudo, inetutils with its ftp and telnet, wget, pciutils, ...). ssh tools come in via the openssh service, 'herd' via the shepherd service, and the guix service is deleted. BNW (Bayesian Network Webserver) is also deployable through this daemon: bnw-container.scm modernized container: nginx (foreground service), php-fpm, BNW with GN2 handshake (XSS-fixed); web port 8881, ssh 2222 vms/bnw-vm.scm the same as a self-contained qcow2 VM (2 GiB RAM suggested); web 8881, ssh 22 ## Why a privileged daemon? (and the rootless alternative) `guix system container` needs **real root on the host**: * mount/PID/network namespaces are created with real root (no uid mapping involved), * `--network` needs `CAP_NET_ADMIN` to set up the interface, * a shepherd runs *as root* inside the container. A sandboxed (rootless) agent cannot provide any of that, which is precisely why this daemon exists: root listens on a socket and the sandbox sends START/KILL/LIST/LOG. Note that fixing AppArmor on the host (e.g. adding the `k` permission for the `guix-daemon//guix-builder` profile) only unblocks the *build* daemon's builders — it does not change this privilege boundary. **Rootless alternative for smoke tests.** `guix shell -C` runs entirely on unprivileged *user namespaces* and works from an unprivileged account. Two conditions apply: 1. unprivileged user namespaces must be allowed — check `sysctl kernel.apparmor_restrict_unprivileged_userns`; on systems where it is `1` (Ubuntu 24.04 and newer) each program needs an AppArmor profile before it can create user namespaces. Where it is `0` or unset, nothing extra is needed; 2. the profile must live in a writable place — a bare `guix shell -C` fails when `/var/guix/profiles` is read-only. Use an explicit profile: ```sh mkdir -p /tmp/rootless-test guix package -p /tmp/rootless-test/profile -i coreutils # built via the host daemon guix shell -C -p /tmp/rootless-test/profile -- /bin/sh -c 'id' ``` Inside such a container you are still yourself (your own uid, e.g. `uid=1000`), there is no shepherd, no sshd and no `guix system` services — but a stack of foreground daemons (nginx + php-fpm + octave, ports > 1024, data via `--share=...`) runs fine for fast zero-privilege smoke tests. Use both as they fit: keep the daemon for deployment-grade testing of real `guix system` environments (service management, ssh, host keys, production parity), and use the rootless shell path for quick, privilege-free checks straight from the sandbox. Other options researched: * an **unprivileged guix-daemon** (Guix ≥ 2025, "build daemon drops its privileges", see *Build Environment Setup* in the manual) removes the need for a root daemon for *builds* — irrelevant when a root daemon is already running, and it does not help with running *system* containers; * **rootless Podman** also runs containers as an unprivileged user (user namespaces + subordinate UID/GID service) — but gives Docker-style isolation, not `guix system` semantics, and needs subuid/subgid setup; * **PRoot** is the pre-userns fallback (ptrace-based, slow) — only relevant on kernels without user namespace support. ## Files * `bin/run-guix-system-container.c` — the daemon. * `bin/run-guix-container.rb` — unprivileged client; validates the hash locally before talking to the daemon. * `test/` — unit tests for the container-launcher and daemon fingerprint logic (`test/run.sh`). ## Licence BSD 3-Clause -- see [LICENSE](LICENSE).