about summary refs log tree commit diff

Guix system container daemon

When developing system containers it is useful to be able to test them on the underlying host. Particularly when using AI agents. This little tool allows starting system containers through a network call. It validates the originating IP and whether this is a true container.

Still, make sure this only runs on systems that have limited functionality. I use it on my reduced AI box where the agents run in containers.

A daemon that starts Guix system containers (guix system container ... launchers from /gnu/store) on request. You can run it by hand, but it is meant to run as root from shepherd (Guix System) or systemd (Debian). It refuses to start otherwise: check Privileges required.

Network protocol

Line based, one request per connection:

START <hash|/gnu/store/HASH-run-container> [<fingerprint>]\n
→ OK <pid>\n      container started (detached)
→ ERR <reason>\n  rejected
KILL <pid>\n
→ OK killed <pid>\n  SIGKILL sent to a container started by this daemon
→ ERR <reason>\n  unknown pid, not ours, etc.
LIST\n
→ OK\n + one "<pid> <name>" line per running container
LOG <hash|/gnu/store/HASH-run-container>\n
→ OK\n + raw log contents   (boot output of the container)
FINGERPRINT\n
→ OK <fingerprint>\n  the daemon's fingerprint

The daemon runs at most one container at a time: START is refused with ERR already running container <pid> while a previously started container is still alive — KILL it (or --list to find it) first.

The client run-guix-container.rb speaks this protocol:

./run-guix-container.rb --daemon 127.0.0.1:8989 HASH
GUIX_CONTAINER_DAEMON=127.0.0.1:8989 ./run-guix-container.rb HASH
./run-guix-container.rb --daemon 127.0.0.1:8989 --list
./run-guix-container.rb --daemon 127.0.0.1:8989 --kill PID
./run-guix-container.rb --daemon 127.0.0.1:8989 --log HASH

Inspect a running container (as root):

nsenter -at $PID /run/current-system/profile/bin/bash --login

Daemon fingerprint

Start the daemon with --fingerprint WORD, a short word or version like v0.1:

run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1

If no fingerprint is given, the daemon generates a random hash at startup. The fingerprint makes sure everyone talks to the daemon (and thus the containers) they expect:

  • clients can query it (FINGERPRINT command) or pass the expected fingerprint with START <hash> <fingerprint> — a mismatch is refused with ERR fingerprint mismatch
  • the client has --fingerprint WORD, which verifies the daemon via FINGERPRINT before doing anything else
  • every started container receives the fingerprint in its GUIX_CONTAINER_FINGERPRINT environment variable; a container OS can show it at login (e.g. write it to /etc/issue), so an ssh session immediately reveals which daemon generation started it

Shutdown behaviour

On SIGINT or SIGTERM the daemon terminates the containers it started: SIGTERM to all of them, wait up to 10 seconds for them to exit, then SIGKILL any survivors, and exit. Containers started by a previous daemon instance are not tracked and not touched.

Access control

The daemon listens on one address only, given by --listen IP (default 127.0.0.1, i.e. loopback — only local clients can reach it). Pass an explicit address to expose it further, e.g. --listen 0.0.0.0 for all IPv4 interfaces, or a specific address such as --listen 192.168.1.10. IPv6 addresses work too (--listen ::1).

There is no per-client ACL: whoever can reach the listening address may start containers. Restrict access at the network level (bind to a dedicated interface, firewall the port, or use a private network between the host and its containers).

Every request is logged to stderr with a timestamp and client address.

Validation

Before a container script is exec'd, the daemon verifies:

  • the name resolves to exactly HASH-run-container (32-char nix-base32 hash, unique in the store)
  • the file is a direct child of /gnu/store, executable, and immutable (no write bits — Guix store items are 0555 root-owned)
  • the file is structurally a generated Guix system container launcher:
  • first line is a guile shebang (#!/gnu/store/...-guile-.../bin/guile ...) — this also rules out ELF binaries, which start with \x7fELF
  • it calls call-with-container, the container entry point
  • it sets GUIX_NEW_SYSTEM, as generated system containers do
  • it references linux-container

A regular binary, an ELF file, or a script that merely mentions these strings but is not a generated launcher will not pass.

Build & install

cd bin
gcc -O2 -Wall -o run-guix-system-container run-guix-system-container.c
# Debian:
sudo install -o root -g root -m 0755 run-guix-system-container /usr/local/bin/
# Guix System: add it to a channel/package or keep it in a root-owned
# directory, e.g. /run/current-system/profile/bin/

# Run the unit tests:
cd ../test && ./run.sh

Privileges required

Container creation needs real root on the host: CAP_SETUID (to write the container's 65536-entry uid map) and CAP_SYS_ADMIN (to mount its file systems, /proc and /dev). Without them every START succeeds in forking a launcher that dies within milliseconds and leaves nothing behind: the client sees OK <pid> and then a LIST that comes back empty.

So the daemon checks at startup and exits with status 1 when geteuid() != 0 or when CapEff from /proc/self/status lacks CAP_SETUID or CAP_SYS_ADMIN (root in a unit with a narrowed CapabilityBoundingSet=), e.g.:

error: run-guix-system-container must run as root (uid 1000): container
  creation needs CAP_SETUID and CAP_SYS_ADMIN, without them every
  START fails

Container boot output goes to /var/log/guix-containers, or wherever --log-dir DIR (or $GUIX_CONTAINER_LOG_DIR) points -- the flag wins over the variable, and the default above applies when neither is given. If that directory cannot be created or written (read-only /var, hardened unit, a daemon that is not system-wide) the daemon only warns -- containers still run -- but a boot failure then leaves no log to read. With ProtectSystem=strict either point --log-dir at a writable path or set LogsDirectory=guix-containers (equivalently ReadWritePaths=/var/log/guix-containers).

Running it

systemd (Debian)

Basic unit, /etc/systemd/system/guix-container-daemon.service:

[Unit]
Description=Guix system container launcher daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1
Restart=on-failure

[Install]
WantedBy=multi-user.target

A hardened variant — the daemon only needs to read the store, listen on a socket, fork and exec, so most systemd sandboxing can be enabled (note: ProtectSystem=strict would hide /gnu/store unless made read-only-visible, which is the default anyway — keep it read-only):

[Unit]
Description=Guix system container launcher daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1
Restart=on-failure

# Logging goes to stderr; let journald capture it.
StandardOutput=journal
StandardError=journal

# The daemon forks detached containers (setsid), which must survive a
# daemon restart — do not use KillMode=control-group here.
KillMode=process

# Sandbox: the daemon needs /gnu/store (read-only), /dev/null, the
# socket, and nothing else writable.
ProtectSystem=strict
ReadOnlyPaths=/gnu/store
# Container boot logs; without this /var/log is read-only and the
# daemon can only warn that boot output will be lost.
LogsDirectory=guix-containers
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
RestrictSUIDSGID=true

# Network filtering (cgroup BPF): only allow loopback traffic, deny
# everything else — a second layer on top of --listen 127.0.0.1.
IPAddressDeny=any
IPAddressAllow=localhost
# Optionally log traffic from/to the unit:
IPAccounting=yes

[Install]
WantedBy=multi-user.target

Enable and watch the log:

sudo systemctl daemon-reload
sudo systemctl enable --now guix-container-daemon
sudo journalctl -u guix-container-daemon -f     # watch the log

systemd network filtering

Beyond --listen, systemd can restrict the unit's network at the cgroup level with IPAddressAllow= / IPAddressDeny= (enforced with BPF, applies to both directions, and to anything the unit's processes spawn — including the started containers, since they are children of the unit's cgroup):

# allow only loopback, deny everything else
IPAddressDeny=any
IPAddressAllow=localhost

Rules accept addresses, prefixes (192.168.1.0/24), and the special values any and localhost. Order does not matter; the most specific match wins. Combined with RestrictAddressFamilies= this gives belt-and-braces isolation: even a bug that made the daemon bind 0.0.0.0 could not be reached from outside.

Two related options:

  • PrivateNetwork=yes — gives the unit its own network namespace; then nothing, not even the host's loopback, can reach it except via a veth/VRF setup. Usually too strong for this daemon.
  • IPAccounting=yes — counts bytes in/out per unit, useful for auditing.

Note: containers started by the daemon inherit the unit's cgroup and thus its IPAddress rules — if containers need real network access, keep the rules on the daemon minimal or move started containers to a scope with Delegate= semantics.

shepherd (Guix System)

Add a service to the OS configuration:

(simple-service 'guix-container-da shepherd-root-service-type
  (list (shepherd-service
          (provision '(guix-container-daemon))
          (requirement '(user-processes))
          (start #~(make-forkexec-constructor
                    (list #$(file-append my-tools "/bin/run-guix-system-container")
                          "--listen" "127.0.0.1"
                          "--port" "8989"
                          "--fingerprint" "v0.1")
                    #:log-file "/var/log/guix-container-daemon.log"))
          (stop #~(make-kill-destructor)))))

Worked example: a service container other programs talk to

The pattern that matters in practice is a container whose services the host then uses over loopback. --network shares the host's network namespace, so a service listening on a port inside the container is reachable at 127.0.0.1:PORT on the host -- no port forwarding and no NAT. (The daemon's own --port is a different thing: it is the socket you send START to.)

Build first, start second. guix system container ... builds and runs the container in the foreground, which is not what you want when a daemon is going to own it:

# 1. get the launcher's derivation, then build the launcher
guix system container -d --network --share=/srv/share config.scm   # -> /gnu/store/HASH-run-container.drv
guix build /gnu/store/HASH-run-container.drv                       # -> /gnu/store/HASH-run-container

# 2. let the daemon start it (it validates the launcher, detaches it, and
#    enforces one container at a time)
./run-guix-container.rb --daemon 127.0.0.1:8989 CONTAINER_HASH

# 3. it is now running; talk to its services on the host
curl http://127.0.0.1:PORT/...

--share=HOSTDIR mounts a host directory inside the container at the same path. A --share whose host path does not exist at container start is dropped silently -- Guix does not warn -- so create the directory first, and if a container cannot see a shared path, check it exists before looking at the container.

Two timings worth knowing when testing (measured on a Slurm cluster container, 2026-11-12): a container reaches its services in about 2 s from START when the store paths are already built, and about 45 s on its first boot, where activation still has to generate keys and lay out its spool and log directories. If a test client gives up after a few seconds, it is measuring the first case and losing against the second.

Boot output is the way to see what a container is doing: LOG HASH over the protocol, --log HASH from the client, or /var/log/guix-containers/<pid>.log on the host (which the daemon warns it cannot write without LogsDirectory=/ReadWritePaths=, see the unit files below).

Containers and VMs

Container definitions live in services/, VM definitions in vms/. All of them provide ssh and are minimal (hello + ssh server). Containers share the host's whole /gnu/store read-only; the VM image contains only the closure of the system, so it exposes no host file systems at all.

Containers are built and started like this:

guix system container --network services/hello-container.scm
./run-guix-container.rb --daemon 127.0.0.1:8900 <HASH>

The VM is built as a disk image and run under qemu (user-mode networking forwards host port 2223 to guest ssh port 22):

guix system image -t qcow2 vms/hello-vm.scm
cp /gnu/store/...-image.qcow2 /tmp/vm.qcow2 && chmod u+w /tmp/vm.qcow2
qemu-system-x86_64 -m 1024 -drive file=/tmp/vm.qcow2,format=qcow2 \
  -netdev user,id=n0,hostfwd=tcp:127.0.0.1:2223-:22 \
  -device virtio-net-pci,netdev=n0 -nographic

Definition Description


hello-container.scm Bare-bones test container running 'hello' with an ssh server on port 2222. Root login with etc/hello-container-key (host key pinned from etc/ssh-container-key, so the identification never changes). Smoke test: ssh -p 2222 -i etc/hello-container-key root@127.0.0.1 then run /run/current-system/profile/bin/hello.

debug-container.scm Same OS as hello-container, plus the container fingerprint service (shows the starting daemon generation in /etc/issue) and a DBG2 monitor that dumps ps, herd status and syslog into the container log every 20s. Use this to debug boot problems; its host-name is "debug-container".

hello-vm.scm The same minimal OS as a SELF-CONTAINED qcow2 VM. Unlike the containers, the VM's /gnu/store holds only this system's closure (~260 items, not the whole host store) and nothing from the host is mounted. SSH on guest port 22, forwarded to host port 2223. Serial console on ttyS0 for debugging; root password "hello". Host SSH keys are generated at first boot and persist in the (writable) image.

                The packages field is slim on purpose: just bash,
                coreutils, findutils, grep, sed, gawk, tar, gzip,
                shadow, procps, guile and hello (%base-packages would
                add ~400 binaries: nano, man-db, sudo, inetutils with
                its ftp and telnet, wget, pciutils, ...).  ssh tools
                come in via the openssh service, 'herd' via the
                shepherd service, and the guix service is deleted.

BNW (Bayesian Network Webserver) is also deployable through this daemon:

bnw-container.scm    modernized container: nginx (foreground
                     service), php-fpm, BNW with GN2 handshake
                     (XSS-fixed); web port 8881, ssh 2222
vms/bnw-vm.scm       the same as a self-contained qcow2 VM
                     (2 GiB RAM suggested); web 8881, ssh 22

Why a privileged daemon? (and the rootless alternative)

guix system container needs real root on the host:

  • mount/PID/network namespaces are created with real root (no uid mapping involved),
  • --network needs CAP_NET_ADMIN to set up the interface,
  • a shepherd runs as root inside the container.

A sandboxed (rootless) agent cannot provide any of that, which is precisely why this daemon exists: root listens on a socket and the sandbox sends START/KILL/LIST/LOG. Note that fixing AppArmor on the host (e.g. adding the k permission for the guix-daemon//guix-builder profile) only unblocks the build daemon's builders — it does not change this privilege boundary.

Rootless alternative for smoke tests. guix shell -C runs entirely on unprivileged user namespaces and works from an unprivileged account. Two conditions apply:

  1. unprivileged user namespaces must be allowed — check sysctl kernel.apparmor_restrict_unprivileged_userns; on systems where it is 1 (Ubuntu 24.04 and newer) each program needs an AppArmor profile before it can create user namespaces. Where it is 0 or unset, nothing extra is needed;
  2. the profile must live in a writable place — a bare guix shell -C fails when /var/guix/profiles is read-only. Use an explicit profile:
mkdir -p /tmp/rootless-test
guix package -p /tmp/rootless-test/profile -i coreutils   # built via the host daemon
guix shell -C -p /tmp/rootless-test/profile -- /bin/sh -c 'id'

Inside such a container you are still yourself (your own uid, e.g. uid=1000), there is no shepherd, no sshd and no guix system services — but a stack of foreground daemons (nginx + php-fpm + octave, ports > 1024, data via --share=...) runs fine for fast zero-privilege smoke tests.

Use both as they fit: keep the daemon for deployment-grade testing of real guix system environments (service management, ssh, host keys, production parity), and use the rootless shell path for quick, privilege-free checks straight from the sandbox.

Other options researched:

  • an unprivileged guix-daemon (Guix ≥ 2025, "build daemon drops its privileges", see Build Environment Setup in the manual) removes the need for a root daemon for builds — irrelevant when a root daemon is already running, and it does not help with running system containers;
  • rootless Podman also runs containers as an unprivileged user (user namespaces + subordinate UID/GID service) — but gives Docker-style isolation, not guix system semantics, and needs subuid/subgid setup;
  • PRoot is the pre-userns fallback (ptrace-based, slow) — only relevant on kernels without user namespace support.

Files

  • bin/run-guix-system-container.c — the daemon.
  • bin/run-guix-container.rb — unprivileged client; validates the hash locally before talking to the daemon.
  • test/ — unit tests for the container-launcher and daemon fingerprint logic (test/run.sh).

Licence

BSD 3-Clause -- see LICENSE.