1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
|
# Guix system container daemon
When developing system containers it is useful to be able to test them
on the underlying host. Particularly when using AI agents. This little
tool allows starting system containers through a network call. It
validates the originating IP and whether this is a true container.
Still, make sure this only runs on systems that have limited
functionality. I use it on my reduced AI box where the agents run in
containers.
A daemon that starts Guix system containers (`guix system container ...`
launchers from `/gnu/store`) on request. You can run it by hand, but
it is meant to run **as root from shepherd (Guix System) or systemd
(Debian)**. It refuses to start otherwise: check [Privileges
required](#privileges-required).
## Network protocol
Line based, one request per connection:
```
START <hash|/gnu/store/HASH-run-container> [<fingerprint>]\n
→ OK <pid>\n container started (detached)
→ ERR <reason>\n rejected
KILL <pid>\n
→ OK killed <pid>\n SIGKILL sent to a container started by this daemon
→ ERR <reason>\n unknown pid, not ours, etc.
LIST\n
→ OK\n + one "<pid> <name>" line per running container
LOG <hash|/gnu/store/HASH-run-container>\n
→ OK\n + raw log contents (boot output of the container)
FINGERPRINT\n
→ OK <fingerprint>\n the daemon's fingerprint
```
The daemon runs **at most one container at a time**: `START` is
refused with `ERR already running container <pid>` while a previously
started container is still alive — `KILL` it (or `--list` to find it)
first.
The client `run-guix-container.rb` speaks this protocol:
```sh
./run-guix-container.rb --daemon 127.0.0.1:8989 HASH
GUIX_CONTAINER_DAEMON=127.0.0.1:8989 ./run-guix-container.rb HASH
./run-guix-container.rb --daemon 127.0.0.1:8989 --list
./run-guix-container.rb --daemon 127.0.0.1:8989 --kill PID
./run-guix-container.rb --daemon 127.0.0.1:8989 --log HASH
```
Inspect a running container (as root):
```sh
nsenter -at $PID /run/current-system/profile/bin/bash --login
```
## Daemon fingerprint
Start the daemon with `--fingerprint WORD`, a short word or version
like `v0.1`:
```sh
run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1
```
If no fingerprint is given, the daemon generates a **random hash** at
startup. The fingerprint makes sure everyone talks to the daemon (and
thus the containers) they expect:
* clients can query it (`FINGERPRINT` command) or pass the expected
fingerprint with `START <hash> <fingerprint>` — a mismatch is
refused with `ERR fingerprint mismatch`
* the client has `--fingerprint WORD`, which verifies the daemon via
`FINGERPRINT` before doing anything else
* every started container receives the fingerprint in its
`GUIX_CONTAINER_FINGERPRINT` environment variable; a container OS
can show it at login (e.g. write it to `/etc/issue`), so an ssh
session immediately reveals which daemon generation started it
## Shutdown behaviour
On SIGINT or SIGTERM the daemon terminates the containers it started:
SIGTERM to all of them, wait up to **10 seconds** for them to exit,
then SIGKILL any survivors, and exit. Containers started by a
*previous* daemon instance are not tracked and not touched.
## Access control
The daemon listens on **one address only**, given by `--listen IP`
(default `127.0.0.1`, i.e. loopback — only local clients can reach
it). Pass an explicit address to expose it further, e.g.
`--listen 0.0.0.0` for all IPv4 interfaces, or a specific address such
as `--listen 192.168.1.10`. IPv6 addresses work too (`--listen ::1`).
There is no per-client ACL: whoever can reach the listening address
may start containers. Restrict access at the network level (bind to a
dedicated interface, firewall the port, or use a private network
between the host and its containers).
Every request is logged to stderr with a timestamp and client address.
## Validation
Before a container script is exec'd, the daemon verifies:
* the name resolves to exactly `HASH-run-container` (32-char nix-base32
hash, unique in the store)
* the file is a direct child of `/gnu/store`, executable, and
immutable (no write bits — Guix store items are `0555` root-owned)
* the file is structurally a generated Guix system container launcher:
* first line is a guile shebang
(`#!/gnu/store/...-guile-.../bin/guile ...`) — this also rules out
ELF binaries, which start with `\x7fELF`
* it calls `call-with-container`, the container entry point
* it sets `GUIX_NEW_SYSTEM`, as generated system containers do
* it references `linux-container`
A regular binary, an ELF file, or a script that merely mentions
these strings but is not a generated launcher will not pass.
## Build & install
```sh
cd bin
gcc -O2 -Wall -o run-guix-system-container run-guix-system-container.c
# Debian:
sudo install -o root -g root -m 0755 run-guix-system-container /usr/local/bin/
# Guix System: add it to a channel/package or keep it in a root-owned
# directory, e.g. /run/current-system/profile/bin/
# Run the unit tests:
cd ../test && ./run.sh
```
## Privileges required
Container creation needs **real root on the host**: `CAP_SETUID` (to
write the container's 65536-entry uid map) and `CAP_SYS_ADMIN` (to
mount its file systems, `/proc` and `/dev`). Without them every
`START` succeeds in forking a launcher that dies within milliseconds
and leaves nothing behind: the client sees `OK <pid>` and then a
`LIST` that comes back empty.
So the daemon checks at startup and **exits with status 1** when
`geteuid() != 0` or when `CapEff` from `/proc/self/status` lacks
`CAP_SETUID` or `CAP_SYS_ADMIN` (root in a unit with a narrowed
`CapabilityBoundingSet=`), e.g.:
```
error: run-guix-system-container must run as root (uid 1000): container
creation needs CAP_SETUID and CAP_SYS_ADMIN, without them every
START fails
```
Container boot output goes to `/var/log/guix-containers`, or wherever
`--log-dir DIR` (or `$GUIX_CONTAINER_LOG_DIR`) points -- the flag wins over
the variable, and the default above applies when neither is given. If that
directory cannot be created or written (read-only `/var`, hardened unit, a
daemon that is not system-wide) the daemon only warns -- containers still
run -- but a boot failure then leaves no log to read. With
`ProtectSystem=strict` either point `--log-dir` at a writable path or set
`LogsDirectory=guix-containers` (equivalently
`ReadWritePaths=/var/log/guix-containers`).
## Running it
### systemd (Debian)
Basic unit, `/etc/systemd/system/guix-container-daemon.service`:
```ini
[Unit]
Description=Guix system container launcher daemon
After=network.target
[Service]
ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
A hardened variant — the daemon only needs to read the store, listen
on a socket, fork and exec, so most systemd sandboxing can be enabled
(note: `ProtectSystem=strict` would hide `/gnu/store` unless made
read-only-visible, which is the default anyway — keep it `read-only`):
```ini
[Unit]
Description=Guix system container launcher daemon
After=network.target
[Service]
ExecStart=/usr/local/bin/run-guix-system-container --listen 127.0.0.1 --port 8989 --fingerprint v0.1
Restart=on-failure
# Logging goes to stderr; let journald capture it.
StandardOutput=journal
StandardError=journal
# The daemon forks detached containers (setsid), which must survive a
# daemon restart — do not use KillMode=control-group here.
KillMode=process
# Sandbox: the daemon needs /gnu/store (read-only), /dev/null, the
# socket, and nothing else writable.
ProtectSystem=strict
ReadOnlyPaths=/gnu/store
# Container boot logs; without this /var/log is read-only and the
# daemon can only warn that boot output will be lost.
LogsDirectory=guix-containers
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
RestrictSUIDSGID=true
# Network filtering (cgroup BPF): only allow loopback traffic, deny
# everything else — a second layer on top of --listen 127.0.0.1.
IPAddressDeny=any
IPAddressAllow=localhost
# Optionally log traffic from/to the unit:
IPAccounting=yes
[Install]
WantedBy=multi-user.target
```
Enable and watch the log:
```sh
sudo systemctl daemon-reload
sudo systemctl enable --now guix-container-daemon
sudo journalctl -u guix-container-daemon -f # watch the log
```
### systemd network filtering
Beyond `--listen`, systemd can restrict the unit's network at the
cgroup level with `IPAddressAllow=` / `IPAddressDeny=` (enforced with
BPF, applies to both directions, and to anything the unit's processes
spawn — including the started containers, since they are children of
the unit's cgroup):
```ini
# allow only loopback, deny everything else
IPAddressDeny=any
IPAddressAllow=localhost
```
Rules accept addresses, prefixes (`192.168.1.0/24`), and the special
values `any` and `localhost`. Order does not matter; the most
specific match wins. Combined with `RestrictAddressFamilies=` this
gives belt-and-braces isolation: even a bug that made the daemon bind
`0.0.0.0` could not be reached from outside.
Two related options:
* `PrivateNetwork=yes` — gives the unit its own network namespace;
then *nothing*, not even the host's loopback, can reach it except
via a veth/VRF setup. Usually too strong for this daemon.
* `IPAccounting=yes` — counts bytes in/out per unit, useful for
auditing.
Note: containers started by the daemon inherit the unit's cgroup and
thus its IPAddress rules — if containers need real network access,
keep the rules on the daemon minimal or move started containers to a
scope with `Delegate=` semantics.
### shepherd (Guix System)
Add a service to the OS configuration:
```scheme
(simple-service 'guix-container-da shepherd-root-service-type
(list (shepherd-service
(provision '(guix-container-daemon))
(requirement '(user-processes))
(start #~(make-forkexec-constructor
(list #$(file-append my-tools "/bin/run-guix-system-container")
"--listen" "127.0.0.1"
"--port" "8989"
"--fingerprint" "v0.1")
#:log-file "/var/log/guix-container-daemon.log"))
(stop #~(make-kill-destructor)))))
```
## Worked example: a service container other programs talk to
The pattern that matters in practice is a container whose services the *host*
then uses over loopback. `--network` shares the host's network namespace, so a
service listening on a port inside the container is reachable at
`127.0.0.1:PORT` on the host -- no port forwarding and no NAT. (The daemon's
own `--port` is a different thing: it is the socket you send `START` to.)
Build first, start second. `guix system container ...` builds *and* runs the
container in the foreground, which is not what you want when a daemon is going
to own it:
```sh
# 1. get the launcher's derivation, then build the launcher
guix system container -d --network --share=/srv/share config.scm # -> /gnu/store/HASH-run-container.drv
guix build /gnu/store/HASH-run-container.drv # -> /gnu/store/HASH-run-container
# 2. let the daemon start it (it validates the launcher, detaches it, and
# enforces one container at a time)
./run-guix-container.rb --daemon 127.0.0.1:8989 CONTAINER_HASH
# 3. it is now running; talk to its services on the host
curl http://127.0.0.1:PORT/...
```
`--share=HOSTDIR` mounts a host directory inside the container at the same
path. A `--share` whose host path does **not** exist at container start is
dropped silently -- Guix does not warn -- so create the directory first, and if
a container cannot see a shared path, check it exists before looking at the
container.
Two timings worth knowing when testing (measured on a Slurm cluster container,
2026-11-12): a container reaches its services in about **2 s** from `START`
when the store paths are already built, and about **45 s** on its first boot,
where activation still has to generate keys and lay out its spool and log
directories. If a test client gives up after a few seconds, it is measuring the
first case and losing against the second.
Boot output is the way to see what a container is doing: `LOG HASH` over the
protocol, `--log HASH` from the client, or `/var/log/guix-containers/<pid>.log`
on the host (which the daemon warns it cannot write without
`LogsDirectory=`/`ReadWritePaths=`, see the unit files below).
## Containers and VMs
Container definitions live in `services/`, VM definitions in `vms/`.
All of them provide ssh
and are minimal (hello + ssh server). Containers share the host's
whole /gnu/store read-only; the VM image contains only the closure of
the system, so it exposes no host file systems at all.
Containers are built and started like this:
```sh
guix system container --network services/hello-container.scm
./run-guix-container.rb --daemon 127.0.0.1:8900 <HASH>
```
The VM is built as a disk image and run under qemu (user-mode
networking forwards host port 2223 to guest ssh port 22):
```sh
guix system image -t qcow2 vms/hello-vm.scm
cp /gnu/store/...-image.qcow2 /tmp/vm.qcow2 && chmod u+w /tmp/vm.qcow2
qemu-system-x86_64 -m 1024 -drive file=/tmp/vm.qcow2,format=qcow2 \
-netdev user,id=n0,hostfwd=tcp:127.0.0.1:2223-:22 \
-device virtio-net-pci,netdev=n0 -nographic
```
Definition Description
------------------- ------------------------------------------------------------
hello-container.scm Bare-bones test container running 'hello' with an ssh
server on port 2222. Root login with
etc/hello-container-key (host key pinned from
etc/ssh-container-key, so the identification never
changes). Smoke test:
ssh -p 2222 -i etc/hello-container-key root@127.0.0.1
then run /run/current-system/profile/bin/hello.
debug-container.scm Same OS as hello-container, plus the container
fingerprint service (shows the starting daemon
generation in /etc/issue) and a DBG2 monitor that
dumps ps, herd status and syslog into the container
log every 20s. Use this to debug boot problems; its
host-name is "debug-container".
hello-vm.scm The same minimal OS as a SELF-CONTAINED qcow2 VM.
Unlike the containers, the VM's /gnu/store holds only
this system's closure (~260 items, not the whole host
store) and nothing from the host is mounted. SSH on
guest port 22, forwarded to host port 2223. Serial
console on ttyS0 for debugging; root password
"hello". Host SSH keys are generated at first boot
and persist in the (writable) image.
The packages field is slim on purpose: just bash,
coreutils, findutils, grep, sed, gawk, tar, gzip,
shadow, procps, guile and hello (%base-packages would
add ~400 binaries: nano, man-db, sudo, inetutils with
its ftp and telnet, wget, pciutils, ...). ssh tools
come in via the openssh service, 'herd' via the
shepherd service, and the guix service is deleted.
BNW (Bayesian Network Webserver) is also deployable through this
daemon:
bnw-container.scm modernized container: nginx (foreground
service), php-fpm, BNW with GN2 handshake
(XSS-fixed); web port 8881, ssh 2222
vms/bnw-vm.scm the same as a self-contained qcow2 VM
(2 GiB RAM suggested); web 8881, ssh 22
## Why a privileged daemon? (and the rootless alternative)
`guix system container` needs **real root on the host**:
* mount/PID/network namespaces are created with real root (no uid
mapping involved),
* `--network` needs `CAP_NET_ADMIN` to set up the interface,
* a shepherd runs *as root* inside the container.
A sandboxed (rootless) agent cannot provide any of that, which is
precisely why this daemon exists: root listens on a socket and the
sandbox sends START/KILL/LIST/LOG. Note that fixing AppArmor on the
host (e.g. adding the `k` permission for the `guix-daemon//guix-builder`
profile) only unblocks the *build* daemon's builders — it does not
change this privilege boundary.
**Rootless alternative for smoke tests.** `guix shell -C` runs entirely
on unprivileged *user namespaces* and works from an unprivileged
account. Two conditions apply:
1. unprivileged user namespaces must be allowed — check
`sysctl kernel.apparmor_restrict_unprivileged_userns`; on systems
where it is `1` (Ubuntu 24.04 and newer) each program needs an
AppArmor profile before it can create user namespaces. Where it is
`0` or unset, nothing extra is needed;
2. the profile must live in a writable place — a bare
`guix shell -C` fails when `/var/guix/profiles` is read-only. Use
an explicit profile:
```sh
mkdir -p /tmp/rootless-test
guix package -p /tmp/rootless-test/profile -i coreutils # built via the host daemon
guix shell -C -p /tmp/rootless-test/profile -- /bin/sh -c 'id'
```
Inside such a container you are still yourself (your own uid, e.g.
`uid=1000`), there
is no shepherd, no sshd and no `guix system` services — but a stack of
foreground daemons (nginx + php-fpm + octave, ports > 1024, data via
`--share=...`) runs fine for fast zero-privilege smoke tests.
Use both as they fit: keep the daemon for deployment-grade testing of
real `guix system` environments (service management, ssh, host keys,
production parity), and use the rootless shell path for quick,
privilege-free checks straight from the sandbox.
Other options researched:
* an **unprivileged guix-daemon** (Guix ≥ 2025, "build daemon drops its
privileges", see *Build Environment Setup* in the manual) removes the
need for a root daemon for *builds* — irrelevant when a root daemon
is already running, and it does not help with running *system*
containers;
* **rootless Podman** also runs containers as an unprivileged user
(user namespaces + subordinate UID/GID service) — but gives
Docker-style isolation, not `guix system` semantics, and needs
subuid/subgid setup;
* **PRoot** is the pre-userns fallback (ptrace-based, slow) — only
relevant on kernels without user namespace support.
## Files
* `bin/run-guix-system-container.c` — the daemon.
* `bin/run-guix-container.rb` — unprivileged client; validates the hash
locally before talking to the daemon.
* `test/` — unit tests for the container-launcher and daemon
fingerprint logic (`test/run.sh`).
## Licence
BSD 3-Clause -- see [LICENSE](LICENSE).
|