#+TITLE: Setup of a Simple Guix Build Farm and Substitute Server
#+AUTHOR: Collin J. Doeger
In the world of reproducible computing, GNU Guix stands out as a pioneering distribution that
enables bit-for-bit reproducible builds and a comprehensive package management system.
However, building software from source for every package can be time-consuming and
resource-intensive. This is where substitute servers play a crucial role, allowing users to
download pre-built binary packages instead of compiling them locally.
Earlier this year [[https://lists.gnu.org/archive/html/guix-devel/2024-07/msg00033.html][I announced on the guix mailing list]] that a new North American based Guix
substitute server and build farm, cuirass.genenetwork.org, was available for general use.
This initiative was made possible by the generous contribution of server hardware and
infrastructure from GeneNetwork.org.
* Why Build Another Substitute Server?
The Guix ecosystem thrives on diversity and decentralization. By establishing additional
substitute servers, we achieve several critical objectives:
- **Improved Build Diversity**: Multiple independent build farms reduce the risk of
single-point-of-failure and increase the verification of build reproducibility.
- **Reduced Latency**: Geographically distributed servers mean faster download times for
users in different regions.
- **Increased Resilience**: If one substitute server is down, users can fall back to
alternatives.
- **Community Contribution**: Each new substitute server strengthens the broader Guix
infrastructure.
This article provides a comprehensive guide to setting up a Guix build farm and substitute
server, drawing inspiration from existing GNU Guix project infrastructure. You can see their
full source code [[https://git.savannah.gnu.org/cgit/guix/maintenance.git][here]].
* Hardware and Infrastructure
The Tennessee Guix Build Farm was made possible through a collaboration with GeneNetwork.org,
who provided the following server specifications:
- **Processor**: Dual AMD EPYC 9274F 24-Core, 48 Thread Processors
- **RAM**: 768 GB DDR5 ECC
- **Storage**: 1 TB SSD
- **Network**: 1 Gbps nic, 100Mbps dedicated connection
- **Location**: University of Tennessee, Knoxville Data Center
These robust specifications allow for efficient package building, caching, and serving of
substitutes for the Guix community.
* Components of the Guix Build Farm
** Cuirass - building packages
Cuirass is the GNU Guix continuous integration software, and is responsible for watching for
changes to one or more VCS repositories (usually [[https://guix.gnu.org/manual/en/html_node/Channels.html][Guix channels]]), executing build jobs for
packages that have changed, and finally, storing build results in its database.
*** Define Cuirass Specs
In order to run Cuirass via the ~cuirass-service-type~, we need to define what we
want Cuirass to build. In the case of guix-na, we want to build the GNU Guix distribution and
its packages, so we declare ~%cuirass-specs~ as a [[https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html][G-Expression]] that will return a list of
[[https://guix.gnu.org/cuirass/manual/html_node/Specifications.html][cuirass specifications]] with a single entry named "guix", which does exactly that!
#+begin_src scheme
(define %cuirass-specs
#~(list (specification
(name "guix")
(priority 0)
(build '(channels guix))
(channels %default-channels))))
#+end_src
If you have custom channels you wish to build, you would add a new ~specification~ to the
list. Cuirass can build more then just channels and their packages, it can also build images,
tarballs, a specific set of packages, a manifest, and more! See the [[https://guix.gnu.org/cuirass/manual/html_node/Specifications.html][Cuirass specification]]
documentation for more details.
Its worth noting that if you're following along by reviewing the full source code of the
Tennessee build farm that there is an additional specification listed (for guix-na itself!).
More details on that in [[*Guix Configuration as a Channel][Guix Configuration as a Channel]].
*** Setup Cuirass Service
Now that we have defined what we want Cuirass to build, we need to specify its guix service
in the ~services~ field of our ~operating-system~ definition, which in turn will run Cuirass.
We are going to setup nginx as a reverse proxy for cuirass later on, so we'll set its host to
localhost, and pass along the specifications we defined earlier.
#+begin_src scheme
(service cuirass-service-type
(cuirass-configuration
(host "localhost")
(specifications %cuirass-specs)))
#+end_src
** Provide Substitutes using Guix Publish
With Cuirass configured and the guix store being populated with package builds as the guix
channel changes, we now turn our attention to serving these builds as substitutes to Guix
users. This is done using [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html][guix publish]], which Guix provides the [[file:~/.org/roam/20221129213953-advent_of_code.org::*Day 2][guix-publish-service-type]],
which is used in the ~services~ field of ~operating-system~ definition.
#+begin_src scheme
(service guix-publish-service-type
(guix-publish-configuration
(port 3000)
(cache "/var/cache/guix/publish")
(ttl (* 90 24 3600))
#+end_src
Similar to Cuirass, access to guix-publish will be provided through nginx as a reverse proxy.
** Anonomize IPs (anonip)
Guix users care about their privacy, and though this is not necessarily a requirement,
anonymizing nginx access logs using the anonip is implemented by all public Guix sponsored
build farms, so keeping with this privacy preserving trend, cuirass.genenetwork.org
implements the same log anonymization.
To anonymize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_node/Log-Rotation.html][anonip-service-type]] will be configured and used, however, we
want to anonymize multiple log files, which means multiple instances of the anonip running.
To assist with this, a helper function ~anonip-service~ is defined.
#+begin_src scheme
(define (anonip-service file)
(service anonip-service-type
(anonip-configuration
(input (format #false "/var/run/anonip/~a" file))
(output (format #false "/var/log/anonip/~a" file)))
#+end_src
Additionally, for services that will leverage these anonymized logs (in our case, only
nginx), it will be necessary to ensure that the appropriate instance of anonip is running
prior to the respective service that will utilize it. To help declare this dependency,
another helper function is defined.
#+begin_src scheme
(define (log-file->anonip-service-name file)
"Return the name of the Anonip service handling FILE, a log file."
(symbol-append 'anonip-/var/log/anonip/ (string->symbol file)))
#+end_src
We also define a list of anonymized log files which will be used later on along side the
~log-file->anonip-service-name~ function in order to define shepherd service dependencies for
nginx.
#+begin_src scheme
(define %anonip-nginx-log-files
;; List of files handled by Anonip for nginx
'("http.access.log"
"https.access.log"))
#+end_src
All that remains is to ensure that for each log file we are anonymizing, we start a
corresponding anonip-service. This can be added to the ~services~ field of our
~operating-system~ declaration.
#+begin_src scheme
(map anonip-service %anonip-nginx-log-files)
#+end_src
** Nginx Reverse Proxy
Nginx is arguably the most complicated part of the setup. This section touches on the
essential details of configuring nginx to act as a reverse proxy for both guix-publish, and
Cuirass.
*** Certbot
We would like to provide https access to cuirass, so we require a tls certificate, which we
will provision using [[https://letsencrypt.org/][letsencrypt]] via the [[https://github.com/certbot/certbot][certbot]] tool. Luckily, Guix provides a
[[https://guix.gnu.org/manual/en/html_node/Certificate-Services.html][certbox-service-type]] which can be used to configure certbot. As with prior services, this is
added to our ~services~ field in our ~operating-system~ configuration.
#+begin_src scheme
(service certbot-service-type
(certbot-configuration
(email "collin@rekahsoft.ca")
(certificates
(list
(certificate-configuration
(domains '("cuirass.genenetwork.org"))
(deploy-hook %nginx-deploy-hook))))))
#+end_src
This service references ~%nginx-deploy-hook~, which we define below. It sends ~SIGHUP~ to
restart nginx when certbot renews certificates so the most recent certificate/s are used.
#+begin_src scheme
(define %nginx-deploy-hook
(program-file
"nginx-deploy-hook"
#~(let ((pid (call-with-input-file "/var/run/nginx/pid" read)))
(kill pid SIGHUP))))
#+end_src
Next we define a function we will use later in the [[*Configure Nginx Server Blocks][Configure Nginx Server Blocks]] section
to lookup a certificate or private key file by host in order to reference them when
configuring Nginx tls.
#+begin_src scheme
(define* (le host #:optional privkey)
(string-append "/etc/letsencrypt/live/"
host "/"
(if privkey "privkey" "fullchain")
".pem"))
#+end_src
*** Configure Nginx Location block for ~guix-publish~
Lets define a function that given a url, produces the appropriate nginx location blocks to
enable guix-publish running on some provided URL.
, starting from the definition below that will return a list
of ~nginx-location-configuration~ blocks
#+begin_src scheme
(define (publish-locations url)
"Return the nginx location blocks for 'guix publish' running on URL."
(list (nginx-location-configuration
...) ...)
#+end_src
Lets look and explain the purpose of each location-configuration.
- ~/nix-cache-info~
#+begin_src scheme
(nginx-location-configuration
(uri "/nix-cache-info")
(body
(list
(string-append
"proxy_pass " url "/nix-cache-info;")
;; Cache this file since that's always the first thing we ask
;; for.
"proxy_cache static;"
"proxy_cache_valid 200 100d;" ; cache hits for a looong time.
"proxy_cache_valid any 5m;" ; cache misses/others for 5 min.
"proxy_ignore_client_abort on;"
;; We need to hide and ignore the Set-Cookie header to enable
;; caching.
"proxy_hide_header Set-Cookie;"
"proxy_ignore_headers Set-Cookie;")))
#+end_src
- ~/nar/~
#+begin_src scheme
(nginx-location-configuration
(uri "/nar/")
(body
(list
(string-append "proxy_pass " url ";")
"client_body_buffer_size 256k;"
;; Be more tolerant of delays when fetching a nar.
"proxy_read_timeout 60s;"
"proxy_send_timeout 60s;"
;; Enable caching for nar files, to avoid reconstructing and
;; recompressing archives.
"proxy_cache nar;"
"proxy_cache_valid 200 30d;" ; cache hits for 1 month
"proxy_cache_valid 504 3m;" ; timeout, when hydra.gnu.org is overloaded
"proxy_cache_valid any 1h;" ; cache misses/others for 1h.
"proxy_ignore_client_abort on;"
;; Nars are already compressed.
"gzip off;"
;; We need to hide and ignore the Set-Cookie header to enable
;; caching.
"proxy_hide_header Set-Cookie;"
"proxy_ignore_headers Set-Cookie;"
;; Provide a 'content-length' header so that 'guix
;; substitute-binary' knows upfront how much it is downloading.
;; "add_header Content-Length $body_bytes_sent;"
)))
#+end_src
- ~~ \\.narinfo$~
#+begin_src scheme
(nginx-location-configuration
(uri "~ \\.narinfo$")
(body
(list
;; Since 'guix publish' has its own caching, and since it relies
;; on the atime of cached narinfos to determine whether a
;; narinfo can be removed from the cache, don't do any caching
;; here.
(string-append "proxy_pass " url ";")
;; For HTTP pipelining. This has a dramatic impact on
;; performance.
"client_body_buffer_size 128k;"
;; Narinfos requests are short, serve many of them on a
;; connection.
"keepalive_requests 600;"
;; Do not tolerate slowness of hydra.gnu.org when fetching
;; narinfos: better return 504 quickly than wait forever.
"proxy_connect_timeout 10s;"
"proxy_read_timeout 10s;"
"proxy_send_timeout 10s;"
;; 'guix publish --ttl' produces a 'Cache-Control' header for
;; use by 'guix substitute'. Let it through rather than use
;; nginx's "expire" directive since the expiration time defined
;; by 'guix publish' is the right one.
"proxy_pass_header Cache-Control;"
"proxy_ignore_client_abort on;"
;; We need to hide and ignore the Set-Cookie header to enable
;; caching.
"proxy_hide_header Set-Cookie;"
"proxy_ignore_headers Set-Cookie;")))
#+end_src
- ~/file/~
#+begin_src scheme
;; Content-addressed files served by 'guix publish'.
(nginx-location-configuration
(uri "/file/")
(body
(list
(string-append "proxy_pass " url ";")
"proxy_cache cas;"
"proxy_cache_valid 200 200d;" ; cache hits
"proxy_cache_valid any 5m;" ; cache misses/others
"proxy_ignore_client_abort on;")))
#+end_src
- ~/robots.txt~
First, lets define a string ~publish-robots.txt~, that we'll configure Nginx to serve on the
~/robots.txt~ route to prevent good-faith crawlers from downloading substitutes.
#+begin_src scheme
(define publish-robots.txt
;; Try to prevent good-faith crawlers from downloading substitutes. Allow
;; indexing the root—which is expected to be static or cheap—to remain visible
;; in search engine results for, e.g., 'Guix CI'.
"\
User-agent: *\r
Disallow: /\r
Allow: /$\r
\r
")
#+end_src
To serve this ~robots.txt~ we use a g-exp to store its contents as a file in the guix
store, to be served by Nginx on the ~/robots.txt~ route.
#+begin_src scheme
;; Try to prevent good-faith crawlers from downloading substitutes.
(nginx-location-configuration (uri "= /robots.txt") (body (list #~(string-append "try_files "
#$(plain-file "robots.txt" publish-robots.txt) " =404;") "root /;")))
#+end_src
*** TODO Nginx locations (FIND BETTER NAME)
#+begin_src scheme
(define (balg02-locations publish-url)
"Return nginx location blocks with 'guix publish' reachable at
PUBLISH-URL."
(append (publish-locations publish-url)
(list
;; Cuirass.
(nginx-location-configuration
(uri "/")
(body (list "proxy_pass http://localhost:8081;"
;; ;; See
;; ;; .
;; (string-append
;; "add_header Onion-Location http://" %ci-onion
;; "$request_uri;")
)))
(nginx-location-configuration
(uri "~ ^/admin")
(body
(list "if ($ssl_client_verify != SUCCESS) { return 403; } proxy_pass http://localhost:8081;")))
(nginx-location-configuration
(uri "/static")
(body
(list
"proxy_pass http://localhost:8081;"
;; Cuirass adds a 'Cache-Control' header, honor it.
"proxy_cache static;"
"proxy_cache_valid 200 2d;"
"proxy_cache_valid any 10m;"
"proxy_ignore_client_abort on;")))
(nginx-location-configuration
(uri "/download") ;Cuirass "build products"
(body
(list
"proxy_pass http://localhost:8081;"
"expires 10d;" ;override 'Cache-Control'
"proxy_cache static;"
"proxy_cache_valid 200 30d;"
"proxy_cache_valid any 10m;"
"proxy_ignore_client_abort on;")))
(nginx-location-configuration ;certbot
(uri "/.well-known")
(body (list "root /var/www;"))))))
#+end_src
*** TODO Configure Nginx Server Blocks
#+begin_src scheme
(define %publish-url "http://localhost:3000")
(define %tls-settings
(list
;; Make sure SSL is disabled.
"ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;"
;; Disable weak cipher suites.
"ssl_ciphers HIGH:!aNULL:!MD5;"
"ssl_prefer_server_ciphers on;"
;; Use our own DH parameters created with:
;; openssl dhparam -out dhparams.pem 2048
;; as suggested at .
"ssl_dhparam /etc/dhparams.pem;"))
#+end_src
#+begin_src scheme
(define %balg02-servers
(list
;; Redirect domains that don't explicitly support HTTP (below) to HTTPS.
(nginx-server-configuration
(listen '("80"))
(raw-content
(list "return 308 https://$host$request_uri;")))
;; Domains that still explicitly support plain HTTP.
(nginx-server-configuration
(listen '("80"))
(server-name `("cuirass.genenetwork.org"
;;
"~[0-9]$"))
(locations (balg02-locations %publish-url))
(raw-content
(list
"access_log /var/run/anonip/http.access.log;"
"proxy_set_header X-Forwarded-Host $host;"
"proxy_set_header X-Forwarded-Port $server_port;"
"proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;")))
;; HTTPS servers
(nginx-server-configuration
(listen '("443 ssl"))
(server-name '("cuirass.genenetwork.org"))
(ssl-certificate (le "cuirass.genenetwork.org"))
(ssl-certificate-key (le "cuirass.genenetwork.org" 'key))
(locations (balg02-locations %publish-url))
(raw-content
(append
%tls-settings
(list
"access_log /var/run/anonip/https.access.log;"
"proxy_set_header X-Forwarded-Host $host;"
"proxy_set_header X-Forwarded-Port $server_port;"
"proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
;; TODO:
;; For Cuirass admin interface authentication
;; "ssl_client_certificate /etc/ssl-ca/certs/ca.crt;"
;; "ssl_verify_client optional;"
))))))
#+end_src
#+begin_src scheme
(define %extra-content
(list
"default_type application/octet-stream;"
"sendfile on;"
(accept-languages)
;; Maximum chunk size to send. Partly this is a workaround for
;; , but also the nginx docs mention that
;; "Without the limit, one fast connection may seize the worker
;; process entirely."
;;
"sendfile_max_chunk 1m;"
"keepalive_timeout 65;"
;; Use HTTP 1.1 to talk to the backend so we benefit from keep-alive
;; connections and chunked transfer encoding. The latter allows us to
;; make sure we do not cache partial downloads.
"proxy_http_version 1.1;"
;; The 'inactive' parameter for caching is not very useful in our
;; case: all that matters is that LRU sweeping happens when 'max_size'
;; is hit.
;; cache for nar files
"proxy_cache_path /var/cache/nginx/nar"
" levels=2"
" inactive=8d" ; inactive keys removed after 8d
" keys_zone=nar:4m" ; nar cache meta data: ~32K keys
" max_size=10g;" ; total cache data size max
;; cache for content-addressed files
"proxy_cache_path /var/cache/nginx/cas"
" levels=2"
" inactive=180d" ; inactive keys removed after 180d
" keys_zone=cas:8m" ; nar cache meta data: ~64K keys
" max_size=50g;" ; total cache data size max
;; cache for build logs
"proxy_cache_path /var/cache/nginx/logs"
" levels=2"
" inactive=60d" ; inactive keys removed after 60d
" keys_zone=logs:8m" ; narinfo meta data: ~64K keys
" max_size=4g;" ; total cache data size max
;; cache for static data
"proxy_cache_path /var/cache/nginx/static"
" levels=1"
" inactive=10d" ; inactive keys removed after 10d
" keys_zone=static:1m" ; nar cache meta data: ~8K keys
" max_size=200m;" ; total cache data size max
;; If Hydra cannot honor these delays, then something is wrong and
;; we'd better drop the connection and return 504.
"proxy_connect_timeout 10s;"
"proxy_read_timeout 10s;"
"proxy_send_timeout 10s;"
;; Cache timeouts for a little while to avoid increasing pressure.
"proxy_cache_valid 504 30s;"))
#+end_src
#+begin_src scheme
(define %nginx-configuration
(nginx-configuration
(server-blocks %balg02-servers)
(server-names-hash-bucket-size 128)
(modules
(list
;; Module to redirect users to the localized pages of their choice.
(file-append nginx-accept-language-module
"/etc/nginx/modules/ngx_http_accept_language_module.so")))
(global-directives
'((worker_processes . 16)
(pcre_jit . on)
(events . ((worker_connections . 1024)))))
(extra-content
(string-join %extra-content "\n"))
(shepherd-requirement
(map log-file->anonip-service-name
%anonip-nginx-log-files))))
#+end_src
*** Cache activation
To ensure the nginx cache folder exists on the file-system prior to the first run of nginx, we
create a [[https://guix.gnu.org/manual/en/html_node/Service-Reference.html][simple-service]] that creates the ~/var/cache/nginx~ folder upon system activation.
#+begin_src scheme
(define %nginx-cache-activation
;; Make sure /var/cache/nginx exists on the first run.
(simple-service 'nginx-/var/cache/nginx
activation-service-type
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(mkdir-p "/var/cache/nginx")))))
#+end_src
This service can then be added to the ~services~ field of our ~operating-system~
configuration, finalization our configuration of nginx.
** Setup guix-daemon
- Allow for substitutes from this server
- Adjust guix-daemon configuration (timeouts, # of build accounts, # of cores to use)
#+begin_src scheme
(define* (guix-daemon-config #:key (max-jobs 5) (cores 4)
(build-accounts-to-max-jobs-ratio 4)
(authorized-keys '())
(substitute-urls '()))
(guix-configuration
(substitute-urls substitute-urls)
(authorized-keys authorized-keys)
;; We don't want to let builds get stuck for too long, but we still want
;; to allow building things that can take a while (eg. 3h). Adjust as necessary.
(max-silent-time 3600)
(timeout (* 6 3600))
(log-compression 'gzip) ;be friendly to 'guix publish' users
(build-accounts (* build-accounts-to-max-jobs-ratio max-jobs))
(extra-options (list "--max-jobs" (number->string max-jobs)
"--cores" (number->string cores)
"--gc-keep-derivations"))))
#+end_src
#+begin_src scheme
(modify-services %base-services
(guix-service-type config => (guix-daemon-config
#:substitute-urls
'("https://cuirass.genenetwork.org")
#:max-jobs 20
#:cores 4
#:authorized-keys
(cons
(local-file "../../../.pubkeys/guix/cuirass.genenetwork.org.pub")
%default-authorized-guix-keys)
#:build-accounts-to-max-jobs-ratio 5)))
#+end_src
*** TODO Setup Client Certificates for Cuirass Administration
** Installation
We've left out other aspects of defining our ~operating-system~ configuration, as its [[https://guix.gnu.org/manual/devel/en/html_node/operating_002dsystem-Reference.html][well
documented]] by the Guix project, and varies depending on the specifics of your machine (for
instance, file-systems, users/groups, initrd-modules, etc..). Once assembled, the
~operating-system~ configuration can be placed in a file ~system.scm~, which defines a guix
system configured with Cuirass, guix-publish, guix-daemon, and nginx, setup to act as a
single-node build farm and substitute server! It can be trivially installed following the
[[https://guix.gnu.org/manual/en/html_node/Proceeding-with-the-Installation.html][manual Guix System installation documentation]].
#+begin_src shell
guix system init system.scm /mnt
#+end_src
Here ~/mnt~ is where the target root file-system is mounted after being prepared for
installation. See our [[https://git.genenetwork.org/guix-north-america/tree/docs/initial-setup.org][Initial Setup Documentation]] for the specifics of preparing the disks
for Guix installation in our case.
*** Remotely Bootstrapping Guix from Debian
In our case with the Tennessee Build Farm, physical access was inconvenient due to travel
distance, so installation needed to be completed remotely. The target server already had
Debian running on it, configured with serial access available out-of-band via [[https://en.wikipedia.org/wiki/Dell_DRAC][Dell iDRAC]] as
well as ssh access to Debian. Another ssd was available for use, and was the target of our
Guix installation.
Guix can be [[https://guix.gnu.org/manual/en/html_node/Binary-Installation.html][installed on foreign distributions]], which is well documented, so its not covered
here, but is the first step in bootstrapping Guix from Debian.
**** Guix Configuration as a Channel
Once Guix (the package manager) is installed on Debian, we need to ensure partition our
drives as required (which again varies, so will not be covered here). Next we need to make
our ~operating-system~ configuration available so we can complete bootstrapping Guix. The
most straightforward way to do so is to just copy ~system.scm~ file we defined earlier.
Another way is to capture the configuration in a git repository, and [[https://guix.gnu.org/manual/en/html_node/Creating-a-Channel.html][make it a Guix channel]].
This enables great shared tracking of system changes over time (outside of [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html][guix system
list-generations ...]]), and is what was done while bootstrapping our installation.
We choose to include a ~channels.scm~ file as part of the configuration channel that pins the
versions of software that will be used with our deployment. So, in order to boot the server
using an ~operating-system~ defined in [[https://git.genenetwork.org/guix-north-america/][our configuration channel]], we first download the
~channels.scm~ file.
#+begin_src shell
curl -O https://git.genenetwork.org/guix-north-america/plain/channels.scm
#+end_src
We then create a temporary ~bootstrap.scm~ file that contains a references to the
~operating-system~ we defined in our configuration channel.
#+begin_src scheme
(@ (guix-na config balg02) balg02)
#+end_src
We then use ~guix time-machine~ to specify these channels when installing Guix onto the
system.
#+begin_src shell
guix time-machine -C channels.scm -- system init bootstrap.scm /mnt
#+end_src
Subsequent updates to the system can be done without using the ~bootstrap.scm~ file. For
instance, say the guix channel is updated in ~channels.scm~. To apply this change to the
server, the new channels would need to be pulled, and the system reconfigured.
#+begin_src shell
sudo -i guix pull -C <(curl https://git.genenetwork.org/guix-north-america/plain/channels.scm)
sudo -i guix system reconfigure -e '(@ (guix-na config balg02) balg02)'
#+end_src
One caveat to using ~-e|--expression~ is that currently this expression is not stored along
side the guix system generation, which makes it not possible to know which ~operating-system~
configuration was used from a given channel (reported upstream as issue [[https://issues.guix.gnu.org/54631][#54631]]). To work
around this for the time being, a file containing this expression can be used (just like was
used for bootstrapping).
* Challenges and Lessons Learned
Setting up a public Guix substitute server is not without its challenges:
1. **Performance Tuning**: Configuring Cuirass and the Guix daemon to efficiently use
available resources required careful optimization.
2. **Privacy Considerations**: Implementing IP anonymization with anonip was crucial to
protect user privacy.
3. **Bandwidth and Storage Management**: Implementing intelligent caching strategies to
manage storage and network resources.
Luckily, many of these challenges had already been sorted out by existing Guix build farms,
making this endeavor much easier.
The biggest challenge was remote installation, where one hiccup was not realizing the
~megaraid_sas~ module was needed for our root ssd used for Guix, resulting in a failed first
boot following bootstrapping. Luckily this was resolved by booting to Debian and
reconfigure-ing Guix after adjusting our ~initrd-modules~ to include ~megaraid_sas~.
* Future Roadmap
Looking ahead, we have several goals for the cuirass.genenetwork.org substitute server:
- Collaborate with Guix maintainers to potentially include this server in the included list
of default Guix substitute servers
- Expand build coverage to include more architectures and specialized packages
- Implement more sophisticated monitoring and performance tracking
- Explore potential partnerships with other academic and research institutions
* Conclusion
The Tennessee Guix Build Farm represents more than just a technical infrastructure project.
It embodies the spirit of open-source collaboration, community-driven development, and the
principles of reproducible computing. By providing a robust, privacy-conscious substitute
server, we hope to contribute to the growth and accessibility of the GNU Guix ecosystem.
We invite other organizations, universities, and community members to consider setting up
their own substitute servers. Each new node makes the Guix network stronger, more resilient,
and more accessible.
**Acknowledgments**: Special thanks to GeneNetwork.org for their hardware support and the GNU
Guix community for their ongoing innovation in package management and reproducible systems.