path: root/docs/blog-Tennnessee-build-farm.org

#+TITLE: Setup of a Simple Guix Build Farm and Substitute Server
#+AUTHOR: Collin J. Doering

# FIXME: in a perfect world, I would set text wrapping automatically for latex minted like so:
# -*- org-latex-minted-options: '(("breaklines" . "true") ("breakanywhere" . "true")); -*-
# However, this currently fails (likely due to a bug?)

#+LATEX_HEADER: \usepackage[margin=1.5cm]{geometry}
#+LATEX_HEADER: \usepackage{xcolor}
#+LATEX_HEADER: \definecolor{link}{HTML}{506060}
#+LATEX_HEADER: \hypersetup{colorlinks=true,citecolor=link,filecolor=link,linkcolor=link,urlcolor=link}

#+LATEX_HEADER_EXTRA: \usepackage{mdframed}
#+LATEX_HEADER_EXTRA: \BeforeBeginEnvironment{minted}{\begin{mdframed}}
#+LATEX_HEADER_EXTRA: \AfterEndEnvironment{minted}{\end{mdframed}}


#+latex: \clearpage

In the world of reproducible computing, GNU Guix stands out as a pioneering distribution that
enables bit-for-bit reproducible builds and a comprehensive package management system.
However, building software from source for every package can be time-consuming and
resource-intensive. This is where substitute servers play a crucial role, allowing users to
download pre-built binary packages instead of compiling them locally.

Here we built a dedicated build farm and substitute server in just few lines of code!

Guix hands us the programmer's dream of deploying a full server with its services and
orchestration as a program! Not only can the 'program' be adapted to other machines -- across
the planet -- the deployment also benefits from Guix reproducibility guarantees, including
natural roll-backs of full and partial deployment. The federated nature of these build farms
is increasingly important - in fact, I invite you to roll-your-own based on this article.

Earlier this year [[https://lists.gnu.org/archive/html/guix-devel/2024-07/msg00033.html][I announced on the guix mailing list]] that a new North American based Guix
substitute server and build farm, cuirass.genenetwork.org, was available for general use.
Having a server in the USA increases the speed and reduces the latency for North American
downloaders of Guix and all 20,000+ built free software packages. It also provides redundancy
in case something happens with other substitute servers.

* Why Build Another Substitute Server?

The Guix ecosystem thrives on diversity and decentralization. By establishing additional
substitute servers, we achieve several critical objectives:

- **Improved Build Diversity**: Multiple independent build farms reduce the risk of
  single-point-of-failure and increase the verification of build reproducibility.
- **Reduced Latency**: Geographically distributed servers mean faster download times for
  users in different regions.
- **Increased Resilience**: If one substitute server is down, users can fall back to
  alternatives.
- **Community Contribution**: Each new substitute server strengthens the broader Guix
  infrastructure.

This article provides a comprehensive guide to setting up a Guix build farm and substitute
server, drawing inspiration from existing GNU Guix project infrastructure. You can see their
full source code [[https://git.savannah.gnu.org/cgit/guix/maintenance.git][here]].

* TODO Hardware and Infrastructure

The Tennessee Guix Build Farm was made possible through a collaboration with GeneNetwork.org,
who provided the following server specifications:

- **Processor**: Dual AMD EPYC 9274F 24-Core, 48 Thread Processors
- **RAM**: 768 GB DDR5 ECC
- **Storage**: 1 TB SSD
- **Network**: 1 Gbps nic, 100Mbps dedicated connection
- **Location**: Memphis TN

These robust specifications allow for efficient package building, caching, and serving of
substitutes for the Guix community.

FIXME: how much do we actually need in terms of cores and RAM? That is if someone else wants to create a build farm.

* Components of the Guix Build Farm

** Cuirass - building packages

Cuirass is the GNU Guix continuous integration software, and is responsible for watching for
changes to one or more VCS repositories (usually [[https://guix.gnu.org/manual/en/html_node/Channels.html][Guix channels]]), executing build jobs for
packages that have changed, and finally, storing build results in its database.

*** Define Cuirass Specs

In order to run Cuirass via the ~cuirass-service-type~, we need to define what we
want Cuirass to build. In the case of guix-na, we want to build the GNU Guix distribution and
its packages, so we declare ~%cuirass-specs~ as a [[https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html][G-Expression]] that will return a list of
[[https://guix.gnu.org/cuirass/manual/html_node/Specifications.html][cuirass specifications]] with a single entry named "guix", which does exactly that!

#+begin_src scheme
  (define %cuirass-specs
    #~(list (specification
               (name "guix")
               (priority 0)
               (build '(channels guix))
               (channels %default-channels))))
#+end_src

If you have custom channels you wish to build, you would add a new ~specification~ to the
list. Cuirass can build more then just channels and their packages, it can also build images,
tarballs, a specific set of packages, a manifest, and more! See the [[https://guix.gnu.org/cuirass/manual/html_node/Specifications.html][Cuirass specification]]
documentation for more details.

Its worth noting that if you're following along by reviewing the full source code of the
Tennessee build farm that there is an additional specification listed (for guix-na itself!).
More details on that in [[*Guix Configuration as a Channel][Guix Configuration as a Channel]].

*** Setup Cuirass Service

Now that we have defined what we want Cuirass to build, we need to specify its guix service
in the ~services~ field of our ~operating-system~ definition, which in turn will run Cuirass.
We are going to setup nginx as a reverse proxy for cuirass later on, so we'll set its host to
localhost, and pass along the specifications we defined earlier.

#+begin_src scheme
  (service cuirass-service-type
           (cuirass-configuration
            (host "localhost")
            (specifications %cuirass-specs)))
#+end_src

** Provide Substitutes using Guix Publish

With Cuirass configured and the guix store being populated with package builds as the guix
channel changes, we now turn our attention to serving these builds as substitutes to Guix
users. This is done using [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html][guix publish]], which Guix provides the [[file:~/.org/roam/20221129213953-advent_of_code.org::*Day 2][guix-publish-service-type]],
which is used in the ~services~ field of ~operating-system~ definition.

#+begin_src scheme
  (service guix-publish-service-type
           (guix-publish-configuration
            (port 3000)
            (cache "/var/cache/guix/publish")
            (ttl (* 90 24 3600))
#+end_src

Similar to Cuirass, access to guix-publish will be provided through nginx as a reverse proxy.

** Anonomize IPs (anonip)

Guix users care about their privacy, and though this is not necessarily a requirement,
anonymizing nginx access logs using the anonip is implemented by all public Guix sponsored
build farms, so keeping with this privacy preserving trend, cuirass.genenetwork.org
implements the same log anonymization.

To anonymize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_node/Log-Rotation.html][anonip-service-type]] will be configured and used, however, we
want to anonymize multiple log files, which means multiple instances of the anonip running.
To assist with this, a helper function ~anonip-service~ is defined.

#+begin_src scheme
  (define (anonip-service file)
    (service anonip-service-type
             (anonip-configuration
              (input (format #false "/var/run/anonip/~a" file))
              (output (format #false "/var/log/anonip/~a" file)))
#+end_src

Additionally, for services that will leverage these anonymized logs (in our case, only
nginx), it will be necessary to ensure that the appropriate instance of anonip is running
prior to the respective service that will utilize it. To help declare this dependency,
another helper function is defined.

#+begin_src scheme
  (define (log-file->anonip-service-name file)
    "Return the name of the Anonip service handling FILE, a log file."
    (symbol-append 'anonip-/var/log/anonip/ (string->symbol file)))
#+end_src

We also define a list of anonymized log files which will be used later on along side the
~log-file->anonip-service-name~ function in order to define shepherd service dependencies for
nginx.

#+begin_src scheme
  (define %anonip-nginx-log-files
    ;; List of files handled by Anonip for nginx
    '("http.access.log"
      "https.access.log"))
#+end_src

All that remains is to ensure that for each log file we are anonymizing, we start a
corresponding anonip-service. This can be added to the ~services~ field of our
~operating-system~ declaration.

#+begin_src scheme
  (map anonip-service %anonip-nginx-log-files)
#+end_src

** Nginx Reverse Proxy

Nginx is arguably the most complicated part of the setup. This section touches on the
essential details of configuring nginx to act as a reverse proxy for both guix-publish, and
Cuirass.

*** Certbot

We would like to provide https access to cuirass, so we require a tls certificate, which we
will provision using [[https://letsencrypt.org/][letsencrypt]] via the [[https://github.com/certbot/certbot][certbot]] tool. Luckily, Guix provides a
[[https://guix.gnu.org/manual/en/html_node/Certificate-Services.html][certbox-service-type]] which can be used to configure certbot. As with prior services, this is
added to our ~services~ field in our ~operating-system~ configuration.

#+begin_src scheme
  (service certbot-service-type
           (certbot-configuration
            (email "collin@rekahsoft.ca")
            (certificates
             (list
              (certificate-configuration
               (domains '("cuirass.genenetwork.org"))
               (deploy-hook %nginx-deploy-hook))))))
#+end_src

This service references ~%nginx-deploy-hook~, which we define below. It sends ~SIGHUP~ to
restart nginx when certbot renews certificates so the most recent certificate/s are used.

#+begin_src scheme
  (define %nginx-deploy-hook
    (program-file
     "nginx-deploy-hook"
     #~(let ((pid (call-with-input-file "/var/run/nginx/pid" read)))
         (kill pid SIGHUP))))
#+end_src

Next we define a function we will use later in the [[*Configure Nginx Server Blocks][Configure Nginx Server Blocks]] section
to lookup a certificate or private key file by host in order to reference them when
configuring Nginx tls.

#+begin_src scheme
  (define* (le host #:optional privkey)
    (string-append "/etc/letsencrypt/live/"
                   host "/"
                   (if privkey "privkey" "fullchain")
                   ".pem"))
#+end_src

*** TODO Configure Nginx Location block for ~guix-publish~

Lets define a function that given a url, produces a list of appropriate nginx location blocks
to enable guix-publish running on some provided URL.

#+begin_src scheme
  (define (publish-locations url)
    "Return the nginx location blocks for 'guix publish' running on URL."
    (list (nginx-location-configuration ...) ...)
#+end_src

Starting from the definition above, lets fill in and explain the purpose of each
location-configuration in the list that will be returned from our function.

- ~/nix-cache-info~

  ~guix-publish~ provides a route ~/nix-cache-info~ which returns text/plain content of
  key/value pairs, used by cooperating clients (which is normally the case, when using the
  ~guix ...~ command). Given away by the reference to [[https://nixos.org/][Nix]] in its name, this route (and some
  aspects of the design of guix-publish) are inspired by nix-serve, and the [[https://nixos.wiki/wiki/Binary_Cache][Nix Binary Cache]].

  At the time of writing this article, the following is returned from
  cuirass.genenetwork.org/nix-cache-info, and will be returned from every guix-published
  based substitute server.

  #+begin_src text
    StoreDir: /gnu/store
    WantMassQuery: 0
    Priority: 100
  #+end_src

  FIXME: Its interesting to note that outside of the expected difference of ~StoreDir~
  varying on Nix and Guix (on nix its ~/nix/store~), ~WantMassQuery~ and ~Priority~ are both
  set to static values for Nix and Guix, but they vary!

  Now that we have some more context on the route, here is the nginx-location-configuration
  we will return to proxy requests appropriately.

  #+begin_src scheme
    (nginx-location-configuration
     (uri "/nix-cache-info")
     (body
      (list
       (string-append
        "proxy_pass " url "/nix-cache-info;")
       ;; Cache this file since that's always the first thing we ask
       ;; for.
       "proxy_cache static;"
       "proxy_cache_valid 200 100d;"     ; cache hits for a looong time.
       "proxy_cache_valid any 5m;"       ; cache misses/others for 5 min.
       "proxy_ignore_client_abort on;"

       ;; We need to hide and ignore the Set-Cookie header to enable
       ;; caching.
       "proxy_hide_header    Set-Cookie;"
       "proxy_ignore_headers Set-Cookie;")))
  #+end_src

- ~~ \\.narinfo$~

  [[https://nix.dev/manual/nix/2.22/protocols/nix-archive][NAR (Nix Archive Format)]] is the format used by Guix for cached substitutes. To get a sense
  for how substitutes are downloaded from a guix-publish substitute server, lets take a
  moment to do so manually for the [[https://packages.guix.gnu.org/packages/hello/2.12.1/][hello]] package.

  First, lets find the store path of the package (but without actually building it).

  #+begin_src shell :results output code :wrap src text :exports both
    guix build --dry-run hello
  #+end_src

  #+RESULTS:
  #+begin_src text
  /gnu/store/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1
  #+end_src

  The result is composed of a few parts:

  - ~/gnu/store/~ :: the guix store path
  - ~8bjy9g0cssjrw9ljz2r8ww1sma95isfj~ :: a hash uniquely identifying the store item
  - ~-hello-2.12.1~ :: the package-name and version, separated by dashes

  We now have enough context to define our route matching anything that ends in ~.narinfo~.

  #+begin_src scheme
    (nginx-location-configuration
     (uri "~ \\.narinfo$")
     (body
      (list
       ;; Since 'guix publish' has its own caching, and since it relies
       ;; on the atime of cached narinfos to determine whether a
       ;; narinfo can be removed from the cache, don't do any caching
       ;; here.
       (string-append "proxy_pass " url ";")

       ;; For HTTP pipelining.  This has a dramatic impact on
       ;; performance.
       "client_body_buffer_size 128k;"

       ;; Narinfos requests are short, serve many of them on a
       ;; connection.
       "keepalive_requests 600;"

       ;; Do not tolerate slowness of hydra.gnu.org when fetching
       ;; narinfos: better return 504 quickly than wait forever.
       "proxy_connect_timeout 10s;"
       "proxy_read_timeout 10s;"
       "proxy_send_timeout 10s;"

       ;; 'guix publish --ttl' produces a 'Cache-Control' header for
       ;; use by 'guix substitute'.  Let it through rather than use
       ;; nginx's "expire" directive since the expiration time defined
       ;; by 'guix publish' is the right one.
       "proxy_pass_header Cache-Control;"

       "proxy_ignore_client_abort on;"

       ;; We need to hide and ignore the Set-Cookie header to enable
       ;; caching.
       "proxy_hide_header    Set-Cookie;"
       "proxy_ignore_headers Set-Cookie;")))
  #+end_src

- ~/nar/~

  As part of defining the nginx-location-configuration for ~*.narinfo~ routes, we started to
  manually fetch a substitute. Here we will continue, using the provided hash to query to
  query the substitute server for a corresponding ~.narinfo~ file.

  #+begin_src shell :results output
    curl https://cuirass.genenetwork.org/8bjy9g0cssjrw9ljz2r8ww1sma95isfj.narinfo
  #+end_src

  #+RESULTS:
  #+begin_example
  StorePath: /gnu/store/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1
  NarHash: sha256:0f94l0bl09i2igkhklzkawqbbdn4kkxl90wbb4y7f0dnni4f6ljh
  NarSize: 235240
  References: 8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1 zvlp3n8iwa1svxmwv4q22pv1pb1c9pjq-glibc-2.39 zzpbp6rr43smwxzvzd4qd317z5j7qblj-gcc-11.4.0-lib
  Deriver: 79dhya6sngg4rf53m1cyxlhn8y4pnw2n-hello-2.12.1.drv
  Signature: 1;balg02;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNjE0ODFDNDUzMkU3RTIyOUEzMDlDREVBRDM2MkE2Qzk4QjU0RkFDNEUyQjA1ODEzQ0ZDOEI1NzQ2RUY0NjYxMiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDQ2NTA3Q0FBNUJFNEY1QUQxRUE0NzUwQzlEMjgzMjQ5NDMwMDQ1OEIzRTM5QUJDOTBFMjZGNkU0MTA0RjMwNCMpCiAgIChzICMwQjdERDlCRUE5ODA0MTkyQ0E2OTUwQzFGRUYzRDdEQ0M3RTMyQzNEMENGNDg3NkY4RkRBMzEwRTUzNkYwNEVBIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOTU3OEFENkNEQjIzQkE1MUY5QzQxODVENUQ1QTMyQTdFRUI0N0FDREQ1NUYxQ0NCOENFRTRFMDU3MEZCRjk2MSMpCiAgICkKICApCiApCg==
  URL: nar/gzip/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1
  Compression: gzip
  FileSize: 73331
  URL: nar/zstd/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1
  Compression: zstd
  FileSize: 65480
  #+end_example

  If the package is not available, this would return a ~404~ not found error. However, in our
  case the substitute is found, and various details about it are provided back to the
  requester. The ~URL~ field is notable, as it will be used next to download the substitute,
  which is why we define another proxied route for ~/nar/~ below.

  #+begin_src scheme
    (nginx-location-configuration
     (uri "/nar/")
     (body
      (list
       (string-append "proxy_pass " url ";")
       "client_body_buffer_size 256k;"

       ;; Be more tolerant of delays when fetching a nar.
       "proxy_read_timeout 60s;"
       "proxy_send_timeout 60s;"

       ;; Enable caching for nar files, to avoid reconstructing and
       ;; recompressing archives.
       "proxy_cache nar;"
       "proxy_cache_valid 200 30d;" ; cache hits for 1 month
       "proxy_cache_valid 504 3m;"  ; timeout, when hydra.gnu.org is overloaded
       "proxy_cache_valid any 1h;"  ; cache misses/others for 1h.

       "proxy_ignore_client_abort on;"

       ;; Nars are already compressed.
       "gzip off;"

       ;; We need to hide and ignore the Set-Cookie header to enable
       ;; caching.
       "proxy_hide_header    Set-Cookie;"
       "proxy_ignore_headers Set-Cookie;"

       ;; Provide a 'content-length' header so that 'guix
       ;; substitute-binary' knows upfront how much it is downloading.
       ;; "add_header Content-Length $body_bytes_sent;"
       )))
  #+end_src

- ~/file/~

  FIXME: provide more context

  #+begin_src scheme
    ;; Content-addressed files served by 'guix publish'.
    (nginx-location-configuration
     (uri "/file/")
     (body
      (list
       (string-append "proxy_pass " url ";")

       "proxy_cache cas;"
       "proxy_cache_valid 200 200d;"          ; cache hits
       "proxy_cache_valid any 5m;"            ; cache misses/others

       "proxy_ignore_client_abort on;")))
  #+end_src

- ~/robots.txt~

  First, lets define a string ~publish-robots.txt~, that we'll configure Nginx to serve on the
  ~/robots.txt~ route to prevent good-faith crawlers from downloading substitutes.

  #+begin_src scheme
    (define publish-robots.txt
      ;; Try to prevent good-faith crawlers from downloading substitutes.  Allow
      ;; indexing the root—which is expected to be static or cheap—to remain visible
      ;; in search engine results for, e.g., 'Guix CI'.
      "\
    User-agent: *\r
    Disallow: /\r
    Allow: /$\r
    \r
    ")
  #+end_src

  To serve this ~robots.txt~ we use a g-exp to store its contents as a file in the guix
  store, to be served by Nginx on the ~/robots.txt~ route.

  #+begin_src scheme
    ;; Try to prevent good-faith crawlers from downloading substitutes.
    (nginx-location-configuration
     (uri "= /robots.txt")
     (body
      (list
       #~(string-append "try_files "
    		    #$(plain-file "robots.txt" publish-robots.txt)
    		    " =404;")
       "root /;")))
  #+end_src

*** TODO Nginx locations (FIND BETTER NAME)

#+begin_src scheme
  (define (balg02-locations publish-url)
    "Return nginx location blocks with 'guix publish' reachable at
  PUBLISH-URL."
    (append (publish-locations publish-url)
            (list
             ;; Cuirass.
             (nginx-location-configuration
              (uri "/")
              (body (list "proxy_pass http://localhost:8081;"
                          ;; ;; See
                          ;; ;; <https://community.torproject.org/onion-services/advanced/onion-location/>.
                          ;; (string-append
                          ;;  "add_header Onion-Location http://" %ci-onion
                          ;;  "$request_uri;")
                          )))
             (nginx-location-configuration
              (uri "~ ^/admin")
              (body
               (list "if ($ssl_client_verify != SUCCESS) { return 403; } proxy_pass http://localhost:8081;")))

             (nginx-location-configuration
              (uri "/static")
              (body
               (list
                "proxy_pass http://localhost:8081;"
                ;; Cuirass adds a 'Cache-Control' header, honor it.
                "proxy_cache static;"
                "proxy_cache_valid 200 2d;"
                "proxy_cache_valid any 10m;"
                "proxy_ignore_client_abort on;")))

             (nginx-location-configuration
              (uri "/download")                     ;Cuirass "build products"
              (body
               (list
                "proxy_pass http://localhost:8081;"
                "expires 10d;"                      ;override 'Cache-Control'
                "proxy_cache static;"
                "proxy_cache_valid 200 30d;"
                "proxy_cache_valid any 10m;"
                "proxy_ignore_client_abort on;")))

             (nginx-location-configuration          ;certbot
              (uri "/.well-known")
              (body (list "root /var/www;"))))))
#+end_src

*** TODO Configure Nginx Server Blocks

#+begin_src scheme
  (define %publish-url "http://localhost:3000")

  (define %tls-settings
    (list
     ;; Make sure SSL is disabled.
     "ssl_protocols       TLSv1.1 TLSv1.2 TLSv1.3;"
     ;; Disable weak cipher suites.
     "ssl_ciphers         HIGH:!aNULL:!MD5;"
     "ssl_prefer_server_ciphers on;"

     ;; Use our own DH parameters created with:
     ;;    openssl dhparam -out dhparams.pem 2048
     ;; as suggested at <https://weakdh.org/sysadmin.html>.
     "ssl_dhparam         /etc/dhparams.pem;"))
#+end_src

#+begin_src scheme
  (define %balg02-servers
    (list
     ;; Redirect domains that don't explicitly support HTTP (below) to HTTPS.
     (nginx-server-configuration
      (listen '("80"))
      (raw-content
       (list "return 308 https://$host$request_uri;")))

     ;; Domains that still explicitly support plain HTTP.
     (nginx-server-configuration
      (listen '("80"))
      (server-name `("cuirass.genenetwork.org"
                     ;; <https://logs.guix.gnu.org/guix/2021-11-20.log#155427>
                     "~[0-9]$"))
      (locations (balg02-locations %publish-url))
      (raw-content
       (list
        "access_log  /var/run/anonip/http.access.log;"
        "proxy_set_header X-Forwarded-Host $host;"
        "proxy_set_header X-Forwarded-Port $server_port;"
        "proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;")))

     ;; HTTPS servers
     (nginx-server-configuration
      (listen '("443 ssl"))
      (server-name '("cuirass.genenetwork.org"))
      (ssl-certificate (le "cuirass.genenetwork.org"))
      (ssl-certificate-key (le "cuirass.genenetwork.org" 'key))
      (locations (balg02-locations %publish-url))
      (raw-content
       (append
        %tls-settings
        (list
         "access_log  /var/run/anonip/https.access.log;"
         "proxy_set_header X-Forwarded-Host $host;"
         "proxy_set_header X-Forwarded-Port $server_port;"
         "proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;"
         ;; TODO:
         ;; For Cuirass admin interface authentication
         ;; "ssl_client_certificate /etc/ssl-ca/certs/ca.crt;"
         ;; "ssl_verify_client optional;"
         ))))))
#+end_src

#+begin_src scheme
  (define %extra-content
    (list
     "default_type  application/octet-stream;"
     "sendfile        on;"

     (accept-languages)

     ;; Maximum chunk size to send.  Partly this is a workaround for
     ;; <http://bugs.gnu.org/19939>, but also the nginx docs mention that
     ;; "Without the limit, one fast connection may seize the worker
     ;; process entirely."
     ;;  <http://nginx.org/en/docs/http/ngx_http_core_module#sendfile_max_chunk>
     "sendfile_max_chunk 1m;"

     "keepalive_timeout  65;"

     ;; Use HTTP 1.1 to talk to the backend so we benefit from keep-alive
     ;; connections and chunked transfer encoding.  The latter allows us to
     ;; make sure we do not cache partial downloads.
     "proxy_http_version 1.1;"

     ;; The 'inactive' parameter for caching is not very useful in our
     ;; case: all that matters is that LRU sweeping happens when 'max_size'
     ;; is hit.

     ;; cache for nar files
     "proxy_cache_path /var/cache/nginx/nar"
     "     levels=2"
     "     inactive=8d"	    ; inactive keys removed after 8d
     "     keys_zone=nar:4m"  ; nar cache meta data: ~32K keys
     "     max_size=10g;"     ; total cache data size max

     ;; cache for content-addressed files
     "proxy_cache_path /var/cache/nginx/cas"
     "     levels=2"
     "     inactive=180d"	    ; inactive keys removed after 180d
     "     keys_zone=cas:8m"  ; nar cache meta data: ~64K keys
     "     max_size=50g;"         ; total cache data size max

     ;; cache for build logs
     "proxy_cache_path /var/cache/nginx/logs"
     "     levels=2"
     "     inactive=60d"          ; inactive keys removed after 60d
     "     keys_zone=logs:8m"     ; narinfo meta data: ~64K keys
     "     max_size=4g;"          ; total cache data size max

     ;; cache for static data
     "proxy_cache_path /var/cache/nginx/static"
     "     levels=1"
     "     inactive=10d"	       ; inactive keys removed after 10d
     "     keys_zone=static:1m"   ; nar cache meta data: ~8K keys
     "     max_size=200m;"        ; total cache data size max

     ;; If Hydra cannot honor these delays, then something is wrong and
     ;; we'd better drop the connection and return 504.
     "proxy_connect_timeout 10s;"
     "proxy_read_timeout 10s;"
     "proxy_send_timeout 10s;"

     ;; Cache timeouts for a little while to avoid increasing pressure.
     "proxy_cache_valid 504 30s;"))
#+end_src

#+begin_src scheme
  (define %nginx-configuration
    (nginx-configuration
     (server-blocks %balg02-servers)
     (server-names-hash-bucket-size 128)
     (modules
      (list
       ;; Module to redirect users to the localized pages of their choice.
       (file-append nginx-accept-language-module
                    "/etc/nginx/modules/ngx_http_accept_language_module.so")))
     (global-directives
      '((worker_processes . 16)
        (pcre_jit . on)
        (events . ((worker_connections . 1024)))))
     (extra-content
      (string-join %extra-content "\n"))
     (shepherd-requirement
      (map log-file->anonip-service-name
           %anonip-nginx-log-files))))
#+end_src

*** Cache activation

To ensure the nginx cache folder exists on the file-system prior to the first run of nginx, we
create a [[https://guix.gnu.org/manual/en/html_node/Service-Reference.html][simple-service]] that creates the ~/var/cache/nginx~ folder upon system activation.

#+begin_src scheme
  (define %nginx-cache-activation
    ;; Make sure /var/cache/nginx exists on the first run.
    (simple-service 'nginx-/var/cache/nginx
                    activation-service-type
                    (with-imported-modules '((guix build utils))
                      #~(begin
                          (use-modules (guix build utils))
                          (mkdir-p "/var/cache/nginx")))))
#+end_src

This service can then be added to the ~services~ field of our ~operating-system~
configuration, finalization our configuration of nginx.

** TODO Setup guix-daemon

- Allow for substitutes from this server
- Adjust guix-daemon configuration (timeouts, # of build accounts, # of cores to use)

#+begin_src scheme
  (define* (guix-daemon-config #:key (max-jobs 5) (cores 4)
                               (build-accounts-to-max-jobs-ratio 4)
                               (authorized-keys '())
                               (substitute-urls '()))
    (guix-configuration
     (substitute-urls substitute-urls)
     (authorized-keys authorized-keys)

     ;; We don't want to let builds get stuck for too long, but we still want
     ;; to allow building things that can take a while (eg. 3h). Adjust as necessary.
     (max-silent-time 3600)
     (timeout (* 6 3600))

     (log-compression 'gzip)               ;be friendly to 'guix publish' users

     (build-accounts (* build-accounts-to-max-jobs-ratio max-jobs))
     (extra-options (list "--max-jobs" (number->string max-jobs)
                          "--cores" (number->string cores)
                          "--gc-keep-derivations"))))
#+end_src

#+begin_src scheme
  (modify-services %base-services
                   (guix-service-type config => (guix-daemon-config
                                                 #:substitute-urls
                                                 '("https://cuirass.genenetwork.org")
                                                 #:max-jobs 20
                                                 #:cores 4
                                                 #:authorized-keys
                                                 (cons
                                                  (local-file "../../../.pubkeys/guix/cuirass.genenetwork.org.pub")
                                                  %default-authorized-guix-keys)
                                                 #:build-accounts-to-max-jobs-ratio 5)))
#+end_src

*** TODO Setup Client Certificates for Cuirass Administration

** Installation

We've left out other aspects of defining our ~operating-system~ configuration, as its [[https://guix.gnu.org/manual/devel/en/html_node/operating_002dsystem-Reference.html][well
documented]] by the Guix project, and varies depending on the specifics of your machine (for
instance, file-systems, users/groups, initrd-modules, etc..). Once assembled, the
~operating-system~ configuration can be placed in a file ~system.scm~, which defines a guix
system configured with Cuirass, guix-publish, guix-daemon, and nginx, setup to act as a
single-node build farm and substitute server! It can be trivially installed following the
[[https://guix.gnu.org/manual/en/html_node/Proceeding-with-the-Installation.html][manual Guix System installation documentation]].

#+begin_src shell
  guix system init system.scm /mnt
#+end_src

Here ~/mnt~ is where the target root file-system is mounted after being prepared for
installation. See our [[https://git.genenetwork.org/guix-north-america/tree/docs/initial-setup.org][Initial Setup Documentation]] for the specifics of preparing the disks
for Guix installation in our case.

*** Remotely Bootstrapping Guix from Debian

In our case with the Tennessee Build Farm, physical access was inconvenient due to travel
distance, so installation needed to be completed remotely. The target server already had
Debian running on it, configured with serial access available out-of-band via [[https://en.wikipedia.org/wiki/Dell_DRAC][Dell iDRAC]] as
well as ssh access to Debian. Another ssd was available for use, and was the target of our
Guix installation.

Guix can be [[https://guix.gnu.org/manual/en/html_node/Binary-Installation.html][installed on foreign distributions]], which is well documented, so its not covered
here, but is the first step in bootstrapping Guix from Debian.

**** Guix Configuration as a Channel

Once Guix (the package manager) is installed on Debian, we need to ensure partition our
drives as required (which again varies, so will not be covered here). Next we need to make
our ~operating-system~ configuration available so we can complete bootstrapping Guix. The
most straightforward way to do so is to just copy ~system.scm~ file we defined earlier.
Another way is to capture the configuration in a git repository, and [[https://guix.gnu.org/manual/en/html_node/Creating-a-Channel.html][make it a Guix channel]].
This enables great shared tracking of system changes over time (outside of [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html][guix system
list-generations ...]]), and is what was done while bootstrapping our installation.

We choose to include a ~channels.scm~ file as part of the configuration channel that pins the
versions of software that will be used with our deployment. So, in order to boot the server
using an ~operating-system~ defined in [[https://git.genenetwork.org/guix-north-america/][our configuration channel]], we first download the
~channels.scm~ file.

#+begin_src shell
  curl -O https://git.genenetwork.org/guix-north-america/plain/channels.scm
#+end_src

We then create a temporary ~bootstrap.scm~ file that contains a references to the
~operating-system~ we defined in our configuration channel.

#+begin_src scheme
  (@ (guix-na config balg02) balg02)
#+end_src

We then use ~guix time-machine~ to specify these channels when installing Guix onto the
system.

#+begin_src shell
  guix time-machine -C channels.scm -- system init bootstrap.scm /mnt
#+end_src

Subsequent updates to the system can be done without using the ~bootstrap.scm~ file. For
instance, say the guix channel is updated in ~channels.scm~. To apply this change to the
server, the new channels would need to be pulled, and the system reconfigured.

#+begin_src shell
  sudo -i guix pull -C <(curl https://git.genenetwork.org/guix-north-america/plain/channels.scm)
  sudo -i guix system reconfigure -e '(@ (guix-na config balg02) balg02)'
#+end_src

One caveat to using ~-e|--expression~ is that currently this expression is not stored along
side the guix system generation, which makes it not possible to know which ~operating-system~
configuration was used from a given channel (reported upstream as issue [[https://issues.guix.gnu.org/54631][#54631]]). To work
around this for the time being, a file containing this expression can be used (just like was
used for bootstrapping).

* Challenges and Lessons Learned

Setting up a public Guix substitute server is not without its challenges:

1. **Performance Tuning**: Configuring Cuirass and the Guix daemon to efficiently use
   available resources required careful optimization.
2. **Privacy Considerations**: Implementing IP anonymization with anonip was crucial to
   protect user privacy.
3. **Bandwidth and Storage Management**: Implementing intelligent caching strategies to
   manage storage and network resources.

Luckily, many of these challenges had already been sorted out by existing Guix build farms,
making this endeavor much easier.

The biggest challenge was remote installation, where one hiccup was not realizing the
~megaraid_sas~ module was needed for our root ssd used for Guix, resulting in a failed first
boot following bootstrapping. Luckily this was resolved by booting to Debian and
reconfigure-ing Guix after adjusting our ~initrd-modules~ to include ~megaraid_sas~.

* Future Roadmap

Looking ahead, we have several goals for the cuirass.genenetwork.org substitute server:

- Collaborate with Guix maintainers to potentially include this server in the included list
  of default Guix substitute servers
- Expand build coverage to include more architectures and specialized packages
- Implement more sophisticated monitoring and performance tracking
- Explore potential partnerships with other academic and research institutions

* Conclusion

The Tennessee Guix Build Farm represents more than just a technical infrastructure project.
It embodies the spirit of open-source collaboration, community-driven development, and the
principles of reproducible computing. By providing a robust, privacy-conscious substitute
server, we hope to contribute to the growth and accessibility of the GNU Guix ecosystem.

We invite other organizations, universities, and community members to consider setting up
their own substitute servers. Each new node makes the Guix network stronger, more resilient,
and more accessible.

** Acknowledgments

We thank the Guix North America team consisting of Collin Doering, Pjotr Prins and Arun
Isaac.

We wish to acknowledge the GNU Guix project, in particular Ludovic Courtès, Efraim Flashner,
Christopher Baines, Ricardo Wurmus, Christine Lemmer-Webber and Andy Wingo for their
incredible work on GNU Guix and GNU Guile and are a continuous inspiration. Without their
input our work would not be possible. We also wish to acknowledge the fantastic [GeneNetwork
and Pangenome teams](https://genenetwork.org/facilities/) for providing support and hardware,
including the spiffy AMD Genoa machine that runs the Tennessee Guix Build Farm at
Memphis-based [Worldspice](https://www.worldspice.net/).