diff options
Diffstat (limited to 'docs/blog-Tennnessee-build-farm.org')
| -rw-r--r-- | docs/blog-Tennnessee-build-farm.org | 92 |
1 files changed, 59 insertions, 33 deletions
diff --git a/docs/blog-Tennnessee-build-farm.org b/docs/blog-Tennnessee-build-farm.org index 0163377..70b5a88 100644 --- a/docs/blog-Tennnessee-build-farm.org +++ b/docs/blog-Tennnessee-build-farm.org @@ -10,7 +10,7 @@ well as substitute availability and improved response times due to server locali * TODO note inspiration, and in cases direct copy from https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin.scm -* Setting up a Minimal Guix Build Farm and Substitute Server +* Setting up a Minimal Guix Build "Farm" and Substitute Server Though a Guix build farm and substitute server could be deployed on any distribution, we naturally chose to use Guix itself. There are a variety of components that provide the @@ -20,9 +20,9 @@ necessary functionality: derivations, packages, etc.. - [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html][guix-publish]] :: Provides substitute archives for consumption by users (indirectly via nginx as a local reverse proxy). -- nginx :: Acts as a reverse proxy for Cuirass and guix-publish. -- certbot :: Fetches ssl certificates so cuirass and substitutes can be served over https. -- anonip :: Anatomizes http logs to preserve user privacy. +- [[https://github.com/nginx/nginx][nginx]] :: Acts as a reverse proxy for Cuirass and guix-publish. +- [[https://github.com/certbot/certbot][certbot]] :: Fetches ssl certificates so cuirass and substitutes can be served over https. +- [[https://github.com/DigitaleGesellschaft/Anonip][anonip]] :: Anatomizes http logs to preserve user privacy. How each of these components are setup is detailed below, component-by-component. You can see the full source-code for the Tennessee build farm at @@ -61,10 +61,10 @@ More details on that in [[*Guix Configuration as a Channel][Guix Configuration a *** Setup Cuirass Service -Now that we have defined what we want Cuirass to build, we need to enable its guix service -which in turn will run Cuirass. We are going to later setup nginx as a reverse proxy for -cuirass, so we'll set its host to localhost, and pass along the specifications we defined -earlier. +Now that we have defined what we want Cuirass to build, we need to specify its guix service +in the ~services~ field of our ~operating-system~ definition, which in turn will run Cuirass. +We are going to setup nginx as a reverse proxy for cuirass later on, so we'll set its host to +localhost, and pass along the specifications we defined earlier. #+begin_src scheme (service cuirass-service-type @@ -77,25 +77,29 @@ earlier. With Cuirass configured and the guix store being populated with package builds as the guix channel changes, we now turn our attention to serving these builds as substitutes to Guix -users. This is done using [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html][guix publish]], which Guix provides the [[file:~/.org/roam/20221129213953-advent_of_code.org::*Day 2][guix-publish-service-type]] in -order to configure and run. +users. This is done using [[https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html][guix publish]], which Guix provides the [[file:~/.org/roam/20221129213953-advent_of_code.org::*Day 2][guix-publish-service-type]], +which is used in the ~services~ field of ~operating-system~ definition. #+begin_src scheme - (service guix-publish-service-type - (guix-publish-configuration - (port 3000) - (cache "/var/cache/guix/publish") - (ttl (* 90 24 3600)) + (service guix-publish-service-type + (guix-publish-configuration + (port 3000) + (cache "/var/cache/guix/publish") + (ttl (* 90 24 3600)) #+end_src +Similar to Cuirass, access to guix-publish will be provided through nginx as a reverse proxy. + ** Anonomize IPs (anonip) -Guix users care about their privacy, and though this is necessarily a requirement, -anonomizing nginx access logs using the anonip-service is implemented by all public Guix -sponsored build farms, so keeping with this privacy preserving trend, cuirass.genenetwork.org -implements the same log anonomization. +Guix users care about their privacy, and though this is not necessarily a requirement, +anonymizing nginx access logs using the anonip is implemented by all public Guix sponsored +build farms, so keeping with this privacy preserving trend, cuirass.genenetwork.org +implements the same log anonymization. -To anonomize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_node/Log-Rotation.html][anonip-service-type]] is configured and used. +To anonymize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_node/Log-Rotation.html][anonip-service-type]] will be configured and used, however, we +want to anonymize multiple log files, which means multiple instances of the anonip running. +To assist with this, a helper function ~anonip-service~ is defined. #+begin_src scheme (define (anonip-service file) @@ -103,18 +107,45 @@ To anonomize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_ (anonip-configuration (input (format #false "/var/run/anonip/~a" file)) (output (format #false "/var/log/anonip/~a" file))) +#+end_src - (define %anonip-log-files - ;; List of files handled by Anonip - '("http.access.log" - "https.access.log")) +Additionally, for services that will leverage these anonymized logs (in our case, only +nginx), it will be necessary to ensure that the appropriate instance of anonip is running +prior to the respective service that will utilize it. To help declare this dependency, +another helper function is defined. +#+begin_src scheme (define (log-file->anonip-service-name file) "Return the name of the Anonip service handling FILE, a log file." (symbol-append 'anonip-/var/log/anonip/ (string->symbol file))) #+end_src -** Certbot +We also define a list of anonymized log files which will be used later on along side the +~log-file->anonip-service-name~ function in order to define shepherd service dependencies for +nginx. + +#+begin_src scheme + (define %anonip-nginx-log-files + ;; List of files handled by Anonip for nginx + '("http.access.log" + "https.access.log")) +#+end_src + +All that remains is to ensure that for each log file we are anonymizing, we start a +corresponding anonip-service. This can be added to the ~services~ field of our +~operating-system~ declaration. + +#+begin_src scheme + (map anonip-service %anonip-nginx-log-files) +#+end_src + +** Nginx Reverse Proxy + +Nginx is arguably the most complicated part of the setup. This section touches on the +essential details of configuring nginx to act as a reverse proxy for both guix-publish, and +Cuirass. + +*** Certbot #+begin_src scheme (define* (le host #:optional privkey) @@ -124,9 +155,7 @@ To anonomize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_ ".pem")) #+end_src -** Nginx Reverse Proxy - -*** abc +*** TODO include details about how nginx is setup #+begin_src scheme (define publish-robots.txt @@ -335,10 +364,7 @@ To anonomize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_ (listen '("80")) (server-name `("cuirass.genenetwork.org" ;; <https://logs.guix.gnu.org/guix/2021-11-20.log#155427> - "~[0-9]$" - ; TODO: onion - ; ,(regexp-quote %ci-onion) - )) + "~[0-9]$")) (locations (balg02-locations %publish-url)) (raw-content (list @@ -451,7 +477,7 @@ To anonomize nginx access logs, the [[https://guix.gnu.org/manual/devel/en/html_ (string-join %extra-content "\n")) (shepherd-requirement (map log-file->anonip-service-name - %anonip-log-files)))) + %anonip-nginx-log-files)))) #+end_src *** Cache activation |