aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorCollin J. Doering2025-01-01 17:58:51 -0500
committerCollin J. Doering2025-01-01 17:58:51 -0500
commit18cc0e082187b857344a931f825b8b0103fa0f07 (patch)
treeb142f17608fc28e239c5e96edef183e6506ebb3b /docs
parent69fd6cacb18c308befa472e526563f9d5e9dc786 (diff)
downloadguix-north-america-18cc0e082187b857344a931f825b8b0103fa0f07.tar.gz
Wrap up and polish nginx section; add sections on "First Boot" and ssl ca/cert generation
Diffstat (limited to 'docs')
-rw-r--r--docs/blog-Tennnessee-build-farm.org429
1 files changed, 297 insertions, 132 deletions
diff --git a/docs/blog-Tennnessee-build-farm.org b/docs/blog-Tennnessee-build-farm.org
index 03ceaab..3a97e17 100644
--- a/docs/blog-Tennnessee-build-farm.org
+++ b/docs/blog-Tennnessee-build-farm.org
@@ -1,10 +1,6 @@
#+TITLE: Setup of a Simple Guix Build Farm and Substitute Server
#+AUTHOR: Collin J. Doering
-# FIXME: in a perfect world, I would set text wrapping automatically for latex minted like so:
-# -*- org-latex-minted-options: '(("breaklines" . "true") ("breakanywhere" . "true")); -*-
-# However, this currently fails (likely due to a bug?)
-
#+LATEX_HEADER: \usepackage[margin=1.5cm]{geometry}
#+LATEX_HEADER: \usepackage{xcolor}
#+LATEX_HEADER: \definecolor{link}{HTML}{506060}
@@ -69,8 +65,16 @@ who provided the following server specifications:
These robust specifications allow for efficient package building, caching, and serving of
substitutes for the Guix community.
+#+begin_quote
FIXME: how much do we actually need in terms of cores and RAM? That is if someone else wants to create a build farm.
+- I find this hard to answer, as using the default build mechanism in cuirass seems to have
+ issues; I get odd stalls on some evaluations that I have not been able to figure out.
+- currently running on 4 cores
+- memory and disk are the most important aspects
+- partly about whats possible to keep up with (re: the guix distribution itself); building a subset is also possible.
+#+end_quote
+
* Components of the Guix Build Farm
** Cuirass - building packages
@@ -231,7 +235,7 @@ configuring Nginx tls.
".pem"))
#+end_src
-*** TODO Configure Nginx Location block for ~guix-publish~
+*** Configure Nginx Location block for ~guix-publish~
Lets define a function that given a url, produces a list of appropriate nginx location blocks
to enable guix-publish running on some provided URL.
@@ -243,7 +247,7 @@ to enable guix-publish running on some provided URL.
#+end_src
Starting from the definition above, lets fill in and explain the purpose of each
-location-configuration in the list that will be returned from our function.
+nginx-location-configuration in the list that will be returned from our function.
- ~/nix-cache-info~
@@ -262,10 +266,6 @@ location-configuration in the list that will be returned from our function.
Priority: 100
#+end_src
- FIXME: Its interesting to note that outside of the expected difference of ~StoreDir~
- varying on Nix and Guix (on nix its ~/nix/store~), ~WantMassQuery~ and ~Priority~ are both
- set to static values for Nix and Guix, but they vary!
-
Now that we have some more context on the route, here is the nginx-location-configuration
we will return to proxy requests appropriately.
@@ -297,7 +297,7 @@ location-configuration in the list that will be returned from our function.
First, lets find the store path of the package (but without actually building it).
- #+begin_src shell :results output code :wrap src text :exports both
+ #+begin_src shell :results output code :wrap src text :exports both :eval no-export
guix build --dry-run hello
#+end_src
@@ -359,7 +359,7 @@ location-configuration in the list that will be returned from our function.
manually fetch a substitute. Here we will continue, using the provided hash to query to
query the substitute server for a corresponding ~.narinfo~ file.
- #+begin_src shell :results output
+ #+begin_src shell :results output :eval no-export
curl https://cuirass.genenetwork.org/8bjy9g0cssjrw9ljz2r8ww1sma95isfj.narinfo
#+end_src
@@ -419,25 +419,6 @@ location-configuration in the list that will be returned from our function.
)))
#+end_src
-- ~/file/~
-
- FIXME: provide more context
-
- #+begin_src scheme
- ;; Content-addressed files served by 'guix publish'.
- (nginx-location-configuration
- (uri "/file/")
- (body
- (list
- (string-append "proxy_pass " url ";")
-
- "proxy_cache cas;"
- "proxy_cache_valid 200 200d;" ; cache hits
- "proxy_cache_valid any 5m;" ; cache misses/others
-
- "proxy_ignore_client_abort on;")))
- #+end_src
-
- ~/robots.txt~
First, lets define a string ~publish-robots.txt~, that we'll configure Nginx to serve on the
@@ -471,57 +452,117 @@ location-configuration in the list that will be returned from our function.
"root /;")))
#+end_src
-*** TODO Nginx locations (FIND BETTER NAME)
+*** Nginx locations and route configuration
+
+Lets define another function that will be used to produce the
+~nginx-location-configurations~'s we will use in our ~operating-system~ configuration,
+including setup for guix-publish (using the previously defined ~publish-locations~
+function above) in addition to setup for Cuirass and Certbot.
#+begin_src scheme
(define (balg02-locations publish-url)
"Return nginx location blocks with 'guix publish' reachable at
PUBLISH-URL."
(append (publish-locations publish-url)
- (list
- ;; Cuirass.
- (nginx-location-configuration
- (uri "/")
- (body (list "proxy_pass http://localhost:8081;"
- ;; ;; See
- ;; ;; <https://community.torproject.org/onion-services/advanced/onion-location/>.
- ;; (string-append
- ;; "add_header Onion-Location http://" %ci-onion
- ;; "$request_uri;")
- )))
- (nginx-location-configuration
- (uri "~ ^/admin")
- (body
- (list "if ($ssl_client_verify != SUCCESS) { return 403; } proxy_pass http://localhost:8081;")))
-
- (nginx-location-configuration
- (uri "/static")
- (body
- (list
- "proxy_pass http://localhost:8081;"
- ;; Cuirass adds a 'Cache-Control' header, honor it.
- "proxy_cache static;"
- "proxy_cache_valid 200 2d;"
- "proxy_cache_valid any 10m;"
- "proxy_ignore_client_abort on;")))
-
- (nginx-location-configuration
- (uri "/download") ;Cuirass "build products"
- (body
- (list
- "proxy_pass http://localhost:8081;"
- "expires 10d;" ;override 'Cache-Control'
- "proxy_cache static;"
- "proxy_cache_valid 200 30d;"
- "proxy_cache_valid any 10m;"
- "proxy_ignore_client_abort on;")))
-
- (nginx-location-configuration ;certbot
- (uri "/.well-known")
- (body (list "root /var/www;"))))))
-#+end_src
-
-*** TODO Configure Nginx Server Blocks
+ (list ...)))
+#+end_src
+
+Similar to beforehand, starting from the definition above, lets fill in and explain the
+purpose of each nginx-location-configuration in the list that will be returned from our
+function.
+
+- ~/~
+
+ Cuirass and its routes should be accessible, which exactly what the below
+ nginx-configuration does! It proxies traffic to the ~guix-publish~ service (which is
+ assumed to be running at its default location).
+
+ #+begin_src scheme
+ (nginx-location-configuration
+ (uri "/")
+ (body (list "proxy_pass http://localhost:8081;")))
+ #+end_src
+
+- ~/admin~
+
+ As stated in the [[https://guix.gnu.org/cuirass/manual/cuirass.html#Authentication][Cuirass manual]], "Cuirass does not provide its own authentication
+ mechanism; by default, any user can do anything via its web interface. To restrict this to
+ only authorized users, one approach is to proxy the Cuirass web site via a web server such
+ as Nginx and configure the web server to require client certificate verification for pages
+ under the ‘/admin’ prefix."
+
+ Here we do exactly as recommended by the manual: restrict access to ~/admin~ Cuirass routes
+ by checking ssl client certificate validity, proxying to cuirass when successful, and
+ returning 403 Unauthorized otherwise.
+
+ #+begin_src scheme
+ (nginx-location-configuration
+ (uri "~ ^/admin")
+ (body
+ (list "if ($ssl_client_verify != SUCCESS) { return 403; } proxy_pass http://localhost:8081;")))
+ #+end_src
+
+- ~/static~
+
+ Cuirass serves a series of static js/css/media files for its web interface. These are
+ available via the ~/static~ route, which we add a ~nginx-location-configuration~ for below.
+
+ #+begin_src scheme
+ (nginx-location-configuration
+ (uri "/static")
+ (body
+ (list
+ "proxy_pass http://localhost:8081;"
+ ;; Cuirass adds a 'Cache-Control' header, honor it.
+ "proxy_cache static;"
+ "proxy_cache_valid 200 2d;"
+ "proxy_cache_valid any 10m;"
+ "proxy_ignore_client_abort on;")))
+ #+end_src
+
+- ~/download~
+
+ Cuirass allows users to download build products (eg. system images). To enable this we must
+ proxy ~/download~ routes to cuirass.
+
+ #+begin_src scheme
+ (nginx-location-configuration
+ (uri "/download") ;Cuirass "build products"
+ (body
+ (list
+ "proxy_pass http://localhost:8081;"
+ "expires 10d;" ;override 'Cache-Control'
+ "proxy_cache static;"
+ "proxy_cache_valid 200 30d;"
+ "proxy_cache_valid any 10m;"
+ "proxy_ignore_client_abort on;")))
+ #+end_src
+
+ This concludes the configuration of routes for Cuirass.
+
+- ~/.well-known~
+
+ Now we configure the ~/.well-known~ route, to enable Certbot certificate rotation. Namely,
+ Certbot uses the ~/.well-known~ route to serve its challenge/response files.
+
+ #+begin_src scheme
+ (nginx-location-configuration ;certbot
+ (uri "/.well-known")
+ (body (list "root /var/www;")))
+ #+end_src
+
+ For more details, see the [[https://letsencrypt.org/docs/challenge-types/]["HTTP-01 challenge" section of the Lets Encrypt Challenge Types]]
+ documentation. For Guix specific details, check out the [[https://guix.gnu.org/manual/en/html_node/Certificate-Services.html][certbot-service-type]] documentation.
+
+*** Configure Nginx Server Blocks
+
+Now that we have a function (~balg02-locations~) that given a guix-publish url to proxy, will
+output a list of ~nginx-location-configuration~'s for Cuirass, guix-publish, and certbot, we
+set our attention on setting up the necessary ~nginx-server-configuration~'s for our build
+farm.
+
+First, we define a few constants that will be reused across the http and https server
+configurations later on.
#+begin_src scheme
(define %publish-url "http://localhost:3000")
@@ -540,59 +581,87 @@ location-configuration in the list that will be returned from our function.
"ssl_dhparam /etc/dhparams.pem;"))
#+end_src
+Next, we'll define our ~nginx-server-configuration~'s, starting with the following definition
+which we will fill in one ~nginx-server-configuration~ at a time.
+
#+begin_src scheme
(define %balg02-servers
+ (list ...))
+#+end_src
+
+Our first ~nginx-server-configuration~ in our list is forward-looking: it puts in place a
+http to https redirect for sites we may add in the future.
+
+#+begin_src scheme
+ ;; Redirect domains that don't explicitly support HTTP (below) to HTTPS.
+ (nginx-server-configuration
+ (listen '("80"))
+ (raw-content
+ (list "return 308 https://$host$request_uri;")))
+#+end_src
+
+Our second, configures nginx to listen for HTTP traffic on its default port for our domain.
+Here we leverage the ~balg02-locations~ function we defined earlier, and make sure to send
+our log files through anonip which we previously configured.
+
+#+begin_src scheme
+ ;; Domains that still explicitly support plain HTTP.
+ (nginx-server-configuration
+ (listen '("80"))
+ (server-name `("cuirass.genenetwork.org"
+ ;; <https://logs.guix.gnu.org/guix/2021-11-20.log#155427>
+ "~[0-9]$"))
+ (locations (balg02-locations %publish-url))
+ (raw-content
(list
- ;; Redirect domains that don't explicitly support HTTP (below) to HTTPS.
- (nginx-server-configuration
- (listen '("80"))
- (raw-content
- (list "return 308 https://$host$request_uri;")))
-
- ;; Domains that still explicitly support plain HTTP.
- (nginx-server-configuration
- (listen '("80"))
- (server-name `("cuirass.genenetwork.org"
- ;; <https://logs.guix.gnu.org/guix/2021-11-20.log#155427>
- "~[0-9]$"))
- (locations (balg02-locations %publish-url))
- (raw-content
- (list
- "access_log /var/run/anonip/http.access.log;"
- "proxy_set_header X-Forwarded-Host $host;"
- "proxy_set_header X-Forwarded-Port $server_port;"
- "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;")))
-
- ;; HTTPS servers
- (nginx-server-configuration
- (listen '("443 ssl"))
- (server-name '("cuirass.genenetwork.org"))
- (ssl-certificate (le "cuirass.genenetwork.org"))
- (ssl-certificate-key (le "cuirass.genenetwork.org" 'key))
- (locations (balg02-locations %publish-url))
- (raw-content
- (append
- %tls-settings
- (list
- "access_log /var/run/anonip/https.access.log;"
- "proxy_set_header X-Forwarded-Host $host;"
- "proxy_set_header X-Forwarded-Port $server_port;"
- "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
- ;; TODO:
- ;; For Cuirass admin interface authentication
- ;; "ssl_client_certificate /etc/ssl-ca/certs/ca.crt;"
- ;; "ssl_verify_client optional;"
- ))))))
+ "access_log /var/run/anonip/http.access.log;"
+ "proxy_set_header X-Forwarded-Host $host;"
+ "proxy_set_header X-Forwarded-Port $server_port;"
+ "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;")))
#+end_src
+Our last ~nginx-server-configuration~ configures nginx to liston for HTTPS traffic on its
+default port (443); it is nearly identical to the previous ~nginx-server-configuration~, but
+makes use of the ~le~ lets encrypt helper function we defined earlier to lookup the
+appropriate ssh certificate or key.
+
+Additionally, it specifies a SSL/TLS CA certificate to use for client certificate
+verification. Later on in [[*Setup Certificate Authority and Client Certificates for Cuirass Administration][Setup Certificate Authority and Client Certificates for Cuirass
+Administration]], we'll provide some details on how the CA and client certificates can be
+generated.
+
+#+begin_src scheme
+ ;; HTTPS servers
+ (nginx-server-configuration
+ (listen '("443 ssl"))
+ (server-name '("cuirass.genenetwork.org"))
+ (ssl-certificate (le "cuirass.genenetwork.org"))
+ (ssl-certificate-key (le "cuirass.genenetwork.org" 'key))
+ (locations (balg02-locations %publish-url))
+ (raw-content
+ (append
+ %tls-settings
+ (list
+ "access_log /var/run/anonip/https.access.log;"
+ "proxy_set_header X-Forwarded-Host $host;"
+ "proxy_set_header X-Forwarded-Port $server_port;"
+ "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
+ ;; For Cuirass admin interface authentication
+ "ssl_client_certificate /etc/ssl-ca/certs/ca.crt;"
+ "ssl_verify_client optional;"))))
+#+end_src
+
+*** Nginx Service Definition
+
+We are nearing the conclusion of configuring nginx! Last up we will define some extra content
+which will be included with our nginx configuration.
+
#+begin_src scheme
(define %extra-content
(list
"default_type application/octet-stream;"
"sendfile on;"
- (accept-languages)
-
;; Maximum chunk size to send. Partly this is a workaround for
;; <http://bugs.gnu.org/19939>, but also the nginx docs mention that
;; "Without the limit, one fast connection may seize the worker
@@ -649,16 +718,13 @@ location-configuration in the list that will be returned from our function.
"proxy_cache_valid 504 30s;"))
#+end_src
+Finally, all of our hard work comes together with a simple ~nginx-configuration~.
+
#+begin_src scheme
(define %nginx-configuration
(nginx-configuration
(server-blocks %balg02-servers)
(server-names-hash-bucket-size 128)
- (modules
- (list
- ;; Module to redirect users to the localized pages of their choice.
- (file-append nginx-accept-language-module
- "/etc/nginx/modules/ngx_http_accept_language_module.so")))
(global-directives
'((worker_processes . 16)
(pcre_jit . on)
@@ -730,9 +796,7 @@ configuration, finalization our configuration of nginx.
#:build-accounts-to-max-jobs-ratio 5)))
#+end_src
-*** TODO Setup Client Certificates for Cuirass Administration
-
-** Installation
+* Installation
We've left out other aspects of defining our ~operating-system~ configuration, as its [[https://guix.gnu.org/manual/devel/en/html_node/operating_002dsystem-Reference.html][well
documented]] by the Guix project, and varies depending on the specifics of your machine (for
@@ -742,7 +806,7 @@ system configured with Cuirass, guix-publish, guix-daemon, and nginx, setup to a
single-node build farm and substitute server! It can be trivially installed following the
[[https://guix.gnu.org/manual/en/html_node/Proceeding-with-the-Installation.html][manual Guix System installation documentation]].
-#+begin_src shell
+#+begin_src shell :eval no
guix system init system.scm /mnt
#+end_src
@@ -750,7 +814,7 @@ Here ~/mnt~ is where the target root file-system is mounted after being prepared
installation. See our [[https://git.genenetwork.org/guix-north-america/tree/docs/initial-setup.org][Initial Setup Documentation]] for the specifics of preparing the disks
for Guix installation in our case.
-*** Remotely Bootstrapping Guix from Debian
+** Remotely Bootstrapping Guix from Debian
In our case with the Tennessee Build Farm, physical access was inconvenient due to travel
distance, so installation needed to be completed remotely. The target server already had
@@ -761,7 +825,7 @@ Guix installation.
Guix can be [[https://guix.gnu.org/manual/en/html_node/Binary-Installation.html][installed on foreign distributions]], which is well documented, so its not covered
here, but is the first step in bootstrapping Guix from Debian.
-**** Guix Configuration as a Channel
+*** Guix Configuration as a Channel
Once Guix (the package manager) is installed on Debian, we need to ensure partition our
drives as required (which again varies, so will not be covered here). Next we need to make
@@ -776,7 +840,7 @@ versions of software that will be used with our deployment. So, in order to boot
using an ~operating-system~ defined in [[https://git.genenetwork.org/guix-north-america/][our configuration channel]], we first download the
~channels.scm~ file.
-#+begin_src shell
+#+begin_src shell :eval no
curl -O https://git.genenetwork.org/guix-north-america/plain/channels.scm
#+end_src
@@ -790,7 +854,7 @@ We then create a temporary ~bootstrap.scm~ file that contains a references to th
We then use ~guix time-machine~ to specify these channels when installing Guix onto the
system.
-#+begin_src shell
+#+begin_src shell :eval no
guix time-machine -C channels.scm -- system init bootstrap.scm /mnt
#+end_src
@@ -798,7 +862,7 @@ Subsequent updates to the system can be done without using the ~bootstrap.scm~ f
instance, say the guix channel is updated in ~channels.scm~. To apply this change to the
server, the new channels would need to be pulled, and the system reconfigured.
-#+begin_src shell
+#+begin_src shell :eval no
sudo -i guix pull -C <(curl https://git.genenetwork.org/guix-north-america/plain/channels.scm)
sudo -i guix system reconfigure -e '(@ (guix-na config balg02) balg02)'
#+end_src
@@ -809,6 +873,107 @@ configuration was used from a given channel (reported upstream as issue [[https:
around this for the time being, a file containing this expression can be used (just like was
used for bootstrapping).
+* First Boot
+
+There are some opportunities to improve our deployment, as currently it requires manual
+intervention after the initial installation. Here we'll detail whats necessary.
+
+** Setup dhparams after installation
+
+Earlier in [[*Configure Nginx Server Blocks][Configure Nginx Server Blocks]], we specified our own DH parameters, as suggested by
+https://weakdh.org/sysadmin.html. We need to create them after the first boot of our build farm.
+
+#+begin_src shell :eval no
+ openssl dhparam -out /etc/dhparams.pem 2048
+#+end_src
+
+** Initial SSL/TLS Certificate/s Generation
+
+In order for the ~nginx-service~ to run successfully, it needs SSL/TLS certificates, which
+upon initial boot will not be available. This causes a cascade of service failures until the
+initial certificates are generated. However, there is a chicken-and-egg problem - the nginx
+service needs to be running to provide http access to the ~/.well-known~ route so certbot can
+complete. This can be worked around by providing a self-signed certificate and key
+temporarily for each certificate/key refereed to, which in our case is only one.
+
+#+begin_src shell :eval no
+ mkdir -p /etc/letsencrypt/live/cuirass.genenetwork.org
+ openssl req -x509 -nodes -days 1 -newkey rsa:2048 -keyout /etc/letsencrypt/live/cuirass.genenetwork.org/privkey.pem -out /etc/letsencrypt/live/cuirass.genenetwork.org/fullchain.pem -subj "/CN=temporary"
+#+end_src
+
+Following creating a temporary self-signed certificate, we can start nginx.
+
+#+begin_src shell :eval no
+ sudo herd start nginx
+#+end_src
+
+And now, we can force certbot to renew our certificates.
+
+#+begin_src shell :eval no
+ sudo herd start renew-certbot-certificates
+#+end_src
+
+Once this completes, nginx will automatically be restarted (using the ~%nginx-deploy-hook~ we
+setup earlier when setting up [[*Certbot][Certbot]]), and will use the new Let's Encrypt certificates.
+
+If the cuirass service is still in a failing state, it can now be restarted.
+
+** Setup Certificate Authority and Client Certificates for Cuirass Administration
+
+Setup of SSL/TLS client side certificates is beyond the scope of this article, however we
+will briefly mention and demonstrate how the [[https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/etc/new-client-cert.scm][etc/new-client-cert.scm]] script that is part of
+the Cuirass repository (but not in the provided cuirass guix package) can be adjusted to fit
+the needs of most Cuirass deployments.
+
+First fetch the script and make it executable.
+
+#+begin_src shell :eval no
+ curl -O https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/etc/new-client-cert.scm
+ chmod +x new-client-cert.scm
+#+end_src
+
+This script leverages ~guix~ itself to be run, so its necessary to be running on a system
+with guix installed in order to use this script. For more details on how this is done, see
+the note on the [[https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-shell.html][Invoking guix shell]] documentation about "guix shell can also be used as a
+script interpreter".
+
+The script currently needs to be slightly adjusted to alter the ~subject-template~ being
+used, which by default is ~"/C=DE/ST=Berlin/L=Berlin/O=GNU Guix/OU=Cuirass/CN=~a"~ - this
+string defines the openssl subject name used when generating a CA and client certificates,
+where ~~a~ will be substituted for a name later on. Here you may want to adjust the location,
+organization, etc.. to suite your case.
+
+First a Certificate Authority needs to be generated. This should be done from the cuirass
+server itself so the CA key never leaves the server its being used on, and can be done with
+the ~new-client-cert.scm~ script as follows.
+
+#+begin_src shell :eval no
+ ./new-client-cert.scm --generate-ca
+#+end_src
+
+By default, this generates a CA key ~/etc/ssl-ca/private/ca.key~ and certificate
+~/etc/ssl-ca/certs/ca.crt~, so is important to ensure these directories already exist and are
+writable by the user running the script.
+
+#+begin_src shell :eval no
+ mkdir -p /etc/ssl-ca/{private,cert}
+#+end_src
+
+With the CA in place, one or more client certificates can now be generated. First, clients
+will need to supply a Certificate Signing Request (CSR), which should be placed in the
+current working directory and named ~<who>.csr~ where ~<who>~ should be the name of the
+certificate signing request (eg. ~collin.csr~). We can then run the following to generate a
+signed client certificate, which will be outputted as ~<who>.p12~ in the current working
+directory.
+
+#+begin_src shell
+ ./new-client-cert.scm collin
+#+end_src
+
+The ~collin.p12~ [[https://en.wikipedia.org/wiki/PKCS_12][PKCS 12]] formatted certificate file that was generated by the above command
+can now be provided back to the requester, to be installed in their browser or operating
+system, enabling administrative access to cuirass.
+
* Challenges and Lessons Learned
Setting up a public Guix substitute server is not without its challenges: