diff options
Diffstat (limited to 'topics/guix')
| -rw-r--r-- | topics/guix/guix-profiles.gmi | 77 | ||||
| -rw-r--r-- | topics/guix/guix-system-containers-and-how-we-use-them.gmi | 74 |
2 files changed, 151 insertions, 0 deletions
diff --git a/topics/guix/guix-profiles.gmi b/topics/guix/guix-profiles.gmi new file mode 100644 index 0000000..578bb82 --- /dev/null +++ b/topics/guix/guix-profiles.gmi @@ -0,0 +1,77 @@ +# Updating Guix profiles (for production and development) + +This document describes updating Guix for GeneNetwork by using a channel. To develop guix packages see + +=> guix/packages.gmi + +## Executive summary + +To build guix profiles that can run GN2+GN3 we use a so-called channel. + +Note that a recently tested channel can be fetched from cd.genenetwork.org. That should include the recent git hash values. + +=> https://ci.genenetwork.org/channels.scm + +Alternatively put the following into a channels.scm file. +``` +(list (channel + (name 'gn-bioinformatics) + (url "https://gitlab.com/genenetwork/guix-bioinformatics") + (branch "master"))) +``` +Build a profile using +``` +$ guix pull -C channels.scm -p ~/.guix-extra-profiles/genenetwork +``` + +Activate the newly built profile using +``` +$ export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork +$ . $GUIX_PROFILE/etc/profile +``` +Hack away on any GeneNetwork repository as usual. Now, any development environment you create with `guix shell` will be identical to what others on the team and the continuous integration system are using. This development environment is guaranteed to not break regardless of the latest changes to Guix upstream or other externalities. + +## Production (tux01) + +After downloading channels.scm from + +=> https://ci.genenetwork.org/channels.scm + +On production we do something like +``` +tux01:~$ ~/opt/guix-pull/bin/guix pull -C channels.scm -p ~/opt/guix-gn-channel +tux01:~$ ~/opt/guix-gn-channel/bin/guix package -i genenetwork2 -p /usr/local/guix-profiles/gn-latest-yyyymmdd +``` + +Now we can use that profile to run genenetwork2 and genenetwork3: + +``` +source /usr/local/guix-profiles/gn-latest-yyyymmdd/profile +``` + +And everything should be in the PATH, PYTHONPATH etc. + +## Rationale + +From time to time, updates to upstream Guix break our guix-bioinformatics channel. As a result, `guix pull` breaks and our work comes to a standstill for a few days until this can be fixed. While it is important to gradually move towards the latest and greatest, we would like to avoid being at the mercy of externalities and would prefer to update in a more controlled way without interrupting everyone's work. + +To this end, we hard-code the guix-bioinformatics channel to depend on a *specific* commit of upstream Guix that is tied to guix-bioinformatics, for example: + +``` +(list (channel + (name 'gn-bioinformatics) + (url "https://gitlab.com/genenetwork/guix-bioinformatics") + (branch "master") + (commit + "feff05b47c305d8c944499fbc00fd2126f2b881d"))) +``` + +This is why the recommended channels.scm file above does not include a %default-guix-channel. However, this comes with the drawback that your entire system will be stuck at that specific commit of upstream Guix (but not if you use another profile as described above). We highly recommend using a separate `guix pull` profile specifically for GeneNetwork work, as described above. + +This scheme also comes with the added bonus that all members on the team and the continuous integration system will be using exactly the same Guix. Above channels.scm file is only exposed on a *succesful* build. + +## Notes + +We recently had to switch to gitlab because our git server went down on Penguin2. We may move to a cgit solution soon, see + +=> ../issues/cant-use-guix-bioinformatics-with-guix-pull.gmi diff --git a/topics/guix/guix-system-containers-and-how-we-use-them.gmi b/topics/guix/guix-system-containers-and-how-we-use-them.gmi new file mode 100644 index 0000000..26baca8 --- /dev/null +++ b/topics/guix/guix-system-containers-and-how-we-use-them.gmi @@ -0,0 +1,74 @@ +# Guix system containers and how we use them + +Our preferred way to deploy long-running services is to use Guix system containers. Note that Guix system containers are different from guix shell containers (and the older guix environment containers). guix shell containers are meant for interactive use and are a poor fit for long running services. Other non-Guix ways such as running long-running processes in tmuxes, running long-running services from your development repository, etc. are a quick dirty way to get things done. + +Guix system containers are fully described by a configuration file that, among other things, specifies the services run in it. These scheme configuration files are built using `guix system container' and produce a script. This script, when run, starts the described container. + +``` +$ guix system container foo.scm +/gnu/store/9ld75cjg54xwqvsvvgdd38rv3d4x4wzz-run-container +``` + +One cool aspect is that system containers can be easily tested on your own laptop. +A great tutorial can be found at + +=> https://guix.gnu.org/cookbook/en/html_node/Guix-System-Containers.html + +A system container comes with a running shepherd process and running services are explicitly configured. + +Our most important containers are defined in + +=> https://github.com/genenetwork/genenetwork-machines/blob/main/genenetwork-development.scm + +## Share network with the host + +Usually, we want the container to share the network with the host. So, we add the --network flag. +``` +$ guix system container --network foo.scm +``` + +## Adding state to Guix system containers + +Guix system containers are completely ephemeral, that is, they have no persistent state. But often, we have services that need to retain some state. Think a database server needing to persist its database directory, a web server needing to persist its logs, etc. To allow this persistence, we expose (read-only) or share (read-write) directories from the host into the container. +``` +$ guix system container --network --share=/var/lib/foo foo.scm +``` + +## systemd services to manage the container processes + +Now, running these container scripts directly from the command-line, probably from within a tmux, makes for a very fragile deployment. So, we symlink the script into /usr/local/bin and set up a systemd service to manage the container process. + +``` +# ln --force --symbolic $(guix system container --network --share=/var/lib/foo foo.scm) /usr/local/bin/foo-container +``` + +A systemd service file foo-container.service for this container should be put at /etc/systemd/system/. + +``` +[Unit] +Description = Run foo container + +[Service] +ExecStart = /usr/local/bin/foo-container + +[Install] +WantedBy = multi-user.target +``` + +This allows us to start, stop and enable (for starting at boot time) the container easily. + +``` +# guix system start foo-container +# guix system stop foo-container +# guix system enable foo-container +``` + +With our service enabled to start at boot time, we need not worry about reboots. All our containers, and the services contained therein, start up smoothly on boot. + +## Register as garbage collector root + +Finally, we must also tell Guix not to accidentally garbage collect our container or any of its dependencies. To this end, we symlink it into /var/guix/gcroots. + +``` +# ln --force --symbolic /usr/local/bin/foo-container /var/guix/gcroots +``` |