diff options
Diffstat (limited to 'topics/guix')
| -rw-r--r-- | topics/guix/genenetwork-fixating-guix.gmi | 34 | ||||
| -rw-r--r-- | topics/guix/guix-profiles.gmi | 27 | ||||
| -rw-r--r-- | topics/guix/packages.gmi | 16 |
3 files changed, 53 insertions, 24 deletions
diff --git a/topics/guix/genenetwork-fixating-guix.gmi b/topics/guix/genenetwork-fixating-guix.gmi new file mode 100644 index 0000000..844b0fd --- /dev/null +++ b/topics/guix/genenetwork-fixating-guix.gmi @@ -0,0 +1,34 @@ +# Fixating Guix for GN + +The GeneNetwork services depend on a rather complicated Guix deployment. The problem is not guix, but GN itself :) +But we were getting bitten by updates on upstream, as well as updates on our different targets/services. + +# Using channels that affect GN production + +To avoid duplication of work and unknown rabbit holes we decided to fixate guix trunk and other dependencies by using Guix channels. This means all GN development happens on a single version of Guix! That version is defined here: + +=> https://git.genenetwork.org/gn-machines/tree/.guix-channel + +Note that guix-forge and guix-bioinformatics are *also* fixated. The idea is that we only upgrade GN packages in gn-machines itself by inheriting definitions. E.g. + +=> https://git.genenetwork.org/gn-machines/tree/guix/gn-machines/genenetwork.scm + +We will probably get rid of the guix-past and guix-rust-past-crates sub-channels soon by removing those packages that depend on those (genenetwork1 will get its own tree, and @alexm will upgrade the rust packages). + +If someone wants to update guix channel or guix-bioinformatics channel they should not update this file. The one in charge is @fredm. Fred has to be in control because we don't want to break production. It is forbidden to touch this channel file. + +People can patch the packages and gn-machines, but if it involves CI/CD and/or production in any way, Fred will have to know about it. + +# Service level channels + +For individual services, such as genenetwork2, genenetwork3, gn-auth, etc., we have local channel files. These should mirror above gn-machines channel file to make sure we can migrate your code easily. E.g. + +=> https://github.com/genenetwork/genenetwork3/blob/main/.guix-channel + +Should match + +=> https://git.genenetwork.org/gn-machines/tree/.guix-channel + +If that is not the case we have a major problem! So before sending patches to Fred make sure the channels match. + +To be honest, I think we should fetch these channels automagically from gn-machines as a first step. diff --git a/topics/guix/guix-profiles.gmi b/topics/guix/guix-profiles.gmi index 8cf41d8..15f7683 100644 --- a/topics/guix/guix-profiles.gmi +++ b/topics/guix/guix-profiles.gmi @@ -15,9 +15,9 @@ Note that a recently tested channel can be fetched from cd.genenetwork.org. That Alternatively put the following into a channels.scm file. ``` (list (channel - (name 'gn-bioinformatics) - (url "https://git.genenetwork.org/guix-bioinformatics") - (branch "master"))) + (name 'gn-machines) + (url "https://git.genenetwork.org/gn-machines") + (branch "main"))) ``` Build a profile using ``` @@ -55,23 +55,6 @@ And everything should be in the PATH, PYTHONPATH etc. From time to time, updates to upstream Guix break our guix-bioinformatics channel. As a result, `guix pull` breaks and our work comes to a standstill for a few days until this can be fixed. While it is important to gradually move towards the latest and greatest, we would like to avoid being at the mercy of externalities and would prefer to update in a more controlled way without interrupting everyone's work. -To this end, we hard-code the guix-bioinformatics channel to depend on a *specific* commit of upstream Guix that is tied to guix-bioinformatics, for example: +To this end, we hard-code the guix-bioinformatics channel to depend on a specific commit of upstream Guix that is tied to guix-bioinformatics. This is why the recommended channels.scm file above does not include a %default-guix-channel. However, this comes with the drawback that your entire system will be stuck at that specific commit of upstream Guix (but not if you use another profile as described above). We highly recommend using a separate `guix pull` profile specifically for GeneNetwork work, as described above. -``` -(list (channel - (name 'gn-bioinformatics) - (url "https://gitlab.com/genenetwork/guix-bioinformatics") - (branch "master") - (commit - "feff05b47c305d8c944499fbc00fd2126f2b881d"))) -``` - -This is why the recommended channels.scm file above does not include a %default-guix-channel. However, this comes with the drawback that your entire system will be stuck at that specific commit of upstream Guix (but not if you use another profile as described above). We highly recommend using a separate `guix pull` profile specifically for GeneNetwork work, as described above. - -This scheme also comes with the added bonus that all members on the team and the continuous integration system will be using exactly the same Guix. Above channels.scm file is only exposed on a *succesful* build. - -## Notes - -We recently had to switch to gitlab because our git server went down on Penguin2. We may move to a cgit solution soon, see - -=> ../issues/cant-use-guix-bioinformatics-with-guix-pull.gmi +This scheme also comes with the added bonus that all members on the team and the continuous integration system will be using exactly the same Guix. \ No newline at end of file diff --git a/topics/guix/packages.gmi b/topics/guix/packages.gmi index a52f49b..b4a393c 100644 --- a/topics/guix/packages.gmi +++ b/topics/guix/packages.gmi @@ -2,15 +2,27 @@ To deploy GN we have packages in Guix itself (that comes with a distribution), in guix-bioinformatics and in guix-past (for older packages). +When you develop a new package it is best to run against a recent version of guix. Note that with GeneNetwork this is different as we 'fixate' guix at an older version. See + +=> genenetwork-fixating-guix + Typically run a guix pull to get the latest guix: ``` mkdir -p ~/opt -guix package -i guix -p ~/opt/guix +guix pull -p ~/opt/guix-pull +unset GUIX_PROFILE # for Debian +source ~/opt/guix-pull ``` and checkout guix-past and guix-bioinformatics using git. +Note that a codeberg pull may be faster (Guix recently moved main development to Codeberg): + +``` +guix pull -p ~/opt/guix-pull --url=https://codeberg.org/guix/guix +``` + Now Guix should be happy with ``` @@ -20,7 +32,7 @@ genenetwork2 3.11-2.1328932 out /home/wrk/guix-bioinformatics/gn genenetwork3 0.1.0-2.e781996 out /home/wrk/guix-bioinformatics/gn/packages/genenetwork.scm:107:4 ``` -and we can try building +note that using the -L switch is a bit of a hack. Normally we use channels. We can try building ``` ~/opt/guix-pull/bin/guix build -L ~/guix-bioinformatics/ -L ~/guix-past/modules/ genenetwork2 |