gn-machines - Guix configuration for genenetwork machines

This repository houses Guix configuration for GeneNetwork machines and containers.

The git repo lives at https://git.genenetwork.org/gn-machines/

GeneNetwork development container

The GeneNetwork development container is currently run on tux02. It runs continuous integration and continuous deployment services for genenetwork2, genenetwork3 and several other associated projects.

To build and install the container, you will need the guix-bioinformatics and guix-forge channels. Once these channels are pulled and available, on tux02, run

$ ./genenetwork-development-deploy.sh

tux02 is configured with a systemd service to run this container. Restart it.

$ sudo systemctl restart genenetwork-development-container

Virtuoso container for tux01 production

The virtuoso container is run on tux01 production. It runs virtuoso alone.

To build and install the container, you will need the guix-bioinformatics channel. Once guix-bioinformatics is pulled and available, on tux01, run

$ ./virtuoso-deploy.sh

tux01 is configured with a systemd service to run this container. Restart it.

$ sudo systemctl restart virtuoso-container

Secure virtuoso authentication

In containers containing virtuoso instances, it is important to secure authentication by changing default user passwords and disabling unnecessary users. See virtuoso gemtext documentation on passwords for more details.

Getting a Shell into the Container

You can get a shell into the container with something like:

sudo guix container exec 89086 /run/current-system/profile/bin/bash --login

When you start the container, you can get a shell into the container using the nsenter command. You will need the process ID of the container, which you can see on container startup or on your can get with something like:

ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container'

Where /usr/local/bin/genenetwork-development-container is the path used for invoking (running) the system container.

Once you have a listing, get the PID for the shepherd process. You could pipe the output of the command above to grep 'shepherd' to ease your search.

That will give you output of the form:

11869 pts/3    00:00:00 shepherd

From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html) under the packages option, the list of packages installed under the global profile are found in /run/current-system/profile, for example:

/run/current-system/profile/ls /gnu/store

to list the files under /gnu/store

With that knowledge, we can now get a shell using nsenter as follows:

sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
     --init-file /run/current-system/profile/etc/profile

which will give you a bash shell with the PATH environment variable setup correctly to give you access to all packages in the global profile.

Troubleshooting Tips

Use Profiles

When troubleshooting, we need to be using the correct profile that has all the necessary dependencies. Use a channels file to set up a profile. An example of a channel that was fixed at Python 3.9:

(list (channel
       (name 'gn-bioinformatics)
       (url "https://gitlab.com/genenetwork/guix-bioinformatics")
       (branch "master")
       (commit
        "9939feb61ea29881d42628bc58a43886f7da6573"))
      (channel
       (name 'guix-forge)
       (url "https://git.systemreboot.net/guix-forge/")
       (branch "main")
       (introduction
        (make-channel-introduction
         "0432e37b20dd678a02efee21adf0b9525a670310"
         (openpgp-fingerprint
          "7F73 0343 F2F0 9F3C 77BF  79D3 2E25 EE8B 6180 2BB3")))))

Activate a profile by:

export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile

Double-check to confirm that you are using the correct channel using guix describe. The use of profiles is well documented here—it should match your channels.scm file.

View Logs

When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":

tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log

Note that to be able to view log files, you have to have root permissions.

Back-ups

When running borg, you can run into the following error:

root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)

To fix it:

borg config repo id
rm ~/.config/borg/security/REPO_ID/manifest-timestamp
borg delete --cache-only REPO