This repository houses Guix configuration for GeneNetwork machines and
containers.
The git repo lives at https://git.genenetwork.org/gn-machines/
* GeneNetwork development container
The GeneNetwork development container is currently run on /tux02/. It runs
continuous integration and continuous deployment services for
genenetwork2, genenetwork3 and several other associated projects.
To build and install the container, you will need the
[[https://gitlab.com/genenetwork/guix-bioinformatics][guix-bioinformatics]] and [[https://git.systemreboot.net/guix-forge/][guix-forge]] channels. Once these channels are
pulled and available, on /tux02/, run
#+BEGIN_SRC shell
$ ./genenetwork-development-deploy.sh
#+END_SRC
/tux02/ is configured with a systemd service to run this
container. Restart it.
#+BEGIN_SRC shell
$ sudo systemctl restart genenetwork-development-container
#+END_SRC
* Virtuoso container for tux01 production
The virtuoso container is run on /tux01/ production. It runs virtuoso
alone.
To build and install the container, you will need the
guix-bioinformatics channel. Once guix-bioinformatics is pulled and
available, on /tux01/, run
#+begin_src shell
$ ./virtuoso-deploy.sh
#+end_src
/tux01/ is configured with a systemd service to run this
container. Restart it.
#+begin_src shell
$ sudo systemctl restart virtuoso-container
#+end_src
* Secure virtuoso authentication
In containers containing virtuoso instances, it is important to secure
authentication by changing default user passwords and disabling
unnecessary users. See [[https://issues.genenetwork.org/topics/systems/virtuoso][virtuoso gemtext documentation]] on passwords for
more details.
* Getting a Shell into the Container
You can get a shell into the container with something like:
#+BEGIN_SRC sh
sudo guix container exec 89086 /run/current-system/profile/bin/bash --login
#+END_SRC
When you start the container, you can get a shell into the container using the ~nsenter~ command. You will need the process ID of the container, which you can see on container startup or on your can get with something like:
#+BEGIN_SRC sh
ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container'
#+END_SRC
Where =/usr/local/bin/genenetwork-development-container= is the path used for
invoking (running) the system container.
Once you have a listing, get the PID for the =shepherd= process. You could pipe
the output of the command above to ~grep 'shepherd'~ to ease your search.
That will give you output of the form:
#+BEGIN_EXAMPLE
11869 pts/3 00:00:00 shepherd
#+END_EXAMPLE
From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html)
under the *packages* option, the list of packages installed under the global
profile are found in */run/current-system/profile*, for example:
#+BEGIN_SRC sh
/run/current-system/profile/ls /gnu/store
#+END_SRC
to list the files under */gnu/store*
With that knowledge, we can now get a shell using ~nsenter~ as follows:
#+BEGIN_SRC sh
sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
--init-file /run/current-system/profile/etc/profile
#+END_SRC
which will give you a bash shell with the ~PATH~ environment variable setup
correctly to give you access to all packages in the global profile.
* Troubleshooting Tips
** Use Profiles
When troubleshooting, we need to be using the correct profile that has all the necessary dependencies. Use a [[https://ci.genenetwork.org/channels.scm][channels]] file to set up a profile. An example of a channel that was fixed at Python 3.9:
#+begin_src scheme
(list (channel
(name 'gn-bioinformatics)
(url "https://gitlab.com/genenetwork/guix-bioinformatics")
(branch "master")
(commit
"9939feb61ea29881d42628bc58a43886f7da6573"))
(channel
(name 'guix-forge)
(url "https://git.systemreboot.net/guix-forge/")
(branch "main")
(introduction
(make-channel-introduction
"0432e37b20dd678a02efee21adf0b9525a670310"
(openpgp-fingerprint
"7F73 0343 F2F0 9F3C 77BF 79D3 2E25 EE8B 6180 2BB3")))))
#+end_src
Activate a profile by:
#+begin_src bash
export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile
#+end_src
Double-check to confirm that you are using the correct channel using =guix describe=. The use of profiles is well documented [[https://issues.genenetwork.org/topics/guix-profiles][here]]---it should match your channels.scm file.
** View Logs
When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":
#+begin_src sh
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log
#+end_src
Note that to be able to view log files, you have to have root permissions.
** Back-ups
When running borg, you can run into the following error:
#+begin_src text
root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
#+end_src
To fix it:
: borg config repo id
: rm ~/.config/borg/security/REPO_ID/manifest-timestamp
: borg delete --cache-only REPO