This repository houses Guix configuration for GeneNetwork machines and
containers.

The git repo lives at https://git.genenetwork.org/gn-machines/

* GeneNetwork development container

The GeneNetwork development container is run on /tux02/. It runs
continuous integration and continuous deployment services for
genenetwork2, genenetwork3 and several other associated projects.

To build and install the container, you will need the
[[https://gitlab.com/genenetwork/guix-bioinformatics][guix-bioinformatics]] and [[https://git.systemreboot.net/guix-forge/][guix-forge]] channels. Once these channels are
pulled and available, on /tux02/, run
#+BEGIN_SRC shell
$ ./genenetwork-development-deploy.sh
#+END_SRC

/tux02/ is configured with a systemd service to run this
container. Restart it.
#+BEGIN_SRC shell
$ sudo systemctl restart genenetwork-development-container
#+END_SRC

* Virtuoso container for tux01 production

The virtuoso container is run on /tux01/ production. It runs virtuoso
alone.

To build and install the container, you will need the
guix-bioinformatics channel. Once guix-bioinformatics is pulled and
available, on /tux01/, run
#+begin_src shell
  $ ./virtuoso-deploy.sh
#+end_src

/tux01/ is configured with a systemd service to run this
container. Restart it.
#+begin_src shell
  $ sudo systemctl restart virtuoso-container
#+end_src

* Secure virtuoso authentication

In containers containing virtuoso instances, it is important to secure
authentication by changing default user passwords and disabling
unnecessary users. See [[https://issues.genenetwork.org/topics/systems/virtuoso][virtuoso gemtext documentation]] on passwords for
more details.

* Getting a Shell into the Container

You can get a shell into the container with something like:

#+BEGIN_SRC sh
  sudo guix container exec 89086 /run/current-system/profile/bin/bash --login
#+END_SRC

When you start the container, you can get a shell into the container using the
~nsenter~ command. You will need the process ID of the container, which your
can get with something like:

#+BEGIN_SRC sh
  ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container'
#+END_SRC

Where =/usr/local/bin/genenetwork-development-container= is the path used for
invoking (running) the system container.

Once you have a listing, get the PID for the =shepherd= process. You could pipe
the output of the command above to ~grep 'shepherd'~ to ease your search.

That will give you output of the form:

#+BEGIN_EXAMPLE
  11869 pts/3    00:00:00 shepherd
#+END_EXAMPLE

From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html)
under the *packages* option, the list of packages installed under the global
profile are found in */run/current-system/profile*, for example:

#+BEGIN_SRC sh
  /run/current-system/profile/ls /gnu/store
#+END_SRC

to list the files under */gnu/store*

With that knowledge, we can now get a shell using ~nsenter~ as follows:

#+BEGIN_SRC sh
  sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
       --init-file /run/current-system/profile/etc/profile
#+END_SRC

which will give you a bash shell with the ~PATH~ environment variable setup
correctly to give you access to all packages in the global profile.

* Troubleshooting Tips

** Use Profiles

When troubleshooting, we need to be using the correct profile that has all the necessary dependencies.  Use a  [[https://ci.genenetwork.org/channels.scm][channels]] file to set up a profile.  An example of a channel that was fixed at Python 3.9:

#+begin_src scheme
(list (channel
       (name 'gn-bioinformatics)
       (url "https://gitlab.com/genenetwork/guix-bioinformatics")
       (branch "master")
       (commit
        "9939feb61ea29881d42628bc58a43886f7da6573"))
      (channel
       (name 'guix-forge)
       (url "https://git.systemreboot.net/guix-forge/")
       (branch "main")
       (introduction
        (make-channel-introduction
         "0432e37b20dd678a02efee21adf0b9525a670310"
         (openpgp-fingerprint
          "7F73 0343 F2F0 9F3C 77BF  79D3 2E25 EE8B 6180 2BB3")))))
#+end_src

Activate a profile by:

#+begin_src bash
export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile
#+end_src

Double-check to confirm that you are using the correct channel using =guix describe=.  The use of profiles is well documented [[https://issues.genenetwork.org/topics/guix-profiles][here]]---it should match your channels.scm file.

** View Logs

When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":

#+begin_src sh
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log
#+end_src

Note that to be able to view log files, you have to have root permissions.

** Back-ups

When running borg, you can run into the following error:

#+begin_src text
root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
#+end_src

To fix it:

: borg config repo id
: rm ~/.config/borg/security/REPO_ID/manifest-timestamp
: borg delete --cache-only REPO