This repository houses Guix configuration for GeneNetwork machines and
containers.

The git repo lives at https://git.genenetwork.org/gn-machines/

For CI/CD see

=> https://issues.genenetwork.org/topics/systems/ci-cd

For philosophy and (KISS) incremental minimalistic development containers, see:

=> https://issues.genenetwork.org/topics/systems/debug-and-developing-code-with-genenetwork-system-container

These are stored in the './minimal' directory of this repo. See the [[./minimal/README.md][README]].

* GeneNetwork development container (aka CI/CD)

The GeneNetwork development container is currently run on /tux02/.
It runs continuous integration and continuous deployment services for genenetwork2, genenetwork3 and several other associated projects.

To build and install the container, you will need the
[[https://gitlab.com/genenetwork/guix-bioinformatics][guix-bioinformatics]] and [[https://git.systemreboot.net/guix-forge/][guix-forge]] channels. Once these channels are
pulled and available, on /tux02/, run

#+BEGIN_SRC shell
$ ./genenetwork-development-deploy.sh
#+END_SRC

It will try to create symlinks at the end. You can do this as root (too).

Note we current run as aruni user and the latest guix is the user profile.

#+BEGIN_SRC shell
aruni@tux02:~/gn-machines$ which guix
/home/aruni/.config/guix/current/bin/guix
aruni@tux02:~/gn-machines$ guix describe
Generation 46   Mar 27 2025 23:39:42    (current)
#+END_SRC

/tux02/ is configured with a systemd service to run this
container. Restart it.
#+BEGIN_SRC shell
$ sudo systemctl restart genenetwork-development-container
#+END_SRC

* Virtuoso container for tux01 production

The virtuoso container is run on /tux01/ production. It runs virtuoso
alone.

To build and install the container, you will need the
guix-bioinformatics channel. Once guix-bioinformatics is pulled and
available, on /tux01/, run
#+begin_src shell

  $ ./virtuoso-deploy.sh
#+end_src

/tux01/ is configured with a systemd service to run this
container. Restart it.
#+begin_src shell
  $ sudo systemctl restart virtuoso-container
#+end_src

* Secure virtuoso authentication

In containers containing virtuoso instances, it is important to secure
authentication by changing default user passwords and disabling
unnecessary users. See [[https://issues.genenetwork.org/topics/systems/virtuoso][virtuoso gemtext documentation]] on passwords for
more details.

* Getting a Shell into the Container

You can get a shell into the container with something like:

#+BEGIN_SRC sh
  sudo guix container exec 89086 /run/current-system/profile/bin/bash --login
#+END_SRC

When you start the container, you can get a shell into the container using the ~nsenter~ command. You will need the process ID of the container, which you can see on container startup or on your can get with something like:

#+BEGIN_SRC sh
  ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container' | grep 'shepherd'
#+END_SRC

Where =/usr/local/bin/genenetwork-development-container= is the path used for
invoking (running) the system container.

That will give you output of the form:

#+BEGIN_EXAMPLE
  11869 pts/3    00:00:00 shepherd
#+END_EXAMPLE

From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html)
under the *packages* option, the list of packages installed under the global
profile are found in */run/current-system/profile*, for example:

#+BEGIN_SRC sh
  /run/current-system/profile/ls /gnu/store
#+END_SRC

to list the files under */gnu/store*

With that knowledge, we can now get a shell using ~nsenter~ as follows:

#+BEGIN_SRC sh
  sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
       --init-file /run/current-system/profile/etc/profile
#+END_SRC

which will give you a bash shell with the ~PATH~ environment variable setup
correctly to give you access to all packages in the global profile.

* Troubleshooting Tips

** Use Profiles

When troubleshooting, we need to be using the correct profile that has all the necessary dependencies.  Use a  [[https://ci.genenetwork.org/channels.scm][channels]] file to set up a profile.  An example of a channel that was fixed at Python 3.9:

#+begin_src scheme
(list (channel
       (name 'gn-bioinformatics)
       (url "https://gitlab.com/genenetwork/guix-bioinformatics")
       (branch "master")
       (commit
        "9939feb61ea29881d42628bc58a43886f7da6573"))
      (channel
       (name 'guix-forge)
       (url "https://git.systemreboot.net/guix-forge/")
       (branch "main")
       (introduction
        (make-channel-introduction
         "0432e37b20dd678a02efee21adf0b9525a670310"
         (openpgp-fingerprint
          "7F73 0343 F2F0 9F3C 77BF  79D3 2E25 EE8B 6180 2BB3")))))
#+end_src

Activate a profile by:

#+begin_src bash
export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile
#+end_src

Double-check to confirm that you are using the correct channel using =guix describe=.  The use of profiles is well documented [[https://issues.genenetwork.org/topics/guix-profiles][here]]---it should match your channels.scm file.

** View Logs

When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":

#+begin_src sh
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log
#+end_src

Note that to be able to view log files, you have to have root permissions.

** Back-ups

When running borg, you can run into the following error:

#+begin_src text
root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
#+end_src

To fix it:

: borg config repo id
: rm ~/.config/borg/security/REPO_ID/manifest-timestamp
: borg delete --cache-only REPO