about summary refs log tree commit diff
This repository houses Guix configuration for GeneNetwork machines and
containers.

The git repo lives at https://git.genenetwork.org/gn-machines/

For CI/CD see

=> https://issues.genenetwork.org/topics/systems/ci-cd

For philosophy and (KISS) incremental minimalistic development containers, see:

=> https://issues.genenetwork.org/topics/systems/debug-and-developing-code-with-genenetwork-system-container

These are stored in the './minimal' directory of this repo. See the [[./minimal/README.md][README]].

* GeneNetwork development container (aka CI/CD)

The GeneNetwork development container is currently run on /tux02/.
It runs continuous integration and continuous deployment services for genenetwork2, genenetwork3 and several other associated projects.

To build and install the container, you will need the
[[https://gitlab.com/genenetwork/guix-bioinformatics][guix-bioinformatics]] and [[https://git.systemreboot.net/guix-forge/][guix-forge]] channels. Once these channels are
pulled and available, on /tux02/, run

#+BEGIN_SRC shell
$ ./genenetwork-development-deploy.sh
#+END_SRC

It will try to create symlinks at the end. You can do this as root (too).

Note we current run as aruni user and the latest guix is the user profile.

#+BEGIN_SRC shell
aruni@tux02:~/gn-machines$ which guix
/home/aruni/.config/guix/current/bin/guix
aruni@tux02:~/gn-machines$ guix describe
Generation 46   Mar 27 2025 23:39:42    (current)
#+END_SRC

/tux02/ is configured with a systemd service to run this
container. Restart it.
#+BEGIN_SRC shell
$ sudo systemctl restart genenetwork-development-container
#+END_SRC

* Virtuoso container for tux01 production

The virtuoso container is run on /tux01/ production. It runs virtuoso
alone.

To build and install the container, you will need the
guix-bioinformatics channel. Once guix-bioinformatics is pulled and
available, on /tux01/, run
#+begin_src shell

  $ ./virtuoso-deploy.sh
#+end_src

/tux01/ is configured with a systemd service to run this
container. Restart it.
#+begin_src shell
  $ sudo systemctl restart virtuoso-container
#+end_src

* Secure virtuoso authentication

In containers containing virtuoso instances, it is important to secure
authentication by changing default user passwords and disabling
unnecessary users. See [[https://issues.genenetwork.org/topics/systems/virtuoso][virtuoso gemtext documentation]] on passwords for
more details.

* Getting a Shell into the Container

You can get a shell into the container with something like:

#+BEGIN_SRC sh
  sudo guix container exec 89086 /run/current-system/profile/bin/bash --login
#+END_SRC

When you start the container, you can get a shell into the container using the ~nsenter~ command. You will need the process ID of the container, which you can see on container startup or by inspecting the output of the following ~ps~ command:

#+BEGIN_SRC sh
  ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container' | grep 'shepherd'
#+END_SRC

Where =/usr/local/bin/genenetwork-development-container= is the path used for
invoking (running) the system container.

That will give you output of the form:

#+BEGIN_EXAMPLE
  11869 pts/3    00:00:00 shepherd
#+END_EXAMPLE

From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html)
under the *packages* option, the list of packages installed under the global
profile are found in */run/current-system/profile*, for example:

#+BEGIN_SRC sh
  /run/current-system/profile/ls /gnu/store
#+END_SRC

to list the files under */gnu/store*

With that knowledge, we can now get a shell using ~nsenter~ as follows:

#+BEGIN_SRC sh
  sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
       --init-file /run/current-system/profile/etc/profile
#+END_SRC

which will give you a bash shell with the ~PATH~ environment variable setup
correctly to give you access to all packages in the global profile.

* Troubleshooting Tips

** Use Profiles

When troubleshooting, we need to be using the correct profile that has all the necessary dependencies.  Use a  [[https://ci.genenetwork.org/channels.scm][channels]] file to set up a profile.  An example of a channel that was fixed at Python 3.9:

#+begin_src scheme
(list (channel
       (name 'gn-bioinformatics)
       (url "https://gitlab.com/genenetwork/guix-bioinformatics")
       (branch "master")
       (commit
        "9939feb61ea29881d42628bc58a43886f7da6573"))
      (channel
       (name 'guix-forge)
       (url "https://git.systemreboot.net/guix-forge/")
       (branch "main")
       (introduction
        (make-channel-introduction
         "0432e37b20dd678a02efee21adf0b9525a670310"
         (openpgp-fingerprint
          "7F73 0343 F2F0 9F3C 77BF  79D3 2E25 EE8B 6180 2BB3")))))
#+end_src

Activate a profile by:

#+begin_src bash
export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile
#+end_src

Double-check to confirm that you are using the correct channel using =guix describe=.  The use of profiles is well documented [[https://issues.genenetwork.org/topics/guix-profiles][here]]---it should match your channels.scm file.

** View Logs

When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":

#+begin_src sh
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log
#+end_src

Note that to be able to view log files, you have to have root permissions.

** Back-ups

When running borg, you can run into the following error:

#+begin_src text
root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
#+end_src

To fix it:

: borg config repo id
: rm ~/.config/borg/security/REPO_ID/manifest-timestamp
: borg delete --cache-only REPO

* Debugging a container started by run-guix-container (GeneCup example)

The newer containers (GeneCup, BNW, cgit, julia-notebook) are built with
=guix system container= and started through the small
=run-guix-system-container= daemon in the =run-guix-container= repo (a
sibling checkout), so a sandboxed agent without root can start them.  =genenetwork/containers/genecup-container.scm=
is the worked example below; the same layout applies to the others.

** Build

Three checkouts must be on the load path; =-L= must point at the
directory that *contains* the module trees (=forge/=, =gn/=,
=genenetwork/=), which differs per host:

#+begin_src sh
  # factory layout: checkouts under <root>/code/, guix-forge as a sub-checkout
  base=<root>/code; forge=$base/contrib/guix-forge/guix
  # flat layout (e.g. balg01): ~/services/genecup-service/{genecup,
  # gn-machines, guix-bioinformatics, guix-forge} side by side
  #   base=$HOME/services/genecup-service; forge=$base/guix-forge/guix

  guix system container --network \
    -L $base/guix-bioinformatics -L $base/gn-machines -L $forge \
    --expose=$HOME/.config/gemini/credentials=/run/secrets/gemini-credentials \
    $base/gn-machines/genenetwork/containers/genecup-container.scm
#+end_src

A wrong =-L= is silent and misleading: the service module's own
=(forge anubis)= import fails, the module loads empty, and you get

#+begin_example
  guix/ui.scm:1034:18: error: genecup-service-type: unbound variable
#+end_example

so check the paths first; this must print the service types:

#+begin_src sh
  guix repl -L $base/guix-bioinformatics -L $base/gn-machines -L $forge <<'EOF'
  (use-modules (genenetwork services genecup))
  (display (list genecup-service-type genecup-nginx-service-type))
  (newline)
  EOF
#+end_src

The Gemini key is a secret and is **not** in the store: the launcher
bind-mounts it read-only at =/run/secrets/gemini-credentials= and the
genecup activation copies it to =/var/lib/genecup/.config/gemini/credentials=
(owner =genecup=, mode =0400= -- the app calls =SystemExit= on any other
mode).  Without =--expose= the container still runs, minus the LLM
classification.

** Start and get in

#+begin_src sh
  cd <checkout>/run-guix-container          # the client lives here
  ./bin/run-guix-container.rb --daemon 127.0.0.1:8989 --fingerprint <FP> <HASH>
  ./bin/run-guix-container.rb --daemon 127.0.0.1:8989 --fingerprint <FP> --list
  ./bin/run-guix-container.rb --daemon 127.0.0.1:8989 --fingerprint <FP> --log <HASH>

  ssh -p 2223 -i ~/hello-container-key root@127.0.0.1
#+end_src

=--log= is the container's boot output; the daemon refuses to start
without root (it cannot write the container's uid map), and warns at
startup when =/var/log/guix-containers= is not writable, in which case
that output is lost.  Ports are on the host: 4201 nginx (+Anubis), 4202
the Flask app, 2223 ssh.

** Where to look

Inside the container (the =ssh= above), as root:

| path                                          | contents                                                                                                       |
|-----------------------------------------------+----------------------------------------------------------------------------------------------------------------|
| =/var/log/genecup.log=                        | the app: =GEMINI_API_KEY=set=, =Gemini API call: ... (gemini-3-flash-preview)= + prompt, =Gemini response: ...=, =Gemini batch attempt 1/3 failed (ClientError): ...=, traceback, =Gemini batch: N of M sentences labelled= |
| =/var/log/gemini-check.log=                   | one line from the boot check: =Gemini classification available: key in ... works, gemini-3-flash-preview answered (HTTP 200)=, or =... call failed (HTTP 404): {...}= |
| =/var/log/messages=                           | shepherd: service start failures and their exceptions                                                           |
| =/var/log/anubis.log=, =/var/log/nginx/*=     | only relevant when the page does not load at all                                                                |
| =/var/lib/genecup/.config/gemini/credentials= | must be =-r-------- genecup genecup=; wrong owner/mode means no key                                            |
| =/var/lib/genecup/=                           | =userspub.sqlite= and the per-user result folders                                                              |

#+begin_src sh
  grep -n Gemini /var/log/genecup.log | tail -40
  grep -A40 "Gemini batch failed after" /var/log/genecup.log   # last traceback
  cat /var/log/gemini-check.log
  herd status genecup; herd status gemini-check
#+end_src

Two traps: the app's stdout is block-buffered when shepherd redirects it
to the log (the service sets =PYTHONUNBUFFERED=1=, older builds did
not, so their log lags behind the page), and only *successful*
classifications are cached (in memory), so reloading the same sentence
page reuses a cached result while a fresh search forces a new API call.

** "Gemini classification not available"

The stress view used to print one heading for three different
situations.  It now names them:

- *no key* -- "no API key at =~/.config/gemini/credentials= (missing,
  unreadable or not mode 0400)", heading "not classified: no Gemini API key";
- *call failed* -- the exception, the model, the attempt count, and a
  pointer to =/var/log/genecup.log= and =/var/log/gemini-check.log=;
- *no labels* -- the model answered but without a Cellular Stress /
  Organismal Stress label, quoting the raw response.

The boot check in =/var/log/gemini-check.log= calls the model
(=gemini-3-flash-preview=, the same constant as =server.py=) rather than
just listing =/v1beta/models=: a key can list a model and still not be
allowed to call it (preview gating, project/quota/region), which shows up
as exactly this page heading.

To test the model from a shell (substitute the key file the container
uses):

#+begin_src sh
  curl -sS -X POST -H 'Content-Type: application/json' \
    -d '{"contents":[{"parts":[{"text":"Classify the following sentence as Cellular Stress or Organismal Stress and answer with the label only: Heat shock protein 70 protects cells from oxidative stress."}]}]}' \
    "https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent?key=$(cat ~/.config/gemini/credentials)"
#+end_src

** Tests

#+begin_src sh
  # build with the key exposed, start via the daemon, assert credential
  # mode, the boot-check verdict, GEMINI_API_KEY=set and the app on 4202
  genenetwork/services/test/run_genecup_container_tests.sh [--keep]
  # same, for a container built without a key
  GEMINI_KEY=/nonexistent genenetwork/services/test/run_genecup_container_tests.sh --expect-missing
#+end_src

Checkout locations and the daemon address are auto-detected (override
with =base=, =GUIX_BIOINF=, =GN_MACHINES=, =GUIX_FORGE=, =DAEMON=,
=DAEMON_FINGERPRINT=, =SSH_PORT=, =CLIENT_KEY=).  The script kills any
container the daemon is already running and harvests the logs into
=genenetwork/services/test/logs/= before killing its own.