Fallback and debugging

3 files changed, 129 insertions, 14 deletions

diff --git a/issues/systems/fallback-errors.gmi b/issues/systems/fallback-errors.gmi
index e0111de..3c44e50 100644
--- a/issues/systems/fallback-errors.gmi
+++ b/issues/systems/fallback-errors.gmi
@@ -4,7 +4,7 @@ This page tracks outstanding problems with
 
 => https://fallback.genenetwork.org/
 
-Note that some issues may be related to produciton (i.e. the same).
+Note that some issues may be related to production (i.e. the same). Note also I created my own development VM in just an hour, so I don't have to mess with the fallback while others are using it.
 
 # Tags
 
@@ -17,15 +17,16 @@ Note that some issues may be related to produciton (i.e. the same).
 # Tasks
 
 * [X] 502 timeout errors
-* [ ] Rqtl2 not working
+* [X] Rqtl2 not working
 * [ ] 413 error
 * [ ] Monitor service - both systemd and sheepdog
+* [ ] Files in /tmp
 
-## 502 timeout errors
+## 502 timeout errors (fixed)
 
 These should no longer happen. It required relaxing timeouts on nginx and gunicorn.
 
-## Rqtl2/HK/Pairscan not working
+## Rqtl2/HK/Pairscan not working (fixed)
 
 URL error on result of
 
@@ -61,7 +62,7 @@ and in the code we mostly see another addition of the slash - except for this pa
 
 => https://en.wikipedia.org/wiki/URL
 
-# R/qtl JSONDecodeError
+# R/qtl JSONDecodeError (fixed)
 
 After adding the slash to GN3 URLs (see above) led to the next error:
 
@@ -119,7 +120,9 @@ So, for some reason Rscript is not in the path for GN3. So I added that to the p
 
 =>  /topics/systems/debug-and-developing-code-with-genenetwork-system-container.gmi
 
-## QTLReaper showing font problem
+That way I was able to troubleshoot the paths and added them to the containers.
+
+## QTLReaper showing font problem (fixed)
 
 Try HK on
 
@@ -144,19 +147,40 @@ COUR_FILE = "./gn2/wqflask/static/fonts/courbd.ttf"
 
 so most likely GN2 is not running from that actually source dir.
 
-## Pair scan runs, but does not show image
+## Pair scan runs, but does not show image (fixed)
 
 Pair scan shows a table on the fallback, but not the image. On CD and production no pair scan results!
 
+=> https://gndev.genenetwork.org/show_trait?trait_id=10150&dataset=BXDPublish
+
+Takes about a minute for 26 mice. The csv files are generated in /tmp/gn3, but no image. The relevant code is
+
+```
+   <div class="qtlcharts" id="chart_container">
+        <div id="pairscan_chart"></div>
+    </div>
+```
+
+The problem is with Karl's D3 panels, we get an error 'TypeError: d3.scaleLinear is not a function'. That 1.7.1 release is a bit old, so let's try updating that first. After updating to 1.8.4 we have the same error. D3 is at 3.5.17 - that is 2016(!).
+
+Now the latest versions of D3 do *not* provide the actual JS through github. The page
+
+=> https://d3js.org/getting-started
+
+has links to
+
+=> https://d3js.org/d3.v7.js
+=> https://d3js.org/d3.v7.min.js
+
+We can't use those because the SHA may change. So we have to host a local copy. That is fine.
+
 ## 413 Request Entity Too Large
 
 On mapping from
 
 => https://fallback.genenetwork.org/show_trait?trait_id=10002&dataset=HET3-ITPPublish
 
-## Purescript genome browser icons not showing
-
-## Files generated in /tmp
+## Files generated in /tmp (later)
 
 Not unique file names appearing in TMPDIR:
 
@@ -168,9 +192,22 @@ Not unique file names appearing in TMPDIR:
 -rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 835 Apr  2 07:03 -logP.png
 ```
 
+These are created by the HK run. Harmless for now, but if two people run HK at the same time we'll get a conflict.
+
 Unique file names, but lingering on file system - note they are root because I created them outside a container
 
 ```
 -rw-r--r-- 1 root root   10M Apr  2 07:03 4OJCzmGV.cross
 -rw-r--r-- 1 root root   10M Apr  2 07:03 6cExgwkd.cross
 ```
+
+# Purescript browser buttons not showing (fixed)
+
+The following button does not display:
+
+```
+                       <button id="scrollLeft"  type="button" >
+                          <i class="fas fa-arrow-left"></i>
+```
+
+it is part of font-awesome and apparently not loaded properly. Actually the stylesheet is loaded twice(!) in mapping_results.html AND we use two different class names (fas and fa). Changing the class name fixed it.
diff --git a/topics/systems/debug-and-developing-code-with-genenetwork-system-container.gmi b/topics/systems/debug-and-developing-code-with-genenetwork-system-container.gmi
index d52090e..2e3a83c 100644
--- a/topics/systems/debug-and-developing-code-with-genenetwork-system-container.gmi
+++ b/topics/systems/debug-and-developing-code-with-genenetwork-system-container.gmi
@@ -41,6 +41,12 @@ root@genenetwork /# ls /export/source/fallback-debug/
   genenetwork3/
 ```
 
+Another useful command is running a command from outside the VM:
+
+```
+nsenter -at 1359047 /run/current-system/profile/bin/tail -f /var/log/gunicorn-genenetwork3.log
+```
+
 Next, after making a note of the port and paths with `ps xau`, we stop the running GN3 instance with
 
 ```
@@ -55,13 +61,22 @@ Now we can start GN3 properly. We can reuse the gunicorn setup above, but for de
 
 Note the added quotes. The command will fail with 'No module named gn3'. Good! Now to load the source dir we need to make it visible. We'll use $SOURCES for that.
 
-Through shepherd find the profile in use
+Through above shepherd command find the profile in use by printing pola-wrapper. You can use a local Ruby command, such as
+
+```
+/gnu/store/7ri578qarmn1cj2inl243xar6p7j1vxh-ruby-3.1.4/bin/ruby -n -e '$_ =~ /(\"\S+-profile)/; p $1 if $1' < /gnu/store/d7idvpad0d49l7647ci046w3xiz58xym-gunicorn-genenetwork3-pola-wrapper
+"\"/gnu/store/yi76sybwqql4ky60yahv91z57srb2fr0-profile
+```
+
+and set the profile to match
 
 ```
-export PROFILE=/gnu/store/yi76sybwqql4ky60yahv91z57srb2fr0-profile/lib/python3.10/site-packages/
+export PROFILE=/gnu/store/yi76sybwqql4ky60yahv91z57srb2fr0-profile
 ```
 
-This worked loading the PYTHONPATH worker source path with `--chdir`!
+> Note: you can find ruby in /gnu/store in the VM as it shares with the host
+
+This worked running gunicorn directly and loading the PYTHONPATH worker source path with `--chdir`!
 
 ```
 herd stop gunicorn-genenetwork3
@@ -120,6 +135,26 @@ export OAUTH2_ACCESS_TOKEN_GENERATOR="tests.unit.auth.test_token.gen_token"
 PATH=$PATH:$PROFILE/bin R_LIBS_USER=$PROFILE/site-library PYTHONPATH=$PROFILE/lib/python3.10/site-packages pytest
 ```
 
+# Locating scripts and fonts
+
+In Guix VMs we can not assume the start dir is the base of the source code - as we do in development. Paths may need to be written and that happens at the package level. E.g.
+
+=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=6303fe046b944a380704eb02144cca777fee689b
+
+The great thing is that, once it is done, it will always work with Guix! It is also a security feature that the full store is hard coded.
+
+For fonts we have
+
+```
+gn2/utility/Plot.py
+44:VERDANA_FILE = "./gn2/wqflask/static/fonts/verdana.ttf"
+45:COUR_FILE = "./gn2/wqflask/static/fonts/courbd.ttf"
+```
+
+and rewrite them with
+
+=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=1a6522f3a5a7d0990e50fe5dd91f286f62e083e7
+
 # Fixing shared paths
 
 By default both GN2 and GN3 run as containers. We pass temporary files through the file sytem, so let's try and fix that first. The good news is that they only have to share the TMPDIR. First we share a new directory under /var/tmp for the system container. Next we have to tell GN2 and GN3 system containers to use /var/tmp/gn2. This was done in commit
@@ -149,3 +184,46 @@ rqtl_wrapper = current_app.config["RQTL_WRAPPER"]
 ```
 
 http://127.0.0.1:8893/api/menu/generate/json
+
+# Speeding up channel builds
+
+The routine of `guix pull` for channel builds is quite elaborate in a development phase. Also, for every change you'll have to update the remote guix-bioinformatics repository - and that is not so great either. One solution is to *remove* the guix-bioinformatics channel and use the -L switch on a local repo instead.
+
+```
+git clone https://git.genenetwork.org/guix-bioinformatics/
+```
+
+Now you need to remove guix-bioinformatics from `.config/guix/channels.scm` and now do a guix pull using the local repo:
+
+```
+guix pull -p ~/opt/guix-dev
+```
+
+To get a channel list from an existing guix try `guix describe -f channel`. That can be pasted into a channels file. Anyway, you should end up with something like this:
+
+```
+~/opt/guix-dev/bin/guix describe
+Generation 1    Apr 02 2024 10:13:37    (current)
+  guix b0b988c
+    repository URL: https://git.savannah.gnu.org/git/guix.git
+    branch: master
+    commit: b0b988c41c9e0e591274495a1b2d6f27fcdae15a
+  guix-forge 6c622a6
+    repository URL: https://git.systemreboot.net/guix-forge/
+    branch: main
+    commit: 6c622a67051c22eeefe37eedb17d427fbb70c99b
+  guix-past 921f845
+    repository URL: https://gitlab.inria.fr/guix-hpc/guix-past
+    branch: master
+    commit: 921f845dc0dec9f052dcda479a15e787f9fd5b0a
+```
+
+Now we can use guix bioinformatics with
+
+```
+~/opt/guix-dev/bin/guix search -L ~/guix-bioinformatics genenetwork
+```
+
+and it should show the genenetwork stack. Now you can build a machine using -L. See for example:
+
+=> gndev-deploy
diff --git a/topics/systems/fire-up-genenetwork-system-container.gmi b/topics/systems/fire-up-genenetwork-system-container.gmi
index d751fa3..6778dee 100644
--- a/topics/systems/fire-up-genenetwork-system-container.gmi
+++ b/topics/systems/fire-up-genenetwork-system-container.gmi
@@ -98,7 +98,7 @@ apt-get install libnginx-mod-stream
 Inside the contianer, the first time you may check the certificates
 
 ```
-acme renew
+/usr/bin/acme renew
 ```
 
 After that you should be able to run