Merge /home/gn2/gene

1 files changed, 843 insertions, 14 deletions

diff --git a/doc/README.org b/doc/README.org
index f6ab6a52..3754dbbd 100644
--- a/doc/README.org
+++ b/doc/README.org
@@ -1,28 +1,857 @@
-#+TITLE: Installing GeneNetwork services with GNU Guix
+
+#+TITLE: Installing GeneNetwork services
 
 * Table of Contents                                                     :TOC:
  - [[#introduction][Introduction]]
- - [[#binary-deployment][Binary deployment]]
- - [[#from-source-deployment][From source deployment]]
+ - [[#quick-installation-recipe][Quick installation recipe]]
+   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
+   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
+   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
+   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]
+ - [[#run-mysql-server][Run MySQL server]]
+ - [[#gn2-dependency-graph][GN2 Dependency Graph]]
+ - [[#source-deployment][Source deployment]]
+   - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]]
+   - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]]
+ - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]]
+   - [[#update-to-recent-guix][Update to recent guix]]
+   - [[#install-gn2][Install GN2]]
+   - [[#run-gn2][Run GN2]]
+ - [[#trouble-shooting][Trouble shooting]]
+   - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]]
+   - [[#error-can-not-find-directory-homegn2_data][ERROR: can not find directory $HOME/gn2_data]]
+   - [[#cant-run-a-module][Can't run a module]]
+ - [[#irc-session][IRC session]]
 
 * Introduction
 
-Large system deployments tend to get very complex. In this document we
-explain the GeneNetwork deployment system which is based on GNU Guix
-(see Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]).
+If you want to understand the architecture of GN2 read
+[[Architecture.org]].  The rest of this document is mostly on deployment
+of GN2.
+
+Large system deployments can get very [[http://biobeat.org/gn2.svg][complex]]. In this document we
+explain the GeneNetwork version 2 (GN2) reproducible deployment system
+which is based on GNU Guix (see also Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix
+system can be used to install GN with all its files and dependencies.
+
+The official installation path is from a checked out version of the
+main Guix package tree and that of the Genenetwork package
+tree. Current supported versions can be found as the SHA values of
+'gn-latest' branches of [[https://github.com/genenetwork/guix-bioinformatics/tree/gn-latest][Guix bioinformatics]] and [[https://github.com/genenetwork/guix/tree/gn-latest][GNU Guix main]].
+
+For a full view of runtime dependencies as defined by GNU Guix, see
+the [[#gn2-dependency-graph][GN2 Dependency Graph]].
+
+* Quick installation recipe
+
+This is a recipe for quick and dirty installation of GN2. For
+convenience everything is installed as root, though in reality only
+GNU Guix has to be installed as root. I tested this recipe on a fresh
+install of Debian 8.3.0 (in KVM) though it should work on any modern
+Linux distribution (including CentOS). For more elaborate installation
+instructions see [[#source-deployment][Source deployment]].
+
+Note that GN2 consists of an approx. 5 GB installation including
+database.
+
+** Step 1: Install GNU Guix
+
+Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow
+[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially you have to download and unpack the tar ball
+(which creates directories in /gnu and /var/guix), add build users and
+group (Guix builds software as unpriviliged users) and run the Guix
+daemon after fixing the paths (also known as the 'profile').
+
+Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission
+to download binaries from the GNU server) and you should be able to
+install the hello package using binary packages (no building)
+
+#+begin_src bash
+export PATH=~/.guix-profile/bin:$PATH
+guix pull
+guix package -i hello --dry-run
+#+end_src
+
+Which should show something like
+
+: The following files would be downloaded:
+:   /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
+:   ...
+#+end_src
+
+means binary installs.  The actual installation command of 'hello' is
+
+#+begin_src bash
+guix package -i hello
+hello
+  Hello, world!
+#+end_src
 
-* Binary deployment
+If you actually see things building it means that Guix is not yet
+properly installed and up-to-date, i.e., the key is missing or you
+need to do a 'guix pull'. Press Ctrl-C to interrupt.
 
-NYA
+If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get
+rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]].
 
-* From source deployment
+** Step 2: Checkout the GN2 git repositories
 
-GNU Guix allows for [[https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org][reproducible deployment]] based on a checked out
-Guix repository - use gn-stable for that:
+To fixate the software dependency graph GN2 uses git repositories of
+Guix packages. First install git if it is missing
 
-#+begin_src sh   :lang bash
+#+begin_src bash
+guix package -i git
+export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
+#+end_src
+
+check out the git repsitories (gn-latest branch)
+
+#+begin_src bash
+cd ~
 mkdir genenetwork
 cd genenetwork
-git checkout https://github.com/genenetwork/guix.git gn-stable-guix
-git checkout https://github.com/genenetwork/guix-bioinformatics.git
+git clone --branch gn-latest https://github.com/genenetwork/guix-bioinformatics
+git clone --branch gn-latest --recursive https://github.com/genenetwork/guix guix-gn-latest
+cd guix-gn-latest
+#+end_src bash
+
+** Step 3: Authorize the GN Guix server
+
+GN2 has its own GNU Guix binary distribution server. To trust it you have
+to add the following key 
+
+#+begin_src scheme
+(public-key
+ (ecc
+  (curve Ed25519)
+  (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#)
+ )
+)
+#+end_src
+
+by pasting it into the command
+
+#+begin_src bash
+guix archive --authorize
+#+end_src
+
+and hit Ctrl-D.
+
+Now you can use the substitute server to install GN2 binaries. 
+
+** Step 4: Install and run GN2 
+
+Since this is a quick and dirty install we are going to override the
+GNU Guix package path by pointing the package path to our repository:
+
+#+begin_src bash
+rm /root/.config/guix/latest
+ln -s ~/genenetwork/guix-gn-latest/ /root/.config/guix/latest
+#+end_src
+
+Now check whether you can find the GN2 package with
+
+#+begin_src bash
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2
+  genenetwork2    2.0-a8fcff4     out     gn/packages/genenetwork.scm:144:2
+#+end_src
+
+(ignore the source file newer then ... messages, this is caused by the
+/root/.config/guix/latest override).
+
+And install with
+
+#+begin_src bash
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \
+  guix package -i genenetwork2 \
+  --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" \
+  --fallback
+#+end_src
+
+Note: the order of the substitute url's may make a difference in speed
+(put the one first that is fastest for your location and time of day).
+
+After installation you should be able to run genenetwork2 after updating
+the Guix suggested environment vars. Check the output of
+
+#+begin_src bash
+guix package --search-paths
+export PYTHONPATH="/root/.guix-profile/lib/python2.7/site-packages"
+export R_LIBS_SITE="/root/.guix-profile/site-library/"
+#+end_src
+
+and copy-paste the listed exports into the terminal before running:
+
+#+begin_src bash
+genenetwork2
+#+end_src
+
+It will complain that the database is missing. See the next section on
+running MySQL server for downloading and installing a MySQL GN2
+database. After installing the database restart genenetwork2 and point
+your browser at [[http://localhost:5003/]].
+
+End of the GN2 installation recipe!
+
+* Run MySQL server
+
+At this point we require the underlying distribution to install and
+run mysqld. Currently we have two databases for deployment,
+'db_webqtl_s' is the small testing database containing experiments
+from BXD mice and 'db_webqtl_plant' which contains all plant related
+material.
+
+Download one database from
+
+http://files.genenetwork.org/raw_database/
+https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip
+
+Check the md5sum.
+
+After installation inflate the database binary in the MySQL directory
+(this installation path is subject to change soon) 
+
+: chown -R mysql:mysql db_webqtl_s/
+: chmod 700 db_webqtl_s/
+: chmod 660 db_webqtl_s/*
+
+restart MySQL service (mysqld). Login as root and
+
+: mysql> show databases;
+: +--------------------+
+: | Database           |
+: +--------------------+
+: | information_schema |
+: | db_webqtl_s        |
+: | mysql              |
+: | performance_schema |
+: +--------------------+
+
+Set permissions and match password in your settings file below:
+
+: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password';
+
+Note that if the mysql connection is not working, try connecting to
+the IP address and check server firewall, hosts.allow and mysql IP
+configuration.
+
+Note for the plant database you can rename it to db_webqtl_s, or
+change the settings in etc/default_settings.py to match your path.
+
+* GN2 Dependency Graph
+
+Graph of all runtime dependencies as installed by GNU Guix.
+
+#+ATTR_HTML: :title GN2_graph
+[[http://biobeat.org/gn2.svg]]
+
+* Source deployment
+
+This section gives a more elaborate instruction for installing GN2
+from source.
+
+First execute above 4 steps:
+
+   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
+   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
+   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
+   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]
+
+
+** Run your own copy of GN2
+
+At some point you may want to fix the source code. Assuming you have
+Guix and Genenetwork2 installed (as described above) clone the GN2
+repository from https://github.com/genenetwork/genenetwork2.
+
+Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
+R_LIBS_SITE are set) from the information given by guix:
+
+: guix package --search-paths
+
+Inside the repository:
+
+: cd genenetwork2
+: ./bin/genenetwork2 
+
+Will fire up your local repo http://localhost:5003/ using the  
+settings in ./etc/default_settings.py. These settings may 
+not reflect your system. To override settings create your own from a copy of
+default_settings.py and pass it into GN2 with
+
+: ./bin/genenetwork2 $HOME/my_settings.py
+
+and everything *should* work (note the full path to the settings
+file). This way we develop against the exact same dependency graph of
+software.
+
+If something is not working, take a hint from the settings file
+that comes in the Guix installation. It sits in something like
+
+: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py
+
+** Set up nginx port forwarding
+
+nginx can be used as a reverse proxy for GN2. For example, we want to
+expose GN2 on port 80 while it is running on port 5003. Essentially
+the configuration looks like
+
+#+begin_src js
+    server {
+        listen 80;
+        server_name test-gn2.genenetwork.org;
+        access_log  logs/test-gn2.access.log;
+
+        proxy_connect_timeout       3000;
+        proxy_send_timeout          3000;
+        proxy_read_timeout          3000;
+        send_timeout                3000;
+
+        location / {
+            proxy_set_header   Host      $http_host;
+            proxy_set_header   Connection keep-alive;
+            proxy_set_header   X-Real-IP $remote_addr;
+            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header   X-Forwarded-Host $server_name;
+            proxy_pass         http://127.0.0.1:5003;
+        }
+}
+#+end_src js
+
+Install the nginx webserver (as root)
+
+: guix package -i nginx
+
+The nginx example configuration examples can be found in the Guix
+store through
+
+: ls -l /root/.guix-profile/sbin/nginx
+: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx
+
+Use that path
+
+: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
+:   fastcgi.conf            koi-win             scgi_params
+:   fastcgi.conf.default    mime.types          scgi_params.default
+:   fastcgi_params          mime.types.default  uwsgi_params
+:   fastcgi_params.default  nginx.conf          uwsgi_params.default
+:   koi-utf                 nginx.conf.default  win-utf
+
+And copy any relevant files to /etc/nginx.  A configuration file for
+GeneNetwork (reverse proxy) port forwarding can be found in the source
+repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
+(still as root)
+: cp ./etc/nginx-genenetwork.conf /etc/nginx/
+
+Make dirs
+
+: mkdir -p /var/spool/nginx/logs
+
+Add users
+
+: adduser nobody ; addgroup nobody 
+
+Run nginx
+
+: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx
+
+* Source deployment and other information on reproducibility
+
+See the document [[GUIX-Reproducible-from-source.org]].
+
+** Update to recent guix
+
+We now compile Guix from scratch.
+
+Create, install and run a recent version of the guix-daemon by
+compiling the guix repository you have installed with git in
+step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after
+
+: cd ~/genenetwork/guix-gn-latest
+
+Make sure to restart the guix daemon and run guix client from this
+directory.
+
+** Install GN2
+
+Reinstall genenetwork2 using the new tree
+
+#+begin_src bash
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org"
+#+end_src bash
+
+Note the use of ./pre-inst-env here!
+
+Actually, it should be the same installation as in step 4, so nothing
+gets downloaded.
+
+** Run GN2
+
+Make a note of the paths with
+
+#+begin_src bash
+./pre-inst-env guix package --search-paths
+#+end_src bash
+
+After setting the paths for the server
+
+#+begin_src bash
+export PATH=~/.guix-profile/bin:$PATH
+export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
+export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
+export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
+export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
+export XDG_DATA_DIRS="$HOME/.guix-profile/share"
+export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
+#+end_src bash
+
+run the main script (in ~/.guix-profile/bin)
+
+#+begin_src bash
+genenetwork2
+#+end_src bash
+
+will start the default server which listens on port 5003, i.e.,
+http://localhost:5003/.
+
+OK, we are where we were before with step 4. Only difference is that we 
+used our own compiled guix server.
+
+* Trouble shooting
+
+** ImportError: No module named jinja2
+
+If you have all the Guix packages installed this error points out that
+the environment variables are not set. Copy-paste the paths into your
+terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
+information given by guix:
+
+: guix package --search-paths
+
+On one system:
+
+: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
+: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
+: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
+
+and perhaps a few more. 
+** ERROR: can not find directory $HOME/gn2_data
+
+The default settings file looks in your $HOME/gn2_data. Since these
+files come with a Guix installation you should take a hint from the
+values in the installed version of default_settings.py (see above in
+this document).
+
+** Can't run a module
+
+In rare cases, development modules are not brought in with Guix
+because no source code is available. This can lead to missing modules
+on a running server. Please check with the authors when a module
+is missing.
+* IRC session
+
+Here an IRC session where we installed GN2 from scratch using GNU Guix
+and a download of the test database.
+
+#+begin_src
+<pjotrp> time to get binary install sorted :)  [07:03]
+<pjotrp> Guix is designed for distributed installation servers
+<pjotrp> we have one on guix.genenetwork.org
+<pjotrp> it contains all the prebuild packages
+<pjotrp> for GN
+<user01> okay  [07:04]
+<pjotrp> let's step back however  [07:05]
+<pjotrp> I presume the environment is set with all guix package --search-paths
+<pjotrp> right?
+<user01> yep
+<user01> set to the ones in ~/.guix-profile/
+<pjotrp> good, and you are in gn-latest-guix repo  [07:06]
+<user01> yep  [07:07]
+<pjotrp> git log shows 
+
+Author: David Thompson <dthompson2@worcester.edu>
+Date:   Sun Mar 27 21:20:19 2016 -0400
+
+<user01> yes
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix
+         package -A genenetwork2  [07:08]
+<pjotrp> shows
+
+genenetwork2    2.0-a8fcff4     out ../guix-bioinformatics/gn/packages/genenetwork.scm:144:2
+genenetwork2-database-small     1.0     out ../guix-bioinformatics/gn/packages/genenetwork.scm:270:4
+genenetwork2-files-small        1.0     out ../guix-bioinformatics/gn/packages/genenetwork.scm:228:4
+
+<user01> yeah  [07:09]
+<pjotrp> OK, we are in sync. This means we should be able to install the exact
+         same software
+<pjotrp> I need to start up my guix daemon - I usually run it in a screen
+<pjotrp> screen -S guix-daemon
+<user01> hah, I don't have screen installed yet  [07:11]
+<pjotrp> comes with guix ;)  [07:12]
+<pjotrp> no worries, you can run it any way you want
+<pjotrp> $HOME/.guix-profile/bin/guix-daemon --build-users-group=guixbuild 
+<user01> then something's weird, because it says I don't have it
+<pjotrp> oh, you need to install it first  [07:13]
+<pjotrp> guix package -A screen
+<pjotrp> screen  4.3.1   out     gnu/packages/screen.scm:34:2
+<pjotrp> but you can skip this install, for now
+<user01> alright  [07:14]
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix
+         package -i genenetwork2 --dry-run
+<pjotrp> substitute: updating list of substitutes from
+         'https://mirror.hydra.gnu.org'...  79.1%
+<pjotrp> you see that?
+<pjotrp> followed by  [07:15]
+substitute: updating list of substitutes from
+'https://hydra.gnu.org'... 100.0%
+The following derivations would be built:
+   /gnu/store/rk7nw0rjqqsha958m649wrykadx6mmhl-profile.drv
+
+/gnu/store/7b0qjybvfx8syzvfs7p5rdablwhbkbvs-module-import-compiled.drv
+   /gnu/store/cy9zahbbf23d3cqyy404lk9f50z192kp-module-import.drv
+   /gnu/store/ibdn603i8grf0jziy5gjsly34wx82lmk-gtk-icon-themes.drv
+
+<pjotrp> which should have the same HASH values /gnu/store/7b0qjybvf... etc.
+                                                                        [07:16]
+<user01> profile has a different hash
+<pjotrp> but the next ones?
+<user01> they're the same
+<pjotrp> not sure why profile differs. Do you see the contact with
+         mirror.hydra.org?  [07:17]
+<user01> yeah
+<pjotrp> OK, that means you set the key correctly for that one :)
+<pjotrp> alright we are at the same state now. You can see most packages need
+         to be rebuild because they are no longer cached as binaries on hydra
+                                                                        [07:18]
+<pjotrp> things move fast...
+<user01> hehe
+<pjotrp> let me also do the same on my laptop - which I have staged before
+                                                                        [07:19]
+<pjotrp> btw, to set the path I often do  [07:20]
+<pjotrp> export
+         PATH="/home/wrk/.guix-profile/bin:/home/wrk/.guix-profile/sbin":$PATH
+<pjotrp> to keep things like 'screen' from Debian
+<pjotrp> Once past building guix itself that is normally OK  [07:21]
+<user01> ah, okay
+<user01> will do that
+<pjotrp> the guix build requires certain versions of tools, so you don't want
+         to mix foreign tools in  [07:23]
+<user01> makes sense  [07:24]
+<pjotrp> On my laptop I am trying the main updating list of substitutes from
+         'http://hydra.gnu.org'...  10.5%  [07:27]
+<pjotrp> it is a bit slow, but let's see if there is a difference with the
+         mirror
+<pjotrp> you can see there are two servers here. Actually with recent daemons,
+         if the mirror fails it will try the main server  [07:28]
+<pjotrp> I documented the use of a caching server here  [07:29]
+<pjotrp> https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org
+<pjotrp> this is exactly what we are doing now
+<user01> alrighty  [07:35]
+<pjotrp> To see if a remote server has a guix server running it should respond 
+                                                                        [07:36]
+<pjotrp> lynx http://guix.genenetwork.org:8080 --dump
+<pjotrp> Resource not found: /
+<pjotrp> 
+<pjotrp> you see that?
+<user01> yes  [07:37]
+<pjotrp> good. The main hydra server is too slow. So on my laptop I forced
+         using the mirror with  [07:38]
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix
+         package -i genenetwork2 --dry-run
+         --substitute-urls="http://mirror.hydra.gnu.org"
+<pjotrp> 
+<pjotrp> the list looks the same to me  [07:40]
+<user01> me too
+<pjotrp> note that some packages will be built and some downloaded, right?
+                                                                        [07:41]
+<user01> yes
+<pjotrp> atlas is actually a binary on my system  [07:43]
+<pjotrp> I mean in that list
+<pjotrp> so, it should not build. Same as yours?
+<user01> yeah, atlas and r-gtable are the ones to be downloaded
+<pjotrp> You should not have seen that error ;)
+<pjotrp> we should try and install it this way, try  [07:44]
+<pjotrp>  env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix
+         package -i genenetwork2 --cores=4 --max-jobs=4 --keep-going  [07:46]
+<pjotrp> set CPUs and max-jobs to something sensible
+<pjotrp> Does your VM have multiple cores?
+<pjotrp> note you can always press Ctrl-C during install
+<user01> it doesn't, I'll reboot it and give it another core  [07:47]
+<user02> Hey  [07:48]
+<user02> I'm here
+<user02> Will be stepping away for some breakfast
+<pjotrp> Can you do the same as us
+<pjotrp> Can you see the irc log
+<user02> Alright
+<user02> Yes, I can
+<user02> Please email me a copy in five minutes
+<pjotrp> user01: so when I use the GN server  [07:56]
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix
+         package -i genenetwork2 --dry-run
+         --substitute-urls=http://guix.genenetwork.org:8080
+<pjotrp> I don't need to build anything  [07:57]
+<pjotrp> (this won't work for you, yet)
+<pjotrp> to get it to work you need to 'trust' it   [07:58]
+<pjotrp> but, first get the build going
+<pjotrp> I'll have a coffee while you and get building
+<user01> yeah it's doing its thing now  [08:01]
+<pjotrp> cool  [08:02]
+<pjotrp> in a separate terminal you can try and install with the gn mirror
+                                                                        [08:05]
+<pjotrp> I'll  send you the public key and you can paste it as said
+         https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org
+                                                                        [08:06]
+<user01> alright
+<pjotrp> should be in the E-mail  [08:09]
+<pjotrp> getting it working it kinda nasty since the server gives no feedback
+<pjotrp> it works when you see no more in the build list ;)  [08:11]
+<pjotrp> btw, you can install software in parallel. Guix does that.
+<pjotrp> even the same packages
+<pjotrp> so keep building ;)
+<pjotrp> try and do this with Debian...
+<pjotrp> coffee for me  [08:12]
+<user01> the first build failed                                                                        [08:15]
+<pjotrp> OK, Dennis fixed that one yesterday  [08:27]
+<pjotrp> the problem is that sometime source tarballs disappear  [08:28]
+<pjotrp> R is notorious for that
+<user01> haha, that's inconvenient..
+<pjotrp> well, it is good that Guix catches them
+<pjotrp> but we do not cache sources
+<pjotrp> binaries are cached - to some degree - so we don't have to rebuild
+         those  [08:29]
+<pjotrp> time to use the guix cache at guix.genenetwork.org
+<pjotrp> try and install the key (it is in the E-mail)
+<pjotrp> and see what this lists  [08:31]
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix
+         package -i genenetwork2
+         --substitute-urls=http://guix.genenetwork.org:8080 --dry-run
+<pjotrp> should be all binary installs
+<user01> it's not..  [08:32]
+<user01> if I remove --substitute-urls, the list changes, does that mean I
+             have the key set up correctly at least?  [08:33]
+<pjotrp> dunno  [08:35]
+<pjotrp> how many packages does it want to build?
+<pjotrp> should be zero
+<user01> four
+<pjotrp> Ah, that is OK - those are default profile things
+<user01> genenetwork2 is among the ones to be downloaded so  [08:36]
+<pjotrp> remove --dry-run
+<pjotrp> yeah, good sign :)
+<pjotrp> we'll still hit a snag, but run it
+<pjotrp> should be fast
+<user01> doing it  [08:37]
+<user01> it worked!  [08:38]
+<user01> I think  [08:39]
+<pjotrp> heh  [08:40]
+<pjotrp> you mean it is finished?
+<user01> yep
+<pjotrp> type genenetwork2
+<user01> complains about not being able to connect to the database  [08:41]
+<pjotrp> last snag :)
+<pjotrp> no database
+<pjotrp> well, we succeeded in installing a same-byte install of a very
+         complex system :)  [08:42]
+<pjotrp> (always take time to congratulate yourself)
+<pjotrp> now we need to install mysql
+<user01> hehe :)
+<pjotrp> this can be done throug guix or through debian  [08:43]
+<pjotrp> the latter is a bit easier here, so let's do that
+<pjotrp> fun note: you can mix debian and guix
+<pjotrp> Follow instructions on   [08:44]
+<pjotrp>
+         https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-mysql-server
+<pjotrp> apt-get install mysql-common  [08:45]
+<pjotrp> may do it
+<pjotrp> You can also install with guix, but I need to document that
+<pjotrp> btw your internet must be fast :)  [08:46]
+<user01> hehe it is ;)
+<pjotrp> when the database is installed  [08:48]
+<pjotrp> be sure to set the password as instructed  [08:50]
+<pjotrp> when mysql is set the genenetwork2 command should fire up the web
+         server on localhost:5003  [08:58]
+<pjotrp> btw my internet is way slower :)  [09:00]
+<user02> I'm back  [09:04]
+<user02> fixed router firmware upgrade problem
+<user02> unbricking
+<pjotrp> tssk  [09:07]
+<user02> I'll never leave routers to update themselves again  [09:08]
+<user02> self-brick highway
+<user02> Resuming  [09:09]
+<pjotrp> auto-updates are evil
+<pjotrp> always switch them off
+<pjotrp> user02: can you install genenetwork like user has done?  [09:10]
+<pjotrp> pretty well documented here now :)
+<user02> Yes I can  [09:11]
+<user02> Already installed key
+<pjotrp> user02: you are getting binary packages only now?  [09:13]
+<user02> That's the sanest way to go now
+<user02> seriously
+<pjotrp> everything should be pre-built from guix.genenetwork.org
+<pjotrp> you are downloading?
+<user02> yes  [09:15]
+<pjotrp> cool. Maybe an idea to set up a server 
+<pjotrp> for your own use
+<user02> Stuck at downloading preprocesscore
+<pjotrp> should not  [09:24]
+<pjotrp> what does env GUIX_PACKAGE_PATH=../guix-bioinformatics/
+         ./pre-inst-env guix package -i genenetwork2
+         --substitute-urls="http://guix.genenetwork.org:8080" --dry-run
+                                                                        [09:25]
+<pjotrp> say for r-prepocesscore
+<pjotrp> download or build?
+<pjotrp> mine says download  [09:26]
+<user02> it only lists the derivatives to be built
+<user02> nothing else happens  [09:27]
+<pjotrp> OK, so there is a problem
+<pjotrp> your key may not be working
+<pjotrp> everything should be listed as 'to be download'  [09:28]
+<user02> Hmm
+<user02> Ah
+<user02> I know where I messed up
+<pjotrp> where?
+<user02> I did add the key
+<user02> However
+<pjotrp> (I am documenting)
+<user02> I did not tell guix to trust it
+<pjotrp> yes
+<pjotrp> and there is another potential problem
+<user02> Remember the documentation on installing guix?
+<user02> You have to tell guix to trust the default key  [09:29]
+<user02> Right?
+<user02> So in this case
+<pjotrp> read the IRC log
+<user02> That step is mandatory
+<pjotrp> user01: how are you doing?
+<pjotrp> user02:
+         https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org#using-gnu-guix-archive
+                                                                        [09:30]
+<user01> a little bit left on the db download
+<pjotrp> user02: you should see no more building
+<pjotrp> user02: another issue may be that you updated r-preprocesscore
+         package in guix-buinformatics  [09:32]
+<pjotrp> all downstream packages will want to rebuild
+<user02> no, not really
+<user02> It's not even installed
+<pjotrp> checkout a branch of the the old version - make sure we are in synch
+<pjotrp> should be at
+         /gnu/store/y1f3r2xs3fhyadd46nd2aqbr2p9qv2ra-r-biocpreprocesscore-1.32.0
+                                                                        [09:33]
+<pjotrp> 
+<user03> pjotrp: Possibly we should use the archive utility of Guix to do
+        deployment to avoid such out-of-sync differences :)  [09:34]
+<pjotrp> maybe. I did not get archive to update profiles properly  [09:37]
+<pjotrp> Also it is good that they get to understand guix
+         this way
+<pjotrp> carved in stone, eh  [09:38]
+<user02> Yeah, all good  [09:39]
+<user02> My mistake was skipping the guix archive part
+<user02> Can we begin with the install?
+<user02> It's telling me of derivatives that will be downloaded  [09:40]
+<user02> So we're good
+<user02> Here goes
+<pjotrp> yeeha  [09:42]
+<user02> pjotrp, where is this guix.genenetwork.org located at?
+<pjotrp> Tennessee
+<user02> It's...it's....sloooooooowwwwwwwwwwwwww
+<pjotrp> not from Europe
+<pjotrp> is it downloading at all?
+<user02> It should be extended
+<user02> Yes...like at 100KB/s  [09:43]
+<user02> tear-jerker
+<user02> Verizon problems
+<user02> who's the host?
+<pjotrp> I am getting 500Kb/s
+<pjotrp> UT
+<user02> Guix's servers can run off more than one server, right?
+<user02> I'd like to host that particular server here
+<user02> For speed
+<pjotrp> yes
+<user02> Sooner or later
+<user02> It will be a necessity  [09:45]
+<pjotrp> exactly what I am doing - this is our server
+<pjotrp> guix.genenetwork.org:8080
+<user02> All done installing  [09:46]
+<pjotrp> what?
+<user02> Now the databases
+<pjotrp> what do you mean by slow exactly?
+<user02> Yes, it's installed
+<pjotrp> can you run genenetwork2
+<user02> setting variables
+<user02> If I try running it now, it will fail as I don't have the DBs  [09:47]
+<pjotrp> cool - you had a lot of prebuilt packages already
+<pjotrp> OK, follow the instructions I wrote above
+<user01> now everything seems to be working for me :)
+<user02> OK
+<pjotrp> user01: excellent!
+<pjotrp> you see a webserver?
+<user01> yep, can connect to localhost:5003  [09:48]
+<pjotrp> So now you are running a guix copy of GN2
+<pjotrp> you can see where it lives with `which genenetwork2` or ls -l
+         ~/.guix-profile/bin/genenetwork2  [09:49]
+<pjotrp>
+         /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/genenetwork2
+<pjotrp> it is a script
+<pjotrp> written by guix, open it  [09:50]
+<pjotrp> inside it points to paths and our script at
+<pjotrp>
+         /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/.genenetwork2-real
+<pjotrp> if you open that you can see how the webserver is started  [09:51]
+<pjotrp> next step is to run a recent version of GN2
+<user01> okay  [09:52]
+<pjotrp> See
+         https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-your-own-copy-of-gn2
+<pjotrp> but do not checkout that genetwork2_diet
+<pjotrp> we reverted to the main tree
+<pjotrp> clone git@github.com:genenetwork/genenetwork2.git  [09:53]
+<pjotrp> instead and checkout the staging branch 
+<pjotrp> that is effectively my branch  [09:54]
+<pjotrp> when that is done you should be able to fire up the webserver from
+         there  [09:55]
+<pjotrp> using ./bin/genenetwork2
+<user02> now installing DBs
+<user02> Downloading
+<pjotrp> annoyingly the source tree is ~700Mb  [09:56]
+<user02> Can it also be done by installing the guix package
+         genenetwork2-database-small?
+<pjotrp> I changed it in the diet version to 8Mb, but I had to revert
+<user01> I need to make my VM bigger...
+<pjotrp> user02: not ready  [09:57]
+<user02> ok
+<pjotrp> user01: sorry
+<pjotrp> user01: you could mount a local dir inside the VM for development
+<pjotrp> that would allow you to use MAC tools for editing
+<pjotrp> just an idea
+<user01> yeah, I figure I'll do something like that
+<pjotrp> do you use emacs?  [09:58]
+<user01> yep
+<pjotrp> that can also run on remote files over ssh
+<pjotrp> that's an alternative
+<pjotrp> kudos for using emacs :), wdyt user03 
+<user02> 79 minutes to go downloading the db
+<pjotrp> user02: sorry about that  [09:59]
+<pjotrp> it is 2GB
+<user02> user, you can also mount the directory via sshfs
+<user02> Mac OSX runs OpenSSH
+<pjotrp> user02: sopa
+<user02> You can therefore mount a directory outside the VM to the VM via
+         sshfs  [10:00]
+<pjotrp> yes, 3 options now
+<user02> That way, you can set up a VM only for it's logic
+<user02> Apps + the OS it runs  [10:01]
+<user02> For data, let it reside on physical host accessible via sshfs
+<user02> Use this Arch wiki reference:
+         https://wiki.archlinux.org/index.php/SSHFS
+<user02> I edited that last somewhere in 2015, may have been updated since
+         then
+<user01> alright, cool!  [10:04]
+<pjotrp> user01: you are almost done  [10:06]
+<pjotrp> I wrote an elixir package for guix :)
+<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix
+         package -A elixir
+         --substitute-urls="http://guix.genenetwork.org:8080"   [10:08]
+<pjotrp> elixir  1.2.3   out
+         ../guix-bioinformatics/gn/packages/elixir.scm:31:2
+<pjotrp> 
+<pjotrp> I am building it on guix.genenetwork.org right now  [10:09]
+<user01> nice  [10:10]
 #+end_src