path: root/doc/GUIX-Reproducible-from-source.org

#+TITLE: Installing GeneNetwork services with GNU Guix

* Table of Contents                                                     :TOC:
 - [[#introduction][Introduction]]
 - [[#binary-deployment-through-gnu-guix][Binary deployment through GNU Guix]]
   - [[#quick-installation-recipe][Quick installation recipe]]
   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
   - [[#step-4-install-and-run-gn2][Step 4: Install and run GN2]]
 - [[#from-source-deployment][From source deployment]]
 - [[#create-archive][Create archive]]
 - [[#source-deployment][Source deployment]]
   - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]]
   - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]]
 - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]]
   - [[#update-to-recent-guix][Update to recent guix]]
   - [[#install-gn2][Install GN2]]
   - [[#run-gn2][Run GN2]]

* Introduction

Large system deployments tend to get very complex. In this document we
explain the GeneNetwork deployment system which is based on GNU Guix
(see Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]] and the main [[README.org]] doc).

* Binary deployment through GNU Guix
** Quick installation recipe

This is a recipe for quick and dirty installation of GN2. For
convenience everything is installed as root, though in reality only
GNU Guix has to be installed as root. I tested this recipe on a fresh
install of Debian 8.3.0 (in KVM) though it should work on any modern
Linux distribution (including CentOS).

Note that GN2 consists of an approx. 5 GB installation including
database. If you use a virtual machine we recommend to use at least
double.

** Step 1: Install GNU Guix

Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow
[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially, download and unpack the tar ball (which
creates directories in /gnu and /var/guix), add build users and group
(Guix builds software as unpriviliged users) and run the Guix daemon
after fixing the paths (also known as the 'profile').

Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission
to download binaries from the GNU server) and you should be able to
install the hello package using binary packages (no building)

#+begin_src bash
export PATH=~/.guix-profile/bin:$PATH
guix pull
guix package -i hello --dry-run
#+end_src

Which should show something like

: The following files would be downloaded:
:   /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
:   ...
#+end_src

means binary installs.  The actual installation command of 'hello' is

#+begin_src bash
guix package -i hello
hello
  Hello, world!
#+end_src

If you actually see things building it means that Guix is not yet
properly installed and up-to-date, i.e., the key is missing or you
need to do a 'guix pull'. Press Ctrl-C to interrupt.

If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get
rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]].

** Step 2: Checkout the GN2 git repositories

To fixate the software dependency graph GN2 uses git repositories of
Guix packages. First install git if it is missing

#+begin_src bash
guix package -i git
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
#+end_src

check out the git repositories (gn-deploy branch)

#+begin_src bash
cd ~
mkdir genenetwork
cd genenetwork
git clone --branch gn-deploy https://github.com/genenetwork/guix-bioinformatics
git clone --branch gn-deploy --recursive https://github.com/genenetwork/guix guix-gn-deploy
cd guix-gn-deploy
#+end_src bash

To test whether this is working try:

#+begin_src bash
#+end_src bash

** Step 3: Authorize the GN Guix server

GN2 has its own GNU Guix binary distribution server. To trust it you have
to add the following key

#+begin_src scheme
(public-key
 (ecc
  (curve Ed25519)
  (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#)
 )
)
#+end_src

by pasting it into the command

#+begin_src bash
guix archive --authorize
#+end_src

and hit Ctrl-D.

Now you can use the substitute server to install GN2 binaries.

** Step 4: Install and run GN2

Since this is a quick and dirty install we are going to override the
GNU Guix package path by pointing the package path to our repository:

#+begin_src bash
rm /root/.config/guix/latest
ln -s ~/genenetwork/guix-gn-deploy/ /root/.config/guix/latest
#+end_src

Now check whether you can find the GN2 package with

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2
  genenetwork2    2.0-a8fcff4     out     gn/packages/genenetwork.scm:144:2
#+end_src

(ignore the source file newer then ... messages, this is caused by the
/root/.config/guix/latest override).

And install with

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \
  guix package -i genenetwork2 \
  --substitute-urls="http://guix.genenetwork.org"
#+end_src

Note: the order of the substitute url's may make a difference in speed
(put the one first that is fastest for your location and time of day).

Note: if your system starts building or gives an error it may well be
Step 3 did not succeed. The installation should actually be smooth at
this point and only do binary installs (no compiling).

After installation you should be able to run genenetwork2 after updating
the Guix suggested environment vars. Check the output of

#+begin_src bash
guix package --search-paths
export PYTHONPATH="/root/.guix-profile/lib/python3.8/site-packages"
export R_LIBS_SITE="/root/.guix-profile/site-library/"
#+end_src

and copy-paste the listed exports into the terminal before running:

#+begin_src bash
genenetwork2
#+end_src

It will complain that the database is missing. See the next section on
running MySQL server for downloading and installing a MySQL GN2
database. After installing the database restart genenetwork2 and point
your browser at [[http://localhost:5003/]].

End of the GN2 installation recipe!

* From source deployment

GNU Guix allows for [[https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org][reproducible deployment]] based on a checked out
Guix repository - use gn-stable for that:

#+begin_src sh   :lang bash
mkdir genenetwork
cd genenetwork
git checkout https://github.com/genenetwork/guix.git gn-stable-guix
git checkout https://github.com/genenetwork/guix-bioinformatics.git
#+end_src

Next build guix (and run) following the instructions in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][guix-notes]].

Once that is done we can add the guix-bioinformatics path with

: env GUIX_PACKAGE_PATH=../guix-bioinformatics command

So

#+begin_src sh   :lang bash
#+begin_src sh   :lang bash
gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -A genenetwork
genenetwork1    1.0-d622c803b   out     ../guix-bioinformatics/gn/packages/bioinformatics.scm:163:2
genenetwork2    2.0-9e9475053   out     ../guix-bioinformatics/gn/packages/bioinformatics.scm:215:2
#+end_src sh   :lang bash

Install with

#+begin_src sh   :lang bash
gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -i genenetwork2
#+end_src sh   :lang bash

* Create archive

: env GUIX_PACKAGE_PATH=../../genenetwork/guix-bioinformatics/ ./pre-inst-env guix archive --export -r genenetwork2 > guix_gn2-2.0-9e9475053.nar


* Source deployment

This section gives a more elaborate instruction for installing GN2
from source.

First execute above 4 steps:

   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]


** Run your own copy of GN2

At some point you may want to fix the source code. Assuming you have
Guix and Genenetwork2 installed (as described above) clone the GN2
repository from https://github.com/genenetwork/genenetwork2.

Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
R_LIBS_SITE are set) from the information given by guix:

: guix package --search-paths

Inside the repository:

: cd genenetwork2
: ./bin/genenetwork2

Will fire up your local repo http://localhost:5003/ using the
settings in ./etc/default_settings.py. These settings may
not reflect your system. To override settings create your own from a copy of
default_settings.py and pass it into GN2 with

: ./bin/genenetwork2 $HOME/my_settings.py

and everything *should* work (note the full path to the settings
file). This way we develop against the exact same dependency graph of
software.

If something is not working, take a hint from the settings file
that comes in the Guix installation. It sits in something like

: cat ~/.guix-profile/lib/python3.8/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py

** Set up nginx port forwarding

nginx can be used as a reverse proxy for GN2. For example, we want to
expose GN2 on port 80 while it is running on port 5003. Essentially
the configuration looks like

#+begin_src js
    server {
        listen 80;
        server_name test-gn2.genenetwork.org;
        access_log  logs/test-gn2.access.log;

        proxy_connect_timeout       3000;
        proxy_send_timeout          3000;
        proxy_read_timeout          3000;
        send_timeout                3000;

        location / {
            proxy_set_header   Host      $http_host;
            proxy_set_header   Connection keep-alive;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
            proxy_pass         http://127.0.0.1:5003;
        }
}
#+end_src js

Install the nginx webserver (as root)

: guix package -i nginx

The nginx example configuration examples can be found in the Guix
store through

: ls -l /root/.guix-profile/sbin/nginx
: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx

Use that path

: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
:   fastcgi.conf            koi-win             scgi_params
:   fastcgi.conf.default    mime.types          scgi_params.default
:   fastcgi_params          mime.types.default  uwsgi_params
:   fastcgi_params.default  nginx.conf          uwsgi_params.default
:   koi-utf                 nginx.conf.default  win-utf

And copy any relevant files to /etc/nginx.  A configuration file for
GeneNetwork (reverse proxy) port forwarding can be found in the source
repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
(still as root)
: cp ./etc/nginx-genenetwork.conf /etc/nginx/

Make dirs

: mkdir -p /var/spool/nginx/logs

Add users

: adduser nobody ; addgroup nobody

Run nginx

: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx

* Source deployment and other information on reproducibility

See the document [[GUIX-Reproducible-from-source.org]].

** Update to recent guix

We now compile Guix from scratch.

Create, install and run a recent version of the guix-daemon by
compiling the guix repository you have installed with git in
step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after

: cd ~/genenetwork/guix-gn-deploy

Make sure to restart the guix daemon and run guix client from this
directory.

** Install GN2

Reinstall genenetwork2 using the new tree

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org https://mirror.guixsd.org"
#+end_src bash

Note the use of ./pre-inst-env here!

Actually, it should be the same installation as in step 4, so nothing
gets downloaded.

** Run GN2

Make a note of the paths with

#+begin_src bash
./pre-inst-env guix package --search-paths
#+end_src bash

or this should also work if guix is installed

#+begin_src bash
guix package --search-paths
#+end_src bash

After setting the paths for the server

#+begin_src bash
export PATH=~/.guix-profile/bin:$PATH
export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages"
export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
export XDG_DATA_DIRS="$HOME/.guix-profile/share"
export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
#+end_src bash

run the main script (in ~/.guix-profile/bin)

#+begin_src bash
genenetwork2
#+end_src bash

will start the default server which listens on port 5003, i.e.,
http://localhost:5003/.

OK, we are where we were before with step 4. Only difference is that we
used our own compiled guix server.