path: root/doc/README.org

#+TITLE: Installing GeneNetwork services

* Table of Contents                                                     :TOC:
 - [[#introduction][Introduction]]
 - [[#source-deployment][Source deployment]]
   - [[#install-guix][Install guix]]
   - [[#checkout-the-git-repositories][Checkout the git repositories]]
   - [[#update-guix][Update guix]]
   - [[#install-gn2][Install GN2]]
   - [[#run-gn2][Run GN2]]
   - [[#run-mysql-server][Run MySQL server]]
   - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]]
   - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]]
 - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]]
 - [[#trouble-shooting][Trouble shooting]]
   - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]]
   - [[#error-can-not-find-directory-homegn2_data][ERROR: can not find directory $HOME/gn2_data]]
   - [[#cant-run-a-module][Can't run a module]]

* Introduction

Large system deployments can get very complex. In this document we
explain the GeneNetwork version 2 (GN2) reproducible deployment system
which is based on GNU Guix (see also Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix
system can be used to install GN with all its files and dependencies.

The official installation path is from a checked out version of the
main Guix package tree and that of the Genenetwork package
tree. Current supported versions can be found as the SHA values of
'gn-latest' branches of [[https://github.com/genenetwork/guix-bioinformatics/tree/gn-latest][Guix bioinformatics]] and [[https://github.com/genenetwork/guix/tree/gn-latest][GNU Guix main]].

* Source deployment
** Install guix

Deploying from source is also straightforward. Install GNU Guix using
a binary tar ball as described [[https://github.com/pjotrp/guix-notes][here]].

If it works you should be able to install a package with

: guix package -i hello

** Checkout the git repositories

Check out the two relevant guix and guix-bioinformatics git
repositories:

#+begin_src bash
cd ~
mkdir genenetwork
cd genenetwork
git clone --branch gn-latest https://github.com/genenetwork/guix-bioinformatics
git clone --branch gn-latest --recursive https://github.com/genenetwork/guix guix-gn-latest
cd guix-gn-latest
#+end_src bash

** Update guix

At some point you may decide to create, install and run a recent
version of the guix-daemon by compiling the guix repository. Follow
[[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully.

** Install GN2

#+begin_src bash
env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env \
  guix package -i genenetwork2 --fallback 
#+end_src bash

Note that you can use the genenetwork.org guix substitute caching
server at http://guix.genenetwork.org (which speeds up installs
significantly because all packages are pre-built). And/or use the Guix
mirror with option --substitute-urls=http://mirror.guixsd.org

Make a note of the paths with

#+begin_src bash
./pre-inst-env guix package --search-paths
#+end_src bash

** Run GN2

After setting the paths for the server

#+begin_src bash
export PATH=~/.guix-profile/bin:$PATH
export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
export XDG_DATA_DIRS="$HOME/.guix-profile/share"
export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
#+end_src bash

run the main script (in ~/.guix-profile/bin)

#+begin_src bash
genenetwork2
#+end_src bash

will start the default server which listens on port 5003, i.e.,
http://localhost:5003/.

** Run MySQL server

At this point we require the underlying distribution to install
and run mysqld. 

Download one of

http://files.genenetwork.org/raw_database/
https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip

Check the md5sum.

After installation inflate the database binary in the MySQL directory
(this is subject to change soon) 

: chown -R mysql:mysql db_webqtl_s/
: chmod 700 db_webqtl_s/
: chmod 660 db_webqtl_s/*

restart MySQL service (mysqld). Login as root and

: mysql> show databases;
: +--------------------+
: | Database           |
: +--------------------+
: | information_schema |
: | db_webqtl_s        |
: | mysql              |
: | performance_schema |
: +--------------------+

Set permissions and match password in your settings file below:

: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password';

Note that if the mysql connection is not working, try connecting to
the IP address and check server firewall, hosts.allow and mysql IP
configuration.

** Run your own copy of GN2

At some point you may want to fix the source code. Assuming you have
Guix and Genenetwork2 installed (as described above) clone the GN2
repository from https://github.com/genenetwork/genenetwork2_diet

Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
R_LIBS_SITE are set) from the information given by guix:

: guix package --search-paths

Inside the repository:

: cd genenetwork2
: ./bin/genenetwork2 

Will fire up your local repo http://localhost:5003/ using the  
settings in ./etc/default_settings.py. These settings may 
not reflect your system. To override settings create your own from a copy of
default_settings.py and pass it into GN2 with

: ./bin/genenetwork2 $HOME/my_settings.py

and everything *should* work (note the full path to the settings
file). This way we develop against the exact same dependency graph of
software.

If something is not working, take a hint from the settings file
that comes in the Guix installation. It sits in something like

: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py

** Set up nginx port forwarding

Install the webserver (as root)

: guix package -i nginx

The configuration examples can be found in the Guix store through

: ls -l /root/.guix-profile/sbin/nginx
: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx

Use that path

: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
:   fastcgi.conf            koi-win             scgi_params
:   fastcgi.conf.default    mime.types          scgi_params.default
:   fastcgi_params          mime.types.default  uwsgi_params
:   fastcgi_params.default  nginx.conf          uwsgi_params.default
:   koi-utf                 nginx.conf.default  win-utf

And copy any relevant files to /etc/nginx.  A configuration file for
GeneNetwork (reverse proxy) port forwarding can be found in the source
repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
(still as root)

: cp ./etc/nginx-genenetwork.conf /etc/nginx/

Make dirs

: mkdir -p /var/spool/nginx/logs

Add users

: adduser nobody ; addgroup nobody 

Run nginx

: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx

* Source deployment and other information on reproducibility

See the document [[GUIX-Reproducible-from-source.org]].

* Trouble shooting

** ImportError: No module named jinja2

If you have all the Guix packages installed this error points out that
the environment variables are not set. Copy-paste the paths into your
terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
information given by guix:

: guix package --search-paths

On one system:

: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"

and perhaps a few more. 
** ERROR: can not find directory $HOME/gn2_data

The default settings file looks in your $HOME/gn2_data. Since these
files come with a Guix installation you should take a hint from the
values in the installed version of default_settings.py (see above in
this document).

** Can't run a module

In rare cases, development modules are not brought in with Guix
because no source code is available. This can lead to missing modules
on a running server. Please check with the authors when a module
is missing.