#+TITLE: Installing GeneNetwork services with GNU Guix * Table of Contents :TOC: - [[#introduction][Introduction]] - [[#binary-deployment-through-gnu-guix][Binary deployment through GNU Guix]] - [[#quick-installation-recipe][Quick installation recipe]] - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]] - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]] - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]] - [[#step-4-install-and-run-gn2][Step 4: Install and run GN2]] - [[#from-source-deployment][From source deployment]] - [[#create-archive][Create archive]] - [[#source-deployment][Source deployment]] - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]] - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]] - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]] - [[#update-to-recent-guix][Update to recent guix]] - [[#install-gn2][Install GN2]] - [[#run-gn2][Run GN2]] * Introduction Large system deployments tend to get very complex. In this document we explain the GeneNetwork deployment system which is based on GNU Guix (see Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]] and the main [[README.org]] doc). * Binary deployment through GNU Guix ** Quick installation recipe This is a recipe for quick and dirty installation of GN2. For convenience everything is installed as root, though in reality only GNU Guix has to be installed as root. I tested this recipe on a fresh install of Debian 8.3.0 (in KVM) though it should work on any modern Linux distribution (including CentOS). Note that GN2 consists of an approx. 5 GB installation including database. If you use a virtual machine we recommend to use at least double. ** Step 1: Install GNU Guix Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow [[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially, download and unpack the tar ball (which creates directories in /gnu and /var/guix), add build users and group (Guix builds software as unpriviliged users) and run the Guix daemon after fixing the paths (also known as the 'profile'). Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission to download binaries from the GNU server) and you should be able to install the hello package using binary packages (no building) #+begin_src bash export PATH=~/.guix-profile/bin:$PATH guix pull guix package -i hello --dry-run #+end_src Which should show something like : The following files would be downloaded: : /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10 : ... #+end_src means binary installs. The actual installation command of 'hello' is #+begin_src bash guix package -i hello hello Hello, world! #+end_src If you actually see things building it means that Guix is not yet properly installed and up-to-date, i.e., the key is missing or you need to do a 'guix pull'. Press Ctrl-C to interrupt. If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]]. ** Step 2: Checkout the GN2 git repositories To fixate the software dependency graph GN2 uses git repositories of Guix packages. First install git if it is missing #+begin_src bash guix package -i git export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt #+end_src check out the git repositories (gn-deploy branch) #+begin_src bash cd ~ mkdir genenetwork cd genenetwork git clone --branch gn-deploy https://github.com/genenetwork/guix-bioinformatics git clone --branch gn-deploy --recursive https://github.com/genenetwork/guix guix-gn-deploy cd guix-gn-deploy #+end_src bash To test whether this is working try: #+begin_src bash #+end_src bash ** Step 3: Authorize the GN Guix server GN2 has its own GNU Guix binary distribution server. To trust it you have to add the following key #+begin_src scheme (public-key (ecc (curve Ed25519) (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#) ) ) #+end_src by pasting it into the command #+begin_src bash guix archive --authorize #+end_src and hit Ctrl-D. Now you can use the substitute server to install GN2 binaries. ** Step 4: Install and run GN2 Since this is a quick and dirty install we are going to override the GNU Guix package path by pointing the package path to our repository: #+begin_src bash rm /root/.config/guix/latest ln -s ~/genenetwork/guix-gn-deploy/ /root/.config/guix/latest #+end_src Now check whether you can find the GN2 package with #+begin_src bash env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2 genenetwork2 2.0-a8fcff4 out gn/packages/genenetwork.scm:144:2 #+end_src (ignore the source file newer then ... messages, this is caused by the /root/.config/guix/latest override). And install with #+begin_src bash env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \ guix package -i genenetwork2 \ --substitute-urls="http://guix.genenetwork.org" #+end_src Note: the order of the substitute url's may make a difference in speed (put the one first that is fastest for your location and time of day). Note: if your system starts building or gives an error it may well be Step 3 did not succeed. The installation should actually be smooth at this point and only do binary installs (no compiling). After installation you should be able to run genenetwork2 after updating the Guix suggested environment vars. Check the output of #+begin_src bash guix package --search-paths export PYTHONPATH="/root/.guix-profile/lib/python3.8/site-packages" export R_LIBS_SITE="/root/.guix-profile/site-library/" #+end_src and copy-paste the listed exports into the terminal before running: #+begin_src bash genenetwork2 #+end_src It will complain that the database is missing. See the next section on running MySQL server for downloading and installing a MySQL GN2 database. After installing the database restart genenetwork2 and point your browser at [[http://localhost:5003/]]. End of the GN2 installation recipe! * From source deployment GNU Guix allows for [[https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org][reproducible deployment]] based on a checked out Guix repository - use gn-stable for that: #+begin_src sh :lang bash mkdir genenetwork cd genenetwork git checkout https://github.com/genenetwork/guix.git gn-stable-guix git checkout https://github.com/genenetwork/guix-bioinformatics.git #+end_src Next build guix (and run) following the instructions in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][guix-notes]]. Once that is done we can add the guix-bioinformatics path with : env GUIX_PACKAGE_PATH=../guix-bioinformatics command So #+begin_src sh :lang bash #+begin_src sh :lang bash gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -A genenetwork genenetwork1 1.0-d622c803b out ../guix-bioinformatics/gn/packages/bioinformatics.scm:163:2 genenetwork2 2.0-9e9475053 out ../guix-bioinformatics/gn/packages/bioinformatics.scm:215:2 #+end_src sh :lang bash Install with #+begin_src sh :lang bash gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -i genenetwork2 #+end_src sh :lang bash * Create archive : env GUIX_PACKAGE_PATH=../../genenetwork/guix-bioinformatics/ ./pre-inst-env guix archive --export -r genenetwork2 > guix_gn2-2.0-9e9475053.nar * Source deployment This section gives a more elaborate instruction for installing GN2 from source. First execute above 4 steps: - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]] - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]] - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]] - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]] ** Run your own copy of GN2 At some point you may want to fix the source code. Assuming you have Guix and Genenetwork2 installed (as described above) clone the GN2 repository from https://github.com/genenetwork/genenetwork2. Copy-paste the paths into your terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the information given by guix: : guix package --search-paths Inside the repository: : cd genenetwork2 : ./bin/genenetwork2 Will fire up your local repo http://localhost:5003/ using the settings in ./etc/default_settings.py. These settings may not reflect your system. To override settings create your own from a copy of default_settings.py and pass it into GN2 with : ./bin/genenetwork2 $HOME/my_settings.py and everything *should* work (note the full path to the settings file). This way we develop against the exact same dependency graph of software. If something is not working, take a hint from the settings file that comes in the Guix installation. It sits in something like : cat ~/.guix-profile/lib/python3.8/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py ** Set up nginx port forwarding nginx can be used as a reverse proxy for GN2. For example, we want to expose GN2 on port 80 while it is running on port 5003. Essentially the configuration looks like #+begin_src js server { listen 80; server_name test-gn2.genenetwork.org; access_log logs/test-gn2.access.log; proxy_connect_timeout 3000; proxy_send_timeout 3000; proxy_read_timeout 3000; send_timeout 3000; location / { proxy_set_header Host $http_host; proxy_set_header Connection keep-alive; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $server_name; proxy_pass http://127.0.0.1:5003; } } #+end_src js Install the nginx webserver (as root) : guix package -i nginx The nginx example configuration examples can be found in the Guix store through : ls -l /root/.guix-profile/sbin/nginx : lrwxrwxrwx 3 root guixbuild 66 Dec 31 1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx Use that path : ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/ : fastcgi.conf koi-win scgi_params : fastcgi.conf.default mime.types scgi_params.default : fastcgi_params mime.types.default uwsgi_params : fastcgi_params.default nginx.conf uwsgi_params.default : koi-utf nginx.conf.default win-utf And copy any relevant files to /etc/nginx. A configuration file for GeneNetwork (reverse proxy) port forwarding can be found in the source repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc (still as root) : cp ./etc/nginx-genenetwork.conf /etc/nginx/ Make dirs : mkdir -p /var/spool/nginx/logs Add users : adduser nobody ; addgroup nobody Run nginx : /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx * Source deployment and other information on reproducibility See the document [[GUIX-Reproducible-from-source.org]]. ** Update to recent guix We now compile Guix from scratch. Create, install and run a recent version of the guix-daemon by compiling the guix repository you have installed with git in step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after : cd ~/genenetwork/guix-gn-deploy Make sure to restart the guix daemon and run guix client from this directory. ** Install GN2 Reinstall genenetwork2 using the new tree #+begin_src bash env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org https://mirror.guixsd.org" #+end_src bash Note the use of ./pre-inst-env here! Actually, it should be the same installation as in step 4, so nothing gets downloaded. ** Run GN2 Make a note of the paths with #+begin_src bash ./pre-inst-env guix package --search-paths #+end_src bash or this should also work if guix is installed #+begin_src bash guix package --search-paths #+end_src bash After setting the paths for the server #+begin_src bash export PATH=~/.guix-profile/bin:$PATH export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages" export R_LIBS_SITE="$HOME/.guix-profile/site-library/" export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0" export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0" export XDG_DATA_DIRS="$HOME/.guix-profile/share" export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules" #+end_src bash run the main script (in ~/.guix-profile/bin) #+begin_src bash genenetwork2 #+end_src bash will start the default server which listens on port 5003, i.e., http://localhost:5003/. OK, we are where we were before with step 4. Only difference is that we used our own compiled guix server.