diff options
Diffstat (limited to 'doc/README.org')
| -rw-r--r-- | doc/README.org | 857 |
1 files changed, 843 insertions, 14 deletions
diff --git a/doc/README.org b/doc/README.org index f6ab6a52..3754dbbd 100644 --- a/doc/README.org +++ b/doc/README.org @@ -1,28 +1,857 @@ -#+TITLE: Installing GeneNetwork services with GNU Guix + +#+TITLE: Installing GeneNetwork services * Table of Contents :TOC: - [[#introduction][Introduction]] - - [[#binary-deployment][Binary deployment]] - - [[#from-source-deployment][From source deployment]] + - [[#quick-installation-recipe][Quick installation recipe]] + - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]] + - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]] + - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]] + - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]] + - [[#run-mysql-server][Run MySQL server]] + - [[#gn2-dependency-graph][GN2 Dependency Graph]] + - [[#source-deployment][Source deployment]] + - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]] + - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]] + - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]] + - [[#update-to-recent-guix][Update to recent guix]] + - [[#install-gn2][Install GN2]] + - [[#run-gn2][Run GN2]] + - [[#trouble-shooting][Trouble shooting]] + - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]] + - [[#error-can-not-find-directory-homegn2_data][ERROR: can not find directory $HOME/gn2_data]] + - [[#cant-run-a-module][Can't run a module]] + - [[#irc-session][IRC session]] * Introduction -Large system deployments tend to get very complex. In this document we -explain the GeneNetwork deployment system which is based on GNU Guix -(see Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). +If you want to understand the architecture of GN2 read +[[Architecture.org]]. The rest of this document is mostly on deployment +of GN2. + +Large system deployments can get very [[http://biobeat.org/gn2.svg][complex]]. In this document we +explain the GeneNetwork version 2 (GN2) reproducible deployment system +which is based on GNU Guix (see also Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix +system can be used to install GN with all its files and dependencies. + +The official installation path is from a checked out version of the +main Guix package tree and that of the Genenetwork package +tree. Current supported versions can be found as the SHA values of +'gn-latest' branches of [[https://github.com/genenetwork/guix-bioinformatics/tree/gn-latest][Guix bioinformatics]] and [[https://github.com/genenetwork/guix/tree/gn-latest][GNU Guix main]]. + +For a full view of runtime dependencies as defined by GNU Guix, see +the [[#gn2-dependency-graph][GN2 Dependency Graph]]. + +* Quick installation recipe + +This is a recipe for quick and dirty installation of GN2. For +convenience everything is installed as root, though in reality only +GNU Guix has to be installed as root. I tested this recipe on a fresh +install of Debian 8.3.0 (in KVM) though it should work on any modern +Linux distribution (including CentOS). For more elaborate installation +instructions see [[#source-deployment][Source deployment]]. + +Note that GN2 consists of an approx. 5 GB installation including +database. + +** Step 1: Install GNU Guix + +Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow +[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially you have to download and unpack the tar ball +(which creates directories in /gnu and /var/guix), add build users and +group (Guix builds software as unpriviliged users) and run the Guix +daemon after fixing the paths (also known as the 'profile'). + +Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission +to download binaries from the GNU server) and you should be able to +install the hello package using binary packages (no building) + +#+begin_src bash +export PATH=~/.guix-profile/bin:$PATH +guix pull +guix package -i hello --dry-run +#+end_src + +Which should show something like + +: The following files would be downloaded: +: /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10 +: ... +#+end_src + +means binary installs. The actual installation command of 'hello' is + +#+begin_src bash +guix package -i hello +hello + Hello, world! +#+end_src -* Binary deployment +If you actually see things building it means that Guix is not yet +properly installed and up-to-date, i.e., the key is missing or you +need to do a 'guix pull'. Press Ctrl-C to interrupt. -NYA +If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get +rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]]. -* From source deployment +** Step 2: Checkout the GN2 git repositories -GNU Guix allows for [[https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org][reproducible deployment]] based on a checked out -Guix repository - use gn-stable for that: +To fixate the software dependency graph GN2 uses git repositories of +Guix packages. First install git if it is missing -#+begin_src sh :lang bash +#+begin_src bash +guix package -i git +export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt +#+end_src + +check out the git repsitories (gn-latest branch) + +#+begin_src bash +cd ~ mkdir genenetwork cd genenetwork -git checkout https://github.com/genenetwork/guix.git gn-stable-guix -git checkout https://github.com/genenetwork/guix-bioinformatics.git +git clone --branch gn-latest https://github.com/genenetwork/guix-bioinformatics +git clone --branch gn-latest --recursive https://github.com/genenetwork/guix guix-gn-latest +cd guix-gn-latest +#+end_src bash + +** Step 3: Authorize the GN Guix server + +GN2 has its own GNU Guix binary distribution server. To trust it you have +to add the following key + +#+begin_src scheme +(public-key + (ecc + (curve Ed25519) + (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#) + ) +) +#+end_src + +by pasting it into the command + +#+begin_src bash +guix archive --authorize +#+end_src + +and hit Ctrl-D. + +Now you can use the substitute server to install GN2 binaries. + +** Step 4: Install and run GN2 + +Since this is a quick and dirty install we are going to override the +GNU Guix package path by pointing the package path to our repository: + +#+begin_src bash +rm /root/.config/guix/latest +ln -s ~/genenetwork/guix-gn-latest/ /root/.config/guix/latest +#+end_src + +Now check whether you can find the GN2 package with + +#+begin_src bash +env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2 + genenetwork2 2.0-a8fcff4 out gn/packages/genenetwork.scm:144:2 +#+end_src + +(ignore the source file newer then ... messages, this is caused by the +/root/.config/guix/latest override). + +And install with + +#+begin_src bash +env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \ + guix package -i genenetwork2 \ + --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" \ + --fallback +#+end_src + +Note: the order of the substitute url's may make a difference in speed +(put the one first that is fastest for your location and time of day). + +After installation you should be able to run genenetwork2 after updating +the Guix suggested environment vars. Check the output of + +#+begin_src bash +guix package --search-paths +export PYTHONPATH="/root/.guix-profile/lib/python2.7/site-packages" +export R_LIBS_SITE="/root/.guix-profile/site-library/" +#+end_src + +and copy-paste the listed exports into the terminal before running: + +#+begin_src bash +genenetwork2 +#+end_src + +It will complain that the database is missing. See the next section on +running MySQL server for downloading and installing a MySQL GN2 +database. After installing the database restart genenetwork2 and point +your browser at [[http://localhost:5003/]]. + +End of the GN2 installation recipe! + +* Run MySQL server + +At this point we require the underlying distribution to install and +run mysqld. Currently we have two databases for deployment, +'db_webqtl_s' is the small testing database containing experiments +from BXD mice and 'db_webqtl_plant' which contains all plant related +material. + +Download one database from + +http://files.genenetwork.org/raw_database/ +https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip + +Check the md5sum. + +After installation inflate the database binary in the MySQL directory +(this installation path is subject to change soon) + +: chown -R mysql:mysql db_webqtl_s/ +: chmod 700 db_webqtl_s/ +: chmod 660 db_webqtl_s/* + +restart MySQL service (mysqld). Login as root and + +: mysql> show databases; +: +--------------------+ +: | Database | +: +--------------------+ +: | information_schema | +: | db_webqtl_s | +: | mysql | +: | performance_schema | +: +--------------------+ + +Set permissions and match password in your settings file below: + +: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password'; + +Note that if the mysql connection is not working, try connecting to +the IP address and check server firewall, hosts.allow and mysql IP +configuration. + +Note for the plant database you can rename it to db_webqtl_s, or +change the settings in etc/default_settings.py to match your path. + +* GN2 Dependency Graph + +Graph of all runtime dependencies as installed by GNU Guix. + +#+ATTR_HTML: :title GN2_graph +[[http://biobeat.org/gn2.svg]] + +* Source deployment + +This section gives a more elaborate instruction for installing GN2 +from source. + +First execute above 4 steps: + + - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]] + - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]] + - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]] + - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]] + + +** Run your own copy of GN2 + +At some point you may want to fix the source code. Assuming you have +Guix and Genenetwork2 installed (as described above) clone the GN2 +repository from https://github.com/genenetwork/genenetwork2. + +Copy-paste the paths into your terminal (mainly so PYTHON_PATH and +R_LIBS_SITE are set) from the information given by guix: + +: guix package --search-paths + +Inside the repository: + +: cd genenetwork2 +: ./bin/genenetwork2 + +Will fire up your local repo http://localhost:5003/ using the +settings in ./etc/default_settings.py. These settings may +not reflect your system. To override settings create your own from a copy of +default_settings.py and pass it into GN2 with + +: ./bin/genenetwork2 $HOME/my_settings.py + +and everything *should* work (note the full path to the settings +file). This way we develop against the exact same dependency graph of +software. + +If something is not working, take a hint from the settings file +that comes in the Guix installation. It sits in something like + +: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py + +** Set up nginx port forwarding + +nginx can be used as a reverse proxy for GN2. For example, we want to +expose GN2 on port 80 while it is running on port 5003. Essentially +the configuration looks like + +#+begin_src js + server { + listen 80; + server_name test-gn2.genenetwork.org; + access_log logs/test-gn2.access.log; + + proxy_connect_timeout 3000; + proxy_send_timeout 3000; + proxy_read_timeout 3000; + send_timeout 3000; + + location / { + proxy_set_header Host $http_host; + proxy_set_header Connection keep-alive; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Host $server_name; + proxy_pass http://127.0.0.1:5003; + } +} +#+end_src js + +Install the nginx webserver (as root) + +: guix package -i nginx + +The nginx example configuration examples can be found in the Guix +store through + +: ls -l /root/.guix-profile/sbin/nginx +: lrwxrwxrwx 3 root guixbuild 66 Dec 31 1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx + +Use that path + +: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/ +: fastcgi.conf koi-win scgi_params +: fastcgi.conf.default mime.types scgi_params.default +: fastcgi_params mime.types.default uwsgi_params +: fastcgi_params.default nginx.conf uwsgi_params.default +: koi-utf nginx.conf.default win-utf + +And copy any relevant files to /etc/nginx. A configuration file for +GeneNetwork (reverse proxy) port forwarding can be found in the source +repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc +(still as root) +: cp ./etc/nginx-genenetwork.conf /etc/nginx/ + +Make dirs + +: mkdir -p /var/spool/nginx/logs + +Add users + +: adduser nobody ; addgroup nobody + +Run nginx + +: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx + +* Source deployment and other information on reproducibility + +See the document [[GUIX-Reproducible-from-source.org]]. + +** Update to recent guix + +We now compile Guix from scratch. + +Create, install and run a recent version of the guix-daemon by +compiling the guix repository you have installed with git in +step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after + +: cd ~/genenetwork/guix-gn-latest + +Make sure to restart the guix daemon and run guix client from this +directory. + +** Install GN2 + +Reinstall genenetwork2 using the new tree + +#+begin_src bash +env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" +#+end_src bash + +Note the use of ./pre-inst-env here! + +Actually, it should be the same installation as in step 4, so nothing +gets downloaded. + +** Run GN2 + +Make a note of the paths with + +#+begin_src bash +./pre-inst-env guix package --search-paths +#+end_src bash + +After setting the paths for the server + +#+begin_src bash +export PATH=~/.guix-profile/bin:$PATH +export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages" +export R_LIBS_SITE="$HOME/.guix-profile/site-library/" +export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0" +export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0" +export XDG_DATA_DIRS="$HOME/.guix-profile/share" +export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules" +#+end_src bash + +run the main script (in ~/.guix-profile/bin) + +#+begin_src bash +genenetwork2 +#+end_src bash + +will start the default server which listens on port 5003, i.e., +http://localhost:5003/. + +OK, we are where we were before with step 4. Only difference is that we +used our own compiled guix server. + +* Trouble shooting + +** ImportError: No module named jinja2 + +If you have all the Guix packages installed this error points out that +the environment variables are not set. Copy-paste the paths into your +terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the +information given by guix: + +: guix package --search-paths + +On one system: + +: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages" +: export R_LIBS_SITE="$HOME/.guix-profile/site-library/" +: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0" + +and perhaps a few more. +** ERROR: can not find directory $HOME/gn2_data + +The default settings file looks in your $HOME/gn2_data. Since these +files come with a Guix installation you should take a hint from the +values in the installed version of default_settings.py (see above in +this document). + +** Can't run a module + +In rare cases, development modules are not brought in with Guix +because no source code is available. This can lead to missing modules +on a running server. Please check with the authors when a module +is missing. +* IRC session + +Here an IRC session where we installed GN2 from scratch using GNU Guix +and a download of the test database. + +#+begin_src +<pjotrp> time to get binary install sorted :) [07:03] +<pjotrp> Guix is designed for distributed installation servers +<pjotrp> we have one on guix.genenetwork.org +<pjotrp> it contains all the prebuild packages +<pjotrp> for GN +<user01> okay [07:04] +<pjotrp> let's step back however [07:05] +<pjotrp> I presume the environment is set with all guix package --search-paths +<pjotrp> right? +<user01> yep +<user01> set to the ones in ~/.guix-profile/ +<pjotrp> good, and you are in gn-latest-guix repo [07:06] +<user01> yep [07:07] +<pjotrp> git log shows + +Author: David Thompson <dthompson2@worcester.edu> +Date: Sun Mar 27 21:20:19 2016 -0400 + +<user01> yes +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix + package -A genenetwork2 [07:08] +<pjotrp> shows + +genenetwork2 2.0-a8fcff4 out ../guix-bioinformatics/gn/packages/genenetwork.scm:144:2 +genenetwork2-database-small 1.0 out ../guix-bioinformatics/gn/packages/genenetwork.scm:270:4 +genenetwork2-files-small 1.0 out ../guix-bioinformatics/gn/packages/genenetwork.scm:228:4 + +<user01> yeah [07:09] +<pjotrp> OK, we are in sync. This means we should be able to install the exact + same software +<pjotrp> I need to start up my guix daemon - I usually run it in a screen +<pjotrp> screen -S guix-daemon +<user01> hah, I don't have screen installed yet [07:11] +<pjotrp> comes with guix ;) [07:12] +<pjotrp> no worries, you can run it any way you want +<pjotrp> $HOME/.guix-profile/bin/guix-daemon --build-users-group=guixbuild +<user01> then something's weird, because it says I don't have it +<pjotrp> oh, you need to install it first [07:13] +<pjotrp> guix package -A screen +<pjotrp> screen 4.3.1 out gnu/packages/screen.scm:34:2 +<pjotrp> but you can skip this install, for now +<user01> alright [07:14] +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix + package -i genenetwork2 --dry-run +<pjotrp> substitute: updating list of substitutes from + 'https://mirror.hydra.gnu.org'... 79.1% +<pjotrp> you see that? +<pjotrp> followed by [07:15] +substitute: updating list of substitutes from +'https://hydra.gnu.org'... 100.0% +The following derivations would be built: + /gnu/store/rk7nw0rjqqsha958m649wrykadx6mmhl-profile.drv + +/gnu/store/7b0qjybvfx8syzvfs7p5rdablwhbkbvs-module-import-compiled.drv + /gnu/store/cy9zahbbf23d3cqyy404lk9f50z192kp-module-import.drv + /gnu/store/ibdn603i8grf0jziy5gjsly34wx82lmk-gtk-icon-themes.drv + +<pjotrp> which should have the same HASH values /gnu/store/7b0qjybvf... etc. + [07:16] +<user01> profile has a different hash +<pjotrp> but the next ones? +<user01> they're the same +<pjotrp> not sure why profile differs. Do you see the contact with + mirror.hydra.org? [07:17] +<user01> yeah +<pjotrp> OK, that means you set the key correctly for that one :) +<pjotrp> alright we are at the same state now. You can see most packages need + to be rebuild because they are no longer cached as binaries on hydra + [07:18] +<pjotrp> things move fast... +<user01> hehe +<pjotrp> let me also do the same on my laptop - which I have staged before + [07:19] +<pjotrp> btw, to set the path I often do [07:20] +<pjotrp> export + PATH="/home/wrk/.guix-profile/bin:/home/wrk/.guix-profile/sbin":$PATH +<pjotrp> to keep things like 'screen' from Debian +<pjotrp> Once past building guix itself that is normally OK [07:21] +<user01> ah, okay +<user01> will do that +<pjotrp> the guix build requires certain versions of tools, so you don't want + to mix foreign tools in [07:23] +<user01> makes sense [07:24] +<pjotrp> On my laptop I am trying the main updating list of substitutes from + 'http://hydra.gnu.org'... 10.5% [07:27] +<pjotrp> it is a bit slow, but let's see if there is a difference with the + mirror +<pjotrp> you can see there are two servers here. Actually with recent daemons, + if the mirror fails it will try the main server [07:28] +<pjotrp> I documented the use of a caching server here [07:29] +<pjotrp> https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org +<pjotrp> this is exactly what we are doing now +<user01> alrighty [07:35] +<pjotrp> To see if a remote server has a guix server running it should respond + [07:36] +<pjotrp> lynx http://guix.genenetwork.org:8080 --dump +<pjotrp> Resource not found: / +<pjotrp> +<pjotrp> you see that? +<user01> yes [07:37] +<pjotrp> good. The main hydra server is too slow. So on my laptop I forced + using the mirror with [07:38] +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix + package -i genenetwork2 --dry-run + --substitute-urls="http://mirror.hydra.gnu.org" +<pjotrp> +<pjotrp> the list looks the same to me [07:40] +<user01> me too +<pjotrp> note that some packages will be built and some downloaded, right? + [07:41] +<user01> yes +<pjotrp> atlas is actually a binary on my system [07:43] +<pjotrp> I mean in that list +<pjotrp> so, it should not build. Same as yours? +<user01> yeah, atlas and r-gtable are the ones to be downloaded +<pjotrp> You should not have seen that error ;) +<pjotrp> we should try and install it this way, try [07:44] +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix + package -i genenetwork2 --cores=4 --max-jobs=4 --keep-going [07:46] +<pjotrp> set CPUs and max-jobs to something sensible +<pjotrp> Does your VM have multiple cores? +<pjotrp> note you can always press Ctrl-C during install +<user01> it doesn't, I'll reboot it and give it another core [07:47] +<user02> Hey [07:48] +<user02> I'm here +<user02> Will be stepping away for some breakfast +<pjotrp> Can you do the same as us +<pjotrp> Can you see the irc log +<user02> Alright +<user02> Yes, I can +<user02> Please email me a copy in five minutes +<pjotrp> user01: so when I use the GN server [07:56] +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix + package -i genenetwork2 --dry-run + --substitute-urls=http://guix.genenetwork.org:8080 +<pjotrp> I don't need to build anything [07:57] +<pjotrp> (this won't work for you, yet) +<pjotrp> to get it to work you need to 'trust' it [07:58] +<pjotrp> but, first get the build going +<pjotrp> I'll have a coffee while you and get building +<user01> yeah it's doing its thing now [08:01] +<pjotrp> cool [08:02] +<pjotrp> in a separate terminal you can try and install with the gn mirror + [08:05] +<pjotrp> I'll send you the public key and you can paste it as said + https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org + [08:06] +<user01> alright +<pjotrp> should be in the E-mail [08:09] +<pjotrp> getting it working it kinda nasty since the server gives no feedback +<pjotrp> it works when you see no more in the build list ;) [08:11] +<pjotrp> btw, you can install software in parallel. Guix does that. +<pjotrp> even the same packages +<pjotrp> so keep building ;) +<pjotrp> try and do this with Debian... +<pjotrp> coffee for me [08:12] +<user01> the first build failed [08:15] +<pjotrp> OK, Dennis fixed that one yesterday [08:27] +<pjotrp> the problem is that sometime source tarballs disappear [08:28] +<pjotrp> R is notorious for that +<user01> haha, that's inconvenient.. +<pjotrp> well, it is good that Guix catches them +<pjotrp> but we do not cache sources +<pjotrp> binaries are cached - to some degree - so we don't have to rebuild + those [08:29] +<pjotrp> time to use the guix cache at guix.genenetwork.org +<pjotrp> try and install the key (it is in the E-mail) +<pjotrp> and see what this lists [08:31] +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix + package -i genenetwork2 + --substitute-urls=http://guix.genenetwork.org:8080 --dry-run +<pjotrp> should be all binary installs +<user01> it's not.. [08:32] +<user01> if I remove --substitute-urls, the list changes, does that mean I + have the key set up correctly at least? [08:33] +<pjotrp> dunno [08:35] +<pjotrp> how many packages does it want to build? +<pjotrp> should be zero +<user01> four +<pjotrp> Ah, that is OK - those are default profile things +<user01> genenetwork2 is among the ones to be downloaded so [08:36] +<pjotrp> remove --dry-run +<pjotrp> yeah, good sign :) +<pjotrp> we'll still hit a snag, but run it +<pjotrp> should be fast +<user01> doing it [08:37] +<user01> it worked! [08:38] +<user01> I think [08:39] +<pjotrp> heh [08:40] +<pjotrp> you mean it is finished? +<user01> yep +<pjotrp> type genenetwork2 +<user01> complains about not being able to connect to the database [08:41] +<pjotrp> last snag :) +<pjotrp> no database +<pjotrp> well, we succeeded in installing a same-byte install of a very + complex system :) [08:42] +<pjotrp> (always take time to congratulate yourself) +<pjotrp> now we need to install mysql +<user01> hehe :) +<pjotrp> this can be done throug guix or through debian [08:43] +<pjotrp> the latter is a bit easier here, so let's do that +<pjotrp> fun note: you can mix debian and guix +<pjotrp> Follow instructions on [08:44] +<pjotrp> + https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-mysql-server +<pjotrp> apt-get install mysql-common [08:45] +<pjotrp> may do it +<pjotrp> You can also install with guix, but I need to document that +<pjotrp> btw your internet must be fast :) [08:46] +<user01> hehe it is ;) +<pjotrp> when the database is installed [08:48] +<pjotrp> be sure to set the password as instructed [08:50] +<pjotrp> when mysql is set the genenetwork2 command should fire up the web + server on localhost:5003 [08:58] +<pjotrp> btw my internet is way slower :) [09:00] +<user02> I'm back [09:04] +<user02> fixed router firmware upgrade problem +<user02> unbricking +<pjotrp> tssk [09:07] +<user02> I'll never leave routers to update themselves again [09:08] +<user02> self-brick highway +<user02> Resuming [09:09] +<pjotrp> auto-updates are evil +<pjotrp> always switch them off +<pjotrp> user02: can you install genenetwork like user has done? [09:10] +<pjotrp> pretty well documented here now :) +<user02> Yes I can [09:11] +<user02> Already installed key +<pjotrp> user02: you are getting binary packages only now? [09:13] +<user02> That's the sanest way to go now +<user02> seriously +<pjotrp> everything should be pre-built from guix.genenetwork.org +<pjotrp> you are downloading? +<user02> yes [09:15] +<pjotrp> cool. Maybe an idea to set up a server +<pjotrp> for your own use +<user02> Stuck at downloading preprocesscore +<pjotrp> should not [09:24] +<pjotrp> what does env GUIX_PACKAGE_PATH=../guix-bioinformatics/ + ./pre-inst-env guix package -i genenetwork2 + --substitute-urls="http://guix.genenetwork.org:8080" --dry-run + [09:25] +<pjotrp> say for r-prepocesscore +<pjotrp> download or build? +<pjotrp> mine says download [09:26] +<user02> it only lists the derivatives to be built +<user02> nothing else happens [09:27] +<pjotrp> OK, so there is a problem +<pjotrp> your key may not be working +<pjotrp> everything should be listed as 'to be download' [09:28] +<user02> Hmm +<user02> Ah +<user02> I know where I messed up +<pjotrp> where? +<user02> I did add the key +<user02> However +<pjotrp> (I am documenting) +<user02> I did not tell guix to trust it +<pjotrp> yes +<pjotrp> and there is another potential problem +<user02> Remember the documentation on installing guix? +<user02> You have to tell guix to trust the default key [09:29] +<user02> Right? +<user02> So in this case +<pjotrp> read the IRC log +<user02> That step is mandatory +<pjotrp> user01: how are you doing? +<pjotrp> user02: + https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org#using-gnu-guix-archive + [09:30] +<user01> a little bit left on the db download +<pjotrp> user02: you should see no more building +<pjotrp> user02: another issue may be that you updated r-preprocesscore + package in guix-buinformatics [09:32] +<pjotrp> all downstream packages will want to rebuild +<user02> no, not really +<user02> It's not even installed +<pjotrp> checkout a branch of the the old version - make sure we are in synch +<pjotrp> should be at + /gnu/store/y1f3r2xs3fhyadd46nd2aqbr2p9qv2ra-r-biocpreprocesscore-1.32.0 + [09:33] +<pjotrp> +<user03> pjotrp: Possibly we should use the archive utility of Guix to do + deployment to avoid such out-of-sync differences :) [09:34] +<pjotrp> maybe. I did not get archive to update profiles properly [09:37] +<pjotrp> Also it is good that they get to understand guix + this way +<pjotrp> carved in stone, eh [09:38] +<user02> Yeah, all good [09:39] +<user02> My mistake was skipping the guix archive part +<user02> Can we begin with the install? +<user02> It's telling me of derivatives that will be downloaded [09:40] +<user02> So we're good +<user02> Here goes +<pjotrp> yeeha [09:42] +<user02> pjotrp, where is this guix.genenetwork.org located at? +<pjotrp> Tennessee +<user02> It's...it's....sloooooooowwwwwwwwwwwwww +<pjotrp> not from Europe +<pjotrp> is it downloading at all? +<user02> It should be extended +<user02> Yes...like at 100KB/s [09:43] +<user02> tear-jerker +<user02> Verizon problems +<user02> who's the host? +<pjotrp> I am getting 500Kb/s +<pjotrp> UT +<user02> Guix's servers can run off more than one server, right? +<user02> I'd like to host that particular server here +<user02> For speed +<pjotrp> yes +<user02> Sooner or later +<user02> It will be a necessity [09:45] +<pjotrp> exactly what I am doing - this is our server +<pjotrp> guix.genenetwork.org:8080 +<user02> All done installing [09:46] +<pjotrp> what? +<user02> Now the databases +<pjotrp> what do you mean by slow exactly? +<user02> Yes, it's installed +<pjotrp> can you run genenetwork2 +<user02> setting variables +<user02> If I try running it now, it will fail as I don't have the DBs [09:47] +<pjotrp> cool - you had a lot of prebuilt packages already +<pjotrp> OK, follow the instructions I wrote above +<user01> now everything seems to be working for me :) +<user02> OK +<pjotrp> user01: excellent! +<pjotrp> you see a webserver? +<user01> yep, can connect to localhost:5003 [09:48] +<pjotrp> So now you are running a guix copy of GN2 +<pjotrp> you can see where it lives with `which genenetwork2` or ls -l + ~/.guix-profile/bin/genenetwork2 [09:49] +<pjotrp> + /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/genenetwork2 +<pjotrp> it is a script +<pjotrp> written by guix, open it [09:50] +<pjotrp> inside it points to paths and our script at +<pjotrp> + /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/.genenetwork2-real +<pjotrp> if you open that you can see how the webserver is started [09:51] +<pjotrp> next step is to run a recent version of GN2 +<user01> okay [09:52] +<pjotrp> See + https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-your-own-copy-of-gn2 +<pjotrp> but do not checkout that genetwork2_diet +<pjotrp> we reverted to the main tree +<pjotrp> clone git@github.com:genenetwork/genenetwork2.git [09:53] +<pjotrp> instead and checkout the staging branch +<pjotrp> that is effectively my branch [09:54] +<pjotrp> when that is done you should be able to fire up the webserver from + there [09:55] +<pjotrp> using ./bin/genenetwork2 +<user02> now installing DBs +<user02> Downloading +<pjotrp> annoyingly the source tree is ~700Mb [09:56] +<user02> Can it also be done by installing the guix package + genenetwork2-database-small? +<pjotrp> I changed it in the diet version to 8Mb, but I had to revert +<user01> I need to make my VM bigger... +<pjotrp> user02: not ready [09:57] +<user02> ok +<pjotrp> user01: sorry +<pjotrp> user01: you could mount a local dir inside the VM for development +<pjotrp> that would allow you to use MAC tools for editing +<pjotrp> just an idea +<user01> yeah, I figure I'll do something like that +<pjotrp> do you use emacs? [09:58] +<user01> yep +<pjotrp> that can also run on remote files over ssh +<pjotrp> that's an alternative +<pjotrp> kudos for using emacs :), wdyt user03 +<user02> 79 minutes to go downloading the db +<pjotrp> user02: sorry about that [09:59] +<pjotrp> it is 2GB +<user02> user, you can also mount the directory via sshfs +<user02> Mac OSX runs OpenSSH +<pjotrp> user02: sopa +<user02> You can therefore mount a directory outside the VM to the VM via + sshfs [10:00] +<pjotrp> yes, 3 options now +<user02> That way, you can set up a VM only for it's logic +<user02> Apps + the OS it runs [10:01] +<user02> For data, let it reside on physical host accessible via sshfs +<user02> Use this Arch wiki reference: + https://wiki.archlinux.org/index.php/SSHFS +<user02> I edited that last somewhere in 2015, may have been updated since + then +<user01> alright, cool! [10:04] +<pjotrp> user01: you are almost done [10:06] +<pjotrp> I wrote an elixir package for guix :) +<pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix + package -A elixir + --substitute-urls="http://guix.genenetwork.org:8080" [10:08] +<pjotrp> elixir 1.2.3 out + ../guix-bioinformatics/gn/packages/elixir.scm:31:2 +<pjotrp> +<pjotrp> I am building it on guix.genenetwork.org right now [10:09] +<user01> nice [10:10] #+end_src |