Merge branch 'master' of github.com:genenetwork/genenetwork2

7 files changed, 438 insertions, 892 deletions

diff --git a/README.md b/README.md
index 85042e6a..8221ad65 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
-[![DOI](https://zenodo.org/badge/5591/genenetwork/genenetwork2.svg)](https://zenodo.org/badge/latestdoi/5591/genenetwork/genenetwork2)
+[![DOI](https://zenodo.org/badge/5591/genenetwork/genenetwork2.svg)](https://zenodo.org/badge/latestdoi/5591/genenetwork/genenetwork2) [![JOSS](http://joss.theoj.org/papers/10.21105/joss.00025/status.svg)](http://joss.theoj.org/papers/10.21105/joss.00025)
 
-# GENENETWORK
+# GeneNetwork
 
 This repository contains the source code for the GeneNetwork (GN)
 server http://gn2.genenetwork.org/ (version 2 aka GN2). GN is a Web
@@ -35,10 +35,12 @@ testing using [mechanize](https://github.com/genenetwork/genenetwork2/tree/maste
 ## Documentation
 
 User documentation can be found
-[here](http://gn2.genenetwork.org/help).  The database schema is
-shared with GN1 and described
+[here](http://gn2.genenetwork.org/help).  The architecture of the
+software stack is described [here](./doc/Architecture.org).  The
+database schema is (still) shared with GN1 and currently described
 [here](http://www.genenetwork.org/webqtl/main.py?FormID=schemaShowPage). Software
-documentation is being expanded in the [source code repository](https://github.com/genenetwork/genenetwork2/tree/master/doc).
+documentation is being expanded in the
+[source code repository](https://github.com/genenetwork/genenetwork2/tree/master/doc).
 
 ## Contributing
 
@@ -66,7 +68,12 @@ For more information visit http://www.genenetwork.org/
 
 ## Cite
 
-You can cite this software using
+[![JOSS](http://joss.theoj.org/papers/10.21105/joss.00025/status.svg)](http://joss.theoj.org/papers/10.21105/joss.00025)
+
+GeneNetwork was published in the Journal of Open Source Software as 'GeneNetwork: framework for web-based genetics' by Zachary Sloan, Danny Arends, Karl W. Broman, Arthur Centeno, Nicholas Furlotte, Harm Nijveen, Lei Yan, Xiang Zhou, Robert W. WIlliams and Pjotr Prins 
+
+You may also cite the software using
+
 [![DOI](https://zenodo.org/badge/5591/genenetwork/genenetwork2.svg)](https://zenodo.org/badge/latestdoi/5591/genenetwork/genenetwork2). 
 
 ## Contact
diff --git a/VERSION b/VERSION
new file mode 100644
index 00000000..cd5ac039
--- /dev/null
+++ b/VERSION
@@ -0,0 +1 @@
+2.0
diff --git a/doc/Architecture.org b/doc/Architecture.org
new file mode 100644
index 00000000..c263d3b9
--- /dev/null
+++ b/doc/Architecture.org
@@ -0,0 +1,15 @@
+* GeneNetwork Architecture
+
+** Introduction
+
+This document describes the architecture of GN2. Because GN2 is
+evolving, only a high-level overview is given here.
+
+** Webserver
+
+The webserver is built on [[http://flask.pocoo.org/][Python flask]] and this GN2 source code can be
+found on [[https://github.com/genenetwork/genenetwork2/tree/master/wqflask/wqflask][github]] in the wqflask directory. The routing tables are
+defined in [[https://github.com/genenetwork/genenetwork2/blob/master/wqflask/wqflask/views.py][views.py]]. For example the main page is loaded from a
+template named [[https://github.com/genenetwork/genenetwork2/blob/master/wqflask/wqflask/templates/index_page.htm][index_page.html]] in the [[https://github.com/genenetwork/genenetwork2/tree/master/wqflask/wqflask/templates][templates]] directory. In the 
+template you can find get the form gets filled by a Javascript
+routine defined in [[https://github.com/genenetwork/genenetwork2/blob/master/wqflask/wqflask/static/new/javascript/dataset_select_menu.js][data_select_menu.js]].
diff --git a/doc/README.org b/doc/README.org
index 278dd31c..3754dbbd 100644
--- a/doc/README.org
+++ b/doc/README.org
@@ -3,24 +3,33 @@
 
 * Table of Contents                                                     :TOC:
  - [[#introduction][Introduction]]
+ - [[#quick-installation-recipe][Quick installation recipe]]
+   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
+   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
+   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
+   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]
+ - [[#run-mysql-server][Run MySQL server]]
+ - [[#gn2-dependency-graph][GN2 Dependency Graph]]
  - [[#source-deployment][Source deployment]]
-   - [[#install-guix][Install guix]]
-   - [[#checkout-the-git-repositories][Checkout the git repositories]]
-   - [[#update-guix][Update guix]]
-   - [[#install-gn2][Install GN2]]
-   - [[#run-gn2][Run GN2]]
-   - [[#run-mysql-server][Run MySQL server]]
    - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]]
    - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]]
  - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]]
+   - [[#update-to-recent-guix][Update to recent guix]]
+   - [[#install-gn2][Install GN2]]
+   - [[#run-gn2][Run GN2]]
  - [[#trouble-shooting][Trouble shooting]]
    - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]]
    - [[#error-can-not-find-directory-homegn2_data][ERROR: can not find directory $HOME/gn2_data]]
    - [[#cant-run-a-module][Can't run a module]]
+ - [[#irc-session][IRC session]]
 
 * Introduction
 
-Large system deployments can get very complex. In this document we
+If you want to understand the architecture of GN2 read
+[[Architecture.org]].  The rest of this document is mostly on deployment
+of GN2.
+
+Large system deployments can get very [[http://biobeat.org/gn2.svg][complex]]. In this document we
 explain the GeneNetwork version 2 (GN2) reproducible deployment system
 which is based on GNU Guix (see also Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix
 system can be used to install GN with all its files and dependencies.
@@ -30,20 +39,72 @@ main Guix package tree and that of the Genenetwork package
 tree. Current supported versions can be found as the SHA values of
 'gn-latest' branches of [[https://github.com/genenetwork/guix-bioinformatics/tree/gn-latest][Guix bioinformatics]] and [[https://github.com/genenetwork/guix/tree/gn-latest][GNU Guix main]].
 
-* Source deployment
-** Install guix
+For a full view of runtime dependencies as defined by GNU Guix, see
+the [[#gn2-dependency-graph][GN2 Dependency Graph]].
+
+* Quick installation recipe
+
+This is a recipe for quick and dirty installation of GN2. For
+convenience everything is installed as root, though in reality only
+GNU Guix has to be installed as root. I tested this recipe on a fresh
+install of Debian 8.3.0 (in KVM) though it should work on any modern
+Linux distribution (including CentOS). For more elaborate installation
+instructions see [[#source-deployment][Source deployment]].
+
+Note that GN2 consists of an approx. 5 GB installation including
+database.
+
+** Step 1: Install GNU Guix
+
+Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow
+[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially you have to download and unpack the tar ball
+(which creates directories in /gnu and /var/guix), add build users and
+group (Guix builds software as unpriviliged users) and run the Guix
+daemon after fixing the paths (also known as the 'profile').
+
+Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission
+to download binaries from the GNU server) and you should be able to
+install the hello package using binary packages (no building)
+
+#+begin_src bash
+export PATH=~/.guix-profile/bin:$PATH
+guix pull
+guix package -i hello --dry-run
+#+end_src
+
+Which should show something like
+
+: The following files would be downloaded:
+:   /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
+:   ...
+#+end_src
 
-Deploying from source is also straightforward. Install GNU Guix using
-a binary tar ball as described [[https://github.com/pjotrp/guix-notes][here]].
+means binary installs.  The actual installation command of 'hello' is
 
-If it works you should be able to install a package with
+#+begin_src bash
+guix package -i hello
+hello
+  Hello, world!
+#+end_src
 
-: guix package -i hello
+If you actually see things building it means that Guix is not yet
+properly installed and up-to-date, i.e., the key is missing or you
+need to do a 'guix pull'. Press Ctrl-C to interrupt.
 
-** Checkout the git repositories
+If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get
+rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]].
 
-Check out the two relevant guix and guix-bioinformatics git
-repositories:
+** Step 2: Checkout the GN2 git repositories
+
+To fixate the software dependency graph GN2 uses git repositories of
+Guix packages. First install git if it is missing
+
+#+begin_src bash
+guix package -i git
+export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
+#+end_src
+
+check out the git repsitories (gn-latest branch)
 
 #+begin_src bash
 cd ~
@@ -54,24 +115,343 @@ git clone --branch gn-latest --recursive https://github.com/genenetwork/guix gui
 cd guix-gn-latest
 #+end_src bash
 
-** Update guix
+** Step 3: Authorize the GN Guix server
+
+GN2 has its own GNU Guix binary distribution server. To trust it you have
+to add the following key 
+
+#+begin_src scheme
+(public-key
+ (ecc
+  (curve Ed25519)
+  (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#)
+ )
+)
+#+end_src
+
+by pasting it into the command
+
+#+begin_src bash
+guix archive --authorize
+#+end_src
+
+and hit Ctrl-D.
+
+Now you can use the substitute server to install GN2 binaries. 
+
+** Step 4: Install and run GN2 
+
+Since this is a quick and dirty install we are going to override the
+GNU Guix package path by pointing the package path to our repository:
+
+#+begin_src bash
+rm /root/.config/guix/latest
+ln -s ~/genenetwork/guix-gn-latest/ /root/.config/guix/latest
+#+end_src
+
+Now check whether you can find the GN2 package with
+
+#+begin_src bash
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2
+  genenetwork2    2.0-a8fcff4     out     gn/packages/genenetwork.scm:144:2
+#+end_src
+
+(ignore the source file newer then ... messages, this is caused by the
+/root/.config/guix/latest override).
+
+And install with
+
+#+begin_src bash
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \
+  guix package -i genenetwork2 \
+  --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" \
+  --fallback
+#+end_src
+
+Note: the order of the substitute url's may make a difference in speed
+(put the one first that is fastest for your location and time of day).
+
+After installation you should be able to run genenetwork2 after updating
+the Guix suggested environment vars. Check the output of
+
+#+begin_src bash
+guix package --search-paths
+export PYTHONPATH="/root/.guix-profile/lib/python2.7/site-packages"
+export R_LIBS_SITE="/root/.guix-profile/site-library/"
+#+end_src
+
+and copy-paste the listed exports into the terminal before running:
+
+#+begin_src bash
+genenetwork2
+#+end_src
+
+It will complain that the database is missing. See the next section on
+running MySQL server for downloading and installing a MySQL GN2
+database. After installing the database restart genenetwork2 and point
+your browser at [[http://localhost:5003/]].
+
+End of the GN2 installation recipe!
+
+* Run MySQL server
+
+At this point we require the underlying distribution to install and
+run mysqld. Currently we have two databases for deployment,
+'db_webqtl_s' is the small testing database containing experiments
+from BXD mice and 'db_webqtl_plant' which contains all plant related
+material.
+
+Download one database from
+
+http://files.genenetwork.org/raw_database/
+https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip
+
+Check the md5sum.
+
+After installation inflate the database binary in the MySQL directory
+(this installation path is subject to change soon) 
+
+: chown -R mysql:mysql db_webqtl_s/
+: chmod 700 db_webqtl_s/
+: chmod 660 db_webqtl_s/*
+
+restart MySQL service (mysqld). Login as root and
+
+: mysql> show databases;
+: +--------------------+
+: | Database           |
+: +--------------------+
+: | information_schema |
+: | db_webqtl_s        |
+: | mysql              |
+: | performance_schema |
+: +--------------------+
+
+Set permissions and match password in your settings file below:
+
+: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password';
+
+Note that if the mysql connection is not working, try connecting to
+the IP address and check server firewall, hosts.allow and mysql IP
+configuration.
+
+Note for the plant database you can rename it to db_webqtl_s, or
+change the settings in etc/default_settings.py to match your path.
+
+* GN2 Dependency Graph
+
+Graph of all runtime dependencies as installed by GNU Guix.
+
+#+ATTR_HTML: :title GN2_graph
+[[http://biobeat.org/gn2.svg]]
+
+* Source deployment
+
+This section gives a more elaborate instruction for installing GN2
+from source.
+
+First execute above 4 steps:
+
+   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
+   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
+   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
+   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]
+
+
+** Run your own copy of GN2
+
+At some point you may want to fix the source code. Assuming you have
+Guix and Genenetwork2 installed (as described above) clone the GN2
+repository from https://github.com/genenetwork/genenetwork2.
+
+Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
+R_LIBS_SITE are set) from the information given by guix:
+
+: guix package --search-paths
+
+Inside the repository:
+
+: cd genenetwork2
+: ./bin/genenetwork2 
+
+Will fire up your local repo http://localhost:5003/ using the  
+settings in ./etc/default_settings.py. These settings may 
+not reflect your system. To override settings create your own from a copy of
+default_settings.py and pass it into GN2 with
+
+: ./bin/genenetwork2 $HOME/my_settings.py
+
+and everything *should* work (note the full path to the settings
+file). This way we develop against the exact same dependency graph of
+software.
+
+If something is not working, take a hint from the settings file
+that comes in the Guix installation. It sits in something like
+
+: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py
+
+** Set up nginx port forwarding
+
+nginx can be used as a reverse proxy for GN2. For example, we want to
+expose GN2 on port 80 while it is running on port 5003. Essentially
+the configuration looks like
+
+#+begin_src js
+    server {
+        listen 80;
+        server_name test-gn2.genenetwork.org;
+        access_log  logs/test-gn2.access.log;
+
+        proxy_connect_timeout       3000;
+        proxy_send_timeout          3000;
+        proxy_read_timeout          3000;
+        send_timeout                3000;
+
+        location / {
+            proxy_set_header   Host      $http_host;
+            proxy_set_header   Connection keep-alive;
+            proxy_set_header   X-Real-IP $remote_addr;
+            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header   X-Forwarded-Host $server_name;
+            proxy_pass         http://127.0.0.1:5003;
+        }
+}
+#+end_src js
+
+Install the nginx webserver (as root)
+
+: guix package -i nginx
+
+The nginx example configuration examples can be found in the Guix
+store through
+
+: ls -l /root/.guix-profile/sbin/nginx
+: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx
+
+Use that path
+
+: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
+:   fastcgi.conf            koi-win             scgi_params
+:   fastcgi.conf.default    mime.types          scgi_params.default
+:   fastcgi_params          mime.types.default  uwsgi_params
+:   fastcgi_params.default  nginx.conf          uwsgi_params.default
+:   koi-utf                 nginx.conf.default  win-utf
+
+And copy any relevant files to /etc/nginx.  A configuration file for
+GeneNetwork (reverse proxy) port forwarding can be found in the source
+repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
+(still as root)
+: cp ./etc/nginx-genenetwork.conf /etc/nginx/
+
+Make dirs
+
+: mkdir -p /var/spool/nginx/logs
+
+Add users
+
+: adduser nobody ; addgroup nobody 
+
+Run nginx
+
+: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx
+
+* Source deployment and other information on reproducibility
+
+See the document [[GUIX-Reproducible-from-source.org]].
+
+** Update to recent guix
 
-At some point you may decide to create, install and run a recent
-version of the guix-daemon by compiling the guix repository. Follow
-[[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully.
+We now compile Guix from scratch.
+
+Create, install and run a recent version of the guix-daemon by
+compiling the guix repository you have installed with git in
+step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after
+
+: cd ~/genenetwork/guix-gn-latest
+
+Make sure to restart the guix daemon and run guix client from this
+directory.
 
 ** Install GN2
 
+Reinstall genenetwork2 using the new tree
+
 #+begin_src bash
-env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env \
-  guix package -i genenetwork2 --fallback 
+env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org"
 #+end_src bash
 
-Note that you can use the genenetwork.org guix substitute caching
-server at http://guix.genenetwork.org:8080 (which speeds up installs
-significantly because all packages are pre-built). Here an IRC session
-where we installed GN2 from scratch using GNU Guix and a download
-of the test database:
+Note the use of ./pre-inst-env here!
+
+Actually, it should be the same installation as in step 4, so nothing
+gets downloaded.
+
+** Run GN2
+
+Make a note of the paths with
+
+#+begin_src bash
+./pre-inst-env guix package --search-paths
+#+end_src bash
+
+After setting the paths for the server
+
+#+begin_src bash
+export PATH=~/.guix-profile/bin:$PATH
+export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
+export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
+export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
+export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
+export XDG_DATA_DIRS="$HOME/.guix-profile/share"
+export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
+#+end_src bash
+
+run the main script (in ~/.guix-profile/bin)
+
+#+begin_src bash
+genenetwork2
+#+end_src bash
+
+will start the default server which listens on port 5003, i.e.,
+http://localhost:5003/.
+
+OK, we are where we were before with step 4. Only difference is that we 
+used our own compiled guix server.
+
+* Trouble shooting
+
+** ImportError: No module named jinja2
+
+If you have all the Guix packages installed this error points out that
+the environment variables are not set. Copy-paste the paths into your
+terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
+information given by guix:
+
+: guix package --search-paths
+
+On one system:
+
+: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
+: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
+: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
+
+and perhaps a few more. 
+** ERROR: can not find directory $HOME/gn2_data
+
+The default settings file looks in your $HOME/gn2_data. Since these
+files come with a Guix installation you should take a hint from the
+values in the installed version of default_settings.py (see above in
+this document).
+
+** Can't run a module
+
+In rare cases, development modules are not brought in with Guix
+because no source code is available. This can lead to missing modules
+on a running server. Please check with the authors when a module
+is missing.
+* IRC session
+
+Here an IRC session where we installed GN2 from scratch using GNU Guix
+and a download of the test database.
 
 #+begin_src
 <pjotrp> time to get binary install sorted :)  [07:03]
@@ -475,204 +855,3 @@ The following derivations would be built:
 <pjotrp> I am building it on guix.genenetwork.org right now  [10:09]
 <user01> nice  [10:10]
 #+end_src
-
-** Run GN2
-
-Make a note of the paths with
-
-#+begin_src bash
-./pre-inst-env guix package --search-paths
-#+end_src bash
-
-After setting the paths for the server
-
-#+begin_src bash
-export PATH=~/.guix-profile/bin:$PATH
-export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
-export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
-export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
-export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
-export XDG_DATA_DIRS="$HOME/.guix-profile/share"
-export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
-#+end_src bash
-
-run the main script (in ~/.guix-profile/bin)
-
-#+begin_src bash
-genenetwork2
-#+end_src bash
-
-will start the default server which listens on port 5003, i.e.,
-http://localhost:5003/.
-
-** Run MySQL server
-
-At this point we require the underlying distribution to install
-and run mysqld. 
-
-Download one of
-
-http://files.genenetwork.org/raw_database/
-https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip
-
-Check the md5sum.
-
-After installation inflate the database binary in the MySQL directory
-(this is subject to change soon) 
-
-: chown -R mysql:mysql db_webqtl_s/
-: chmod 700 db_webqtl_s/
-: chmod 660 db_webqtl_s/*
-
-restart MySQL service (mysqld). Login as root and
-
-: mysql> show databases;
-: +--------------------+
-: | Database           |
-: +--------------------+
-: | information_schema |
-: | db_webqtl_s        |
-: | mysql              |
-: | performance_schema |
-: +--------------------+
-
-Set permissions and match password in your settings file below:
-
-: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password';
-
-Note that if the mysql connection is not working, try connecting to
-the IP address and check server firewall, hosts.allow and mysql IP
-configuration.
-
-** Run your own copy of GN2
-
-At some point you may want to fix the source code. Assuming you have
-Guix and Genenetwork2 installed (as described above) clone the GN2
-repository from https://github.com/genenetwork/genenetwork2.
-
-Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
-R_LIBS_SITE are set) from the information given by guix:
-
-: guix package --search-paths
-
-Inside the repository:
-
-: cd genenetwork2
-: ./bin/genenetwork2 
-
-Will fire up your local repo http://localhost:5003/ using the  
-settings in ./etc/default_settings.py. These settings may 
-not reflect your system. To override settings create your own from a copy of
-default_settings.py and pass it into GN2 with
-
-: ./bin/genenetwork2 $HOME/my_settings.py
-
-and everything *should* work (note the full path to the settings
-file). This way we develop against the exact same dependency graph of
-software.
-
-If something is not working, take a hint from the settings file
-that comes in the Guix installation. It sits in something like
-
-: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py
-
-** Set up nginx port forwarding
-
-nginx can be used as a reverse proxy for GN2. For example, we want to
-expose GN2 on port 80 while it is running on port 5003. Essentially
-the configuration looks like
-
-#+begin_src js
-    server {
-        listen 80;
-        server_name test-gn2.genenetwork.org;
-        access_log  logs/test-gn2.access.log;
-
-        proxy_connect_timeout       3000;
-        proxy_send_timeout          3000;
-        proxy_read_timeout          3000;
-        send_timeout                3000;
-
-        location / {
-            proxy_set_header   Host      $http_host;
-            proxy_set_header   Connection keep-alive;
-            proxy_set_header   X-Real-IP $remote_addr;
-            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header   X-Forwarded-Host $server_name;
-            proxy_pass         http://127.0.0.1:5003;
-        }
-}
-#+end_src js
-
-Install the nginx webserver (as root)
-
-: guix package -i nginx
-
-The nginx example configuration examples can be found in the Guix
-store through
-
-: ls -l /root/.guix-profile/sbin/nginx
-: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx
-
-Use that path
-
-: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
-:   fastcgi.conf            koi-win             scgi_params
-:   fastcgi.conf.default    mime.types          scgi_params.default
-:   fastcgi_params          mime.types.default  uwsgi_params
-:   fastcgi_params.default  nginx.conf          uwsgi_params.default
-:   koi-utf                 nginx.conf.default  win-utf
-
-And copy any relevant files to /etc/nginx.  A configuration file for
-GeneNetwork (reverse proxy) port forwarding can be found in the source
-repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
-(still as root)
-: cp ./etc/nginx-genenetwork.conf /etc/nginx/
-
-Make dirs
-
-: mkdir -p /var/spool/nginx/logs
-
-Add users
-
-: adduser nobody ; addgroup nobody 
-
-Run nginx
-
-: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx
-
-* Source deployment and other information on reproducibility
-
-See the document [[GUIX-Reproducible-from-source.org]].
-
-* Trouble shooting
-
-** ImportError: No module named jinja2
-
-If you have all the Guix packages installed this error points out that
-the environment variables are not set. Copy-paste the paths into your
-terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
-information given by guix:
-
-: guix package --search-paths
-
-On one system:
-
-: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"
-: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
-: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
-
-and perhaps a few more. 
-** ERROR: can not find directory $HOME/gn2_data
-
-The default settings file looks in your $HOME/gn2_data. Since these
-files come with a Guix installation you should take a hint from the
-values in the installed version of default_settings.py (see above in
-this document).
-
-** Can't run a module
-
-In rare cases, development modules are not brought in with Guix
-because no source code is available. This can lead to missing modules
-on a running server. Please check with the authors when a module
-is missing.
diff --git a/doc/joss/2016/paper.md b/doc/joss/2016/paper.md
index 9ac323d2..6b640836 100644
--- a/doc/joss/2016/paper.md
+++ b/doc/joss/2016/paper.md
@@ -17,7 +17,8 @@ authors:
   - name: Arthur Centeno
     orcid: 0000-0003-3142-2081
     affiliation: University of Tennessee Health Science Center, USA
-  - name: Nick Furlotte
+  - name: Nicholas Furlotte
+    orcid: 0000-0002-9096-6276
   - name: Harm Nijveen
     orcid: 0000-0002-9167-4945
     affiliation: Wageningen University, The Netherlands
diff --git a/doc/old/gn_installation_notes.txt b/doc/old/gn_installation_notes.txt
deleted file mode 100644
index efea0309..00000000
--- a/doc/old/gn_installation_notes.txt
+++ /dev/null
@@ -1,346 +0,0 @@
-UFW - default firewall confirguation tool for Ubuntu; eases iptables firewall configuration
-
-Enable UFW:
-sudo ufw enable
-Check status of UFW:
-sudo ufw status verbose
-
-Limit ssh:
-sudo ufw limit ssh
-
-Enable http, https:
-sudo ufw allow http
-sudo ufw allow https
-
-===========================================
-
-http://evilrouters.net/2011/02/18/using-etckeeper-with-git-on-ubuntu/
-
-Install etckeeper:
-sudo apt-get install etckeeper
-
-Edit /etc/etckeeper/etckeeper.conf to comment out VCS="bzr" and comment in VCS="git"
-
-Initialize repository:
-sudo etckeeper init
-
-Do initial commit:
-sudo etckeeper commit "Initial commit."
-
-===========================================
-
-Get git:
-
-sudo apt-get install git
-
-===========================================
-
-Clone code from git repository:
-git clone http://github.com/zsloan/genenetwork.git gene 
-
-Pull from branch in git repository:
-git pull origin flask(or whatever the branch is)
-
-**"apt-cache search" searches for text in the description for various packages, while
-"apt-file search" searches for files within the packages**
-
-Search for package with a specified file that can be installed with apt-get
-apt-file search _______
-
-============================================
-
-Create trash directory:
-mkdir ~/.trash
-
-============================================
-
-Using dpkg
-
-Check which system packages are already installed:
-dpkg -l | less
-
-===========================================
-
-Installing virtualenv:
-sudo pip install virtualenv
-
-Tell virtualenv which directory to use:
-virtualenv ve27
-
-Start up virtual environment:
-source ~/ve27/bin/activate
-
-===========================================
-
-Install libmysqlclient-dev (Mysql-Python dependency)
-sudo apt-get install libmysqlclient-dev
-
-Install python-dev (numpy dependency)
-sudo apt-get install python-dev
-
-Install scipy dependencies:
-sudo apt-get install libatlas-base-dev gfortran g++
-
-===========================================
-
-Install pip:
-sudo apt-get install python-pip
-
-REMEMBER TO SOURCE VE BEFORE INSTALLING 
-
-Comment out in requirements.txt:
-Reaper
-numarray
-
-Before installing from requirements.txt, install numpy separately:
-pip install numpy==1.7.0 (or whatever version we're using)
-
-Install from requirements.txt (after activating virtualenv):
-pip install -r gene/doc/requirements.txt
-
-===========================================
-
-Installing QTL Reaper
-
-wget http://downloads.sourceforge.net/project/qtlreaper/qtlreaper/1.1.1/qtlreaper-1.1.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fqtlreaper%2Ffiles%2Flatest%2Fdownload&ts=1358975786&use_mirror=iweb
-mv -v qtlreaper-1.1.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fqtlreaper%2Ffiles%2Flatest%2Fdownload&ts=1358975786&use_mirror=iweb qtlreaper-1.1.1.tar.gz
-tar xvf qtlreaper-1.1.1.tar.gz (to unzip)
-mkdir /home/zas1024/ve27/include/python2.7/Reaper
-python setup.py install
-
-===========================================
-
-Installing numarray 1.5.2
-
-wget http://downloads.sourceforge.net/project/numpy/Old%20Numarray/1.5.2/numarray-1.5.2.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fnumpy%2Ffiles%2FOld%2520Numarray%2F1.5.2%2Fnumarray-1.5.2.tar.gz%2Fdownload&ts=1358978306&use_mirror=superb-dca2
-mv -v numarray-1.5.2.tar.gz?r=http:%2F%2Fsourceforge.net%2Fprojects%2Fnumpy%2Ffiles%2FOld%20Numarray%2F1.5.2%2Fnumarray-1.5.2.tar.gz%2Fdownload numarray-1.5.2.tar.gz
-tar xvf numarray-1.5.2.tar.gz
-sudo python setup.py install
-
-===========================================
-
-Installing nginx
-
-sudo -s
-nginx=stable # use nginx=development for latest development version
-echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
-apt-get update 
-apt-get install nginx
-
-Create configuration file in ~/gene/wqflask/other_config/nginx_conf/ (modeled off of the others)
-Create symbolic link to config file in /etc/nginx/sites-enabled/:
-ln -s dir_to_link_to linking_dir
-
-Run nginx:
-sudo /usr/sbin/nginx
-
-============================================
-
-Copy over zach_settings.py to /home/zas1024 directory (or whatever is home directory)
-
-export WQFLASK_SETTINGS=~/gene/wqflask/cfg/zach_settings.py (or wherever file is located)
-
-============================================
-
-Install MySQL Server
-
-sudo apt-get install mysql-server
-
-mkdir /mnt/big
-fdisk /dev/sdb
-m: for help
-n: new partion
-...
-w: write to table and exit
-
-Start MySQL server:
-service mysql start
-
-Stop MySQL server:
-service mysql stop
-
-Change root password:
-mysql> UPDATE mysql.user SET Password=PASSWORD('your password') WHERE User='root';
-
-Setup accounts in MySQL (first need to delete anonymous/non-root accounts):
-#; use mysql;
-#; select * from user;
-#; delete from user where Host!="localhost";
-#; delete from user where User!="root";
-#; update user set Password = Password('yourpassword') where User='root';
-#; GRANT ALL ON *.* TO 'yourusername'@'%' IDENTIFIED BY 'yourpassword';
-#; select * from user;
-
-Use Percona to optimise my.cfg file:
-https://tools.percona.com/wizard
-
-To check # CPUs:
-grep -c processor /proc/cpuinfo
-
-To check memory, can use htop
-
-============================================
-
-Check RSA key:
-ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
-
-03:2c:d7:01:01:f0:31:3a:c8:df:e4:98:62:2c:59:d2  root@penguin (RSA)
-
-============================================
-
-Installing redis server
-
-In the dls directory:
-$ wget http://redis.googlecode.com/files/redis-2.6.14.tar.gz
-$ tar xzf redis-2.6.14.tar.gz
-$ cd redis-2.6.14
-$ make
-
-Run redis (from inside redis-2.6.14 directory):
-src/redis-server
-
-For the rest of the installation instructions, refer to:
-http://redis.io/topics/quickstart
-
-To continue installation (from the README):
-
-In order to install Redis binaries into /usr/local/bin just use:
-
-    % make install
-
-Make install will just install binaries in your system, but will not configure
-init scripts and configuration files in the appropriate place. This is not
-needed if you want just to play a bit with Redis, but if you are installing
-it the proper way for a production system, we have a script doing this
-for Ubuntu and Debian systems:
-
-    % cd utils
-    % ./install_server
-
-The script will ask you a few questions and will setup everything you need
-to run Redis properly as a background daemon that will start again on
-system reboots.
-
-In the redis config file (/etc/redis/6379.conf) uncomment the following line:
-bind 127.0.0.1
-
-This is so only the local computer has access
-
-============================================
-
-Check RSA key:
-ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
-
-03:2c:d7:01:01:f0:31:3a:c8:df:e4:98:62:2c:59:d2  root@penguin (RSA)
-
-============================================
-
-Using Yolk
-
-Install Yolk:
-pip install yolk
-
-Check packages installed in this virtual environment:
-yolk -l
-
-Checks packages that have updates available:
-yolk -U
-
-===========================================
-
-Installing yaml
-
-Install libyaml-dev:
-sudo apt-get install libyaml-dev
-
-Install yaml:
-pip install pyyaml
-
-===========================================
-
-Install MySQL Client
-
-To fix error "mysql_config not found" while installing packages with pip:
-sudo apt-get install libmysqlclient-dev
-
-===========================================
-
-Installing R
-
-sudo apt-get install r-base-dev
-
-===========================================
-
-Installing rpy2
-
-pip install rpy2
-
-===========================================
-
-Install Parallel Python (pp)
-
-wget http://www.parallelpython.com/downloads/pp/pp-1.6.3.tar.gz
-tar xvf pp-1.6.3.tar.gz
-python setup.py install
-
-==========================================
-
-Install requests
-
-pip install requests
-
-==========================================
-
-Install logging_tree
-
-pip install logging_tree
-
-==========================================
-
-Install colordiff (just to make it easier to view differences in files, not necessary for GN)
-
-sudo apt-get install colordiff
-
-==========================================
-
-Install NTP (network time protocol)
-
-sudo apt-get install ntp
-
-==========================================
-
-To get server running:
-
-!If having seemingly inexplicable problems with imports, make sure I've started the environment!
-
-Start up virtual environment:
-source ~/ve27/bin/activate==============
-
-Install requests
-
-pip install requests
-
-=======================
-
-To set WQFLASK_SETTINGS environment variable:
-export WQFLASK_SETTINGS=~/zach_settings.py (or wherever file is located)
-
-To change screen environment variable (if man not working or to get color, for example):
-export TERM=screen
-
-To search for commands in history if necessary:
-history | grep "(whatever is being searched for)"
-
-Run web server:
-/usr/local/nginx/sbin/nginx
-
-Run server:
-python runserver.py
-
-===========================================
-
-To do full upgrade (as opposed to apt-get upgrade)
-sudo aptitude full-upgrade
-
-===========================================
diff --git a/doc/old/notes.txt b/doc/old/notes.txt
deleted file mode 100644
index f8ce2759..00000000
--- a/doc/old/notes.txt
+++ /dev/null
@@ -1,311 +0,0 @@
-Clone code from git repository:
-git clone http://github.com/zsloan/genenetwork.git gene 
-
-Pull from branch in git repository:
-git pull origin flask(or whatever the branch is)
-
-Add all files in current directory branch to be staged to be committed
-git add -A
-
-Git push to my github repository
-git push https://github.com/zsloan/genenetwork.git master
-
-Get the default path being used when a command is typed:
-which _______
-
-Change default path python looks at when importing
-export PYTHONPATH="/home/zas1024/gene/wqflask"
-
-Get the paths checked when a user types a command:
-echo $PATH
-
-Get all environment variables:
-set
-
-
-**** apt-get is an Ubuntu system-wide package manager, while pip is a Python-only system
-for python packages ****
-
-Upgrade system packages for security updates, etc:
-apt-get update
-apt-get dist-upgrade (can probably be used in place of apt-get upgrade)
-apt-get upgrade
-
-**"apt-cache search" searches for text in the description for various packages, while
-"apt-file search" searches for files within the packages**
-
-Show information about given result item returned by apt-cache search:
-apt-cache show _______
-
-Search for package with a specified file that can be installed with apt-get
-apt-file search _______
-
-============================================
-
-Check which system packages are installed:
-dpkg -l | less
-
-============================================
-
-NEVER USE SUDO WITH PIP
-
-To fix error "mysql_config not found":
-sudo apt-get install libmysqlclient-dev
-
-Create requirements.txt file:
-pip freeze > requirements.txt (or whatever filename)
-
-Install from requirements.txt:
-pip install -r gene/wqflask/requirements.txt -t ve27
-
-============================================
-
-To get server running:
-
-!If having seemingly inexplicable problems with imports, make sure I've started the environment!
-
-Note: Virtual environment is not activated when running as root
-
-Start up virtual environment:
-source ~/ve27/bin/activate
-
-Deactivate virtual environment:
-deactivate (just type deactivate)
-
-To set WQFLASK_SETTINGS environment variable:
-export WQFLASK_SETTINGS=~/zach_settings.py (or wherever file is located)
-
-zach_settings.py is the local config file with the DB_URI
-default_settings.py is the config file with the configuration information that would apply
-to any running instance of GeneNetwork
-
-To change screen environment variable (if man not working or to get color, for example):
-export TERM=screen
-
-To search for commands in history if necessary:
-history | grep "(whatever is being searched for)"
-
-Reload web server:
-/usr/sbin/nginx -s reload
-
-Run server:
-python runserver.py
-
-Run sendmail.py
-python send_mail.py
-
-===========================================
-
-UFW - default firewall confirguation tool for Ubuntu; eases iptables firewall configuration
-
-Enable UFW:
-sudo ufw enable
-Check status of UFW:
-sudo ufw status verbose
-
-===========================================
-
-Update user password:
-use mysql;
-UPDATE user SET password=PASSWORD('whatever') WHERE user='webqtl';
-FLUSH PRIVILEGES;
-
-===========================================
-
-Redis things
-
-Restart redis server:
-sudo /etc/init.d/redis_6379 stop
-sudo /etc/init.d/redis_6379 start
-
-===========================================
-
-Run pylint:
-pylint file_name
-
-For options:
-pylint --help 
-
-===========================================
-
-Start screen session:
-
-If "no option -R" or running for first time:
-byobu-select-backend
-2. screen
-
-byobu -RD (to start)
-control-a then :multiuser on
-control-a then :acladd sam
-
-More info on sharing byobu screen with other users:
-http://wiki.networksecuritytoolkit.org/nstwiki/index.php/HowTo_Share_A_Terminal_Session_Using_Screen
-
-control-a c to create channel
-
-type: screen -list for sessions
-screen -r zas1024/25679.byobu
-
-or if only one:
-
-screen -r zas1024/
-
-===========================================
-
-Start up log:
-Go to /tmp and tail -f flask_gn_log
-
-===========================================
-
-Coffeescript Stuff:
-
-coffee -c (filename)
-coffee -c -w . (to watch for changes and recompile in current directory; the "." is for current directory)
-coffee --help (for information about setting options)
-
-===========================================
-
-Unset ASKPASS when trying to git push
-
-unset SSH_ASKPASS
-
-===========================================
-
-htop: Gives information on processes, cpu/memory load, etc
-dstat: Also gives various system information, resource usage, etc
-df: Reports file system disk space usage
-
-===========================================
-
-ll h*: Finds items in the directory starting with h
-
-===========================================
-
-du -hms * | sort -n : Gives size used by different directories
--h: human readable
--m: in megabytes (default)
--s: summarize
-
-===========================================
-
-rm -rfv
-
-===========================================
-
-cp -a (archive; copies recursively and doesn't follow symbol links)
-   -i (interactive, prompts before overwrite)
-   -v (verbose)
-   -b (backup)
-   
-mv (same as above, but with no -a)
-
-===========================================
-
-Add user:
-sudo adduser
-Edit /etc/sudoers to give user root privileges
-
-
-===========================================
-
-tidyp - Improves/beautifies html code
-tidyp -m -i -w 100 index_page.html
-
-===========================================
-
-ps ax - View processes
-ps ux (the "u" is for user)
-
-kill (process #)
-
-===========================================
-
-Installing QTL Reaper
-
-wget http://downloads.sourceforge.net/project/qtlreaper/qtlreaper/1.1.1/qtlreaper-1.1.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fqtlreaper%2Ffiles%2Flatest%2Fdownload&ts=1358975786&use_mirror=iweb
-tar xvf qtlreaper-1.1.1.tar.gz (to unzip)
-python setup.py build
-python setup.py install
-
-===========================================
-
-Using Yolk
-
-Install Yolk:
-pip install yolk
-
-Check packages installed in this virtual environment:
-yolk -l
-
-Checks packages that have updates available:
-yolk -U
-
-===========================================
-
-Python stuff:
-
-Classes should always inherit "object"
-To iterate through dictionary items: for X, Y in MyDictionary.items():
-
-===========================================
-
-Change import path from command line
-export PYTHONPATH=~/gene/wqflask (or whatever)
-
-===========================================
-
-Create symbolic link:
-ln -s dir_to_link_to linking_dir
-
-===========================================
-
-Reboot server
-sudo reboot
-
-===========================================
-
-Using grep:
-
-grep -ir (search string) (directory)
-
--i ignores case
--r searches recursively through subdirectories of directory you specify
-
-===========================================
-
-Command line arguments:
-
-Use argparse to deal with command line arguments (instead of argv or optparse)
-
-===========================================
-
-Change owner/group:
-
-chown zas1024 somefile (change owner of somefile to zas1024)
-chown zas1024: somefile (change owner and group of somefile to zas1024)
-sudo chown root: somefile (change owner/group of somefile to root)
-
-Change permissions:
-
-chmod g-wr (remove write/read privileges for group)
-chmod g+w (add write privileges for group)
-chmod u+rwx (add read, write, and execute privileges for user)
-etc.
-
-===========================================
-
-MySQL Things -
-
-Create fulltext (MATCH AGAINST syntax) index:
-CREATE FULLTEXT INDEX __________ ON Table (Column);
-
-If <4 character searches not working:
-Add ft_min_word_len = 2 (to make te minimum characters 2) to config file
-Rebuild relevant tables with fulltext indexes with "REPAIR TABLE _______ QUICK"
-
-===========================================
-
-To delete all .pyc files (for example if there's some issue you're having trouble resolving):
-find . -name '*.pyc' -delete
-http://stackoverflow.com/questions/785519/how-do-i-remove-all-pyc-files-from-a-project
-