1 files changed, 347 insertions, 0 deletions
diff --git a/topics/installation.gmi b/topics/installation.gmi
new file mode 100644
index 0000000..74bfe91
--- /dev/null
+++ b/topics/installation.gmi
@@ -0,0 +1,347 @@
+# Installation
+
+* Introduction
+
+Large system deployments can get very [[http://genenetwork.org/environments/][complex]]. In this document we
+explain the GeneNetwork version 2 (GN2) reproducible deployment system
+which is based on GNU Guix (see also [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix
+system can be used to install GN with all its files and dependencies.
+
+The official installation path is from a checked out version of the
+main Guix package tree and that of the Genenetwork package
+tree. Current supported versions can be found as the SHA values of
+'gn-latest' branches of [[https://gitlab.com/genenetwork/guix-bioinformatics][Guix bioinformatics]] and [[https://gitlab.com/genenetwork/guix][GNU Guix]].
+
+For a full view of runtime dependencies as defined by GNU Guix, see
+an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]].
+
+* Check list
+
+To run GeneNetwork the following services need to function:
+
+1. [ ] GNU Guix with a guix profile for genenetwork2
+1. [ ] A path to the (static) genotype files
+1. [ ] Gn-proxy for authentication
+1. [ ] The genenetwork3 service
+1. [ ] Redis
+1. [ ] Mariadb
+
+* Installing Guix packages
+
+Make sure to install GNU Guix using the binary download instructions
+on the main website. Follow the instructions on
+[[GUIX-Reproducible-from-source.org]] to download pre-built binaries. Note
+the download amounts to several GBs of data. Debian-derived distros
+may support
+
+: apt-get install guix
+
+* Creating a GNU Guix profile
+
+We run a GNU Guix channel with packages at [[https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics][guix-bioinformatics]]. The README has instructions for hosting a channel, but typically we use the GUIX_PACKAGE_PATH instead. First upgrade to a recent guix with
+
+: mkdir ~/opt
+: guix pull -p ~/opt/guix-pull
+
+It should upgrade (ignore the locales warnings).  You can optionally specify the specific git checkout of guix with
+
+: guix pull -p ~/opt/guix-pull --commit=f04883d
+
+which is useful when you need to roll back to an earlier version (sometimes our channel goes out of sync). Next, we install GeneNetwork2 with
+
+: source ~/opt/guix-pull/etc/profile
+: git clone https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.git ~/guix-bioinformatics
+: cd ~/guix-bioinformatics
+: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics guix package -i genenetwork2 -p ~/opt/genenetwork2
+
+you probably also need guix-past (the upstream channel for older packages):
+
+: git clone https://gitlab.inria.fr/guix-hpc/guix-past.git ~/guix-past
+: cd ~/guix-past
+: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics:$HOME/guix-past/modules ~/opt/guix-pull/bin/guix package -i genenetwork2 -p ~/opt/genenetwork2
+
+ignore the warnings. Guix should install the software without trying
+to build everything. If you system insists on building all packages,
+try the `--dry-run` switch and fix the [[https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html][substitutes]]. You may add the
+`--substitute-urls="http://guix.genenetwork.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org"` switch.
+
+The guix.genenetwork.org has most of our packages pre-built(!). To use
+it on your own machine the public key is
+
+#+begin_src scheme
+(public-key
+ (ecc
+  (curve Ed25519)
+  (q #E50F005E6DA2F85749B9AA62C8E86BB551CE2B541DC578C4DBE613B39EC9E750#)))
+#+end_src
+
+Once we have a GNU Guix profile, a running database (see below) and the file storage,
+we should be ready to fire up GeneNetwork:
+
+* Running GN2
+
+Check out the source with git:
+
+: git clone git@github.com:genenetwork/genenetwork2.git
+: cd genenetwork2
+
+Run GN2 with above Guix profile
+
+: export GN2_PROFILE=$HOME/opt/genenetwork2
+: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5012 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 etc/default_settings.py -gunicorn-dev
+
+the debug and logging switches can be particularly useful when
+developing GN2. Location and files are the current ones for Penguin2.
+
+It may be useful to tunnel the web server to your local browser with
+an ssh tunnel:
+
+If you want to test a service running on the server on a certain
+port (say 8202) use
+
+  ssh -L 8202:127.0.0.1:8202 -f -N myname@penguin2.genenetwork.org
+
+And browse on your local machine to http://localhost:8202/
+
+* Run gn-proxy
+
+GeneNetwork requires a separate gn-proxy server which handles
+authorisation and access control. For instructions see the
+[[https://github.com/genenetwork/gn-proxy][README]]. Note it may already be running on our servers!
+
+* Run Redis
+
+Redis part of GN2 deployment and will be started by the ./bin/genenetwork2
+startup script.
+
+* Run MariaDB server
+** Install MariaDB with GNU GUIx
+
+These are the steps you can take to install a fresh installation of
+mariadb (which comes as part of the GNU Guix genenetwork2 install).
+
+As root configure the Guix profile
+
+: . ~/opt/genenetwork2/etc/profile
+
+and run for example
+
+#+BEGIN_SRC bash
+adduser mariadb && addgroup mariadb
+mkdir -p /export2/mariadb/database
+chown mariadb.mariadb -R /export2/mariadb/
+mkdir -p /var/run/mysqld
+chown mariadb.mariadb /var/run/mysqld
+su mariadb
+mysql --version
+  mysql  Ver 15.1 Distrib 10.1.45-MariaDB, for Linux (x86_64) using readline 5.1
+mysql_install_db --user=mariadb --datadir=/export2/mariadb/database
+mysqld -u mariadb --datadir=/exportdb/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048"
+#+END_SRC
+
+If you want to run as root you may have to set
+
+: /etc/my.cnf
+: [mariadbd]
+: user=root
+
+You also need to set
+
+: ft_min_word_len = 3
+
+To make sure word text searches (shh) work and rebuild the tables if
+required.
+
+To check error output in a file on start-up run with something like
+
+: mariadbd -u mariadb --console  --explicit_defaults_for_timestamp  --datadir=/gnu/mariadb --log-error=~/test.log
+
+Other tips are that Guix installs mariadbd in your profile, so this may work
+
+: /home/user/.guix-profile/bin/mariadbd -u mariadb --explicit_defaults_for_timestamp  --datadir=/gnu/mariadb
+
+When you get errors like:
+
+: qlalchemy.exc.IntegrityError: (_mariadb_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint')
+
+you may need to set
+
+: set foreign_key_checks=0
+
+** Load the small database in MySQL
+
+At this point we require the underlying distribution to install and
+run mysqld (see next section for GNU Guix). Currently we have two databases for deployment,
+'db_webqtl_s' is the small testing database containing experiments
+from BXD mice and 'db_webqtl_plant' which contains all plant related
+material.
+
+Download one database from
+
+http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk
+
+After installation unzip the database binary in the MySQL directory
+
+#+BEGIN_SRC sh
+cd ~/mysql
+p7zip -d db_webqtl_s.7z
+chown -R mysql:mysql db_webqtl_s/
+chmod 700 db_webqtl_s/
+chmod 660 db_webqtl_s/*
+#+END_SRC
+
+restart MySQL service (mysqld). Login as root
+
+: mysql_upgrade -u root --force
+
+: myslq -u root
+
+and
+
+: mysql> show databases;
+: +--------------------+
+: | Database           |
+: +--------------------+
+: | information_schema |
+: | db_webqtl_s        |
+: | mysql              |
+: | performance_schema |
+: +--------------------+
+
+Set permissions and match password in your settings file below:
+
+: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'webqtl';
+
+You may need to change "localhost" to whatever domain you are
+connecting from (mysql will give an error).
+
+Note that if the mysql connection is not working, try connecting to
+the IP address and check server firewall, hosts.allow and mysql IP
+configuration (see below).
+
+Note for the plant database you can rename it to db_webqtl_s, or
+change the settings in etc/default_settings.py to match your path.
+
+* Get genotype files
+
+The script looks for genotype files. You can find them in
+http://ipfs.genenetwork.org/ipfs/QmXQy3DAUWJuYxubLHLkPMNCEVq1oV7844xWG2d1GSPFPL
+
+#+BEGIN_SRC sh
+mkdir -p $HOME/genotype_files
+cd $HOME/genotype_files
+
+#+END_SRC
+
+* GN2 Dependency Graph
+
+List of all runtime dependencies for GN2 as installed by GNU Guix.
+
+https://genenetwork.org/environments/
+
+* Working with the GN2 source code
+
+See [[development.org]].
+
+* Read more
+
+If you want to understand the architecture of GN2 read
+[[Architecture.org]].  The rest of this document is mostly on deployment
+of GN2.
+
+* Trouble shooting
+
+** ImportError: No module named jinja2
+
+If you have all the Guix packages installed this error points out that
+the environment variables are not set. Copy-paste the paths into your
+terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
+information given by guix:
+
+: guix package --search-paths
+
+On one system:
+
+: export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages"
+: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
+: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
+
+and perhaps a few more.
+** ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype'
+
+The default settings file looks in your $HOME/gn2_data. Since these
+files come with a Guix installation you should take a hint from the
+values in the installed version of default_settings.py (see above in
+this document).
+
+You can use the GENENETWORK_FILES switch to set the datadir, for example
+
+: env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=/gnu/data/gn2_data ./bin/genenetwork2
+
+** Can't run a module
+
+In rare cases, development modules are not brought in with Guix
+because no source code is available. This can lead to missing modules
+on a running server. Please check with the authors when a module
+is missing.
+** Rpy2 error 'show' now found
+
+This error
+
+: __show = rpy2.rinterface.baseenv.get("show")
+: LookupError: 'show' not found
+
+means that R was updated in your path, and that Rpy2 needs to be
+recompiled against this R - don't you love informative messages?
+
+In our case it means that GN's PYTHONPATH is not in sync with
+R_LIBS_SITE. Please check your GNU Guix GN2 installation paths,
+you man need to reinstall. Note that this may be the point you
+may want to start using profiles (see profile section).
+
+** Mysql can't connect server through socket ERROR
+
+The following error
+
+: sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")')
+
+means that MySQL is trying to connect locally to a non-existent MySQL
+server, something you may see in a container. Typically replicated with something like
+
+: mysql -h localhost
+
+try to connect over the network interface instead, e.g.
+
+: mysql -h 127.0.0.1
+
+if that works run genenetwork after setting SQL_URI to something like
+
+: export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s
+
+* NOTES
+
+** Deploying GN2 official
+
+Let's see how fast we can deploy a second copy of GN2.
+
+- [ ] Base install
+  + [ ] First install a Debian server with GNU Guix on board
+  + [ ] Get Guix build going
+    - [ ] Build the correct version of Guix
+    - [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics
+    - [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable
+  + [ ] Create a gn2 user and home with space
+  + [ ] Install redis
+    - [ ] add to systemd
+    - [ ] update redis.cnf
+    - [ ] update database
+  + [ ] Install mariadb (currently debian mariadb-server)
+    - [ ] add to systemd
+    - [ ] system stop mysql
+    - [ ] update mysql.cnf
+    - [ ] update database (see gn-services/services/mariadb.md)
+    - [ ] check tables
+  + [ ] run gn2
+  + [ ] update nginx
+  + [ ] install genenetwork3
+    - [ ] add to systemd