diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README.org | 371 |
1 files changed, 2 insertions, 369 deletions
diff --git a/doc/README.org b/doc/README.org index d4dc1027..575fd42e 100644 --- a/doc/README.org +++ b/doc/README.org @@ -1,371 +1,4 @@ #+TITLE: Installing GeneNetwork services -* Table of Contents :TOC: - - [[#introduction][Introduction]] - - [[#check-list][Check list]] - - [[#installing-guix-packages][Installing Guix packages]] - - [[#creating-a-gnu-guix-profile][Creating a GNU Guix profile]] - - [[#running-gn2][Running GN2]] - - [[#run-gn-proxy][Run gn-proxy]] - - [[#run-redis][Run Redis]] - - [[#run-mariadb-server][Run MariaDB server]] - - [[#install-mariadb-with-gnu-guix][Install MariaDB with GNU GUIx]] - - [[#load-the-small-database-in-mysql][Load the small database in MySQL]] - - [[#get-genotype-files][Get genotype files]] - - [[#gn2-dependency-graph][GN2 Dependency Graph]] - - [[#working-with-the-gn2-source-code][Working with the GN2 source code]] - - [[#read-more][Read more]] - - [[#trouble-shooting][Trouble shooting]] - - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]] - - [[#error-can-not-find-directory-homegn2_data-or-can-not-find-directory-homegenotype_filesgenotype][ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype']] - - [[#cant-run-a-module][Can't run a module]] - - [[#rpy2-error-show-now-found][Rpy2 error 'show' now found]] - - [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]] - - [[#notes][NOTES]] - - [[#deploying-gn2-official][Deploying GN2 official]] - -* Introduction - -Large system deployments can get very [[http://genenetwork.org/environments/][complex]]. In this document we -explain the GeneNetwork version 2 (GN2) reproducible deployment system -which is based on GNU Guix (see also [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix -system can be used to install GN with all its files and dependencies. - -The official installation path is from a checked out version of the -main Guix package tree and that of the Genenetwork package -tree. Current supported versions can be found as the SHA values of -'gn-latest' branches of [[https://gitlab.com/genenetwork/guix-bioinformatics][Guix bioinformatics]] and [[https://gitlab.com/genenetwork/guix][GNU Guix]]. - -For a full view of runtime dependencies as defined by GNU Guix, see -an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]]. - -* Check list - -To run GeneNetwork the following services need to function: - -1. [ ] GNU Guix with a guix profile for genenetwork2 -1. [ ] A path to the (static) genotype files -1. [ ] Gn-proxy for authentication -1. [ ] The genenetwork3 service -1. [ ] Redis -1. [ ] Mariadb - -* Installing Guix packages - -Make sure to install GNU Guix using the binary download instructions -on the main website. Follow the instructions on -[[GUIX-Reproducible-from-source.org]] to download pre-built binaries. Note -the download amounts to several GBs of data. Debian-derived distros -may support - -: apt-get install guix - -* Creating a GNU Guix profile - -We run a GNU Guix channel with packages at [[https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics][guix-bioinformatics]]. The README has instructions for hosting a channel, but typically we use the GUIX_PACKAGE_PATH instead. First upgrade to a recent guix with - -: mkdir ~/opt -: guix pull -p ~/opt/guix-pull - -It should upgrade (ignore the locales warnings). You can optionally specify the specific git checkout of guix with - -: guix pull -p ~/opt/guix-pull --commit=f04883d - -which is useful when you need to roll back to an earlier version (sometimes our channel goes out of sync). Next, we install GeneNetwork2 with - -: source ~/opt/guix-pull/etc/profile -: git clone https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.git ~/guix-bioinformatics -: cd ~/guix-bioinformatics -: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics guix package -i genenetwork2 -p ~/opt/genenetwork2 - -you probably also need guix-past (the upstream channel for older packages): - -: git clone https://gitlab.inria.fr/guix-hpc/guix-past.git ~/guix-past -: cd ~/guix-past -: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics:$HOME/guix-past/modules ~/opt/guix-pull/bin/guix package -i genenetwork2 -p ~/opt/genenetwork2 - -ignore the warnings. Guix should install the software without trying -to build everything. If you system insists on building all packages, -try the `--dry-run` switch and fix the [[https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html][substitutes]]. You may add the -`--substitute-urls="http://guix.genenetwork.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org"` switch. - -The guix.genenetwork.org has most of our packages pre-built(!). To use -it on your own machine the public key is - -#+begin_src scheme -(public-key - (ecc - (curve Ed25519) - (q #E50F005E6DA2F85749B9AA62C8E86BB551CE2B541DC578C4DBE613B39EC9E750#))) -#+end_src - -Once we have a GNU Guix profile, a running database (see below) and the file storage, -we should be ready to fire up GeneNetwork: - -* Running GN2 - -Check out the source with git: - -: git clone git@github.com:genenetwork/genenetwork2.git -: cd genenetwork2 - -Run GN2 with above Guix profile - -: export GN2_PROFILE=$HOME/opt/genenetwork2 -: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5012 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 etc/default_settings.py -gunicorn-dev - -the debug and logging switches can be particularly useful when -developing GN2. Location and files are the current ones for Penguin2. - -It may be useful to tunnel the web server to your local browser with -an ssh tunnel: - -If you want to test a service running on the server on a certain -port (say 8202) use - - ssh -L 8202:127.0.0.1:8202 -f -N myname@penguin2.genenetwork.org - -And browse on your local machine to http://localhost:8202/ - -* Run gn-proxy - -GeneNetwork requires a separate gn-proxy server which handles -authorisation and access control. For instructions see the -[[https://github.com/genenetwork/gn-proxy][README]]. Note it may already be running on our servers! - -* Run Redis - -Redis part of GN2 deployment and will be started by the ./bin/genenetwork2 -startup script. - -* Run MariaDB server -** Install MariaDB with GNU GUIx - -These are the steps you can take to install a fresh installation of -mariadb (which comes as part of the GNU Guix genenetwork2 install). - -As root configure the Guix profile - -: . ~/opt/genenetwork2/etc/profile - -and run for example - -#+BEGIN_SRC bash -adduser mariadb && addgroup mariadb -mkdir -p /export2/mariadb/database -chown mariadb.mariadb -R /export2/mariadb/ -mkdir -p /var/run/mysqld -chown mariadb.mariadb /var/run/mysqld -su mariadb -mysql --version - mysql Ver 15.1 Distrib 10.1.45-MariaDB, for Linux (x86_64) using readline 5.1 -mysql_install_db --user=mariadb --datadir=/export2/mariadb/database -mysqld -u mariadb --datadir=/exportdb/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048" -#+END_SRC - -If you want to run as root you may have to set - -: /etc/my.cnf -: [mariadbd] -: user=root - -You also need to set - -: ft_min_word_len = 3 - -To make sure word text searches (shh) work and rebuild the tables if -required. - -To check error output in a file on start-up run with something like - -: mariadbd -u mariadb --console --explicit_defaults_for_timestamp --datadir=/gnu/mariadb --log-error=~/test.log - -Other tips are that Guix installs mariadbd in your profile, so this may work - -: /home/user/.guix-profile/bin/mariadbd -u mariadb --explicit_defaults_for_timestamp --datadir=/gnu/mariadb - -When you get errors like: - -: qlalchemy.exc.IntegrityError: (_mariadb_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint') - -you may need to set - -: set foreign_key_checks=0 - -** Load the small database in MySQL - -At this point we require the underlying distribution to install and -run mysqld (see next section for GNU Guix). Currently we have two databases for deployment, -'db_webqtl_s' is the small testing database containing experiments -from BXD mice and 'db_webqtl_plant' which contains all plant related -material. - -Download one database from - -http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk - -After installation unzip the database binary in the MySQL directory - -#+BEGIN_SRC sh -cd ~/mysql -p7zip -d db_webqtl_s.7z -chown -R mysql:mysql db_webqtl_s/ -chmod 700 db_webqtl_s/ -chmod 660 db_webqtl_s/* -#+END_SRC - -restart MySQL service (mysqld). Login as root - -: mysql_upgrade -u root --force - -: myslq -u root - -and - -: mysql> show databases; -: +--------------------+ -: | Database | -: +--------------------+ -: | information_schema | -: | db_webqtl_s | -: | mysql | -: | performance_schema | -: +--------------------+ - -Set permissions and match password in your settings file below: - -: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'webqtl'; - -You may need to change "localhost" to whatever domain you are -connecting from (mysql will give an error). - -Note that if the mysql connection is not working, try connecting to -the IP address and check server firewall, hosts.allow and mysql IP -configuration (see below). - -Note for the plant database you can rename it to db_webqtl_s, or -change the settings in etc/default_settings.py to match your path. - -* Get genotype files - -The script looks for genotype files. You can find them in -http://ipfs.genenetwork.org/ipfs/QmXQy3DAUWJuYxubLHLkPMNCEVq1oV7844xWG2d1GSPFPL - -#+BEGIN_SRC sh -mkdir -p $HOME/genotype_files -cd $HOME/genotype_files - -#+END_SRC - -* GN2 Dependency Graph - -List of all runtime dependencies for GN2 as installed by GNU Guix. - -https://genenetwork.org/environments/ - -* Working with the GN2 source code - -See [[development.org]]. - -* Read more - -If you want to understand the architecture of GN2 read -[[Architecture.org]]. The rest of this document is mostly on deployment -of GN2. - -* Trouble shooting - -** ImportError: No module named jinja2 - -If you have all the Guix packages installed this error points out that -the environment variables are not set. Copy-paste the paths into your -terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the -information given by guix: - -: guix package --search-paths - -On one system: - -: export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages" -: export R_LIBS_SITE="$HOME/.guix-profile/site-library/" -: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0" - -and perhaps a few more. -** ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype' - -The default settings file looks in your $HOME/gn2_data. Since these -files come with a Guix installation you should take a hint from the -values in the installed version of default_settings.py (see above in -this document). - -You can use the GENENETWORK_FILES switch to set the datadir, for example - -: env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=/gnu/data/gn2_data ./bin/genenetwork2 - -** Can't run a module - -In rare cases, development modules are not brought in with Guix -because no source code is available. This can lead to missing modules -on a running server. Please check with the authors when a module -is missing. -** Rpy2 error 'show' now found - -This error - -: __show = rpy2.rinterface.baseenv.get("show") -: LookupError: 'show' not found - -means that R was updated in your path, and that Rpy2 needs to be -recompiled against this R - don't you love informative messages? - -In our case it means that GN's PYTHONPATH is not in sync with -R_LIBS_SITE. Please check your GNU Guix GN2 installation paths, -you man need to reinstall. Note that this may be the point you -may want to start using profiles (see profile section). - -** Mysql can't connect server through socket ERROR - -The following error - -: sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")') - -means that MySQL is trying to connect locally to a non-existent MySQL -server, something you may see in a container. Typically replicated with something like - -: mysql -h localhost - -try to connect over the network interface instead, e.g. - -: mysql -h 127.0.0.1 - -if that works run genenetwork after setting SQL_URI to something like - -: export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s - -* NOTES - -** Deploying GN2 official - -Let's see how fast we can deploy a second copy of GN2. - -- [ ] Base install - + [ ] First install a Debian server with GNU Guix on board - + [ ] Get Guix build going - - [ ] Build the correct version of Guix - - [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics - - [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable - + [ ] Create a gn2 user and home with space - + [ ] Install redis - - [ ] add to systemd - - [ ] update redis.cnf - - [ ] update database - + [ ] Install mariadb (currently debian mariadb-server) - - [ ] add to systemd - - [ ] system stop mysql - - [ ] update mysql.cnf - - [ ] update database (see gn-services/services/mariadb.md) - - [ ] check tables - + [ ] run gn2 - + [ ] update nginx - + [ ] install genenetwork3 - - [ ] add to systemd +This information has moved to https://issues.genenetwork.org/topics/deployment +and https://issues.genenetwork.org/topics/installation. |