From 2e4ca1f6732d14c0ce0d953f1c08ecb000a94747 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Thu, 30 Apr 2020 13:44:52 -0500 Subject: GN2 install checklist --- doc/README.org | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) (limited to 'doc') diff --git a/doc/README.org b/doc/README.org index 828ed2cd..b0f1c56b 100644 --- a/doc/README.org +++ b/doc/README.org @@ -658,24 +658,24 @@ The following derivations would be built: Let's see how fast we can deploy a second copy of GN2. -- [-] Base install - + [X] First install a Debian server with GNU Guix on board - + [X] Get Guix build going - - [X] Build the correct version of Guix - - [X] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics - - [X] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable - + [X] Create a gn2 user and home with space - + [X] Install redis (currently debian) - - [X] add to systemd - - [X] update redis.cnf - - [X] update database - + [X] Install mariadb (currently debian mariadb-server) - - [X] add to systemd - - [X] system stop mysql - - [X] update mysql.cnf - - [X] update database (see gn-services/services/mariadb.md) - - [X] check tables +- [ ] Base install + + [ ] First install a Debian server with GNU Guix on board + + [ ] Get Guix build going + - [ ] Build the correct version of Guix + - [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics + - [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable + + [ ] Create a gn2 user and home with space + + [ ] Install redis (currently debian) + - [ ] add to systemd + - [ ] update redis.cnf + - [ ] update database + + [ ] Install mariadb (currently debian mariadb-server) + - [ ] add to systemd + - [ ] system stop mysql + - [ ] update mysql.cnf + - [ ] update database (see gn-services/services/mariadb.md) + - [ ] check tables + [ ] run gn2 (rust-qtlreaper not working) - + [X] update nginx + + [ ] update nginx + [ ] install genenetwork3 - [ ] add to systemd -- cgit v1.2.3 From 918dcce5f42fcbe9327eb30bac06a69000a1fc8b Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Sat, 25 Jul 2020 16:47:24 +0100 Subject: Updating docs and show how to include a local Python module --- README.md | 21 +-- bin/genenetwork2 | 4 +- doc/README.org | 482 ++++++---------------------------------------------- doc/development.org | 55 ++++++ 4 files changed, 114 insertions(+), 448 deletions(-) (limited to 'doc') diff --git a/README.md b/README.md index 47519118..a2a4625a 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,8 @@ developers). See the [installation docs](doc/README.org). ## Run -Once installed GN2 can be run online through a browser interface +Once having installed GN2 it can be run through a browser +interface ```sh genenetwork2 @@ -35,18 +36,12 @@ For full examples (you may need to set a number of environment variables), including running scripts and a Python REPL, also see the startup script [./bin/genenetwork2](https://github.com/genenetwork/genenetwork2/blob/testing/bin/genenetwork2). -Also Mysql and Elasticsearch need to be running, see +Also mariadb and redis need to be running, see [INSTALL](./doc/README.org). ## Testing -We are building 'Mechanical Rob' automated testing using Python -[requests](https://github.com/genenetwork/genenetwork2/tree/master/test/lib) -which can be run with something like - -```sh -env GN2_PROFILE=~/opt/gn-latest ./bin/genenetwork2 ./etc/default_settings.py -c ../test/requests/test-website.py -a http://localhost:5003 -``` +We are building 'Mechanical Rob' automated testing using Python. The GN2_PROFILE is the Guix profile that contains all dependencies. The ./bin/genenetwork2 script sets up the environment @@ -73,10 +68,10 @@ Contribute to GN2 source code by forking the [github repository](https://github.com/genenetwork/genenetwork2/) with git and sending us pull requests. -For development GN2 has a -[mailing list](http://listserv.uthsc.edu/mailman/listinfo/genenetwork-dev) -and an active IRC channel #genenetwork on freenode.net with a -[web interface](http://webchat.freenode.net/). +For development GN2 has a [mailing +list](http://listserv.uthsc.edu/mailman/listinfo/genenetwork-dev) and +an active IRC channel #genenetwork on freenode.net with a [web +interface](http://webchat.freenode.net/). ## License diff --git a/bin/genenetwork2 b/bin/genenetwork2 index 3ae08e0a..dd6db7d6 100755 --- a/bin/genenetwork2 +++ b/bin/genenetwork2 @@ -121,7 +121,7 @@ if [ -z $GN2_PROFILE ]; then read -p "PRESS [ENTER] TO CONTINUE..." else export PATH=$GN2_PROFILE/bin:$PATH - export PYTHONPATH="$GN2_PROFILE/lib/python2.7/site-packages" # never inject another PYTHONPATH!! + export PYTHONPATH="$PYTHON_GN_PATH:$GN2_PROFILE/lib/python2.7/site-packages" # never inject another PYTHONPATH!! export R_LIBS_SITE=$GN2_PROFILE/site-library export JS_GUIX_PATH=$GN2_PROFILE/share/genenetwork2/javascript export GUIX_GTK3_PATH="$GN2_PROFILE/lib/gtk-3.0" @@ -154,7 +154,7 @@ if [ ! -d $R_LIBS_SITE ] ; then fi # We may change this one: -export PYTHONPATH=$GN2_BASE_DIR/wqflask:$PYTHONPATH +export PYTHONPATH=$PYTHON_GN_PATH:$GN2_BASE_DIR/wqflask:$PYTHONPATH # Our UNIX TMPDIR defaults to /tmp - change this on a shared server if [ -z $TMPDIR ]; then diff --git a/doc/README.org b/doc/README.org index c1b1c2b6..46df03c7 100644 --- a/doc/README.org +++ b/doc/README.org @@ -4,9 +4,12 @@ - [[#introduction][Introduction]] - [[#install][Install]] - [[#running-gn2][Running GN2]] + - [[#run-gn-proxy][Run gn-proxy]] + - [[#run-redis][Run Redis]] - [[#run-mariadb-server][Run MariaDB server]] - [[#install-mariadb-with-gnu-guix][Install MariaDB with GNU GUIx]] - [[#load-the-small-database-in-mysql][Load the small database in MySQL]] + - [[#get-genotype-files][Get genotype files]] - [[#gn2-dependency-graph][GN2 Dependency Graph]] - [[#working-with-the-gn2-source-code][Working with the GN2 source code]] - [[#read-more][Read more]] @@ -16,7 +19,6 @@ - [[#cant-run-a-module][Can't run a module]] - [[#rpy2-error-show-now-found][Rpy2 error 'show' now found]] - [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]] - - [[#irc-session][IRC session]] - [[#notes][NOTES]] - [[#deploying-gn2-official][Deploying GN2 official]] @@ -57,28 +59,39 @@ or you can set environment variables to override individual parameters, e.g. the debug and logging switches can be particularly useful when developing GN2. -* Running Redis +* Run gn-proxy -Install redis. Make sure you add the setting: +GeneNetwork requires a separate gn-proxy server which handles +authorisation and access control. For instructions see the [[https://github.com/genenetwork/gn-proxy][README]]. -: appendonly yes +* Run Redis + +Redis part of GN2 deployment and will be started by the ./bin/genenetwork2 +startup script. * Run MariaDB server ** Install MariaDB with GNU GUIx -/Note: we moved to MariaDB/ - These are the steps you can take to install a fresh installation of mariadb (which comes as part of the GNU Guix genenetwork2 install). -As root configure and run +As root configure the Guix profile + +: . ~/opt/genenetwork2/etc/profile + +and run for example #+BEGIN_SRC bash adduser mariadb && addgroup mariadb -mysqld --datadir=/home/mariadb/database --initialize-insecure -mkdir -p /var/run/mariadbd -chown mariadb.mariadb /var/run/mariadbd -mysqld -u mariadb --datadir=/home/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048" +mkdir -p /export2/mariadb/database +chown mariadb.mariadb -R /export2/mariadb/ +mkdir -p /var/run/mysqld +chown mariadb.mariadb /var/run/mysqld +su mariadb +mysql --version + mysql Ver 15.1 Distrib 10.1.45-MariaDB, for Linux (x86_64) using readline 5.1 +mysql_install_db --user=mariadb --datadir=/export2/mariadb/database +mysqld -u mariadb --datadir=/exportdb/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048" #+END_SRC If you want to run as root you may have to set @@ -120,21 +133,22 @@ material. Download one database from -[[http://files.genenetwork.org/raw_database/]] +http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk -[[https://s3.amazonaws.com/genenetwork2/db_webqtl_s.zip]] +After installation unzip the database binary in the MySQL directory -Check the md5sum. - -After installation inflate the database binary in the MySQL directory - -: cd ~/mysql -: chown -R mysql:mysql db_webqtl_s/ -: chmod 700 db_webqtl_s/ -: chmod 660 db_webqtl_s/* +#+BEGIN_SRC sh +cd ~/mysql +p7zip -d db_webqtl_s.7z +chown -R mysql:mysql db_webqtl_s/ +chmod 700 db_webqtl_s/ +chmod 660 db_webqtl_s/* +#+END_SRC restart MySQL service (mysqld). Login as root +: mysql_upgrade -u root --force + : myslq -u root and @@ -151,7 +165,7 @@ and Set permissions and match password in your settings file below: -: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'mysql_password'; +: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'webqtl'; You may need to change "localhost" to whatever domain you are connecting from (mysql will give an error). @@ -163,6 +177,17 @@ configuration (see below). Note for the plant database you can rename it to db_webqtl_s, or change the settings in etc/default_settings.py to match your path. +* Get genotype files + +The script looks for genotype files. You can find them in +http://ipfs.genenetwork.org/ipfs/QmXQy3DAUWJuYxubLHLkPMNCEVq1oV7844xWG2d1GSPFPL + +#+BEGIN_SRC sh +mkdir -p $HOME/genotype_files +cd $HOME/genotype_files + +#+END_SRC + * GN2 Dependency Graph Graph of all runtime dependencies as installed by GNU Guix. @@ -249,415 +274,6 @@ if that works run genenetwork after setting SQL_URI to something like : export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s - -* IRC session - -Here an IRC session where we installed GN2 from scratch using GNU Guix -and a download of the test database. - -#+begin_src - time to get binary install sorted :) [07:03] - Guix is designed for distributed installation servers - we have one on guix.genenetwork.org - it contains all the prebuild packages - for GN - okay [07:04] - let's step back however [07:05] - I presume the environment is set with all guix package --search-paths - right? - yep - set to the ones in ~/.guix-profile/ - good, and you are in gn-deploy-guix repo [07:06] - yep [07:07] - git log shows - -Author: David Thompson -Date: Sun Mar 27 21:20:19 2016 -0400 - - yes - env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix - package -A genenetwork2 [07:08] - shows - -genenetwork2 2.0-a8fcff4 out ../guix-bioinformatics/gn/packages/genenetwork.scm:144:2 -genenetwork2-database-small 1.0 out ../guix-bioinformatics/gn/packages/genenetwork.scm:270:4 -genenetwork2-files-small 1.0 out ../guix-bioinformatics/gn/packages/genenetwork.scm:228:4 - - yeah [07:09] - OK, we are in sync. This means we should be able to install the exact - same software - I need to start up my guix daemon - I usually run it in a screen - screen -S guix-daemon - hah, I don't have screen installed yet [07:11] - comes with guix ;) [07:12] - no worries, you can run it any way you want - $HOME/.guix-profile/bin/guix-daemon --build-users-group=guixbuild - then something's weird, because it says I don't have it - oh, you need to install it first [07:13] - guix package -A screen - screen 4.3.1 out gnu/packages/screen.scm:34:2 - but you can skip this install, for now - alright [07:14] - env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix - package -i genenetwork2 --dry-run - substitute: updating list of substitutes from - 'https://mirror.hydra.gnu.org'... 79.1% - you see that? - followed by [07:15] -substitute: updating list of substitutes from -'https://hydra.gnu.org'... 100.0% -The following derivations would be built: - /gnu/store/rk7nw0rjqqsha958m649wrykadx6mmhl-profile.drv - -/gnu/store/7b0qjybvfx8syzvfs7p5rdablwhbkbvs-module-import-compiled.drv - /gnu/store/cy9zahbbf23d3cqyy404lk9f50z192kp-module-import.drv - /gnu/store/ibdn603i8grf0jziy5gjsly34wx82lmk-gtk-icon-themes.drv - - which should have the same HASH values /gnu/store/7b0qjybvf... etc. - [07:16] - profile has a different hash - but the next ones? - they're the same - not sure why profile differs. Do you see the contact with - mirror.hydra.org? [07:17] - yeah - OK, that means you set the key correctly for that one :) - alright we are at the same state now. You can see most packages need - to be rebuild because they are no longer cached as binaries on hydra - [07:18] - things move fast... - hehe - let me also do the same on my laptop - which I have staged before - [07:19] - btw, to set the path I often do [07:20] - export - PATH="/home/wrk/.guix-profile/bin:/home/wrk/.guix-profile/sbin":$PATH - to keep things like 'screen' from Debian - Once past building guix itself that is normally OK [07:21] - ah, okay - will do that - the guix build requires certain versions of tools, so you don't want - to mix foreign tools in [07:23] - makes sense [07:24] - On my laptop I am trying the main updating list of substitutes from - 'http://hydra.gnu.org'... 10.5% [07:27] - it is a bit slow, but let's see if there is a difference with the - mirror - you can see there are two servers here. Actually with recent daemons, - if the mirror fails it will try the main server [07:28] - I documented the use of a caching server here [07:29] - https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org - this is exactly what we are doing now - alrighty [07:35] - To see if a remote server has a guix server running it should respond - [07:36] - lynx http://guix.genenetwork.org:8080 --dump - Resource not found: / - - you see that? - yes [07:37] - good. The main hydra server is too slow. So on my laptop I forced - using the mirror with [07:38] - env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix - package -i genenetwork2 --dry-run - --substitute-urls="http://mirror.hydra.gnu.org" - - the list looks the same to me [07:40] - me too - note that some packages will be built and some downloaded, right? - [07:41] - yes - atlas is actually a binary on my system [07:43] - I mean in that list - so, it should not build. Same as yours? - yeah, atlas and r-gtable are the ones to be downloaded - You should not have seen that error ;) - we should try and install it this way, try [07:44] - env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix - package -i genenetwork2 --cores=4 --max-jobs=4 --keep-going [07:46] - set CPUs and max-jobs to something sensible - Does your VM have multiple cores? - note you can always press Ctrl-C during install - it doesn't, I'll reboot it and give it another core [07:47] - Hey [07:48] - I'm here - Will be stepping away for some breakfast - Can you do the same as us - Can you see the irc log - Alright - Yes, I can - Please email me a copy in five minutes - user01: so when I use the GN server [07:56] - env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix - package -i genenetwork2 --dry-run - --substitute-urls=http://guix.genenetwork.org:8080 - I don't need to build anything [07:57] - (this won't work for you, yet) - to get it to work you need to 'trust' it [07:58] - but, first get the build going - I'll have a coffee while you and get building - yeah it's doing its thing now [08:01] - cool [08:02] - in a separate terminal you can try and install with the gn mirror - [08:05] - I'll send you the public key and you can paste it as said - https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org - [08:06] - alright - should be in the E-mail [08:09] - getting it working it kinda nasty since the server gives no feedback - it works when you see no more in the build list ;) [08:11] - btw, you can install software in parallel. Guix does that. - even the same packages - so keep building ;) - try and do this with Debian... - coffee for me [08:12] - the first build failed [08:15] - OK, Dennis fixed that one yesterday [08:27] - the problem is that sometime source tarballs disappear [08:28] - R is notorious for that - haha, that's inconvenient.. - well, it is good that Guix catches them - but we do not cache sources - binaries are cached - to some degree - so we don't have to rebuild - those [08:29] - time to use the guix cache at guix.genenetwork.org - try and install the key (it is in the E-mail) - and see what this lists [08:31] - env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix - package -i genenetwork2 - --substitute-urls=http://guix.genenetwork.org --dry-run - should be all binary installs - it's not.. [08:32] - if I remove --substitute-urls, the list changes, does that mean I - have the key set up correctly at least? [08:33] - dunno [08:35] - how many packages does it want to build? - should be zero - four - Ah, that is OK - those are default profile things - genenetwork2 is among the ones to be downloaded so [08:36] - remove --dry-run - yeah, good sign :) - we'll still hit a snag, but run it - should be fast - doing it [08:37] - it worked! [08:38] - I think [08:39] - heh [08:40] - you mean it is finished? - yep - type genenetwork2 - complains about not being able to connect to the database [08:41] - last snag :) - no database - well, we succeeded in installing a same-byte install of a very - complex system :) [08:42] - (always take time to congratulate yourself) - now we need to install mysql - hehe :) - this can be done throug guix or through debian [08:43] - the latter is a bit easier here, so let's do that - fun note: you can mix debian and guix - Follow instructions on [08:44] - - https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-mysql-server - apt-get install mysql-common [08:45] - may do it - You can also install with guix, but I need to document that - btw your internet must be fast :) [08:46] - hehe it is ;) - when the database is installed [08:48] - be sure to set the password as instructed [08:50] - when mysql is set the genenetwork2 command should fire up the web - server on localhost:5003 [08:58] - btw my internet is way slower :) [09:00] - I'm back [09:04] - fixed router firmware upgrade problem - unbricking - tssk [09:07] - I'll never leave routers to update themselves again [09:08] - self-brick highway - Resuming [09:09] - auto-updates are evil - always switch them off - user02: can you install genenetwork like user has done? [09:10] - pretty well documented here now :) - Yes I can [09:11] - Already installed key - user02: you are getting binary packages only now? [09:13] - That's the sanest way to go now - seriously - everything should be pre-built from guix.genenetwork.org - you are downloading? - yes [09:15] - cool. Maybe an idea to set up a server - for your own use - Stuck at downloading preprocesscore - should not [09:24] - what does env GUIX_PACKAGE_PATH=../guix-bioinformatics/ - ./pre-inst-env guix package -i genenetwork2 - --substitute-urls="http://guix.genenetwork.org" --dry-run - [09:25] - say for r-prepocesscore - download or build? - mine says download [09:26] - it only lists the derivatives to be built - nothing else happens [09:27] - OK, so there is a problem - your key may not be working - everything should be listed as 'to be download' [09:28] - Hmm - Ah - I know where I messed up - where? - I did add the key - However - (I am documenting) - I did not tell guix to trust it - yes - and there is another potential problem - Remember the documentation on installing guix? - You have to tell guix to trust the default key [09:29] - Right? - So in this case - read the IRC log - That step is mandatory - user01: how are you doing? - user02: - https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org#using-gnu-guix-archive - [09:30] - a little bit left on the db download - user02: you should see no more building - user02: another issue may be that you updated r-preprocesscore - package in guix-buinformatics [09:32] - all downstream packages will want to rebuild - no, not really - It's not even installed - checkout a branch of the the old version - make sure we are in synch - should be at - /gnu/store/y1f3r2xs3fhyadd46nd2aqbr2p9qv2ra-r-biocpreprocesscore-1.32.0 - [09:33] - - pjotrp: Possibly we should use the archive utility of Guix to do - deployment to avoid such out-of-sync differences :) [09:34] - maybe. I did not get archive to update profiles properly [09:37] - Also it is good that they get to understand guix - this way - carved in stone, eh [09:38] - Yeah, all good [09:39] - My mistake was skipping the guix archive part - Can we begin with the install? - It's telling me of derivatives that will be downloaded [09:40] - So we're good - Here goes - yeeha [09:42] - pjotrp, where is this guix.genenetwork.org located at? - Tennessee - It's...it's....sloooooooowwwwwwwwwwwwww - not from Europe - is it downloading at all? - It should be extended - Yes...like at 100KB/s [09:43] - tear-jerker - Verizon problems - who's the host? - I am getting 500Kb/s - UT - Guix's servers can run off more than one server, right? - I'd like to host that particular server here - For speed - yes - Sooner or later - It will be a necessity [09:45] - exactly what I am doing - this is our server - guix.genenetwork.org:8080 - All done installing [09:46] - what? - Now the databases - what do you mean by slow exactly? - Yes, it's installed - can you run genenetwork2 - setting variables - If I try running it now, it will fail as I don't have the DBs [09:47] - cool - you had a lot of prebuilt packages already - OK, follow the instructions I wrote above - now everything seems to be working for me :) - OK - user01: excellent! - you see a webserver? - yep, can connect to localhost:5003 [09:48] - So now you are running a guix copy of GN2 - you can see where it lives with `which genenetwork2` or ls -l - ~/.guix-profile/bin/genenetwork2 [09:49] - - /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/genenetwork2 - it is a script - written by guix, open it [09:50] - inside it points to paths and our script at - - /gnu/store/1kma5xszvzsvmbb4k699h7gvdncw901i-genenetwork2-2.0-a8fcff4/bin/.genenetwork2-real - if you open that you can see how the webserver is started [09:51] - next step is to run a recent version of GN2 - okay [09:52] - See - https://github.com/genenetwork/genenetwork2/tree/staging/doc#run-your-own-copy-of-gn2 - but do not checkout that genetwork2_diet - we reverted to the main tree - clone git@github.com:genenetwork/genenetwork2.git [09:53] - instead and checkout the staging branch - that is effectively my branch [09:54] - when that is done you should be able to fire up the webserver from - there [09:55] - using ./bin/genenetwork2 - now installing DBs - Downloading - annoyingly the source tree is ~700Mb [09:56] - Can it also be done by installing the guix package - genenetwork2-database-small? - I changed it in the diet version to 8Mb, but I had to revert - I need to make my VM bigger... - user02: not ready [09:57] - ok - user01: sorry - user01: you could mount a local dir inside the VM for development - that would allow you to use MAC tools for editing - just an idea - yeah, I figure I'll do something like that - do you use emacs? [09:58] - yep - that can also run on remote files over ssh - that's an alternative - kudos for using emacs :), wdyt user03 - 79 minutes to go downloading the db - user02: sorry about that [09:59] - it is 2GB - user, you can also mount the directory via sshfs - Mac OSX runs OpenSSH - user02: sopa - You can therefore mount a directory outside the VM to the VM via - sshfs [10:00] - yes, 3 options now - That way, you can set up a VM only for it's logic - Apps + the OS it runs [10:01] - For data, let it reside on physical host accessible via sshfs - Use this Arch wiki reference: - https://wiki.archlinux.org/index.php/SSHFS - I edited that last somewhere in 2015, may have been updated since - then - alright, cool! [10:04] - user01: you are almost done [10:06] - I wrote an elixir package for guix :) - env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix - package -A elixir - --substitute-urls="http://guix.genenetwork.org" [10:08] - elixir 1.2.3 out - ../guix-bioinformatics/gn/packages/elixir.scm:31:2 - - I am building it on guix.genenetwork.org right now [10:09] - nice [10:10] -#+end_src - * NOTES ** Deploying GN2 official @@ -671,7 +287,7 @@ Let's see how fast we can deploy a second copy of GN2. - [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics - [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable + [ ] Create a gn2 user and home with space - + [ ] Install redis (currently debian) + + [ ] Install redis - [ ] add to systemd - [ ] update redis.cnf - [ ] update database @@ -681,7 +297,7 @@ Let's see how fast we can deploy a second copy of GN2. - [ ] update mysql.cnf - [ ] update database (see gn-services/services/mariadb.md) - [ ] check tables - + [ ] run gn2 (rust-qtlreaper not working) + + [ ] run gn2 + [ ] update nginx + [ ] install genenetwork3 - [ ] add to systemd diff --git a/doc/development.org b/doc/development.org index 5e6e318b..e65ccd58 100644 --- a/doc/development.org +++ b/doc/development.org @@ -41,3 +41,58 @@ JS_GN_PATH (3) is for development purposes. By default is is set to $HOME/genenetwork/javascript. Say you are working on an updated version of a JS module not yet in (1) you can simply check out that module in that path and it should show up. + +* Python modules + +Python modules are automatically found in the Guix profile. + +For development purposes it may be useful to try some Python package. +Obviously this is only a temporary measure and when you decide to +include the package it should be packaged in [[http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics][our GNU Guix software +stack]]! + +To add packages you need to make sure the correct Python is used (currently +Python 2.7) to install a package. E.g.. + +#+BEGIN_SRC sh +python --version + Python 2.7.16 +pip --version + pip 18.1 from /usr/lib/python2.7/dist-packages/pip (python 2.7) +#+END_SRC + +You can install a Python package locally with pip, e.g. + +#+BEGIN_SRC sh +pip install hjson +#+END_SRC + +This installed in ~$HOME/.local/lib/python2.7/site-packages~. To add +the search path for GeneNetwork use the environment variable + +#+BEGIN_SRC sh +export PYTHON_GN_PATH=$HOME/.local/lib/python2.7/site-packages +#+END_SRC + +Now you should be able to do + +#+BEGIN_SRC python +import hjson +#+END_SRC + +In fact you can kick off a Python shell with something like + +#+BEGIN_SRC python +env SERVER_PORT=5013 WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG \ + SQL_URI=mysql://gn2:webqtl@localhost/db_webqtl_s \ + GN2_PROFILE=~/opt/genenetwork2 \ + ./bin/genenetwork2 ./etc/default_settings.py -c +Python 2.7.17 (default, Jan 1 1970, 00:00:01) +[GCC 7.5.0] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> import hjson +#+END_SRC + +It should now also work in GN2. + +* TODO External tools -- cgit v1.2.3 From e7c02e570635f6aebe2e2596bfb2e6231a7569e8 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Mon, 27 Jul 2020 00:31:12 +0300 Subject: Document how to run tests * doc/testing.org: - Update how Mechanical Robs works. Python replaced Ruby as the testing tool - Add section that describes how to run unit tests --- doc/testing.org | 66 +++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 45 insertions(+), 21 deletions(-) (limited to 'doc') diff --git a/doc/testing.org b/doc/testing.org index 1d5cc8b8..d5ab117d 100644 --- a/doc/testing.org +++ b/doc/testing.org @@ -1,43 +1,67 @@ #+TITLE: Testing GN2 * Table of Contents :TOC: - - [[#introduction][Introduction]] - - [[#run-tests][Run tests]] - - [[#setup][Setup]] - - [[#running][Running]] +- [[#introduction][Introduction]] +- [[#run-tests][Run tests]] + - [[#setup][Setup]] + - [[#running][Running]] * Introduction -For integration testing we currently use the brilliant Ruby Mechanize -gem against the small database; a setup we call mechanical Rob because -it emulates someone clicking through the website and checking results. +For integration testing, we currently use [[https://github.com/genenetwork/genenetwork2/tree/testing/test/requests][Mechanica Rob]] against the +small [[https://github.com/genenetwork/genenetwork2/blob/testing/doc/database.org][database]]; a setup we call Mechanical Rob because it emulates +someone clicking through the website and checking results. -These scripts invoke calls to a running webserver and test the -response. If a page changes or is broken tests will break and we are -informed. In principle, Mechanical Rob is run before code merges are -committed to the main server. +These scripts invoke calls to a running webserver and test the response. +If a page changes or breaks, tests will fail. In principle, Mechanical +Rob runs before code merges get committed to the main server. -In the future we may move to Python mechanize - it'll be easy to mix -the Ruby and Python versions. +For unit tests, we use python's =unittest= framework. Coverage reports +get generated using [[https://coverage.readthedocs.io/en/coverage-5.2.1/][coverage.py]] which you could also use to run +unit tests. When adding new functionality, it is advisable to add +unit tests. * Run tests ** Setup -Mechanize is not yet included in Guix deployment. +Everything required for testing is already package with guix: +: ./pre-ins-env guix package -i genenetwork2 -p ~/opt/genenetwork2 ** Running -Run the tests from the root of the genenetwork2 source tree as, for -example, +Run the tests from the root of the genenetwork2 source tree as. Ensure +that Redis and Mariadb are running. -: ./bin/test-website http://localhost:5003/ (default) +To run Mechanical Rob: +: time env GN2_PROFILE=~/opt/genenetwork2 TMPDIR=~/tmp SERVER_PORT=5004 GENENETWORK_FILES=/gnu/data/gn2_data/ ./bin/genenetwork2 ./etc/default_settings.py -c ~/projects/genenetwork2/test/requests/test-website.py -a http://localhost:5004 -If you are using the small deployment database you can use +Use these aliases for the following examples. -: ./bin/test-website --skip -n +#+begin_src sh +alias runpython="env GN2_PROFILE=~/opt/gn-latest TMPDIR=/tmp SERVER_PORT=5004 GENENETWORK_FILES=/gnu/data/gn2_data/ ./bin/genenetwork2 -To run individual tests on localhost you can do +alias runcmd="time env GN2_PROFILE=~/opt/gn-latest TMPDIR=//tmp SERVER_PORT=5004 GENENETWORK_FILES=/gnu/data/gn2_data/ ./bin/genenetwork2 ./etc/default_settings.py -cli" +#+end_src -: ruby -Itest -Itest/lib test/lib/mapping.rb --name="/Mapping/" +You could use them in your =.bashrc= or =.zshrc= file. + +To run unit tests: + +: runpython -m unittest discover -v + +Or alternatively using the coverage tool: + +: runcmd coverage run -m unittest discover -v + +To generate a html coverage report in =wqflask/coverage_html_report/= + +: runcmd coverage html + +To output the report to =STDOUT=: + +: runcmd coverage report + +All the configs for running the coverage tool are in +=wqflask/.coveragerc= -- cgit v1.2.3 From b32004bc517d9a58be52662297e541c23f60355b Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Thu, 20 Aug 2020 16:01:54 +0300 Subject: Replace all references to python2.7 with python3.8 Run: ``` find . -type f -print0 | xargs -0 sed -i \ "s|lib/python2.7/site-packages|lib/python3.8/site-packages|g" ``` --- bin/genenetwork2 | 6 +++--- doc/GUIX-Reproducible-from-source.org | 6 +++--- doc/README.org | 2 +- doc/development.org | 4 ++-- etc/default_settings.py | 2 +- 5 files changed, 10 insertions(+), 10 deletions(-) (limited to 'doc') diff --git a/bin/genenetwork2 b/bin/genenetwork2 index dd6db7d6..5f4e0f9a 100755 --- a/bin/genenetwork2 +++ b/bin/genenetwork2 @@ -60,7 +60,7 @@ GN2_ID=$(cat /etc/hostname):$(basename $GN2_BASE_DIR) echo GN2_BASE_DIR=$GN2_BASE_DIR -GUIX_SITE=$GN2_BASE_DIR/lib/python2.7/site-packages +GUIX_SITE=$GN2_BASE_DIR/lib/python3.8/site-packages if [ -d $GUIX_SITE ]; then echo INFO: GN2 is running from GNU Guix GN2_BASE_DIR=$GUIX_SITE @@ -105,7 +105,7 @@ echo GN2_SETTINGS=$settings # if [ -z $ELASTICSEARCH_PROFILE ]; then # echo -e "WARNING: Elastic Search profile has not been set - use ELASTICSEARCH_PROFILE"; # else -# PYTHONPATH="$PYTHONPATH${PYTHONPATH:+:}$ELASTICSEARCH_PROFILE/lib/python2.7/site-packages" +# PYTHONPATH="$PYTHONPATH${PYTHONPATH:+:}$ELASTICSEARCH_PROFILE/lib/python3.8/site-packages" # fi if [ -z $GN2_PROFILE ] ; then @@ -121,7 +121,7 @@ if [ -z $GN2_PROFILE ]; then read -p "PRESS [ENTER] TO CONTINUE..." else export PATH=$GN2_PROFILE/bin:$PATH - export PYTHONPATH="$PYTHON_GN_PATH:$GN2_PROFILE/lib/python2.7/site-packages" # never inject another PYTHONPATH!! + export PYTHONPATH="$GN2_PROFILE/lib/python3.8/site-packages" # never inject another PYTHONPATH!! export R_LIBS_SITE=$GN2_PROFILE/site-library export JS_GUIX_PATH=$GN2_PROFILE/share/genenetwork2/javascript export GUIX_GTK3_PATH="$GN2_PROFILE/lib/gtk-3.0" diff --git a/doc/GUIX-Reproducible-from-source.org b/doc/GUIX-Reproducible-from-source.org index 19e4d14f..fffa9571 100644 --- a/doc/GUIX-Reproducible-from-source.org +++ b/doc/GUIX-Reproducible-from-source.org @@ -167,7 +167,7 @@ the Guix suggested environment vars. Check the output of #+begin_src bash guix package --search-paths -export PYTHONPATH="/root/.guix-profile/lib/python2.7/site-packages" +export PYTHONPATH="/root/.guix-profile/lib/python3.8/site-packages" export R_LIBS_SITE="/root/.guix-profile/site-library/" #+end_src @@ -265,7 +265,7 @@ software. If something is not working, take a hint from the settings file that comes in the Guix installation. It sits in something like -: cat ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py +: cat ~/.guix-profile/lib/python3.8/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py ** Set up nginx port forwarding @@ -380,7 +380,7 @@ After setting the paths for the server #+begin_src bash export PATH=~/.guix-profile/bin:$PATH -export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages" +export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages" export R_LIBS_SITE="$HOME/.guix-profile/site-library/" export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0" export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0" diff --git a/doc/README.org b/doc/README.org index 46df03c7..43c92e3c 100644 --- a/doc/README.org +++ b/doc/README.org @@ -218,7 +218,7 @@ information given by guix: On one system: -: export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages" +: export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages" : export R_LIBS_SITE="$HOME/.guix-profile/site-library/" : export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0" diff --git a/doc/development.org b/doc/development.org index e65ccd58..cd3beea3 100644 --- a/doc/development.org +++ b/doc/development.org @@ -67,11 +67,11 @@ You can install a Python package locally with pip, e.g. pip install hjson #+END_SRC -This installed in ~$HOME/.local/lib/python2.7/site-packages~. To add +This installed in ~$HOME/.local/lib/python3.8/site-packages~. To add the search path for GeneNetwork use the environment variable #+BEGIN_SRC sh -export PYTHON_GN_PATH=$HOME/.local/lib/python2.7/site-packages +export PYTHON_GN_PATH=$HOME/.local/lib/python3.8/site-packages #+END_SRC Now you should be able to do diff --git a/etc/default_settings.py b/etc/default_settings.py index 82c605da..27522187 100644 --- a/etc/default_settings.py +++ b/etc/default_settings.py @@ -19,7 +19,7 @@ # # For GNU Guix deployment also check the paths in # -# ~/.guix-profile/lib/python2.7/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py +# ~/.guix-profile/lib/python3.8/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py import os import sys -- cgit v1.2.3 From 9f49e2dbdf4e06fcb39ed5146560937f228682a7 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 22 Sep 2020 04:31:05 +0300 Subject: Update documentation on testing and CI * README.md: Update section on testing. * doc/docker-container.org: Add it. --- README.md | 10 +++++- doc/docker-container.org | 86 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 doc/docker-container.org (limited to 'doc') diff --git a/README.md b/README.md index 46264252..f1decb3c 100644 --- a/README.md +++ b/README.md @@ -42,9 +42,17 @@ Also mariadb and redis need to be running, see ## Testing To have tests pass, the redis and mariadb instance should be running, because of -asserts sprinkled in the code base(these will be removed in due time). +asserts sprinkled in the code base. + +Right now, the only tests running in CI are unittests. Please make +sure the existing unittests are green when submitting a PR. + +See +[./bin/genenetwork2](https://github.com/genenetwork/genenetwork2/blob/testing/doc/docker-container.org) +for more details. #### Mechanical Rob + We are building 'Mechanical Rob' automated testing using Python [requests](https://github.com/genenetwork/genenetwork2/tree/testing/test/requests) which can be run with: diff --git a/doc/docker-container.org b/doc/docker-container.org new file mode 100644 index 00000000..3c9864c5 --- /dev/null +++ b/doc/docker-container.org @@ -0,0 +1,86 @@ +#+TITLE: Genenetwork2 Dockerized + +* Table of Contents :TOC: +- [[#introduction][Introduction]] +- [[#creating-the-docker-images][Creating the Docker Images]] +- [[#pushing-to-dockerhub][Pushing to DockerHub]] + +* Introduction + +The CI(Continuous Integration) system for Genenetwork2 uses [[https://github.com/features/actions][Github +Actions]]. As such, it's important to have a way to run tests using +facilities provided by GUIX in a reproducible way. This project +leverages GUIX to generate a docker container from which the unittests +are ran from. + +Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This +document will not get into that. It's assumed that you have a working +dockec setup. + +The rest of this document outlines how the docker container used in +the CI builds were created. + +* Creating the Docker Images + +First create the image by running: + +#+begin_src sh +# For the Python 2 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash genenetwork2 + +# For the Python 3 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash python3-genenetwork2 + #+end_src + +The output will look something similar to: + +: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Load the docker image by running: + +: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Results look something similar to: + +#+begin_export ascii +a93f52b7f565: Loading layer 3.174GB/3.174GB +Loaded image: coreutils-genenetwork2:latest +#+end_export + +Assuming you have a docker instance running, you could always run +commands in it e.g: + +: docker run "coreutils-genenetwork2:latest" python --version + + +* Pushing to DockerHub + +We use DockerHub to store the docker images from which we use on our +CI environment using Github Actions. + +To push to dockerhub, first get the image name by running =docker +images=. Push to dockerhub using a command similar to: + +: docker push bonfacekilz/python2-genenetwork2:latest + +Right now, we have 2 images on DockerHub: + +- https://hub.docker.com/repository/docker/bonfacekilz/python2-genenetwork2: + Contains the python2 version of gn2. Don't use this. Please use the + python3 image! +- https://hub.docker.com/repository/docker/bonfacekilz/python3-genenetwork2: + Contains the python3 version of gn2. -- cgit v1.2.3 From 3ec4eb6b831eaa5adcf32a9fca8a60ea229cc1c4 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Fri, 25 Sep 2020 04:09:35 +0300 Subject: Update docs on CI setup with docker * doc/docker-container.org: Update it. --- doc/docker-container.org | 61 +++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 29 deletions(-) (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org index 3c9864c5..ec91824a 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -15,57 +15,60 @@ are ran from. Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This document will not get into that. It's assumed that you have a working -dockec setup. +docker setup. The rest of this document outlines how the docker container used in -the CI builds were created. +the CI builds was created. * Creating the Docker Images -First create the image by running: +The general idea is that GUIX is used to generate a set of binaries, +which will be added to a base mariaDB image. + +First create the gn2 tar archive by running: #+begin_src sh # For the Python 2 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen genenetwork2 # For the Python 3 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash python3-genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen python3-genenetwork2 #+end_src The output will look something similar to: -: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +: /gnu/store/x3m77vwaqcwba24p5s4lrb7w2ii16lj9-tarball-pack.tar.gz + +Now create a folder from which will host the following dockerfile. You +can name this file Dockerfile. Note that mariadb is the base image +since it already has mariadb installed for us. + +#+begin_src conf :mkdirp yes :tangle ~/docker/Dockerfile +FROM mariadb:latest -Load the docker image by running: +COPY ./gn2.tar.gz /tmp/gn2.tar.gz +RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ + mkdir -p /usr/local/mysql /genotype_files/genotype/json +#+end_src -: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +Build the image(Note the fullstop at the end): -Results look something similar to: +: sudo docker build -t python2-genenetwork2:latest -f Dockerfile . -#+begin_export ascii -a93f52b7f565: Loading layer 3.174GB/3.174GB -Loaded image: coreutils-genenetwork2:latest -#+end_export +To load the image interactively you've just created: + +: docker run -ti "python2-genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: -: docker run "coreutils-genenetwork2:latest" python --version - +: docker run "python2-genenetwork2:latest" python --version * Pushing to DockerHub -- cgit v1.2.3 From 5787a6ec5a54e2c98aed38f4c68291cb65b7f609 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 13 Apr 2021 02:37:13 -0500 Subject: SQL notes --- doc/database.org | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) (limited to 'doc') diff --git a/doc/database.org b/doc/database.org index 5107b660..f3d4d92e 100644 --- a/doc/database.org +++ b/doc/database.org @@ -1368,3 +1368,159 @@ mysql> select * from SnpSource limit 5; Empty set (0.00 sec) Hmmm. This is the test database. Then there are the plink files and VCF files. + +* Optimize SQL? + +We were facing some issues with slow queries. A query +was really slow on Penguin2: + +: time mysql -u webqtlout -pwebqtlout db_webqtl < ~/chunk.sql > /dev/null +: real 0m13.082s +: user 0m0.292s +: sys 0m0.032s + +Runs in 1s on Tux01 and 13s on P2, why is that? The gist of it +was increasing an InnoDB cache size(!) + +Interestingly, Penguin2 is running InnoDB on a much slower storage. +It has more indices that Tux01(?!). Probably due to things we have +been trying to make the datatables faster. + +Meanwhile the query is one with many joins: + +#+begin_src sql +SELECT ProbeSet.Name,ProbeSetXRef.DataId, T4.value, T5.value, T6.value, T7.value, T8.value, T9.value, T10.value, T11.value, T12.value, T14.value, T15.value, T17.value, T18.value, T19.value, T20.value, T21.value, T22.value, T24.value, T25.value, T26.value, T28.value, T29.value, T30.value, T31.value, T35.value, T36.value, T37.value, T39.value, T98.value, T99.value, T100.value, T103.value, T487.value, T105.value, T106.value, T110.value FROM (ProbeSet, ProbeSetXRef, ProbeSetFreeze) + left join ProbeSetData as T4 on T4.Id = ProbeSetXRef.DataId + and T4.StrainId=4 + (...) + left join ProbeSetData as T110 on T110.Id = ProbeSetXRef.DataId + and T110.StrainId=110 + WHERE ProbeSetXRef.ProbeSetFreezeId = ProbeSetFreeze.Id + and ProbeSetFreeze.Name = 'HC_M2_0606_P' + and ProbeSet.Id = ProbeSetXRef.ProbeSetId + order by ProbeSet.Id +#+end_src + +And is blazingly fast on Tux01 and (now) fast enough on Penguin2. + +First I checked the tables for indices and storage type. Next I +checked the difference in configuration. + +** Check tables + +Tables (ProbeSetData, ProbeSet, ProbeSetXRef, ProbeSetFreeze) + +*** ProbeSetData + +Same on Tux01 and P2: + +: show indexes from ProbeSetData ; + ++--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | ++--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| ProbeSetData | 0 | DataId | 1 | Id | A | 47769944 | NULL | NULL | | BTREE | | | +| ProbeSetData | 0 | DataId | 2 | StrainId | A | 5111384047 | NULL | NULL | | BTREE | | | ++--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ + +*** ProbeSetFreeze + +Tux01 has less indexes than P2(!): + ++----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | ++----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 911 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 911 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | Name | 1 | Name | A | 911 | NULL | NULL | YES | BTREE | | | +| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 911 | NULL | NULL | | BTREE | | | ++----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +4 rows in set (0.000 sec) + ++----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | ++----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | Name | 1 | Name | A | 883 | NULL | NULL | YES | BTREE | | | +| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | ShortName | 1 | ShortName | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | ProbeFreezeId | 1 | ProbeFreezeId | A | 441 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | conf_and_public | 1 | confidentiality | A | 3 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | conf_and_public | 2 | public | A | 4 | NULL | NULL | | BTREE | | | ++----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +8 rows in set (0.00 sec) + +*** ProbeSet + +Identical indexes + +*** ProbeSetXRef + +Tux01 has less indexes than P2(!): + +MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; ++--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | ++--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 885 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 47713039 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 47713039 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | Locus_IDX | 1 | Locus | A | 15904346 | NULL | NULL | YES | BTREE | | | ++--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +4 rows in set (0.000 sec) + + +MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; ++--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | ++--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 856 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 46412145 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 46412145 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | ProbeSetId1 | 1 | ProbeSetId | A | 5156905 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | Locus | 1 | Locus | A | 23206072 | NULL | NULL | YES | BTREE | | | ++--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ +5 rows in set (0.00 sec) + +** Check storage + +The database in Tux01 is mounted on NVME. On Penguin2 it +is slower SATA with RAID5. + +Also on Penguin2 the following tables are using InnoDB instead of +MyISAM + +#+begin_src sh +-rw-rw---- 1 mysql mysql 79691776 Oct 15 2019 AccessLog.ibd +-rw-rw---- 1 mysql mysql 196608 Oct 24 2019 Docs.ibd +-rw-rw---- 1 mysql mysql 63673729024 Jul 10 2020 GenoData.ibd +-rw-rw---- 1 mysql mysql 34787557376 Jul 9 2020 ProbeData.ibd +-rw-rw---- 1 mysql mysql 254690721792 Jul 10 2020 ProbeSetData.ibd +-rw-rw---- 1 mysql mysql 32103202816 Jul 9 2020 SnpAll.ibd +-rw-rw---- 1 mysql mysql 98304 May 6 2020 TraitMetadata.ibd +#+end_src + +This [[https://www.liquidweb.com/kb/mysql-performance-myisam-vs-innodb/][article]] suggests that myISAM will be faster for our use case. + +** Configuration + +There was one setting on Tux01 missing on P2 + +: +innodb_buffer_pool_size=1024M + +Running the same query twice (so you can see the warmup after +a restart of MariaDB) + +#+begin_src sh +penguin2:/etc$ time mysql -u webqtlout -pwebqtlout db_webqtl < ~/chunk.sql > ~/test.out +real 0m4.253s +user 0m0.276s +sys 0m0.040s +penguin2:/etc$ time mysql -u webqtlout -pwebqtlout db_webqtl < ~/chunk.sql > ~/test.out +real 0m2.633s +user 0m0.296s +sys 0m0.028s +#+end_src + +That is much better :) -- cgit v1.2.3 From f628f14bae508ec1d86606cfb9fdad6096f8b29f Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 13 Apr 2021 02:41:20 -0500 Subject: SQL notes --- doc/database.org | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/database.org b/doc/database.org index f3d4d92e..cd833b83 100644 --- a/doc/database.org +++ b/doc/database.org @@ -1339,7 +1339,8 @@ The SNP count info for the BXD is calculated like this startMb += stepMb #+end_src -select * from BXDSnpPosition limit 5; +: select * from BXDSnpPosition limit 5; + +------+-----------+-----------+----------+ | Chr | StrainId1 | StrainId2 | Mb | +------+-----------+-----------+----------+ @@ -1435,7 +1436,7 @@ Tux01 has less indexes than P2(!): | ProbeSetFreeze | 0 | Name | 1 | Name | A | 911 | NULL | NULL | YES | BTREE | | | | ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 911 | NULL | NULL | | BTREE | | | +----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -4 rows in set (0.000 sec) +: 4 rows in set (0.000 sec) +----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | @@ -1449,7 +1450,7 @@ Tux01 has less indexes than P2(!): | ProbeSetFreeze | 1 | conf_and_public | 1 | confidentiality | A | 3 | NULL | NULL | | BTREE | | | | ProbeSetFreeze | 1 | conf_and_public | 2 | public | A | 4 | NULL | NULL | | BTREE | | | +----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -8 rows in set (0.00 sec) +: 8 rows in set (0.00 sec) *** ProbeSet @@ -1459,7 +1460,7 @@ Identical indexes Tux01 has less indexes than P2(!): -MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; +: MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; +--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ @@ -1468,10 +1469,10 @@ MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; | ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 47713039 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 1 | Locus_IDX | 1 | Locus | A | 15904346 | NULL | NULL | YES | BTREE | | | +--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -4 rows in set (0.000 sec) +: 4 rows in set (0.000 sec) -MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; +: MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; +--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ @@ -1481,7 +1482,7 @@ MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; | ProbeSetXRef | 1 | ProbeSetId1 | 1 | ProbeSetId | A | 5156905 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 1 | Locus | 1 | Locus | A | 23206072 | NULL | NULL | YES | BTREE | | | +--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -5 rows in set (0.00 sec) +: 5 rows in set (0.00 sec) ** Check storage -- cgit v1.2.3 From d6d339adbbdbf18e5acabaa180092184cabacec8 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 13 Apr 2021 02:43:03 -0500 Subject: SQL notes - table layout --- doc/database.org | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) (limited to 'doc') diff --git a/doc/database.org b/doc/database.org index cd833b83..dd068d71 100644 --- a/doc/database.org +++ b/doc/database.org @@ -1417,28 +1417,23 @@ Same on Tux01 and P2: : show indexes from ProbeSetData ; -+--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ProbeSetData | 0 | DataId | 1 | Id | A | 47769944 | NULL | NULL | | BTREE | | | | ProbeSetData | 0 | DataId | 2 | StrainId | A | 5111384047 | NULL | NULL | | BTREE | | | -+--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ *** ProbeSetFreeze Tux01 has less indexes than P2(!): -+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 911 | NULL | NULL | | BTREE | | | | ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 911 | NULL | NULL | | BTREE | | | | ProbeSetFreeze | 0 | Name | 1 | Name | A | 911 | NULL | NULL | YES | BTREE | | | | ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 911 | NULL | NULL | | BTREE | | | -+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ : 4 rows in set (0.000 sec) -+----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 883 | NULL | NULL | | BTREE | | | @@ -1449,7 +1444,6 @@ Tux01 has less indexes than P2(!): | ProbeSetFreeze | 1 | ProbeFreezeId | 1 | ProbeFreezeId | A | 441 | NULL | NULL | | BTREE | | | | ProbeSetFreeze | 1 | conf_and_public | 1 | confidentiality | A | 3 | NULL | NULL | | BTREE | | | | ProbeSetFreeze | 1 | conf_and_public | 2 | public | A | 4 | NULL | NULL | | BTREE | | | -+----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ : 8 rows in set (0.00 sec) *** ProbeSet @@ -1461,19 +1455,17 @@ Identical indexes Tux01 has less indexes than P2(!): : MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; -+--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 885 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 47713039 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 47713039 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 1 | Locus_IDX | 1 | Locus | A | 15904346 | NULL | NULL | YES | BTREE | | | -+--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ : 4 rows in set (0.000 sec) + : MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; -+--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 856 | NULL | NULL | | BTREE | | | @@ -1481,7 +1473,6 @@ Tux01 has less indexes than P2(!): | ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 46412145 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 1 | ProbeSetId1 | 1 | ProbeSetId | A | 5156905 | NULL | NULL | | BTREE | | | | ProbeSetXRef | 1 | Locus | 1 | Locus | A | 23206072 | NULL | NULL | YES | BTREE | | | -+--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ : 5 rows in set (0.00 sec) ** Check storage -- cgit v1.2.3 From 0815b6004a167c5c25f6994aeb31ec3161619c6d Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 13 Apr 2021 02:44:03 -0500 Subject: SQL notes - table layout --- doc/database.org | 56 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) (limited to 'doc') diff --git a/doc/database.org b/doc/database.org index dd068d71..d5462d4e 100644 --- a/doc/database.org +++ b/doc/database.org @@ -1418,32 +1418,32 @@ Same on Tux01 and P2: : show indexes from ProbeSetData ; | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | -+--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -| ProbeSetData | 0 | DataId | 1 | Id | A | 47769944 | NULL | NULL | | BTREE | | | -| ProbeSetData | 0 | DataId | 2 | StrainId | A | 5111384047 | NULL | NULL | | BTREE | | | +|--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------| +| ProbeSetData | 0 | DataId | 1 | Id | A | 47769944 | NULL | NULL | | BTREE | | | +| ProbeSetData | 0 | DataId | 2 | StrainId | A | 5111384047 | NULL | NULL | | BTREE | | | *** ProbeSetFreeze Tux01 has less indexes than P2(!): | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | -+----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 911 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 911 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 0 | Name | 1 | Name | A | 911 | NULL | NULL | YES | BTREE | | | -| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 911 | NULL | NULL | | BTREE | | | +|----------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------| +| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 911 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 911 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | Name | 1 | Name | A | 911 | NULL | NULL | YES | BTREE | | | +| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 911 | NULL | NULL | | BTREE | | | : 4 rows in set (0.000 sec) | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | -+----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 883 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 883 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 0 | Name | 1 | Name | A | 883 | NULL | NULL | YES | BTREE | | | -| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 883 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 1 | ShortName | 1 | ShortName | A | 883 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 1 | ProbeFreezeId | 1 | ProbeFreezeId | A | 441 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 1 | conf_and_public | 1 | confidentiality | A | 3 | NULL | NULL | | BTREE | | | -| ProbeSetFreeze | 1 | conf_and_public | 2 | public | A | 4 | NULL | NULL | | BTREE | | | +|----------------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------| +| ProbeSetFreeze | 0 | PRIMARY | 1 | Id | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | FullName | 1 | FullName | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 0 | Name | 1 | Name | A | 883 | NULL | NULL | YES | BTREE | | | +| ProbeSetFreeze | 1 | NameIndex | 1 | Name2 | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | ShortName | 1 | ShortName | A | 883 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | ProbeFreezeId | 1 | ProbeFreezeId | A | 441 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | conf_and_public | 1 | confidentiality | A | 3 | NULL | NULL | | BTREE | | | +| ProbeSetFreeze | 1 | conf_and_public | 2 | public | A | 4 | NULL | NULL | | BTREE | | | : 8 rows in set (0.00 sec) *** ProbeSet @@ -1456,23 +1456,23 @@ Tux01 has less indexes than P2(!): : MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | -+--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 885 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 47713039 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 47713039 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 1 | Locus_IDX | 1 | Locus | A | 15904346 | NULL | NULL | YES | BTREE | | | +|--------------+------------+------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------| +| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 885 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 47713039 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 47713039 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | Locus_IDX | 1 | Locus | A | 15904346 | NULL | NULL | YES | BTREE | | | : 4 rows in set (0.000 sec) : MariaDB [db_webqtl]> show indexes from ProbeSetXRef ; | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | -+--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ -| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 856 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 46412145 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 46412145 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 1 | ProbeSetId1 | 1 | ProbeSetId | A | 5156905 | NULL | NULL | | BTREE | | | -| ProbeSetXRef | 1 | Locus | 1 | Locus | A | 23206072 | NULL | NULL | YES | BTREE | | | +|--------------+------------+-------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------| +| ProbeSetXRef | 0 | ProbeSetId | 1 | ProbeSetFreezeId | A | 856 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | ProbeSetId | 2 | ProbeSetId | A | 46412145 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 0 | DataId_IDX | 1 | DataId | A | 46412145 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | ProbeSetId1 | 1 | ProbeSetId | A | 5156905 | NULL | NULL | | BTREE | | | +| ProbeSetXRef | 1 | Locus | 1 | Locus | A | 23206072 | NULL | NULL | YES | BTREE | | | : 5 rows in set (0.00 sec) ** Check storage -- cgit v1.2.3 From 45fc1da136b78bac906aad013686a9530f68bd5e Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 11 May 2021 10:33:49 +0300 Subject: doc: docker-container: Default to python3-genenetwork2 for examples --- doc/docker-container.org | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org index ec91824a..c894c4b5 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -59,11 +59,11 @@ RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ Build the image(Note the fullstop at the end): -: sudo docker build -t python2-genenetwork2:latest -f Dockerfile . +: sudo docker build -t python3-genenetwork2:latest -f Dockerfile . To load the image interactively you've just created: -: docker run -ti "python2-genenetwork2:latest" bash +: docker run -ti "python3-genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: @@ -78,7 +78,7 @@ CI environment using Github Actions. To push to dockerhub, first get the image name by running =docker images=. Push to dockerhub using a command similar to: -: docker push bonfacekilz/python2-genenetwork2:latest +: docker push bonfacekilz/python3-genenetwork2:latest Right now, we have 2 images on DockerHub: -- cgit v1.2.3 From c22fdbd6a128ab9c9ddfc14299115a40cec37f07 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Wed, 26 May 2021 18:33:27 +0300 Subject: doc: docker-container: Rename python3-genenetwork2 to genenetwork2 See: https://is.gd/gHJvhe --- doc/docker-container.org | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org index c894c4b5..ef0d71fc 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -32,13 +32,13 @@ First create the gn2 tar archive by running: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ - screen genenetwork2 + screen python2-genenetwork2 # For the Python 3 version: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ - screen python3-genenetwork2 + screen genenetwork2 #+end_src The output will look something similar to: @@ -59,16 +59,16 @@ RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ Build the image(Note the fullstop at the end): -: sudo docker build -t python3-genenetwork2:latest -f Dockerfile . +: sudo docker build -t genenetwork2:latest -f Dockerfile . To load the image interactively you've just created: -: docker run -ti "python3-genenetwork2:latest" bash +: docker run -ti "genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: -: docker run "python2-genenetwork2:latest" python --version +: docker run "genenetwork2:latest" python --version * Pushing to DockerHub @@ -78,7 +78,7 @@ CI environment using Github Actions. To push to dockerhub, first get the image name by running =docker images=. Push to dockerhub using a command similar to: -: docker push bonfacekilz/python3-genenetwork2:latest +: docker push bonfacekilz/genenetwork2:latest Right now, we have 2 images on DockerHub: -- cgit v1.2.3 From a7f2e8dea0a84a70aec29159a9ae35bd55ba047d Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Thu, 17 Jun 2021 08:52:17 +0300 Subject: doc: Add results of investing rpy2 blocking issue in gn2 --- doc/rpy2-performance.org | 170 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 doc/rpy2-performance.org (limited to 'doc') diff --git a/doc/rpy2-performance.org b/doc/rpy2-performance.org new file mode 100644 index 00000000..29b99ba0 --- /dev/null +++ b/doc/rpy2-performance.org @@ -0,0 +1,170 @@ +* Python-Rpy2 performance issues with genenetwork2 + +At one point, genenetwork2 was down. A possible cause was that it +wrote into the log file in an infinite loop due to rpy2, so a solution +was to empty it. Currently, as a work around, rpy2 is disabled by +removing it's imports. This affects WGCNA/ CTL imports and commenting +out Biweight Midcorrelation option in the trait page. See: + +- [[https://github.com/genenetwork/genenetwork2/commit/1baf5f7611909c651483208184c5fbf7d4a7a088][1baf5f7]] +- [[https://github.com/genenetwork/genenetwork2/commit/afee4d625248565857df98d3510f680ae6204864][afee4d6]] +- [[https://github.com/genenetwork/genenetwork2/commit/c458bf0ad731e5e5fd9cbd0686936b3a441bae63][c458bf0]] +- [[https://github.com/genenetwork/genenetwork2/commit/d31f3f763471b19559ca74e73b52b3cb5e7153ce][d31f3f7]] + +** Reproducing the problem + +I went back to commit #b8408cea. With regards to logs, I never +experienced any log issue. Perhaps it's because of how I start my +server: + +: env SERVER_PORT=5004 TMPDIR=/home/bonface/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG GENENETWORK_FILES=/home/bonface/data/genotype_files/ GN2_PROFILE=/home/bonface/opt/python3-genenetwork2 ./scripts/run_debug.sh + +However, when loading the homepage, I occasionally ran into this trace: + +#+begin_export ascii +DEBUG:wqflask.views:.check_access_permissions: @app.before_request check_access_permissions +DEBUG:wqflask.views:.shutdown_session: remove db_session +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Error: ignoring SIGPIPE signal + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In addition: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Warning messages: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: 1: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE, : +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: library '/home/bonface/R/x86_64-unknown-linux-gnu-library/4.0' contains no packages + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: 2: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE, : +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: library '/home/bonface/R/x86_64-unknown-linux-gnu-library/4.0' contains no packages + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Fatal error: unable to initialize the JIT + + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: + *** caught segfault *** + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: address (nil), cause 'memory not mapped' + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: +Possible actions: +1: abort (with core dump, if enabled) +2: normal R exit +3: exit R without saving workspace +4: exit R saving workspace + +Selection: + +#+end_export + +This blocks the flask service. Seems to be related to: [[https://github.com/rpy2/rpy2/issues/769][rpy2-issue#769]] +and [[https://github.com/rpy2/rpy2/issues/809][rpy2-issue#809]]. I tried to reproduce this problem using some endpoint: + +#+begin_src python +@app.route("/test") + def test(): + from rpy2 import robjects as ro + from rpy2 import rinterface + from threading import Thread + + def rpy2_init_simple(): + rinterface.initr_simple() + + thread = Thread(target=rpy2_init_simple) + thread.start() + return "This is a test after importing rpy2" +#+end_src + +which generates this trace: + +#+begin_export ascii +/home/bonface/opt/python3-genenetwork2/lib/python3.8/site-packages/rpy2/rinterface.py:955: UserWarning: R is not initialized by the main thread. + Its taking over SIGINT cannot be reversed here, and as a + consequence the embedded R cannot be interrupted with Ctrl-C. + Consider (re)setting the signal handler of your choice from + the main thread. +warnings.warn( +DEBUG:wqflask.views:.shutdown_session: remove db_session + +#+end_export + +Modifying the endpoint to: + +#+begin_src python +@app.route("/test") + def test(): + import wqflask.correlation.show_corr_results + import wqflask.ctl.ctl_analysis + import time + from wqflask.correlation.correlation_functions import cal_zero_order_corr_for_tiss + + print("Sleeping for 3 seconds") + time.sleep(3) + return "This is a test after importing rpy2" +#+end_src + +and refreshing the page a couple of times, I get: + +#+begin_export ascii +DEBUG:wqflask.views:.check_access_permissions: @app.before_request check_access_ +permissions +Sleeping for 3 seconds +DEBUG:wqflask.views:.shutdown_session: remove db_session +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Error: ignoring SIGPI +PE signal + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In addition: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Warning messages: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: 1: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In (function (package +, help, pos = 2, lib.loc = NULL, character.only = FALSE, : +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: library '/home/bonfa +ce/R/x86_64-unknown-linux-gnu-library/4.0' contains no packages + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: 2: +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: In (function (package +, help, pos = 2, lib.loc = NULL, character.only = FALSE, : +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: library '/home/bonfa +ce/R/x86_64-unknown-linux-gnu-library/4.0' contains no packages + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: +*** caught segfault *** + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: address (nil), cause +'memory not mapped' + +WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: +Possible actions: +1: abort (with core dump, if enabled) +2: normal R exit +3: exit R without saving workspace +4: exit R saving workspace + +Selection: [2021-06-16 13:11:00 +0300] [18657] [INFO] Handling signal: winch +[2021-06-16 13:11:00 +0300] [18657] [INFO] Handling signal: winch +[2021-06-16 13:13:02 +0300] [18657] [INFO] Handling signal: winch +#+end_export + +However, this seems to be non-deterministic, in the sense that I can't really pin what causes the above. I've tried to write a Locust Test that simulates users hitting that endpoint: + +#+begin_src python +"""Load test a single trait page""" +from locust import HttpUser, task, between + + + class LoadTest(HttpUser): + wait_time = between(1, 2.5) + + @task + def fetch_trait(self): + """Fetch a single trait""" + self.client.get("/test") +#+end_src -- cgit v1.2.3 From f7027d4c87786da0d505e0e8e1da23d6e55f6d80 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Thu, 17 Jun 2021 10:09:38 +0300 Subject: doc: rpy2-performance: Replace begin_export with begin_src --- doc/rpy2-performance.org | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/rpy2-performance.org b/doc/rpy2-performance.org index 29b99ba0..ddcac81a 100644 --- a/doc/rpy2-performance.org +++ b/doc/rpy2-performance.org @@ -21,7 +21,7 @@ server: However, when loading the homepage, I occasionally ran into this trace: -#+begin_export ascii +#+begin_src DEBUG:wqflask.views:.check_access_permissions: @app.before_request check_access_permissions DEBUG:wqflask.views:.shutdown_session: remove db_session WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: Error: ignoring SIGPIPE signal @@ -58,7 +58,7 @@ Possible actions: Selection: -#+end_export +#+end_src This blocks the flask service. Seems to be related to: [[https://github.com/rpy2/rpy2/issues/769][rpy2-issue#769]] and [[https://github.com/rpy2/rpy2/issues/809][rpy2-issue#809]]. I tried to reproduce this problem using some endpoint: @@ -80,7 +80,7 @@ and [[https://github.com/rpy2/rpy2/issues/809][rpy2-issue#809]]. I tried to repr which generates this trace: -#+begin_export ascii +#+begin_src /home/bonface/opt/python3-genenetwork2/lib/python3.8/site-packages/rpy2/rinterface.py:955: UserWarning: R is not initialized by the main thread. Its taking over SIGINT cannot be reversed here, and as a consequence the embedded R cannot be interrupted with Ctrl-C. @@ -89,7 +89,7 @@ which generates this trace: warnings.warn( DEBUG:wqflask.views:.shutdown_session: remove db_session -#+end_export +#+end_src Modifying the endpoint to: @@ -108,7 +108,7 @@ Modifying the endpoint to: and refreshing the page a couple of times, I get: -#+begin_export ascii +#+begin_src DEBUG:wqflask.views:.check_access_permissions: @app.before_request check_access_ permissions Sleeping for 3 seconds @@ -136,7 +136,7 @@ WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: library '/home/bonfa ce/R/x86_64-unknown-linux-gnu-library/4.0' contains no packages WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: -*** caught segfault *** +\*** caught segfault *** WARNING:rpy2.rinterface_lib.callbacks:R[write to console]: address (nil), cause 'memory not mapped' @@ -151,9 +151,11 @@ Possible actions: Selection: [2021-06-16 13:11:00 +0300] [18657] [INFO] Handling signal: winch [2021-06-16 13:11:00 +0300] [18657] [INFO] Handling signal: winch [2021-06-16 13:13:02 +0300] [18657] [INFO] Handling signal: winch -#+end_export +#+end_src -However, this seems to be non-deterministic, in the sense that I can't really pin what causes the above. I've tried to write a Locust Test that simulates users hitting that endpoint: +However, this seems to be non-deterministic, in the sense that I can't +really pin what causes the above. I've tried to write a Locust Test +that simulates users hitting that endpoint: #+begin_src python """Load test a single trait page""" @@ -168,3 +170,4 @@ from locust import HttpUser, task, between """Fetch a single trait""" self.client.get("/test") #+end_src + -- cgit v1.2.3 From a992e2038424c2acdf11078356204847d128c8c6 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Thu, 17 Jun 2021 10:11:53 +0300 Subject: docs: rpy2-performance: Indicate version of python-rpy2 being used --- doc/rpy2-performance.org | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/rpy2-performance.org b/doc/rpy2-performance.org index ddcac81a..177c427e 100644 --- a/doc/rpy2-performance.org +++ b/doc/rpy2-performance.org @@ -1,10 +1,10 @@ * Python-Rpy2 performance issues with genenetwork2 At one point, genenetwork2 was down. A possible cause was that it -wrote into the log file in an infinite loop due to rpy2, so a solution -was to empty it. Currently, as a work around, rpy2 is disabled by -removing it's imports. This affects WGCNA/ CTL imports and commenting -out Biweight Midcorrelation option in the trait page. See: +wrote into the log file in an infinite loop due to rpy2(v3.4.4), so a +solution was to empty it. Currently, as a work around, rpy2 is +disabled by removing it's imports. This affects WGCNA/ CTL imports and +commenting out Biweight Midcorrelation option in the trait page. See: - [[https://github.com/genenetwork/genenetwork2/commit/1baf5f7611909c651483208184c5fbf7d4a7a088][1baf5f7]] - [[https://github.com/genenetwork/genenetwork2/commit/afee4d625248565857df98d3510f680ae6204864][afee4d6]] -- cgit v1.2.3 From a5981b8e71380b171b210d55da58e5037b455a6c Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Thu, 17 Jun 2021 10:20:39 +0300 Subject: doc: rpy2-performance: Add a section exploring possible solutions --- doc/rpy2-performance.org | 9 +++++++++ 1 file changed, 9 insertions(+) (limited to 'doc') diff --git a/doc/rpy2-performance.org b/doc/rpy2-performance.org index 177c427e..8f917ca0 100644 --- a/doc/rpy2-performance.org +++ b/doc/rpy2-performance.org @@ -171,3 +171,12 @@ from locust import HttpUser, task, between self.client.get("/test") #+end_src + +** A possible solution + +From this [[https://github.com/rpy2/rpy2/issues/809#issuecomment-845923975][comment]], a possible reason for the above traces, is that +from Flask's end, a [[https://tldp.org/LDP/lpg/node20.html][SIGPIPE]] is somehow generated by our Python +code. However, at this particular point, the R thread just happens to +be running, and R can't handle this correctly. This seems to have been +fixed in this [[https://github.com/rpy2/rpy2/pull/810][PR]] with a this [[https://github.com/rpy2/rpy2/issues/809#issuecomment-851618215][explanation]]. On our end, to have these +changes, we have to update our python-rpy2 version. -- cgit v1.2.3 From d6937e74b85a4fc44153530520774836eed60fe6 Mon Sep 17 00:00:00 2001 From: zsloan Date: Mon, 28 Jun 2021 18:31:29 +0000 Subject: Added doc for creating guix profile, written by Bonface --- doc/guix_profile_setup.org | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 doc/guix_profile_setup.org (limited to 'doc') diff --git a/doc/guix_profile_setup.org b/doc/guix_profile_setup.org new file mode 100644 index 00000000..c397377c --- /dev/null +++ b/doc/guix_profile_setup.org @@ -0,0 +1,39 @@ +* Setting up GUIX profile for GN + +First create a guix profile with the latest packages: + +: ~/opt/guix/bin/guix pull + +This will create a profile with the latest packages under`~/.config/guix/current` + +Now you have the latest guix. Check: `$HOME/.config/guix/current/bin/guix --version` + +At this point, it's worth mentioning that installing +python3-genenetwork using `$HOME/.config/guix/current/bin/guix` should +work; but let's use the dev version(since that may come handy in +time), and it's a nice thing to know. + +Next, we ensure that the appropriate GUILEPATHS are set: + +: export GUILE_LOAD_PATH=$HOME/.config/guix/current/share/guile/site/3.0/ +: export GUILE_LOAD_COMPILED_PATH=$HOME/.config/guix/current/lib/guile/3.0/site-ccache/ + +Get into the container: + +: $HOME/.config/guix/current/bin/guix environment -C guix --ad-hoc bash gcc-toolchain +: ./bootstrap +: ./configure --localstatedir=/var --sysconfdir=/etc + +Check that everything works: + +: make check + +Clean up and build: + +: make clean-go +: make -j 4 +: exit + +Install Python3 (substitute paths when necessary): + +: env GUIX_PACKAGE_PATH='/home/zas1024/guix-bioinformatics:/home/zas1024/guix-past/modules' $HOME/.config/guix/current/bin/guix install python3-genenetwork2 -p ~/opt/python3-genenetwork2 --substitute-urls="http://guix.genenetwork.org https://berlin.guixsd.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org" -- cgit v1.2.3 From fe345c9f2e99be748511f6889420d6560a553a37 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 24 Aug 2021 09:55:05 +0200 Subject: README: install with guix profile --- doc/README.org | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 10 deletions(-) (limited to 'doc') diff --git a/doc/README.org b/doc/README.org index 43c92e3c..fb1781aa 100644 --- a/doc/README.org +++ b/doc/README.org @@ -2,7 +2,8 @@ * Table of Contents :TOC: - [[#introduction][Introduction]] - - [[#install][Install]] + - [[#installing-guix-packages][Installing Guix packages]] + - [[#creating-a-gnu-guix-profile][Creating a GNU Guix profile]] - [[#running-gn2][Running GN2]] - [[#run-gn-proxy][Run gn-proxy]] - [[#run-redis][Run Redis]] @@ -37,27 +38,79 @@ tree. Current supported versions can be found as the SHA values of For a full view of runtime dependencies as defined by GNU Guix, see an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]]. -* Install +* Installing Guix packages Make sure to install GNU Guix using the binary download instructions on the main website. Follow the instructions on [[GUIX-Reproducible-from-source.org]] to download pre-built binaries. Note -the download amounts to several GBs of data. +the download amounts to several GBs of data. Debian-derived distros +may support + +: apt-get install guix + +* Creating a GNU Guix profile + +We run a GNU Guix channel with packages at [[https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics][guix-bioinformatics]]. The +README has instructions for hosting a channel, but typically we use +the GUIX_PACKAGE_PATH instead. First upgrade to a recent guix with + +: mkdir ~/opt +: guix pull -p ~/opt/guix-pull + +It should upgrade (ignore the locales warnings). You can optionally +specify the specific git checkout of guix with + +: guix pull -p ~/opt/guix-pull --commit=f04883d + +which is useful when you ned to roll back to an earlier version +(sometimes our channel goes out of sync). Next, we install +GeneNetwork2 with + +: source ~/opt/guix-pull/etc/profile +: git clone https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.git ~/guix-bioinformatics +: cd ~/guix-bioinformatics +: git pull +: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics guix package -i genenetwork2 -p ~/opt/genenetwork2 + +you probably also need guix-past (the upstream channel for older packages): + +: git clone https://gitlab.inria.fr/guix-hpc/guix-past.git ~/guix-past +: cd ~/guix-past +: git pull +: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics:$HOME/guix-past/modules ~/opt/guix-pull/bin/guix package -i genenetwork2 -p ~/opt/genenetwork2 + +ignore the warnings. Guix should install the software without trying +to build everything. If you system insists on building all packages, +try the `--dry-run` switch and fix the [[https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html][substitutes]]. You may add the +`--substitute-urls="http://guix.genenetwork.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org"` switch. + +The guix.genenetwork.org has most of our packages pre-built(!). To use +it on your own machine the public key is + +#+begin_src scheme +(public-key + (ecc + (curve Ed25519) + (q #E50F005E6DA2F85749B9AA62C8E86BB551CE2B541DC578C4DBE613B39EC9E750#))) +#+end_src + +Once we have a GNU Guix profile, a running database (see below) and the file storage, +we should be ready to fire up GeneNetwork: * Running GN2 -Default settings for GN2 are listed in a file called -[[../etc/default_settings.py][default_settings.py]]. You can copy this file and pass it as a new -parameter to the genenetwork2 command, e.g. +Check out the source with git: -: genenetwork2 mysettings.py +: git clone git@github.com:genenetwork/genenetwork2.git +: cd genenetwork2 -or you can set environment variables to override individual parameters, e.g. +Run GN2 with above Guix profile -: env SERVER_PORT=5004 SQL_URI=mysql://user:pwd@dbhostname/db_webqtl genenetwork2 +: export GN2_PROFILE=$HOME/opt/genenetwork2 +: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5002 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 -gunicorn-dev the debug and logging switches can be particularly useful when -developing GN2. +developing GN2. Location and files are the current ones for Penguin2. * Run gn-proxy -- cgit v1.2.3 From 8a8dcba497e8f4210329f42cd4b4e766d95a36fc Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 24 Aug 2021 10:54:01 +0200 Subject: README: on installing GN2 --- doc/README.org | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/README.org b/doc/README.org index fb1781aa..1236016e 100644 --- a/doc/README.org +++ b/doc/README.org @@ -2,6 +2,7 @@ * Table of Contents :TOC: - [[#introduction][Introduction]] + - [[#check-list][Check list]] - [[#installing-guix-packages][Installing Guix packages]] - [[#creating-a-gnu-guix-profile][Creating a GNU Guix profile]] - [[#running-gn2][Running GN2]] @@ -38,6 +39,17 @@ tree. Current supported versions can be found as the SHA values of For a full view of runtime dependencies as defined by GNU Guix, see an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]]. +* Check list + +To run GeneNetwork the following services need to function: + +1. [ ] GNU Guix with a guix profile for genenetwork2 +1. [ ] A path to the (static) genotype files +1. [ ] Gn-proxy for authentication +1. [ ] The genenetwork3 service +1. [ ] Redis +1. [ ] Mariadb + * Installing Guix packages Make sure to install GNU Guix using the binary download instructions @@ -107,15 +119,26 @@ Check out the source with git: Run GN2 with above Guix profile : export GN2_PROFILE=$HOME/opt/genenetwork2 -: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5002 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 -gunicorn-dev +: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5012 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 etc/default_settings.py -gunicorn-dev the debug and logging switches can be particularly useful when developing GN2. Location and files are the current ones for Penguin2. +It may be useful to tunnel the web server to your local browser with +an ssh tunnel: + +If you want to test a service running on the server on a certain +port (say 8202) use + + ssh -L 8202:127.0.0.1:8202 -f -N myname@penguin2.genenetwork.org + +And browse on your local machine to http://localhost:8202/ + * Run gn-proxy GeneNetwork requires a separate gn-proxy server which handles -authorisation and access control. For instructions see the [[https://github.com/genenetwork/gn-proxy][README]]. +authorisation and access control. For instructions see the +[[https://github.com/genenetwork/gn-proxy][README]]. Note it may already be running on our servers! * Run Redis -- cgit v1.2.3 From 7a15d24a6598f30801dd897ddc72d3773641e7bd Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 5 Oct 2021 11:32:17 +0300 Subject: doc: docker-container.org: Remove python2 gn2 docker set-up command --- doc/docker-container.org | 7 ------- 1 file changed, 7 deletions(-) (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org index ef0d71fc..79b8272f 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -28,13 +28,6 @@ which will be added to a base mariaDB image. First create the gn2 tar archive by running: #+begin_src sh -# For the Python 2 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack --no-grafts\ - -S /gn2-profile=/ \ - screen python2-genenetwork2 - -# For the Python 3 version: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ -- cgit v1.2.3 From ba98ef026544d4437e65a7bd248ff9591296b48e Mon Sep 17 00:00:00 2001 From: Frederick Muriuki Muriithi Date: Wed, 13 Oct 2021 09:37:19 +0300 Subject: Add some documentation for generating heatmaps Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/non-clustered-heatmaps-and-flipping.gmi * Add some documentation on generating the heatmaps, that would be useful for the end user. --- doc/heatmap-generation.org | 34 ++++++++++++++++++++++++++++++++ doc/images/gn2_header_collections.png | Bin 0 -> 7890 bytes doc/images/heatmap_form.png | Bin 0 -> 9363 bytes doc/images/heatmap_with_hover_tools.png | Bin 0 -> 42578 bytes 4 files changed, 34 insertions(+) create mode 100644 doc/heatmap-generation.org create mode 100644 doc/images/gn2_header_collections.png create mode 100644 doc/images/heatmap_form.png create mode 100644 doc/images/heatmap_with_hover_tools.png (limited to 'doc') diff --git a/doc/heatmap-generation.org b/doc/heatmap-generation.org new file mode 100644 index 00000000..a697c70b --- /dev/null +++ b/doc/heatmap-generation.org @@ -0,0 +1,34 @@ +#+STARTUP: inlineimages +#+TITLE: Heatmap Generation +#+AUTHOR: Muriithi Frederick Muriuki + +* Generating Heatmaps + +Like a lot of other features, the heatmap generation requires an existing collection. If none exists, see [[][Creating a new collection]] for how to create a new collection. + +Once you have a collection, you can navigate to the collections page by clicking on the "Collections" link in the header + + +[[./images/gn2_header_collections.png]] + +From that page, pick the collection that you want to work with by clicking on its name on the collections table. + +That takes you to that collection's page, where you can select the data that you want to use to generate the heatmap. + +** Selecting Orientation + +Once you have selected the data, select the orientation of the heatmap you want generated. You do this by selecting either *"vertical"* or *"horizontal"* in the heatmaps form: + +[[./images/heatmap_form.png]] + +Once you have selected the orientation, click on the "Generate Heatmap" button as in the image above. + +The heatmap generation might take a while, but once it is done, an image shows up above the data table. + +** Downloading the PNG copy of the Heatmap + +Once the heatmap image is shown, hovering over it, displays some tools to interact with the image. + +To download, hover over the heatmap image, and click on the "Download plot as png" icon as shown. + +[[./images/heatmap_with_hover_tools.png]] diff --git a/doc/images/gn2_header_collections.png b/doc/images/gn2_header_collections.png new file mode 100644 index 00000000..ac23f9c1 Binary files /dev/null and b/doc/images/gn2_header_collections.png differ diff --git a/doc/images/heatmap_form.png b/doc/images/heatmap_form.png new file mode 100644 index 00000000..163fbb60 Binary files /dev/null and b/doc/images/heatmap_form.png differ diff --git a/doc/images/heatmap_with_hover_tools.png b/doc/images/heatmap_with_hover_tools.png new file mode 100644 index 00000000..4ab79f99 Binary files /dev/null and b/doc/images/heatmap_with_hover_tools.png differ -- cgit v1.2.3