Merge pull request #356 from pjotrp/gn-testing

Updated instructions and SERVER_PORT for single flask server

5 files changed, 96 insertions, 28 deletions

diff --git a/README.md b/README.md
index 59645994..523eeb76 100644
--- a/README.md
+++ b/README.md
@@ -27,9 +27,18 @@ genenetwork2
 
 (default is http://localhost:5003/). For full examples (you'll need to
 set a number of environment variables), including running scripts and
-a Python REPL, see the startup script
+a Python REPL, for example
+
+```sh
+env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=~/data/gn2_data ./bin/genenetwork2
+```
+
+see the startup script
 [./bin/genenetwork2](https://github.com/genenetwork/genenetwork2/blob/testing/bin/genenetwork2).
 
+Also Mysql and Elasticsearch need to be running, see
+[INSTALL](./doc/README.org).
+
 ## Testing
 
 We are building 'Mechanical Rob' automated testing using Python
diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md
index d15cad1a..9ad75aa0 100644
--- a/RELEASE-NOTES.md
+++ b/RELEASE-NOTES.md
@@ -1,4 +1,4 @@
-## ChangeLog v2.11 (date unknown)
+## ChangeLog v2.11 pre-release (date unknown)
 
 This is a massive bug fix release with many improvements. For contributions
 see
@@ -6,12 +6,37 @@ see
 and
 [commits](https://github.com/genenetwork/genenetwork2/commits/master).
 
+The current GNU Guix checkout is at
+
+https://gitlab.com/genenetwork/guix-bioinformatics/tree/gn-latest-20190213
+
+and
+
+https://gitlab.com/genenetwork/guix/tree/gn-latest-20190213
+
+With these branches a binary install can be fetched with
+
+    env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package --substitute-urls="https://berlin.guixsd.org http://guix.genenetwork.org https://mirror.hydra.gnu.org" -i genenetwork2 -p ~/opt/gn-latest-20190213
+
+The key for guix.genenetwork.org is
+
+```scheme
+(public-key
+ (ecc
+  (curve Ed25519)
+  (q #D54AA5C8CBE268CBC82418AB83709611BAC88B39D44E68C18F6E16197B5B5CA0#)
+  )
+ )
+```
+
+### Migrated PIL to pillow (preparing for Python3 migration)
+
+### Dropped pylmm support
+
 ### Added GEMMA support
 
-* Front-end support 
+* Front-end support
 
 ### Added test framework and unit tests
 
 * Added python integration and unit tests
-
-
diff --git a/doc/README.org b/doc/README.org
index 5dc9e994..620c946c 100644
--- a/doc/README.org
+++ b/doc/README.org
@@ -12,13 +12,15 @@
    - [[#load-the-small-database-in-mysql][Load the small database in MySQL]]
  - [[#gn2-dependency-graph][GN2 Dependency Graph]]
  - [[#working-with-the-gn2-source-code][Working with the GN2 source code]]
+ - [[#running-elasticsearch][Running ElasticSearch]]
+   - [[#systemd][SystemD]]
+ - [[#read-more][Read more]]
  - [[#trouble-shooting][Trouble shooting]]
    - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]]
-   - [[#error-can-not-find-directory-homegn2_data][ERROR: can not find directory $HOME/gn2_data]]
+   - [[#error-can-not-find-directory-homegn2_data-or-can-not-find-directory-homegenotype_filesgenotype][ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype']]
    - [[#cant-run-a-module][Can't run a module]]
    - [[#rpy2-error-show-now-found][Rpy2 error 'show' now found]]
    - [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]]
- - [[#read-more][Read more]]
  - [[#irc-session][IRC session]]
 
 * Introduction
@@ -100,11 +102,13 @@ mysql (which comes as part of the GNU Guix genenetwork2 install).
 
 As root configure and run
 
-:  adduser mysql && addgroup mysql
-:  mysqld --datadir=/var/mysql --initialize-insecure
-:  mkdir -p /var/run/mysqld
-:  chown mysql.mysql ~/mysql /var/run/mysqld
-:  mysqld -u mysql --datadir=/var/mysql --explicit_defaults_for_timestamp -P 12048"
+#+BEGIN_SRC bash
+adduser mysql && addgroup mysql
+mysqld --datadir=/var/mysql --initialize-insecure
+mkdir -p /var/run/mysqld
+chown mysql.mysql ~/mysql /var/run/mysqld
+mysqld -u mysql --datadir=/var/mysql --explicit_defaults_for_timestamp -P 12048"
+#+END_SRC
 
 If you want to run as root you may have to set
 
@@ -192,6 +196,41 @@ http://biogems.info/contrib/genenetwork/gn2.svg
 
 See [[development.org]].
 
+* Running ElasticSearch
+
+In order to start up elasticsearch:
+Penguin - change user to "elasticsearch" and use the following command: "env JAVA_HOME=/opt/jdk-9.0.4 /opt/elasticsearch-6.2.1/bin/elasticsearch"
+
+
+** SystemD
+
+New server - as root run "systemctl restart elasticsearch"
+
+#+BEGIN_SRC
+tux01:/etc/systemd/system# cat elasticsearch.service
+[Unit]
+Description=Run Elasticsearch
+
+[Service]
+ExecStart=/opt/elasticsearch-6.2.1/bin/elasticsearch
+Environment=JAVA_HOME=/opt/jdk-9.0.4
+Environment="ES_JAVA_OPTS=-Xms1g -Xmx8g"
+Environment="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/jdk-9.0.4/bin"
+LimitNOFILE=65536
+StandardOutput=syslog
+StandardError=syslog
+User=elasticsearch
+
+[Install]
+WantedBy=multi-user.target
+#+END_SRC
+
+* Read more
+
+If you want to understand the architecture of GN2 read
+[[Architecture.org]].  The rest of this document is mostly on deployment
+of GN2.
+
 * Trouble shooting
 
 ** ImportError: No module named jinja2
@@ -210,13 +249,17 @@ On one system:
 : export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
 
 and perhaps a few more.
-** ERROR: can not find directory $HOME/gn2_data
+** ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype'
 
 The default settings file looks in your $HOME/gn2_data. Since these
 files come with a Guix installation you should take a hint from the
 values in the installed version of default_settings.py (see above in
 this document).
 
+You can use the GENENETWORK_FILES switch to set the datadir, for example
+
+: env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=/gnu/data/gn2_data ./bin/genenetwork2
+
 ** Can't run a module
 
 In rare cases, development modules are not brought in with Guix
@@ -257,18 +300,6 @@ if that works run genenetwork after setting SQL_URI to something like
 
 : export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s
 
-* Running ElasticSearch
-
-In order to start up elasticsearch:
-Penguin - change user to "elasticsearch" and use the following command: "env JAVA_HOME=/opt/jdk-9.0.4 /opt/elasticsearch-6.2.1/bin/elasticsearch"
-
-New server - as root run "systemctl restart elasticsearch"
-
-* Read more
-
-If you want to understand the architecture of GN2 read
-[[Architecture.org]].  The rest of this document is mostly on deployment
-of GN2.
 
 * IRC session
 
diff --git a/wqflask/runserver.py b/wqflask/runserver.py
index 5f41d04d..7c06356b 100644
--- a/wqflask/runserver.py
+++ b/wqflask/runserver.py
@@ -27,9 +27,11 @@ app_config()
 
 werkzeug_logger = logging.getLogger('werkzeug')
 
+from utility.tools import WEBSERVER_MODE, SERVER_PORT
+
 if WEBSERVER_MODE == 'DEBUG':
     app.run(host='0.0.0.0',
-            port=port,
+            port=SERVER_PORT,
             debug=True,
             use_debugger=False,
             threaded=False,
@@ -38,7 +40,7 @@ if WEBSERVER_MODE == 'DEBUG':
 elif WEBSERVER_MODE == 'DEV':
     werkzeug_logger.setLevel(logging.WARNING)
     app.run(host='0.0.0.0',
-            port=port,
+            port=SERVER_PORT,
             debug=False,
             use_debugger=False,
             threaded=False,
@@ -46,7 +48,7 @@ elif WEBSERVER_MODE == 'DEV':
             use_reloader=True)
 else: # staging/production modes
     app.run(host='0.0.0.0',
-            port=port,
+            port=SERVER_PORT,
             debug=False,
             use_debugger=False,
             threaded=True,
diff --git a/wqflask/utility/tools.py b/wqflask/utility/tools.py
index 86ef2e1e..8b2260f5 100644
--- a/wqflask/utility/tools.py
+++ b/wqflask/utility/tools.py
@@ -232,6 +232,7 @@ GN_VERSION         = get_setting('GN_VERSION')
 HOME               = get_setting('HOME')
 WEBSERVER_MODE     = get_setting('WEBSERVER_MODE')
 GN_SERVER_URL      = get_setting('GN_SERVER_URL')
+SERVER_PORT        = get_setting_int('SERVER_PORT')
 SQL_URI            = get_setting('SQL_URI')
 LOG_LEVEL          = get_setting('LOG_LEVEL')
 LOG_LEVEL_DEBUG    = get_setting_int('LOG_LEVEL_DEBUG')