Updating READMEs

2 files changed, 9 insertions, 388 deletions

diff --git a/README.md b/README.md
index 2e61f0b9..e5a50fac 100644
--- a/README.md
+++ b/README.md
@@ -5,12 +5,7 @@ badge](https://ci.genenetwork.org/badge/genenetwork2.svg)](https://ci.genenetwor
 
 # GeneNetwork
 
-This repository contains the current source code for GeneNetwork (GN2)
-https://genenetwork.org/ (version 2). GN2 is a Web
-2.0-style framework that includes data and computational tools for online genetics and genomic analysis of
-many different populations and many types of molecular, cellular, and physiological data.
-The system is used by scientists and clinicians in the field of precision health care and systems genetics.
-GN and its predecessors have been in operation since Jan 1994, making it one of the longest-lived web services in biomedical research (https://en.wikipedia.org/wiki/GeneNetwork, and see a partial list of publications using GN and its predecessor, WebQTL (https://genenetwork.org/references/).
+This repository contains the current source code for GeneNetwork (GN2) https://genenetwork.org/ (version 2). GN2 is a Web 2.0-style framework that includes data and computational tools for online genetics and genomic analysis of many different populations and many types of molecular, cellular, and physiological data. The system is used by scientists and clinicians in the field of precision health care and systems genetics. GN and its predecessors have been in operation since Jan 1994, making it one of the longest-lived web services in biomedical research (https://en.wikipedia.org/wiki/GeneNetwork, and see a partial list of publications using GN and its predecessor, WebQTL https://genenetwork.org/references/).
 
 ## Install
 
@@ -19,19 +14,17 @@ The recommended installation is with GNU Guix which allows you to deploy GN2 and
 ## Documentation
 
 User documentation can be found [here](https://genenetwork.org/help).  The architecture of the software stack is described [here](./doc/Architecture.org).  The database schema is (still) shared with GN1 and currently described [here](http://www.genenetwork.org/webqtl/main.py?FormID=schemaShowPage). Software documentation is being expanded in the [source code repository](https://github.com/genenetwork/genenetwork2/tree/master/doc).
+
 ## Contributing
 
 Issues can be raised through
-[github](https://github.com/genenetwork/genenetwork2/issues).
+[github](https://issues.genenetwork.org/).
 
 Contribute to GN2 source code by forking the
 [github repository](https://github.com/genenetwork/genenetwork2/) with
 git and sending us pull requests.
 
-For development GN2 has a [mailing
-list](http://listserv.uthsc.edu/mailman/listinfo/genenetwork-dev) and
-an active IRC channel #genenetwork on freenode.net with a [web
-interface](http://webchat.freenode.net/).
+For development GN2 has a room on matrix. Ask us for an invitation.
 
 ## License
 
@@ -41,7 +34,7 @@ Public License 3 (AGPLv3). See [LICENSE.txt](LICENSE.txt).
 
 ## More information
 
-For more information visit http://www.genenetwork.org/
+For more information visit https://genenetwork.org/
 
 ## Cite
 
@@ -49,13 +42,8 @@ For more information visit http://www.genenetwork.org/
 
 GeneNetwork was published in the Journal of Open Source Software as 'GeneNetwork: framework for web-based genetics' by Zachary Sloan, Danny Arends, Karl W. Broman, Arthur Centeno, Nicholas Furlotte, Harm Nijveen, Lei Yan, Xiang Zhou, Robert W. WIlliams and Pjotr Prins
 
-You may also cite the software using
-
-[![DOI](https://zenodo.org/badge/5591/genenetwork/genenetwork2.svg)](https://zenodo.org/badge/latestdoi/5591/genenetwork/genenetwork2).
-
 ## Contact
 
-IRC on #genenetwork on irc.freenode.net.
+E-mail: pjotr.public729@thebird.nl
 
-Code and primary web service managed by Dr. Robert W. Williams and the
-University of Tennessee Health Science Center, Memphis TN, USA.
+Code and primary web service managed by Drs. Robert W. Williams and Pjotr Prins at the University of Tennessee Health Science Center, Memphis TN, USA.
diff --git a/doc/README.org b/doc/README.org
index d4dc1027..575fd42e 100644
--- a/doc/README.org
+++ b/doc/README.org
@@ -1,371 +1,4 @@
 #+TITLE: Installing GeneNetwork services
 
-* Table of Contents                                                     :TOC:
- - [[#introduction][Introduction]]
- - [[#check-list][Check list]]
- - [[#installing-guix-packages][Installing Guix packages]]
- - [[#creating-a-gnu-guix-profile][Creating a GNU Guix profile]]
- - [[#running-gn2][Running GN2]]
- - [[#run-gn-proxy][Run gn-proxy]]
- - [[#run-redis][Run Redis]]
- - [[#run-mariadb-server][Run MariaDB server]]
-   - [[#install-mariadb-with-gnu-guix][Install MariaDB with GNU GUIx]]
-   - [[#load-the-small-database-in-mysql][Load the small database in MySQL]]
- - [[#get-genotype-files][Get genotype files]]
- - [[#gn2-dependency-graph][GN2 Dependency Graph]]
- - [[#working-with-the-gn2-source-code][Working with the GN2 source code]]
- - [[#read-more][Read more]]
- - [[#trouble-shooting][Trouble shooting]]
-   - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]]
-   - [[#error-can-not-find-directory-homegn2_data-or-can-not-find-directory-homegenotype_filesgenotype][ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype']]
-   - [[#cant-run-a-module][Can't run a module]]
-   - [[#rpy2-error-show-now-found][Rpy2 error 'show' now found]]
-   - [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]]
- - [[#notes][NOTES]]
-   - [[#deploying-gn2-official][Deploying GN2 official]]
-
-* Introduction
-
-Large system deployments can get very [[http://genenetwork.org/environments/][complex]]. In this document we
-explain the GeneNetwork version 2 (GN2) reproducible deployment system
-which is based on GNU Guix (see also [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix
-system can be used to install GN with all its files and dependencies.
-
-The official installation path is from a checked out version of the
-main Guix package tree and that of the Genenetwork package
-tree. Current supported versions can be found as the SHA values of
-'gn-latest' branches of [[https://gitlab.com/genenetwork/guix-bioinformatics][Guix bioinformatics]] and [[https://gitlab.com/genenetwork/guix][GNU Guix]].
-
-For a full view of runtime dependencies as defined by GNU Guix, see
-an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]].
-
-* Check list
-
-To run GeneNetwork the following services need to function:
-
-1. [ ] GNU Guix with a guix profile for genenetwork2
-1. [ ] A path to the (static) genotype files
-1. [ ] Gn-proxy for authentication
-1. [ ] The genenetwork3 service
-1. [ ] Redis
-1. [ ] Mariadb
-
-* Installing Guix packages
-
-Make sure to install GNU Guix using the binary download instructions
-on the main website. Follow the instructions on
-[[GUIX-Reproducible-from-source.org]] to download pre-built binaries. Note
-the download amounts to several GBs of data. Debian-derived distros
-may support
-
-: apt-get install guix
-
-* Creating a GNU Guix profile
-
-We run a GNU Guix channel with packages at [[https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics][guix-bioinformatics]]. The README has instructions for hosting a channel, but typically we use the GUIX_PACKAGE_PATH instead. First upgrade to a recent guix with
-
-: mkdir ~/opt
-: guix pull -p ~/opt/guix-pull
-
-It should upgrade (ignore the locales warnings).  You can optionally specify the specific git checkout of guix with
-
-: guix pull -p ~/opt/guix-pull --commit=f04883d
-
-which is useful when you need to roll back to an earlier version (sometimes our channel goes out of sync). Next, we install GeneNetwork2 with
-
-: source ~/opt/guix-pull/etc/profile
-: git clone https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.git ~/guix-bioinformatics
-: cd ~/guix-bioinformatics
-: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics guix package -i genenetwork2 -p ~/opt/genenetwork2
-
-you probably also need guix-past (the upstream channel for older packages):
-
-: git clone https://gitlab.inria.fr/guix-hpc/guix-past.git ~/guix-past
-: cd ~/guix-past
-: env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics:$HOME/guix-past/modules ~/opt/guix-pull/bin/guix package -i genenetwork2 -p ~/opt/genenetwork2
-
-ignore the warnings. Guix should install the software without trying
-to build everything. If you system insists on building all packages,
-try the `--dry-run` switch and fix the [[https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html][substitutes]]. You may add the
-`--substitute-urls="http://guix.genenetwork.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org"` switch.
-
-The guix.genenetwork.org has most of our packages pre-built(!). To use
-it on your own machine the public key is
-
-#+begin_src scheme
-(public-key
- (ecc
-  (curve Ed25519)
-  (q #E50F005E6DA2F85749B9AA62C8E86BB551CE2B541DC578C4DBE613B39EC9E750#)))
-#+end_src
-
-Once we have a GNU Guix profile, a running database (see below) and the file storage,
-we should be ready to fire up GeneNetwork:
-
-* Running GN2
-
-Check out the source with git:
-
-: git clone git@github.com:genenetwork/genenetwork2.git
-: cd genenetwork2
-
-Run GN2 with above Guix profile
-
-: export GN2_PROFILE=$HOME/opt/genenetwork2
-: env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5012 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 etc/default_settings.py -gunicorn-dev
-
-the debug and logging switches can be particularly useful when
-developing GN2. Location and files are the current ones for Penguin2.
-
-It may be useful to tunnel the web server to your local browser with
-an ssh tunnel:
-
-If you want to test a service running on the server on a certain
-port (say 8202) use
-
-  ssh -L 8202:127.0.0.1:8202 -f -N myname@penguin2.genenetwork.org
-
-And browse on your local machine to http://localhost:8202/
-
-* Run gn-proxy
-
-GeneNetwork requires a separate gn-proxy server which handles
-authorisation and access control. For instructions see the
-[[https://github.com/genenetwork/gn-proxy][README]]. Note it may already be running on our servers!
-
-* Run Redis
-
-Redis part of GN2 deployment and will be started by the ./bin/genenetwork2
-startup script.
-
-* Run MariaDB server
-** Install MariaDB with GNU GUIx
-
-These are the steps you can take to install a fresh installation of
-mariadb (which comes as part of the GNU Guix genenetwork2 install).
-
-As root configure the Guix profile
-
-: . ~/opt/genenetwork2/etc/profile
-
-and run for example
-
-#+BEGIN_SRC bash
-adduser mariadb && addgroup mariadb
-mkdir -p /export2/mariadb/database
-chown mariadb.mariadb -R /export2/mariadb/
-mkdir -p /var/run/mysqld
-chown mariadb.mariadb /var/run/mysqld
-su mariadb
-mysql --version
-  mysql  Ver 15.1 Distrib 10.1.45-MariaDB, for Linux (x86_64) using readline 5.1
-mysql_install_db --user=mariadb --datadir=/export2/mariadb/database
-mysqld -u mariadb --datadir=/exportdb/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048"
-#+END_SRC
-
-If you want to run as root you may have to set
-
-: /etc/my.cnf
-: [mariadbd]
-: user=root
-
-You also need to set
-
-: ft_min_word_len = 3
-
-To make sure word text searches (shh) work and rebuild the tables if
-required.
-
-To check error output in a file on start-up run with something like
-
-: mariadbd -u mariadb --console  --explicit_defaults_for_timestamp  --datadir=/gnu/mariadb --log-error=~/test.log
-
-Other tips are that Guix installs mariadbd in your profile, so this may work
-
-: /home/user/.guix-profile/bin/mariadbd -u mariadb --explicit_defaults_for_timestamp  --datadir=/gnu/mariadb
-
-When you get errors like:
-
-: qlalchemy.exc.IntegrityError: (_mariadb_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint')
-
-you may need to set
-
-: set foreign_key_checks=0
-
-** Load the small database in MySQL
-
-At this point we require the underlying distribution to install and
-run mysqld (see next section for GNU Guix). Currently we have two databases for deployment,
-'db_webqtl_s' is the small testing database containing experiments
-from BXD mice and 'db_webqtl_plant' which contains all plant related
-material.
-
-Download one database from
-
-http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk
-
-After installation unzip the database binary in the MySQL directory
-
-#+BEGIN_SRC sh
-cd ~/mysql
-p7zip -d db_webqtl_s.7z
-chown -R mysql:mysql db_webqtl_s/
-chmod 700 db_webqtl_s/
-chmod 660 db_webqtl_s/*
-#+END_SRC
-
-restart MySQL service (mysqld). Login as root
-
-: mysql_upgrade -u root --force
-
-: myslq -u root
-
-and
-
-: mysql> show databases;
-: +--------------------+
-: | Database           |
-: +--------------------+
-: | information_schema |
-: | db_webqtl_s        |
-: | mysql              |
-: | performance_schema |
-: +--------------------+
-
-Set permissions and match password in your settings file below:
-
-: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'webqtl';
-
-You may need to change "localhost" to whatever domain you are
-connecting from (mysql will give an error).
-
-Note that if the mysql connection is not working, try connecting to
-the IP address and check server firewall, hosts.allow and mysql IP
-configuration (see below).
-
-Note for the plant database you can rename it to db_webqtl_s, or
-change the settings in etc/default_settings.py to match your path.
-
-* Get genotype files
-
-The script looks for genotype files. You can find them in
-http://ipfs.genenetwork.org/ipfs/QmXQy3DAUWJuYxubLHLkPMNCEVq1oV7844xWG2d1GSPFPL
-
-#+BEGIN_SRC sh
-mkdir -p $HOME/genotype_files
-cd $HOME/genotype_files
-
-#+END_SRC
-
-* GN2 Dependency Graph
-
-List of all runtime dependencies for GN2 as installed by GNU Guix.
-
-https://genenetwork.org/environments/
-
-* Working with the GN2 source code
-
-See [[development.org]].
-
-* Read more
-
-If you want to understand the architecture of GN2 read
-[[Architecture.org]].  The rest of this document is mostly on deployment
-of GN2.
-
-* Trouble shooting
-
-** ImportError: No module named jinja2
-
-If you have all the Guix packages installed this error points out that
-the environment variables are not set. Copy-paste the paths into your
-terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
-information given by guix:
-
-: guix package --search-paths
-
-On one system:
-
-: export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages"
-: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
-: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
-
-and perhaps a few more.
-** ERROR: 'can not find directory $HOME/gn2_data' or 'can not find directory $HOME/genotype_files/genotype'
-
-The default settings file looks in your $HOME/gn2_data. Since these
-files come with a Guix installation you should take a hint from the
-values in the installed version of default_settings.py (see above in
-this document).
-
-You can use the GENENETWORK_FILES switch to set the datadir, for example
-
-: env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=/gnu/data/gn2_data ./bin/genenetwork2
-
-** Can't run a module
-
-In rare cases, development modules are not brought in with Guix
-because no source code is available. This can lead to missing modules
-on a running server. Please check with the authors when a module
-is missing.
-** Rpy2 error 'show' now found
-
-This error
-
-: __show = rpy2.rinterface.baseenv.get("show")
-: LookupError: 'show' not found
-
-means that R was updated in your path, and that Rpy2 needs to be
-recompiled against this R - don't you love informative messages?
-
-In our case it means that GN's PYTHONPATH is not in sync with
-R_LIBS_SITE. Please check your GNU Guix GN2 installation paths,
-you man need to reinstall. Note that this may be the point you
-may want to start using profiles (see profile section).
-
-** Mysql can't connect server through socket ERROR
-
-The following error
-
-: sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")')
-
-means that MySQL is trying to connect locally to a non-existent MySQL
-server, something you may see in a container. Typically replicated with something like
-
-: mysql -h localhost
-
-try to connect over the network interface instead, e.g.
-
-: mysql -h 127.0.0.1
-
-if that works run genenetwork after setting SQL_URI to something like
-
-: export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s
-
-* NOTES
-
-** Deploying GN2 official
-
-Let's see how fast we can deploy a second copy of GN2.
-
-- [ ] Base install
-  + [ ] First install a Debian server with GNU Guix on board
-  + [ ] Get Guix build going
-    - [ ] Build the correct version of Guix
-    - [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics
-    - [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable
-  + [ ] Create a gn2 user and home with space
-  + [ ] Install redis
-    - [ ] add to systemd
-    - [ ] update redis.cnf
-    - [ ] update database
-  + [ ] Install mariadb (currently debian mariadb-server)
-    - [ ] add to systemd
-    - [ ] system stop mysql
-    - [ ] update mysql.cnf
-    - [ ] update database (see gn-services/services/mariadb.md)
-    - [ ] check tables
-  + [ ] run gn2
-  + [ ] update nginx
-  + [ ] install genenetwork3
-    - [ ] add to systemd
+This information has moved to https://issues.genenetwork.org/topics/deployment
+and https://issues.genenetwork.org/topics/installation.