Rename/move

1 files changed, 321 insertions, 0 deletions

diff --git a/topics/database/setting-up-local-development-database.gmi b/topics/database/setting-up-local-development-database.gmi
new file mode 100644
index 0000000..3b743b9
--- /dev/null
+++ b/topics/database/setting-up-local-development-database.gmi
@@ -0,0 +1,321 @@
+# Setting up Local Development Database
+
+You need to set up a quick local database for development without polluting your environment.
+
+## Method 0 (tunnel)
+
+You can use ssh tunneling to access mysql from your machine. Try something like:
+
+```
+ssh -L 3306:127.0.0.1:3306 -f -N tux02.genenetwork.org
+mysql -uwebqtlout -pwebqtlout -h 127.0.0.1 db_webqtl -A -e "show tables;"
+```
+
+If you already have that port in use on your machine try
+
+```
+ssh -L 3307:127.0.0.1:3306 -f -N tux02.genenetwork.org
+mysql -uwebqtlout -pwebqtlout -h 127.0.0.1 -P 3307 db_webqtl -A -e "show tables;"
+```
+
+To keep the connection alive add something like this to your `~/.ssh/config`
+
+```
+ServerAliveInterval 60
+ServerAliveCountMax 10
+```
+
+For specific hosts you can set it up as
+
+```
+Host tux02
+  HostName tux02.genenetwork.org
+  TCPKeepAlive yes
+  ServerAliveInterval 60
+  user myname
+```
+
+## Method 1 (Using Guix system containers)
+
+Setting up mariadb in a Guix container is the preferred and easier method. But, you need root access to run the container. The genenetwork2 repo comes with a guix system container definition to run MariaDB and Redis. From the genenetwork2 repo, you can build and run the container using:
+```
+$ sudo $(./containers/db-container.sh)
+```
+You should now be able to connect to the database using
+```
+$ mysql --protocol tcp -u root
+```
+Create a database db_webqtl_s
+```
+MariaDB [mysql]> CREATE DATABASE db_webqtl_s;
+```
+Load the small database dump into the database. You may find the small database either on space or tux02 at /home/aruni/gn2.sql.lz
+```
+$ lzip -cd gn2.sql.lz | mysql --protocol tcp -u root db_webqtl_s
+```
+Since this is a develpoment server accessible only from localhost, it is ok to use the root user with no password. Configure your development instance of genenetwork2 with the following SQL_URI.
+```
+SQL_URI="mysql://root@127.0.0.1:3306/db_webqtl_s"
+```
+
+## Method 2 (running a local container)
+
+A local container runs as a normal user. It requires some juggling of dirs to load an existing database. Make sure to not run on the same dirs as another mariadb instance(!)
+Here we run a new database, but you can use an existing DB as was done in
+
+=> precompute-mapping-input-data.gmi
+
+```
+cd ~/tmp/mariadb
+mkdir var
+~/tmp/mariadb$ ~/opt/guix-pull/bin/guix shell -C -N coreutils sed mariadb --share=var=/var --share=/export2/tmp=/tmp
+```
+
+inside the container
+
+```
+mkdir -p /var/lib/data
+mkdir -p /var/lib/mysql
+mkdir /var/run
+```
+
+Initialize with
+
+```
+mysql_install_db
+```
+
+and run, for example
+
+```
+export TMPDIR=/tmp
+mysqld_safe --datadir='/var/lib/mysql/' --port=3307 --user=$USER --group=users --nowatch --socket=/var/run/mysqld/mysqld.sock
+```
+
+Now from the container you should be able to connect with the socket
+
+```
+/export/mysql$ ~/opt/guix-pull/bin/guix shell mysql -- mysql --socket=var/run/mysqld/mysqld.sock -uwebqtlout -pwebqtlout db_webqtl
+```
+
+and
+
+```
+Welcome to the MariaDB monitor.  Commands end with ; or \g.
+Your MariaDB connection id is 3
+Server version: 10.5.12-MariaDB MariaDB Server
+
+Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
+
+Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
+
+MariaDB [(none)]> show databases;
++--------------------+
+| Database           |
++--------------------+
+| information_schema |
+| mysql              |
+| performance_schema |
+| test               |
++--------------------+
+4 rows in set (0.001 sec)
+```
+
+To run/bind on a network interface we can open up with:
+
+```
+export TMPDIR=/tmp
+mysqld_safe --datadir='/var/lib/mysql/' --port=3307 --user=$USER --group=users --nowatch --bind-address 127.0.0.1 --socket=/var/run/mysqld/mysqld.sock
+```
+
+Test with
+
+
+```
+ ~/opt/guix-pull/bin/guix shell mysql -- mysql -uwebqtlout -pwebqtlout db_webqtl -h 127.0.0.1 --port=3307
+```
+
+Please don't use world writeable bind-address 0.0.0.0. If you run it for your own purposes use ssh tunneling instead (see above).
+
+If you need to tweak the server configuration you can load the my.cnf file with the `--defaults-file=var/my.cnf` inside and outside the container.
+
+## Method 3 (Manual method without Guix)
+
+/This is not recommended/
+
+* An assumption is made that the GeneNetwork2 profile is in ~/opt/gn_profiles/gn2_latest for the purposes of this documentation. Please replace as appropriate.
+* We install the database files under ~/genenetwork/mariadb. Change as appropriate.
+
+Set up directories
+
+```
+mkdir -pv ~/genenetwork/mariadb/var/run
+mkdir -pv ~/genenetwork/mariadb/var/lib/data
+mkdir -pv ~/genenetwork/mariadb/var/lib/mysql
+```
+
+Set up default my.cnf
+
+```
+cat <<EOF > ~/genenetwork/mariadb/my.cnf
+[client-server]
+socket=~/genenetwork/mariadb/var/run/mysqld/mysqld.sock
+port=3307
+
+[server]
+user=$(whoami)
+socket=~/genenetwork/mariadb/var/run/mysqld/mysqld.sock
+basedir=~/opt/gn_profiles/gn2_latest
+datadir=~/genenetwork/mariadb/var/lib/data
+ft_min_word_len=3
+EOF
+```
+
+Install the database
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql_install_db \
+    --defaults-file=~/genenetwork/mariadb/my.cnf
+```
+
+Running the daemon:
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysqld_safe \
+    --defaults-file=~/genenetwork/mariadb/my.cnf
+```
+
+Connect to daemon
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf
+```
+
+Set up password for user
+
+```
+MariaDB [(none)]> USE mysql;
+MariaDB [mysql]> ALTER USER '<your-username>'@'localhost' IDENTIFIED BY '<the-new-password>';
+MariaDB [mysql]> FLUSH PRIVILEGES;
+```
+
+Now logout and login again with
+
+```
+$ ~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf --password mysql
+```
+
+enter the newly set password and voila, you are logged in and your user has the password set up.
+
+Now, set up a new user, say webqtlout, and a default database they can connect to
+
+```
+MariaDB [mysql]> CREATE DATABASE webqtlout;
+MariaDB [mysql]> CREATE USER 'webqtlout'@'localhost' IDENTIFIED BY '<some-password>';
+MariaDB [mysql]> GRANT ALL PRIVILEGES ON webqtlout.* TO 'webqtlout'@'localhost';
+```
+
+Now logout, and log back in as the new webqtlout user:
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf \
+    --user=webqtlout --host=localhost --password webqtlout
+```
+
+and enter the password you provided.
+
+### Setting up the Small Database
+
+Download the database from
+
+=> http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk
+
+Say you downloaded the file in ~/Downloads, you can now add the database to your server.
+
+First stop the server:
+
+```
+$ ps aux | grep mysqld # get the process ids
+$ kill -s SIGTERM <pid-of-mysqld> <pid-of-mysqld_safe>
+```
+
+Now extract the database archive in the mysql data directory:
+
+```
+$ cd ~/genenetwork/mariadb/var/lib/data
+$ p7zip -k -d ~/Downloads/db_webqtl_s.7z
+```
+
+Now restart the server:
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysqld_safe \
+    --defaults-file=~/genenetwork/mariadb/my.cnf
+```
+
+Then update the databases
+
+```
+$ ~/opt/gn_profiles/gn2_latest/bin/mysql_upgrade \
+    --defaults-file=~/genenetwork/mariadb/my.cnf \
+    --user=frederick --password --force
+```
+
+and login as the administrative user:
+
+```
+$ ~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf \
+    --user=$(whoami) --password
+```
+
+and grant the privileges to your normal user:
+
+```
+MariaDB [mysql]> GRANT ALL PRIVILEGES ON db_webqtl_s.* TO 'webqtlout'@'localhost';
+```
+
+now logout as the administrative user and log back in as the normal user
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf \
+    --user=webqtlout --host=localhost --password db_webqtlout_s
+
+MariaDB [db_webqtlout_s]> SELECT * FROM ProbeSetData LIMIT 20;
+```
+
+verify you see some data.
+
+### A Note on Connection to the Server
+
+So far, we have been connecting to the server by specifying --defaults-file option, e.g.
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --defaults-file=~/genenetwork/mariadb/my.cnf \
+    --user=webqtlout --host=localhost --password db_webqtlout_s
+```
+
+which allows connection via the unix socket.
+
+We could drop that specification and connect via the port with:
+
+```
+~/opt/gn_profiles/gn2_latest/bin/mysql \
+    --user=webqtlout --host=127.0.0.1 --port=3307 --password db_webqtlout_s
+```
+
+In this version, the host specification was changed from
+```
+--host=localhost
+```
+to
+```
+--host=127.0.0.1
+```
+
+whereas, the **--defaults-file** file specification was dropped and a new **--port** specification was added.