path: root/topics/setting-up-local-development-database.gmi

# Setting up Local Development Database

You need to set up a quick local database for development without polluting your environment.

## Method 0 (tunnel)

You can use ssh tunneling to access mysql from your machine. Try something like:

```
ssh -L 3306:127.0.0.1:3306 -f -N tux02.genenetwork.org
mysql -uwebqtlout -pwebqtlout -h 127.0.0.1 db_webqtl -A -e "show tables;"
```

If you already have that port in use on your machine try

```
ssh -L 3307:127.0.0.1:3306 -f -N tux02.genenetwork.org
mysql -uwebqtlout -pwebqtlout -h 127.0.0.1 -P 3307 db_webqtl -A -e "show tables;"
```

To keep the connection alive add something like this to your `~/.ssh/config`

```
ServerAliveInterval 60
ServerAliveCountMax 10
```

For specific hosts you can set it up as

```
Host tux02
  HostName tux02.genenetwork.org
  TCPKeepAlive yes
  ServerAliveInterval 60
  user myname
```

## Method 1 (Using Guix system containers)

Setting up mariadb in a Guix container is the preferred and easier method. But, you need root access to run the container. The genenetwork2 repo comes with a guix system container definition to run MariaDB and Redis. From the genenetwork2 repo, you can build and run the container using:
```
$ sudo $(./containers/db-container.sh)
```
You should now be able to connect to the database using
```
$ mysql --protocol tcp -u root
```
Create a database db_webqtl_s
```
MariaDB [mysql]> CREATE DATABASE db_webqtl_s;
```
Load the small database dump into the database. You may find the small database either on space or tux02 at /home/aruni/gn2.sql.lz
```
$ lzip -cd gn2.sql.lz | mysql --protocol tcp -u root db_webqtl_s
```
Since this is a develpoment server accessible only from localhost, it is ok to use the root user with no password. Configure your development instance of genenetwork2 with the following SQL_URI.
```
SQL_URI="mysql://root@127.0.0.1:3306/db_webqtl_s"
```

## Method 2 (running a local container)

A local container runs as a normal user. It requires some juggling of dirs to load an existing database. Make sure to not run on the same dirs as another mariadb instance(!)
Here we run a new database, but you can use an existing DB as was done in

=> precompute-mapping-input-data.gmi

```
cd ~/tmp/mariadb
mkdir var
~/tmp/mariadb$ ~/opt/guix-pull/bin/guix shell -C -N coreutils sed mariadb --share=var=/var --share=/export2/tmp=/tmp
```

inside the container

```
mkdir -p /var/lib/data
mkdir -p /var/lib/mysql
mkdir /var/run
```

Initialize with

```
mysql_install_db
```

and run, for example

```
export TMPDIR=/tmp
mysqld_safe --datadir='/var/lib/mysql/' --port=3307 --user=$USER --group=users --nowatch --socket=/var/run/mysqld/mysqld.sock
```

Now from the container you should be able to connect with the socket

```
/export/mysql$ ~/opt/guix-pull/bin/guix shell mysql -- mysql --socket=var/run/mysqld/mysqld.sock -uwebqtlout -pwebqtlout db_webqtl
```

and

```
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.5.12-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.001 sec)
```

To run/bind on a network interface we can open up completely (dangerous) with:

```
export TMPDIR=/tmp
mysqld_safe --datadir='/var/lib/mysql/' --port=3307 --user=$USER --group=users --nowatch --bind-address 0.0.0.0 --socket=/var/run/mysqld/mysqld.sock
```

Test


```
 ~/opt/guix-pull/bin/guix shell mysql -- mysql -uwebqtlout -pwebqtlout db_webqtl -h 127.0.0.1 --port=3307
```

If you need to tweak the server configuration you can load the my.cnf file with the `--defaults-file=var/my.cnf` inside and outside the container.

## Method 3 (Manual method without Guix)

/This is not recommended/

* An assumption is made that the GeneNetwork2 profile is in ~/opt/gn_profiles/gn2_latest for the purposes of this documentation. Please replace as appropriate.
* We install the database files under ~/genenetwork/mariadb. Change as appropriate.

Set up directories

```
mkdir -pv ~/genenetwork/mariadb/var/run
mkdir -pv ~/genenetwork/mariadb/var/lib/data
mkdir -pv ~/genenetwork/mariadb/var/lib/mysql
```

Set up default my.cnf

```
cat <<EOF > ~/genenetwork/mariadb/my.cnf
[client-server]
socket=~/genenetwork/mariadb/var/run/mysqld/mysqld.sock
port=3307

[server]
user=$(whoami)
socket=~/genenetwork/mariadb/var/run/mysqld/mysqld.sock
basedir=~/opt/gn_profiles/gn2_latest
datadir=~/genenetwork/mariadb/var/lib/data
ft_min_word_len=3
EOF
```

Install the database

```
~/opt/gn_profiles/gn2_latest/bin/mysql_install_db \
    --defaults-file=~/genenetwork/mariadb/my.cnf
```

Running the daemon:

```
~/opt/gn_profiles/gn2_latest/bin/mysqld_safe \
    --defaults-file=~/genenetwork/mariadb/my.cnf
```

Connect to daemon

```
~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf
```

Set up password for user

```
MariaDB [(none)]> USE mysql;
MariaDB [mysql]> ALTER USER '<your-username>'@'localhost' IDENTIFIED BY '<the-new-password>';
MariaDB [mysql]> FLUSH PRIVILEGES;
```

Now logout and login again with

```
$ ~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf --password mysql
```

enter the newly set password and voila, you are logged in and your user has the password set up.

Now, set up a new user, say webqtlout, and a default database they can connect to

```
MariaDB [mysql]> CREATE DATABASE webqtlout;
MariaDB [mysql]> CREATE USER 'webqtlout'@'localhost' IDENTIFIED BY '<some-password>';
MariaDB [mysql]> GRANT ALL PRIVILEGES ON webqtlout.* TO 'webqtlout'@'localhost';
```

Now logout, and log back in as the new webqtlout user:

```
~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf \
    --user=webqtlout --host=localhost --password webqtlout
```

and enter the password you provided.

### Setting up the Small Database

Download the database from

=> http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk

Say you downloaded the file in ~/Downloads, you can now add the database to your server.

First stop the server:

```
$ ps aux | grep mysqld # get the process ids
$ kill -s SIGTERM <pid-of-mysqld> <pid-of-mysqld_safe>
```

Now extract the database archive in the mysql data directory:

```
$ cd ~/genenetwork/mariadb/var/lib/data
$ p7zip -k -d ~/Downloads/db_webqtl_s.7z
```

Now restart the server:

```
~/opt/gn_profiles/gn2_latest/bin/mysqld_safe \
    --defaults-file=~/genenetwork/mariadb/my.cnf
```

Then update the databases

```
$ ~/opt/gn_profiles/gn2_latest/bin/mysql_upgrade \
    --defaults-file=~/genenetwork/mariadb/my.cnf \
    --user=frederick --password --force
```

and login as the administrative user:

```
$ ~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf \
    --user=$(whoami) --password
```

and grant the privileges to your normal user:

```
MariaDB [mysql]> GRANT ALL PRIVILEGES ON db_webqtl_s.* TO 'webqtlout'@'localhost';
```

now logout as the administrative user and log back in as the normal user

```
~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf \
    --user=webqtlout --host=localhost --password db_webqtlout_s

MariaDB [db_webqtlout_s]> SELECT * FROM ProbeSetData LIMIT 20;
```

verify you see some data.

### A Note on Connection to the Server

So far, we have been connecting to the server by specifying --defaults-file option, e.g.

```
~/opt/gn_profiles/gn2_latest/bin/mysql \
    --defaults-file=~/genenetwork/mariadb/my.cnf \
    --user=webqtlout --host=localhost --password db_webqtlout_s
```

which allows connection via the unix socket.

We could drop that specification and connect via the port with:

```
~/opt/gn_profiles/gn2_latest/bin/mysql \
    --user=webqtlout --host=127.0.0.1 --port=3307 --password db_webqtlout_s
```

In this version, the host specification was changed from
```
--host=localhost
```
to
```
--host=127.0.0.1
```

whereas, the **--defaults-file** file specification was dropped and a new **--port** specification was added.