1 files changed, 198 insertions, 0 deletions
diff --git a/topics/developing-against-gn.gmi b/topics/developing-against-gn.gmi
new file mode 100644
index 0000000..b94b681
--- /dev/null
+++ b/topics/developing-against-gn.gmi
@@ -0,0 +1,198 @@
+# Developing against GeneNetwork
+
+## Configuration
+
+GeneNetwork2 comes with a [default configuration file](./etc/default_settings.py)
+which can be used as a starting point.
+
+The recommended way to deal with the configurations is to **copy** this default configuration file to a location outside of the repository, say,
+
+```sh
+.../genenetwork2$ cp etc/default_settings.py "${HOME}/configurations/gn2.py"
+```
+
+then change the appropriate values in the new file. You can then pass in the new
+file as the configuration file when launching the application,
+
+```sh
+.../genenetwork2$ bin/genenetwork "${HOME}/configurations/gn2.py" <command-to-run>
+```
+
+The other option is to override the configurations in `etc/default_settings.py`
+by setting the configuration you want to override as an environment variable e.g.
+to override the `SQL_URI` value, you could do something like:
+
+```sh
+.../genenetwork2$ env SQL_URI="mysql://<user>:<passwd>@<host>:<port>/<db_name>" \
+	bin/genenetwork "${HOME}/configurations/gn2.py" <command-to-run>
+```
+
+replacing the placeholders in the angle brackets with appropriate values.
+
+For a detailed breakdown of the configuration variables and their use, see the
+[configuration documentation](doc/configurations.org)
+
+## Run
+
+Once having installed GN2 it can be run through a browser
+interface
+
+```sh
+genenetwork2
+```
+
+A quick example is
+
+```sh
+env GN2_PROFILE=~/opt/gn-latest SERVER_PORT=5300 \
+    GENENETWORK_FILES=~/data/gn2_data/ \
+    GN_PROXY_URL="http://localhost:8080"\
+    GN3_LOCAL_URL="http://localhost:8081"\
+	SPARQL_ENDPOINT=http://localhost:8892/sparql\
+    ./bin/genenetwork2 ./etc/default_settings.py -gunicorn-dev
+```
+
+For full examples (you may need to set a number of environment
+variables), including running scripts and a Python REPL, also see the
+startup script [./bin/genenetwork2](https://github.com/genenetwork/genenetwork2/blob/testing/bin/genenetwork2).
+
+Also mariadb and redis need to be running, see
+[INSTALL](./doc/README.org).
+
+## Debugging
+
+To run the application under the pdb debugger, you can add the `--with-pdb`
+option when launching the application, for example:
+
+```sh
+env GN2_PROFILE=~/opt/gn-latest SERVER_PORT=5300 \
+    GENENETWORK_FILES=~/data/gn2_data/ \
+    GN_PROXY_URL="http://localhost:8080"\
+    GN3_LOCAL_URL="http://localhost:8081"\
+	SPARQL_ENDPOINT=http://localhost:8892/sparql\
+    ./bin/genenetwork2 ./etc/default_settings.py --with-pdb
+```
+
+**NOTE**: This should only ever be run in development.
+**NOTE 2**: You will probably need to tell pdb to continue at least once before
+the system begins serving the pages.
+
+Now, you can add the `breakpoint()` call wherever you need to debug and the
+terminal where you started the application with `--with-pdb` will allow you to
+issue commands to pdb to debug your application.
+
+## Development
+
+It may be useful to pull in the GN3 python modules locally. For this
+use `GN3_PYTHONPATH` environment that gets injected in
+the ./bin/genenetwork2 startup.
+
+A continuously deployed instance of genenetwork2 is available at
+[https://cd.genenetwork.org/](https://cd.genenetwork.org/). This
+instance is redeployed on every commit provided that the [continuous
+integration tests](https://ci.genenetwork.org/jobs/genenetwork2) pass.
+
+## Testing
+
+To have tests pass, the redis and mariadb instance should be running, because of
+asserts sprinkled in the code base.
+
+Right now, the only tests running in CI are unittests. Please make
+sure the existing unittests are green when submitting a PR.
+
+From the root directory of the repository, you can run the tests with something
+like:
+
+```sh
+env GN_PROFILE=~/opt/gn-latest SERVER_PORT=5300 \
+	SQL_URI=<uri-to-override-the-default> \
+	./bin/genenetwork2 ./etc/default_settings.py \
+	-c -m pytest -vv
+```
+
+In the case where you use the default `etc/default_settings.py` configuration file, you can override any setting as demonstrated with the `SQL_URI` setting in the command above.
+
+In order to avoid having to set up a whole host of settings every time with the `env` command, you could copy the `etc/default_settings.py` file to a new location (outside the repository is best), and pass that to `bin/genenetwork2` instead.
+
+See
+[./bin/genenetwork2](https://github.com/genenetwork/genenetwork2/blob/testing/doc/docker-container.org)
+for more details.
+
+#### Mechanical Rob
+
+We are building 'Mechanical Rob' automated testing using Python
+[requests](https://github.com/genenetwork/genenetwork2/tree/testing/test/requests)
+which can be run with:
+
+```sh
+env GN2_PROFILE=~/opt/gn-latest \
+    ./bin/genenetwork2 \
+    GN_PROXY_URL="http://localhost:8080" \
+    GN3_LOCAL_URL="http://localhost:8081 "\
+    ./etc/default_settings.py -c \
+    ../test/requests/test-website.py -a http://localhost:5003
+```
+
+The GN2_PROFILE is the Guix profile that contains all
+dependencies. The ./bin/genenetwork2 script sets up the environment
+and executes test-website.py in a Python interpreter. The -a switch
+says to run all tests and the URL points to the running GN2 http
+server.
+
+#### Unit tests
+
+To run unittests, first `cd` into the genenetwork2 directory:
+
+```sh
+# You can use the coverage tool to run the tests
+# You could omit the -v which makes the output verbose
+runcmd coverage run -m unittest discover -v
+
+# Alternatively, you could run the unittests using:
+runpython -m unittest discover -v
+
+# To generate a report in wqflask/coverage_html_report/:
+runcmd coverage html
+```
+
+The `runcmd` and `runpython` are shell aliases defined in the following way:
+
+```sh
+alias runpython="env GN2_PROFILE=~/opt/gn-latest TMPDIR=/tmp SERVER_PORT=5004 GENENETWORK_FILES=/gnu/data/gn2_data/ GN_PROXY_URL="http://localhost:8080" GN3_LOCAL_URL="http://localhost:8081" ./bin/genenetwork2
+
+alias runcmd="time env GN2_PROFILE=~/opt/gn-latest TMPDIR=//tmp SERVER_PORT=5004 GENENETWORK_FILES=/gnu/data/gn2_data/ GN_PROXY_URL="http://localhost:8080" GN3_LOCAL_URL="http://localhost:8081" ./bin/genenetwork2 ./etc/default_settings.py -cli"
+```
+
+Replace some of the env variables as per your use case.
+
+### Troubleshooting
+
+If the menu does not pop up check your `GN2_BASE_URL`. E.g.
+
+```
+curl http://gn2-pjotr.genenetwork.org/api/v_pre1/gen_dropdown
+```
+
+check the logs. If there is ERROR 1054 (42S22): Unknown column
+'InbredSet.Family' in 'field list' it may be you are trying the small
+database.
+
+### Run Scripts
+
+As part of the profiling effort, some scripts are added to run specific parts of the system under a profiler without running the entire web-server - as such, to run the script, you could do something like:
+
+```
+env HOME=/home/frederick \
+    GN2_PROFILE=~/opt/gn2-latest \
+    GN3_DEV_REPO_PATH=~/genenetwork/genenetwork3 \
+    SQL_URI="mysql://username:password@host-ip:host-port/db_webqtl" \
+	SPARQL_ENDPOINT=http://localhost:8892/sparql\
+    SERVER_PORT=5001 \
+    bin/genenetwork2 ../gn2_settings.py \
+    -cli python3 -m scripts.profile_corrs \
+	../performance_$(date +"%Y%m%dT%H:%M:%S").profile
+```
+
+and you can find the performance metrics at the file specified, in this case, a file starting with `performance_` with the date and time of the run, and ending with `.profile`.
+
+Please replace the environment variables in the sample command above with the appropriate values for your environment.