From 9f49e2dbdf4e06fcb39ed5146560937f228682a7 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 22 Sep 2020 04:31:05 +0300 Subject: Update documentation on testing and CI * README.md: Update section on testing. * doc/docker-container.org: Add it. --- doc/docker-container.org | 86 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 doc/docker-container.org (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org new file mode 100644 index 00000000..3c9864c5 --- /dev/null +++ b/doc/docker-container.org @@ -0,0 +1,86 @@ +#+TITLE: Genenetwork2 Dockerized + +* Table of Contents :TOC: +- [[#introduction][Introduction]] +- [[#creating-the-docker-images][Creating the Docker Images]] +- [[#pushing-to-dockerhub][Pushing to DockerHub]] + +* Introduction + +The CI(Continuous Integration) system for Genenetwork2 uses [[https://github.com/features/actions][Github +Actions]]. As such, it's important to have a way to run tests using +facilities provided by GUIX in a reproducible way. This project +leverages GUIX to generate a docker container from which the unittests +are ran from. + +Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This +document will not get into that. It's assumed that you have a working +dockec setup. + +The rest of this document outlines how the docker container used in +the CI builds were created. + +* Creating the Docker Images + +First create the image by running: + +#+begin_src sh +# For the Python 2 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash genenetwork2 + +# For the Python 3 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash python3-genenetwork2 + #+end_src + +The output will look something similar to: + +: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Load the docker image by running: + +: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Results look something similar to: + +#+begin_export ascii +a93f52b7f565: Loading layer 3.174GB/3.174GB +Loaded image: coreutils-genenetwork2:latest +#+end_export + +Assuming you have a docker instance running, you could always run +commands in it e.g: + +: docker run "coreutils-genenetwork2:latest" python --version + + +* Pushing to DockerHub + +We use DockerHub to store the docker images from which we use on our +CI environment using Github Actions. + +To push to dockerhub, first get the image name by running =docker +images=. Push to dockerhub using a command similar to: + +: docker push bonfacekilz/python2-genenetwork2:latest + +Right now, we have 2 images on DockerHub: + +- https://hub.docker.com/repository/docker/bonfacekilz/python2-genenetwork2: + Contains the python2 version of gn2. Don't use this. Please use the + python3 image! +- https://hub.docker.com/repository/docker/bonfacekilz/python3-genenetwork2: + Contains the python3 version of gn2. -- cgit v1.2.3 From 3ec4eb6b831eaa5adcf32a9fca8a60ea229cc1c4 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Fri, 25 Sep 2020 04:09:35 +0300 Subject: Update docs on CI setup with docker * doc/docker-container.org: Update it. --- doc/docker-container.org | 61 +++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 29 deletions(-) (limited to 'doc') diff --git a/doc/docker-container.org b/doc/docker-container.org index 3c9864c5..ec91824a 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -15,57 +15,60 @@ are ran from. Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This document will not get into that. It's assumed that you have a working -dockec setup. +docker setup. The rest of this document outlines how the docker container used in -the CI builds were created. +the CI builds was created. * Creating the Docker Images -First create the image by running: +The general idea is that GUIX is used to generate a set of binaries, +which will be added to a base mariaDB image. + +First create the gn2 tar archive by running: #+begin_src sh # For the Python 2 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen genenetwork2 # For the Python 3 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash python3-genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen python3-genenetwork2 #+end_src The output will look something similar to: -: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +: /gnu/store/x3m77vwaqcwba24p5s4lrb7w2ii16lj9-tarball-pack.tar.gz + +Now create a folder from which will host the following dockerfile. You +can name this file Dockerfile. Note that mariadb is the base image +since it already has mariadb installed for us. + +#+begin_src conf :mkdirp yes :tangle ~/docker/Dockerfile +FROM mariadb:latest -Load the docker image by running: +COPY ./gn2.tar.gz /tmp/gn2.tar.gz +RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ + mkdir -p /usr/local/mysql /genotype_files/genotype/json +#+end_src -: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +Build the image(Note the fullstop at the end): -Results look something similar to: +: sudo docker build -t python2-genenetwork2:latest -f Dockerfile . -#+begin_export ascii -a93f52b7f565: Loading layer 3.174GB/3.174GB -Loaded image: coreutils-genenetwork2:latest -#+end_export +To load the image interactively you've just created: + +: docker run -ti "python2-genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: -: docker run "coreutils-genenetwork2:latest" python --version - +: docker run "python2-genenetwork2:latest" python --version * Pushing to DockerHub -- cgit v1.2.3