From 9f49e2dbdf4e06fcb39ed5146560937f228682a7 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 22 Sep 2020 04:31:05 +0300 Subject: Update documentation on testing and CI * README.md: Update section on testing. * doc/docker-container.org: Add it. --- doc/docker-container.org | 86 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 doc/docker-container.org (limited to 'doc/docker-container.org') diff --git a/doc/docker-container.org b/doc/docker-container.org new file mode 100644 index 00000000..3c9864c5 --- /dev/null +++ b/doc/docker-container.org @@ -0,0 +1,86 @@ +#+TITLE: Genenetwork2 Dockerized + +* Table of Contents :TOC: +- [[#introduction][Introduction]] +- [[#creating-the-docker-images][Creating the Docker Images]] +- [[#pushing-to-dockerhub][Pushing to DockerHub]] + +* Introduction + +The CI(Continuous Integration) system for Genenetwork2 uses [[https://github.com/features/actions][Github +Actions]]. As such, it's important to have a way to run tests using +facilities provided by GUIX in a reproducible way. This project +leverages GUIX to generate a docker container from which the unittests +are ran from. + +Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This +document will not get into that. It's assumed that you have a working +dockec setup. + +The rest of this document outlines how the docker container used in +the CI builds were created. + +* Creating the Docker Images + +First create the image by running: + +#+begin_src sh +# For the Python 2 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash genenetwork2 + +# For the Python 3 version: +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack -f docker --no-grafts \ + -S /usr/bin=/bin -S /etc/profile=/etc/profile \ + -S /share/genenetwork2=/share/genenetwork2 \ + -S /share/javascript=/share/javascript \ + -S /lib=/lib \ + -S /usr/gn2-profile=/ \ + coreutils bash python3-genenetwork2 + #+end_src + +The output will look something similar to: + +: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Load the docker image by running: + +: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz + +Results look something similar to: + +#+begin_export ascii +a93f52b7f565: Loading layer 3.174GB/3.174GB +Loaded image: coreutils-genenetwork2:latest +#+end_export + +Assuming you have a docker instance running, you could always run +commands in it e.g: + +: docker run "coreutils-genenetwork2:latest" python --version + + +* Pushing to DockerHub + +We use DockerHub to store the docker images from which we use on our +CI environment using Github Actions. + +To push to dockerhub, first get the image name by running =docker +images=. Push to dockerhub using a command similar to: + +: docker push bonfacekilz/python2-genenetwork2:latest + +Right now, we have 2 images on DockerHub: + +- https://hub.docker.com/repository/docker/bonfacekilz/python2-genenetwork2: + Contains the python2 version of gn2. Don't use this. Please use the + python3 image! +- https://hub.docker.com/repository/docker/bonfacekilz/python3-genenetwork2: + Contains the python3 version of gn2. -- cgit v1.2.3 From 3ec4eb6b831eaa5adcf32a9fca8a60ea229cc1c4 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Fri, 25 Sep 2020 04:09:35 +0300 Subject: Update docs on CI setup with docker * doc/docker-container.org: Update it. --- doc/docker-container.org | 61 +++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 29 deletions(-) (limited to 'doc/docker-container.org') diff --git a/doc/docker-container.org b/doc/docker-container.org index 3c9864c5..ec91824a 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -15,57 +15,60 @@ are ran from. Find instructions on how to set docker up inside GUIX [[https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org#run-docker][here]]. This document will not get into that. It's assumed that you have a working -dockec setup. +docker setup. The rest of this document outlines how the docker container used in -the CI builds were created. +the CI builds was created. * Creating the Docker Images -First create the image by running: +The general idea is that GUIX is used to generate a set of binaries, +which will be added to a base mariaDB image. + +First create the gn2 tar archive by running: #+begin_src sh # For the Python 2 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen genenetwork2 # For the Python 3 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics::/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack -f docker --no-grafts \ - -S /usr/bin=/bin -S /etc/profile=/etc/profile \ - -S /share/genenetwork2=/share/genenetwork2 \ - -S /share/javascript=/share/javascript \ - -S /lib=/lib \ - -S /usr/gn2-profile=/ \ - coreutils bash python3-genenetwork2 +env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ + ./pre-inst-env guix pack --no-grafts\ + -S /gn2-profile=/ \ + screen python3-genenetwork2 #+end_src The output will look something similar to: -: /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +: /gnu/store/x3m77vwaqcwba24p5s4lrb7w2ii16lj9-tarball-pack.tar.gz + +Now create a folder from which will host the following dockerfile. You +can name this file Dockerfile. Note that mariadb is the base image +since it already has mariadb installed for us. + +#+begin_src conf :mkdirp yes :tangle ~/docker/Dockerfile +FROM mariadb:latest -Load the docker image by running: +COPY ./gn2.tar.gz /tmp/gn2.tar.gz +RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ + mkdir -p /usr/local/mysql /genotype_files/genotype/json +#+end_src -: docker load --input /gnu/store/dj1xh19jq1l9vwq24w3nay2954x0wabb-docker-pack.tar.gz +Build the image(Note the fullstop at the end): -Results look something similar to: +: sudo docker build -t python2-genenetwork2:latest -f Dockerfile . -#+begin_export ascii -a93f52b7f565: Loading layer 3.174GB/3.174GB -Loaded image: coreutils-genenetwork2:latest -#+end_export +To load the image interactively you've just created: + +: docker run -ti "python2-genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: -: docker run "coreutils-genenetwork2:latest" python --version - +: docker run "python2-genenetwork2:latest" python --version * Pushing to DockerHub -- cgit v1.2.3 From 45fc1da136b78bac906aad013686a9530f68bd5e Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 11 May 2021 10:33:49 +0300 Subject: doc: docker-container: Default to python3-genenetwork2 for examples --- doc/docker-container.org | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'doc/docker-container.org') diff --git a/doc/docker-container.org b/doc/docker-container.org index ec91824a..c894c4b5 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -59,11 +59,11 @@ RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ Build the image(Note the fullstop at the end): -: sudo docker build -t python2-genenetwork2:latest -f Dockerfile . +: sudo docker build -t python3-genenetwork2:latest -f Dockerfile . To load the image interactively you've just created: -: docker run -ti "python2-genenetwork2:latest" bash +: docker run -ti "python3-genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: @@ -78,7 +78,7 @@ CI environment using Github Actions. To push to dockerhub, first get the image name by running =docker images=. Push to dockerhub using a command similar to: -: docker push bonfacekilz/python2-genenetwork2:latest +: docker push bonfacekilz/python3-genenetwork2:latest Right now, we have 2 images on DockerHub: -- cgit v1.2.3 From c22fdbd6a128ab9c9ddfc14299115a40cec37f07 Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Wed, 26 May 2021 18:33:27 +0300 Subject: doc: docker-container: Rename python3-genenetwork2 to genenetwork2 See: https://is.gd/gHJvhe --- doc/docker-container.org | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) (limited to 'doc/docker-container.org') diff --git a/doc/docker-container.org b/doc/docker-container.org index c894c4b5..ef0d71fc 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -32,13 +32,13 @@ First create the gn2 tar archive by running: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ - screen genenetwork2 + screen python2-genenetwork2 # For the Python 3 version: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ - screen python3-genenetwork2 + screen genenetwork2 #+end_src The output will look something similar to: @@ -59,16 +59,16 @@ RUN tar -xzf /tmp/gn2.tar.gz -C / && rm -f /tmp/gn2.tar.gz && \ Build the image(Note the fullstop at the end): -: sudo docker build -t python3-genenetwork2:latest -f Dockerfile . +: sudo docker build -t genenetwork2:latest -f Dockerfile . To load the image interactively you've just created: -: docker run -ti "python3-genenetwork2:latest" bash +: docker run -ti "genenetwork2:latest" bash Assuming you have a docker instance running, you could always run commands in it e.g: -: docker run "python2-genenetwork2:latest" python --version +: docker run "genenetwork2:latest" python --version * Pushing to DockerHub @@ -78,7 +78,7 @@ CI environment using Github Actions. To push to dockerhub, first get the image name by running =docker images=. Push to dockerhub using a command similar to: -: docker push bonfacekilz/python3-genenetwork2:latest +: docker push bonfacekilz/genenetwork2:latest Right now, we have 2 images on DockerHub: -- cgit v1.2.3 From 7a15d24a6598f30801dd897ddc72d3773641e7bd Mon Sep 17 00:00:00 2001 From: BonfaceKilz Date: Tue, 5 Oct 2021 11:32:17 +0300 Subject: doc: docker-container.org: Remove python2 gn2 docker set-up command --- doc/docker-container.org | 7 ------- 1 file changed, 7 deletions(-) (limited to 'doc/docker-container.org') diff --git a/doc/docker-container.org b/doc/docker-container.org index ef0d71fc..79b8272f 100644 --- a/doc/docker-container.org +++ b/doc/docker-container.org @@ -28,13 +28,6 @@ which will be added to a base mariaDB image. First create the gn2 tar archive by running: #+begin_src sh -# For the Python 2 version: -env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ - ./pre-inst-env guix pack --no-grafts\ - -S /gn2-profile=/ \ - screen python2-genenetwork2 - -# For the Python 3 version: env GUIX_PACKAGE_PATH="/home/bonface/projects/guix-bioinformatics:/home/bonface/projects/guix-past/modules" \ ./pre-inst-env guix pack --no-grafts\ -S /gn2-profile=/ \ -- cgit v1.2.3