Frederick Muriuki Muriithi
Use the new code that collects the errors in the CLI client.
|5 hours ago|
|etc||3 weeks ago|
|qc_app||2 weeks ago|
|quality_control||5 hours ago|
|scripts||5 hours ago|
|tests||5 hours ago|
|.gitignore||3 weeks ago|
|MANIFEST.in||2 weeks ago|
|README.org||1 week ago|
|guix.scm||1 week ago|
|manifest.scm||3 weeks ago|
|mypy.ini||3 weeks ago|
|pyproject.toml||3 weeks ago|
|setup.cfg||2 weeks ago|
|setup.py||3 weeks ago|
|wsgi.py||3 weeks ago|
GeneNetwork Quality Control Application
- Project Goals
- Deploying/Installing QC
- Running QC
The project seeks to handle the checking of data files for correct syntax and other errors before allowing the code to be uploaded.
The files are "tab-separated" values (TSV) files, and must conform to the following criteria:
Must be tab-separated
No empty data cells
no data cells with spurious characters like `eeeee`, `5.555iloveguix`, etc.
decimal numbers must conform to the following criteria:
when checking an average file decimal numbers must contain exactly three places to the right side of the dot.
when checking a standard error file decimal numbers must contain six or greater places to the right side of the dot.
there must be a number to the left side of the dot (e.g. 0.55555 is allowed but .55555 is not).
check line endings to make sure they are Unix and not DOS
check strain headers against a source of truth (see strains.csv)
For reproducibility, this project is developed using guix.
To launch a guix shell for development, do
guix shell --container --network --pure --manifest=manifest.scm --share=/some/host/directory=/the/upload/directory
which environment that is isolated from the rest of your system.
We share a host directory with the container (that is writeable by the user that started the web application) to serve as the upload directory for the application.
Run the CLI version
To run the CLI version of the application, we need to set up the correct
which enables us to run the script with
python3 scripts/qc.py --help
Without setting up the
PYTHONPATH environment variable, you might get an error
like the following
$ python3 scripts/qc.py --help Traceback (most recent call last): File "/home/frederick/genenetwork/gnqc_py/scripts/qc.py", line 8, in <module> from quality_control.errors import ParseError ModuleNotFoundError: No module named 'quality_control'
NOTE: Setting up the
PYTHONPATH environment variable is only necessary for
the development environment. It MUST NOT require that the end-user set this up
Run the web version
To run the web-version of the qc app in development mode, you need to set up a few environment variables
export FLASK_APP=wsgi.py export FLASK_ENV=development export QCAPP_INSTANCE_PATH=/path/to/directory/with/config.py
then you can run the application with
You then need to manually start the rq worker(s)
rq worker qcapp_queue
Run tests with:
To run the linter over the code base, run:
pylint tests quality_control qc_app
To check for correct type usage in the application, run:
mypy --show-error-codes .
Generate the docker image file with
guix pack -f docker -S /bin=bin genenetwork-qc
That creates the image file with a path such as:
You can now load this file into docker withe
docker load < /gnu/store/ibz5qlwzv0lyply2by7422z0c6jfaa6s-genenetwork-qc-docker-pack.tar.gz
and from there, you can run the application as detailed in the Running QC: CLI-Version section below
The application can be installed using guix by pointing to the guix.scm file as follows:
guix package [-p /path/to/qc/profile] -f guix.scm
TODO Document deployment details for the web version of GeneNetwork QC better
Install the application as shown in the /fredmanglis/gnqc_py/src/branch/main/Installing%20QC section above.
To run qc against a file, the syntax is:
qc [--strainsfile <strainsfile-path>] [--verbose] <filetype> <filepath>
<filetype>is one of "average" or "standard-error"
<filepath>is either an absolute path to the file, or a path relative to the current working directory
--strainsfileoption is not provided, it will default to the one in the root directory of this repository
--verboseoption is a flag, defaulting to
Falsethat controls the display of optional progress messages
To view the usage information for the application, run
TODO Document usage of the web-UI version of the application
Download the docker image file from the releases page of the application and load it to docker with something like:
docker load < genenetwork-qc-0.0.1-1-oxu472i-docker.tar.gz
genenetwork-qc-0.0.1-1-oxu472i.tar.gz with the actual name of the
release you downloaded
Run the application with something like:
docker run -v /path/to/qnqc_py/tests/test_data:/data -ti \ genenetwork-qc:latest /bin/qc average /data/average_error_at_end_200MB.tsv
/path/to/qnqc_py/tests/test_data with the path to the folder where
the file you want to check is in, and
average_error_at_end_200MB.tsv with the
name of the file you want to check for errors.