aboutsummaryrefslogtreecommitdiff
path: root/qc_app/parse.py
AgeCommit message (Collapse)Author
2024-01-24redis-prefix: Update file validation codeFrederick Muriuki Muriithi
Update the file validation script and routes to use the redis prefix for jobs.
2024-01-20Fetch sample/case names from databaseFrederick Muriuki Muriithi
Fetch the sample/case names from the database rather than from a static file in the repository. Issue: https://issues.genenetwork.org/issues/quality-control/read-samples-from-database-by-species
2023-11-15Linting: Fix linting errors.Frederick Muriuki Muriithi
2022-07-19Rework: Use generic worker script to launch processFrederick Muriuki Muriithi
Use the generic worker script as the interface for launching external processes.
2022-07-06Implement UI for dataset selectionFrederick Muriuki Muriithi
As part of updating the database with the new data, there is a need to select the appropriate dataset that the data belongs to, and this commit provides the UI to assist the user do that.
2022-06-28Check for inconsistent columnsFrederick Muriuki Muriithi
The number columns in each contents line should be equal to the nember of columns in the header line.
2022-06-21Test the progress indication featureFrederick Muriuki Muriithi
2022-06-17Test endpoint '/parse/parse'Frederick Muriuki Muriithi
* Ensure error messages are displayed if a request is made to the '/parse/parse' endpoint with invalid, or missing data.
2022-06-15Fix linting and type errorsFrederick Muriuki Muriithi
2022-06-02Enable user abortion of file parsingFrederick Muriuki Muriithi
Enable the user to abort the background parsing of the file.
2022-06-02Expire the jobs in 14 days by defaultFrederick Muriuki Muriithi
2022-06-02Add an error display to the progress status reportFrederick Muriuki Muriithi
Enable the progress status page to show all the errors found at any point during the processing of the file.
2022-05-27Remove unnecessary print statementFrederick Muriuki Muriithi
2022-05-20Fix minor bugsFrederick Muriuki Muriithi
2022-05-20Handle errorsFrederick Muriuki Muriithi
Implement code to handle errors in the processing of files.
2022-05-19Update Web-UI: use new error collection paradigmFrederick Muriuki Muriithi
- README.org: document how to run scripts manually - manifest.scm: remove python-rq as a dependency - qc_app/jobs.py: rework job launching and processing - qc_app/parse.py: use reworked job processing - qc_app/templates/job_progress.html: display progress correctly - qc_app/templates/parse_results.html: display final results - scripts/worker.py: new worker script
2022-05-02Optimise strain names parsingFrederick Muriuki Muriithi
- Use a way faster way of parsing the strains file
2022-04-28Use sqlite3 to save the jobs metadataFrederick Muriuki Muriithi
* Use sqlite to save the jobs metadata and enable UI update of the progress for large files
2022-04-28Update queuing and display results of file parsingFrederick Muriuki Muriithi
* Make the 'worker' functions free from needing the application context by passing all the details they need as arguments. * Enable the display of parsing results.
2022-04-26Fix issues caught processing the jobsFrederick Muriuki Muriithi
* Create and push the application context for the worker functions * Fix the update of meta fields
2022-04-26Queue file parsing jobsFrederick Muriuki Muriithi
Enable the queuing of file parsing jobs, since the files could be really large and take a long time to parse and present results. * etc/default_config.py: Add default config for redis server * manifest.scm: Add redis, and rq as dependencies * qc_app/__init__.py * qc_app/jobs.py: module to hold utilities for management of the jobs * qc_app/parse.py: Enqueue the job - extract file-parsing code to callable function * qc_app/templates/base.html: Enable addition of extra meta tags * qc_app/templates/job_progress.html: template to display job progress * qc_app/templates/no_such_job.html: template to indicate when a job id is invalid * quality_control/parsing.py: Add the total size parsed so far
2022-04-25Call parsing function. Fix a few issues.Frederick Muriuki Muriithi
* qc_app/entry.py: Pass filetype onward to parsing endpoint * qc_app/parse.py: Call the function(s) necessary to parse a file * quality_control/errors.py: Fix argument passing to super class
2022-04-25Implement file uploadFrederick Muriuki Muriithi