aboutsummaryrefslogtreecommitdiff
path: root/qc_app/jobs.py
AgeCommit message (Collapse)Author
2022-08-04Implement data insertionFrederick Muriuki Muriithi
- Hook up external data insertion script to webserver code - Provide rudimentary status indication - Generalise some job creation details
2022-07-19Rework: Use generic worker script to launch processFrederick Muriuki Muriithi
Use the generic worker script as the interface for launching external processes.
2022-07-19Rename file validation scriptFrederick Muriuki Muriithi
As preparation for building a new generic worker script, this commit renames the file validation script from 'worker.py' to 'validate_file.py' so as to ensure the name conforms better to what the script does.
2022-07-07Add filetype details to database update codeFrederick Muriuki Muriithi
The filetype determines the queries to be run to update the database, therefore, this commit adds filetype information.
2022-06-15Fix linting and type errorsFrederick Muriuki Muriithi
2022-06-02Expire the jobs in 14 days by defaultFrederick Muriuki Muriithi
2022-05-19Update Web-UI: use new error collection paradigmFrederick Muriuki Muriithi
- README.org: document how to run scripts manually - manifest.scm: remove python-rq as a dependency - qc_app/jobs.py: rework job launching and processing - qc_app/parse.py: use reworked job processing - qc_app/templates/job_progress.html: display progress correctly - qc_app/templates/parse_results.html: display final results - scripts/worker.py: new worker script
2022-04-28Use sqlite3 to save the jobs metadataFrederick Muriuki Muriithi
* Use sqlite to save the jobs metadata and enable UI update of the progress for large files
2022-04-28Update queuing and display results of file parsingFrederick Muriuki Muriithi
* Make the 'worker' functions free from needing the application context by passing all the details they need as arguments. * Enable the display of parsing results.
2022-04-26Fix issues caught processing the jobsFrederick Muriuki Muriithi
* Create and push the application context for the worker functions * Fix the update of meta fields
2022-04-26Queue file parsing jobsFrederick Muriuki Muriithi
Enable the queuing of file parsing jobs, since the files could be really large and take a long time to parse and present results. * etc/default_config.py: Add default config for redis server * manifest.scm: Add redis, and rq as dependencies * qc_app/__init__.py * qc_app/jobs.py: module to hold utilities for management of the jobs * qc_app/parse.py: Enqueue the job - extract file-parsing code to callable function * qc_app/templates/base.html: Enable addition of extra meta tags * qc_app/templates/job_progress.html: template to display job progress * qc_app/templates/no_such_job.html: template to indicate when a job id is invalid * quality_control/parsing.py: Add the total size parsed so far