diff options
| author | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-11 17:52:32 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-11 13:43:48 -0600 |
| commit | cc279bf596b138e66d1c75f26d48b4062e310b1c (patch) | |
| tree | 7a81e0c99351cda59e163f7bcfb58648e939c5df /docs | |
| parent | 3632200c2b13045e186b48d6d5a82970e0c3cf1e (diff) | |
| download | gn-uploader-cc279bf596b138e66d1c75f26d48b4062e310b1c.tar.gz | |
docs: record normalisation and QC-job state in the implementation notes
Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-implementation-notes.md | 41 |
1 files changed, 22 insertions, 19 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md index b360f16..c1761dc 100644 --- a/docs/WIP/geno-uploads-implementation-notes.md +++ b/docs/WIP/geno-uploads-implementation-notes.md @@ -169,36 +169,39 @@ dataset_id)` validates a completed `genotype-file-stats` job for the dataset and parses the `FileStats` from its recorded stdout. The collector job's `success_handler` redirects to the `study_design` view with the job id; that view renders the collected description, the genotype symbols, the -study-design/encoding inputs and the missing-value symbol(s). Saving stores the -confirmed values (including `missing_symbols`) in the job metadata (`stage02`) -and redirects back (post/redirect/get). +study-design/encoding inputs and the missing-value symbol(s). Saving puts the +confirmed values (including `missing_symbols`) in the **QC job's** metadata at +launch; the completed collector job is left untouched. Revisits read them back +from the existing QC job (found via `jobs_by_metadata`). + +### Stage 03 normalisation + +`uploader.genotypes.normalise.normalise_file(original, config)` writes a +canonical untransposed matrix (`<stem>__work<suffix>`) next to the original, +dropping comments and `@` directives (already captured in `FileStats`), leaving +the original untouched, and rejecting ragged or empty matrices. +`qc.check_file` now handles only the normal orientation. ### Stage 03 value QC -The collector job records the genotype `file` and collector `config` in its -metadata so later stages can re-read the file. -`uploader.genotypes.qc.check_file` checks every sample value against the Stage -02 encoding and missing-value symbols (empty values and unknown symbols are -reported), and `uploader.scripts.genotypes.qc` is the worker that loads the -collector job, runs the checks and exits non-zero on errors. On Stage 02 save, -the QC job is scheduled and its id recorded in the collector job metadata -(`stage03_job_id`) so re-submitting does not schedule it again. +Stage 02 launches the QC job with `stage02`, `genotype_collector_job_id`, +`file` and `config` in the QC job's metadata. The worker loads its own job, +normalises the file, then runs `uploader.genotypes.qc.check_file` on the work +file with `transposed=False`, exiting non-zero on errors. The completed +collector job is never modified; an existing QC job is reused via +`jobs_by_metadata`. The collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so malformed files stop the chain before Stage 02. `uploader.genotypes.qc.sort_markers` is a placeholder ordering helper -(chromosome, cM, Mb) that emits a warning until the appropriate ordering is +(chromosome, Mb, cM) that emits a warning until the appropriate ordering is established. -### Pending: normalisation, comment scoping and error handlers +### Pending: error handlers -- Stage 03 will untranspose transposed files first, producing a canonical - `work_file` for QC and later stages. -- The collector will only feed comments before the data matrix into - `description`. -- Each stage will register an `error_handler` (FileStats errors shown like the - phenotypes job status page; other failures reported with recovery guidance). +Each stage will register an `error_handler` (FileStats errors shown like the +phenotypes job status page; other failures reported with recovery guidance). ## Relationship to the background-jobs system |
