about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-11 17:52:32 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-11 13:43:48 -0600
commitcc279bf596b138e66d1c75f26d48b4062e310b1c (patch)
tree7a81e0c99351cda59e163f7bcfb58648e939c5df /docs
parent3632200c2b13045e186b48d6d5a82970e0c3cf1e (diff)
downloadgn-uploader-cc279bf596b138e66d1c75f26d48b4062e310b1c.tar.gz
docs: record normalisation and QC-job state in the implementation notes
Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-implementation-notes.md41
1 files changed, 22 insertions, 19 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md
index b360f16..c1761dc 100644
--- a/docs/WIP/geno-uploads-implementation-notes.md
+++ b/docs/WIP/geno-uploads-implementation-notes.md
@@ -169,36 +169,39 @@ dataset_id)` validates a completed `genotype-file-stats` job for the dataset
 and parses the `FileStats` from its recorded stdout. The collector job's
 `success_handler` redirects to the `study_design` view with the job id; that
 view renders the collected description, the genotype symbols, the
-study-design/encoding inputs and the missing-value symbol(s). Saving stores the
-confirmed values (including `missing_symbols`) in the job metadata (`stage02`)
-and redirects back (post/redirect/get).
+study-design/encoding inputs and the missing-value symbol(s). Saving puts the
+confirmed values (including `missing_symbols`) in the **QC job's** metadata at
+launch; the completed collector job is left untouched. Revisits read them back
+from the existing QC job (found via `jobs_by_metadata`).
+
+### Stage 03 normalisation
+
+`uploader.genotypes.normalise.normalise_file(original, config)` writes a
+canonical untransposed matrix (`<stem>__work<suffix>`) next to the original,
+dropping comments and `@` directives (already captured in `FileStats`), leaving
+the original untouched, and rejecting ragged or empty matrices.
+`qc.check_file` now handles only the normal orientation.
 
 ### Stage 03 value QC
 
-The collector job records the genotype `file` and collector `config` in its
-metadata so later stages can re-read the file.
-`uploader.genotypes.qc.check_file` checks every sample value against the Stage
-02 encoding and missing-value symbols (empty values and unknown symbols are
-reported), and `uploader.scripts.genotypes.qc` is the worker that loads the
-collector job, runs the checks and exits non-zero on errors. On Stage 02 save,
-the QC job is scheduled and its id recorded in the collector job metadata
-(`stage03_job_id`) so re-submitting does not schedule it again.
+Stage 02 launches the QC job with `stage02`, `genotype_collector_job_id`,
+`file` and `config` in the QC job's metadata. The worker loads its own job,
+normalises the file, then runs `uploader.genotypes.qc.check_file` on the work
+file with `transposed=False`, exiting non-zero on errors. The completed
+collector job is never modified; an existing QC job is reused via
+`jobs_by_metadata`.
 
 The collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is
 non-empty, so malformed files stop the chain before Stage 02.
 
 `uploader.genotypes.qc.sort_markers` is a placeholder ordering helper
-(chromosome, cM, Mb) that emits a warning until the appropriate ordering is
+(chromosome, Mb, cM) that emits a warning until the appropriate ordering is
 established.
 
-### Pending: normalisation, comment scoping and error handlers
+### Pending: error handlers
 
-- Stage 03 will untranspose transposed files first, producing a canonical
-  `work_file` for QC and later stages.
-- The collector will only feed comments before the data matrix into
-  `description`.
-- Each stage will register an `error_handler` (FileStats errors shown like the
-  phenotypes job status page; other failures reported with recovery guidance).
+Each stage will register an `error_handler` (FileStats errors shown like the
+phenotypes job status page; other failures reported with recovery guidance).
 
 ## Relationship to the background-jobs system