about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-12 16:56:49 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-12 14:37:17 -0600
commit8df138974c36d01439c95d4bd053e6e297ddaed8 (patch)
tree2a5360586a3a21e902fdf00596503d44b54b5e71 /docs
parentda2d62432cebe8ddb638202529b8f1c07b34f78f (diff)
downloadgn-uploader-8df138974c36d01439c95d4bd053e6e297ddaed8.tar.gz
docs: update tasks and todos
- update genotype upload error and test status
- update tasks for heading, value and position validation
- track marker uniqueness, ordering and future warnings
- mark marker uniqueness done
- mark sample ordering done

Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-tasks-and-todos.md42
1 files changed, 28 insertions, 14 deletions
diff --git a/docs/WIP/geno-uploads-tasks-and-todos.md b/docs/WIP/geno-uploads-tasks-and-todos.md
index a3ce68f..353e267 100644
--- a/docs/WIP/geno-uploads-tasks-and-todos.md
+++ b/docs/WIP/geno-uploads-tasks-and-todos.md
@@ -26,6 +26,10 @@ implementation notes.
   (before the header row; before the first data row when transposed). Later
   comments are counted but not used. A `matrix_started` flag per chunk, plus
   merge handling, keeps multiprocessing consistent with single-threaded.
+- ✅ Flag empty/missing-value headings (header row, transposed first column and
+  marker headings), empty/missing-value marker and Chr/cM/Mb values, empty
+  sample values, and negative/non-numeric cM/Mb values early (both
+  orientations); exclude `""` from `geno_symbols`.
 - 💭 Decide whether the final description is also persisted to the database, or
   only written into the R/qtl2 control file.
 - 🚧 **Wire the collector into the upload view** (`add_genotype_records`):
@@ -67,15 +71,24 @@ implementation notes.
   canonical `<stem>__work<suffix>`, keep the original immutable, skip `#`/`@`,
   validate ragged rows (see "Untransposing transposed files").
 - ✅ Perform value-level QC parameterised by Stage 01/02 results: every sample
-  value must be an encoding or missing-value symbol; empty values are errors
-  and matching is case-sensitive.
+  value must be an encoding or missing-value symbol; unknown symbols are errors
+  and matching is case-sensitive. (Empty values are rejected earlier, in
+  Stage 01.)
 - ✅ Restrict checks to the symbols present in the file (checks use the
   confirmed encoding/missing symbols; absent symbols are not required).
-- ⬜ Apply the sample/individual/case ordering rules and present the order for
-  user review and reordering (see design notes).
+- ✅ **Marker uniqueness (hard error):** no two markers on the same chromosome
+  may share a name; names may repeat across chromosomes. Implemented in
+  `uploader.genotypes.qc`.
+- ✅ Apply the sample/individual/case ordering rules and present the order for
+  user review and reordering (one of the last Stage 03 steps; see design notes).
+  The accepted order is carried into the next stage job's metadata; a
+  placeholder Stage 04 job is the current hand-off point.
 - ⬜ Sort/verify records by chromosome (ascending), then Mb (ascending), then cM
-  (ascending) — placeholder ordering with a warning/TODO until the appropriate
-  ordering is known.
+  (ascending) (one of the last Stage 03 steps) — placeholder ordering with a
+  warning/TODO until the appropriate ordering is known.
+- 💭 **Warnings (future):** report high missingness/call rate, monomorphic
+  markers, and male heterozygosity on non-pseudoautosomal X/Y; warnings do not
+  stop the chain.
 
 ## Stages 04–07 — Conversion & ingest
 
@@ -133,17 +146,18 @@ implementation notes.
 
 - ✅ Collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is
   non-empty, so malformed files stop the chain before Stage 02.
-- ⬜ Register an `error_handler` for each stage job node (alongside the
-  `success_handler`): stop the chain and inform the user.
-- ⬜ Display `FileStats.errors` like the phenotypes job status page
-  (`uploader.phenotypes.views.job_status`,
-  `uploader/templates/phenotypes/job-status.html`); for other failures, show
-  diagnostics and reasonable recovery steps.
+- ✅ Register `error_handler`s for the collector and QC jobs (other stages
+  later).
+- ✅ Display structured collector/QC errors on the genotype upload error page;
+  other failures fall back to stdout/stderr with recovery guidance.
 
 ## Testing / fixtures
 
-- ⬜ Generate small sample genotype files for workflow tests: normal and
-  transposed, clean and with errors.
+- ✅ Add small sample genotype files
+  (`tests/uploader/genotypes/test_data/`): normal/transposed, clean, collector
+  errors (column mismatches, empty values, invalid headings/marker/metadata/
+  position values), unknown value errors, ragged rows, and comments/directives.
+- ✅ Tests for `job_errors.errors_from_job` using the fixtures.
 
 ## Housekeeping