diff options
| author | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-12 16:56:49 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-12 14:37:17 -0600 |
| commit | 8df138974c36d01439c95d4bd053e6e297ddaed8 (patch) | |
| tree | 2a5360586a3a21e902fdf00596503d44b54b5e71 /docs | |
| parent | da2d62432cebe8ddb638202529b8f1c07b34f78f (diff) | |
| download | gn-uploader-8df138974c36d01439c95d4bd053e6e297ddaed8.tar.gz | |
docs: update tasks and todos
- update genotype upload error and test status - update tasks for heading, value and position validation - track marker uniqueness, ordering and future warnings - mark marker uniqueness done - mark sample ordering done Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-tasks-and-todos.md | 42 |
1 files changed, 28 insertions, 14 deletions
diff --git a/docs/WIP/geno-uploads-tasks-and-todos.md b/docs/WIP/geno-uploads-tasks-and-todos.md index a3ce68f..353e267 100644 --- a/docs/WIP/geno-uploads-tasks-and-todos.md +++ b/docs/WIP/geno-uploads-tasks-and-todos.md @@ -26,6 +26,10 @@ implementation notes. (before the header row; before the first data row when transposed). Later comments are counted but not used. A `matrix_started` flag per chunk, plus merge handling, keeps multiprocessing consistent with single-threaded. +- ✅ Flag empty/missing-value headings (header row, transposed first column and + marker headings), empty/missing-value marker and Chr/cM/Mb values, empty + sample values, and negative/non-numeric cM/Mb values early (both + orientations); exclude `""` from `geno_symbols`. - 💭 Decide whether the final description is also persisted to the database, or only written into the R/qtl2 control file. - 🚧 **Wire the collector into the upload view** (`add_genotype_records`): @@ -67,15 +71,24 @@ implementation notes. canonical `<stem>__work<suffix>`, keep the original immutable, skip `#`/`@`, validate ragged rows (see "Untransposing transposed files"). - ✅ Perform value-level QC parameterised by Stage 01/02 results: every sample - value must be an encoding or missing-value symbol; empty values are errors - and matching is case-sensitive. + value must be an encoding or missing-value symbol; unknown symbols are errors + and matching is case-sensitive. (Empty values are rejected earlier, in + Stage 01.) - ✅ Restrict checks to the symbols present in the file (checks use the confirmed encoding/missing symbols; absent symbols are not required). -- ⬜ Apply the sample/individual/case ordering rules and present the order for - user review and reordering (see design notes). +- ✅ **Marker uniqueness (hard error):** no two markers on the same chromosome + may share a name; names may repeat across chromosomes. Implemented in + `uploader.genotypes.qc`. +- ✅ Apply the sample/individual/case ordering rules and present the order for + user review and reordering (one of the last Stage 03 steps; see design notes). + The accepted order is carried into the next stage job's metadata; a + placeholder Stage 04 job is the current hand-off point. - ⬜ Sort/verify records by chromosome (ascending), then Mb (ascending), then cM - (ascending) — placeholder ordering with a warning/TODO until the appropriate - ordering is known. + (ascending) (one of the last Stage 03 steps) — placeholder ordering with a + warning/TODO until the appropriate ordering is known. +- 💭 **Warnings (future):** report high missingness/call rate, monomorphic + markers, and male heterozygosity on non-pseudoautosomal X/Y; warnings do not + stop the chain. ## Stages 04–07 — Conversion & ingest @@ -133,17 +146,18 @@ implementation notes. - ✅ Collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so malformed files stop the chain before Stage 02. -- ⬜ Register an `error_handler` for each stage job node (alongside the - `success_handler`): stop the chain and inform the user. -- ⬜ Display `FileStats.errors` like the phenotypes job status page - (`uploader.phenotypes.views.job_status`, - `uploader/templates/phenotypes/job-status.html`); for other failures, show - diagnostics and reasonable recovery steps. +- ✅ Register `error_handler`s for the collector and QC jobs (other stages + later). +- ✅ Display structured collector/QC errors on the genotype upload error page; + other failures fall back to stdout/stderr with recovery guidance. ## Testing / fixtures -- ⬜ Generate small sample genotype files for workflow tests: normal and - transposed, clean and with errors. +- ✅ Add small sample genotype files + (`tests/uploader/genotypes/test_data/`): normal/transposed, clean, collector + errors (column mismatches, empty values, invalid headings/marker/metadata/ + position values), unknown value errors, ragged rows, and comments/directives. +- ✅ Tests for `job_errors.errors_from_job` using the fixtures. ## Housekeeping |
