# Genotype uploads — tasks, todos & progress - **Feature:** genotypes upload - **Last updated:** 2026-11-04 - **Related:** [design notes](geno-uploads-design-notes.md) · [implementation notes](geno-uploads-implementation-notes.md) · background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`) Legend: ✅ done · 🚧 in progress · ⬜ todo · 💭 consideration (not committed) This is the single place to avoid tasks falling through the cracks. Design rationale lives in the design notes; what actually got built lives in the implementation notes. ## Stage 01 — Collect info - ✅ Statistics collector implemented (`uploader/scripts/genotypes/preprocess_csv_collect_info.py`). - ✅ Single- and multi-processed paths produce identical `FileStats`. - ✅ Unit tests for accumulation, merge and equivalence (`tests/genotypes/`). - ✅ Fix threshold / header-skip / `blank_lines` defects. - ✅ **Collect comment lines into a multiline description string**: strip the configured comment character, trim surrounding whitespace, keep one line per comment and join with newlines into `FileStats.description`. Still to do: settle the Stage 02 textbox styling. - ✅ Only feed comments **before the data matrix starts** into the description (before the header row; before the first data row when transposed). Later comments are counted but not used. A `matrix_started` flag per chunk, plus merge handling, keeps multiprocessing consistent with single-threaded. - ✅ Flag empty/missing-value headings (header row, transposed first column and marker headings), empty/missing-value marker and Chr/cM/Mb values, empty sample values, and negative/non-numeric cM/Mb values early (both orientations); exclude `""` from `geno_symbols`. - 💭 Decide whether the final description is also persisted to the database, or only written into the R/qtl2 control file. - 🚧 **Wire the collector into the upload view** (`add_genotype_records`): - ✅ Resolve the uploaded file for both the resumable and direct upload paths, via `uploader.files.resolve_uploaded_file`. - ✅ Read the form's metadata (`markers_field`, `chromosomes_field`, `centimorgans_field`, `megabases_field`, `transposed`, `file-separator`, `file-comment-char`, `file-na`) and map it to the collector's config keys (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`, `comment.char`, `na_strings`) via `uploader.genotypes.collector_config`. - ✅ Run the collector as a `gn_libs.jobs` background job (job type `genotype-file-stats`); the job status/summary pages show its output and errors. - ✅ Persist/return the resulting `FileStats` for the next stage (Stage 02 reads it from the collector job). ## Stage 02 — Study design & genotype encoding - ✅ Present the collected comment lines (comment character stripped) as the study/dataset description for review and editing. - ✅ Present the genotype symbols discovered in Stage 01 for encoding. - ✅ Only symbols that actually occur in the file are offered. - ✅ Seed labels from `@` lines when present (`mat`, `pat`, `het`, `unk`). - ✅ Missing `@` lines (all or some) do **not** fail the stage (empty defaults). - ✅ `@`-derived labels are a starting point only; the user can change them. Presence is not validation. - ✅ Allow the user to specify missing-value symbol(s). - ✅ Seed `dot_geno_fields.unk` into the missing-value symbols. - ⬜ Revisit the Stage 02 UI's three-character symbol limit: symbols may be longer and need not be single characters (low priority). - ✅ Capture the confirmed encoding/study design for downstream stages (carried in the QC job metadata at launch, key `stage02`; the collector job is left untouched). ## Stage 03 — Final QC - ✅ Untranspose transposed files as the first action of Stage 03: write a canonical `__work`, keep the original immutable, skip `#`/`@`, validate ragged rows (see "Untransposing transposed files"). - ✅ Perform value-level QC parameterised by Stage 01/02 results: every sample value must be an encoding or missing-value symbol; unknown symbols are errors and matching is case-sensitive. (Empty values are rejected earlier, in Stage 01.) - ✅ Restrict checks to the symbols present in the file (checks use the confirmed encoding/missing symbols; absent symbols are not required). - ✅ **Marker uniqueness (hard error):** no two markers on the same chromosome may share a name; names may repeat across chromosomes. Implemented in `uploader.genotypes.qc`. - ✅ Apply the sample/individual/case ordering rules and present the order for user review and reordering (one of the last Stage 03 steps; see design notes). The accepted order is carried into the next stage job's metadata; a placeholder Stage 04 job is the current hand-off point. - ⬜ Sort/verify records by chromosome (ascending), then Mb (ascending), then cM (ascending) (one of the last Stage 03 steps) — placeholder ordering with a warning/TODO until the appropriate ordering is known. - 💭 **Warnings (future):** report high missingness/call rate, monomorphic markers, and male heterozygosity on non-pseudoautosomal X/Y; warnings do not stop the chain. ## Stages 04–07 — Conversion & ingest - ⬜ Stage 04: convert to R/qtl2 bundle. - ⬜ Stage 05: ingest the R/qtl2 bundle. - ⬜ Stage 06: generate `.geno` (and other artefacts), preserving sample order and ordering records by chromosome then position, both ascending (see Stage 03). - ⬜ Stage 07: pre-compute mapping(s). ## Untransposing transposed files - ✅ Decide placement: first step of Stage 03, before QC. - ✅ Implement normalisation at the start of the Stage 03 job: write a canonical `__work`, keep the original immutable, skip `#`/`@`, validate ragged rows. The path is deterministic, so it is not recorded. - ✅ Run Stage 03 QC against the work file with `transposed=False`; QC handles only the normal orientation. ## Refactoring / technical debt - 💭 **Deferred:** once `add_genotype_records` grows to cover the later stages, refactor it into a monadic pipeline rather than a series of guard clauses — resolve the uploaded file, then chain each stage with `.then(...)`, ending with `.either(_handle_error_path, _handle_success_path)`. Deferred until the view is large enough for the split to pay off. ## Background jobs - ⬜ Implement the stage-job chain using each job's `success_handler` (as in the phenotypes `load_data_to_database`/`load_phenotypes_success_handler` pattern): proceed on success, and stop/report diagnostics on failure (see design notes). - ✅ Completed jobs are immutable: Stage 02 state is carried in the QC job metadata at launch, not written back to the collector job. - ✅ Reuse an existing next-stage job via `jobs_by_metadata` with namespaced keys (`genotype_collector_job_id` + `job-type`). - ⬜ Make stage transitions idempotent: revisiting a completed job's status URL must not re-run a non-idempotent stage; present the previously entered information instead (see design notes). - ⬜ Unique claim / serialised transition for non-idempotent stages (deferred). - ⬜ Add a cleanup daemon/service that periodically calls `delete_expired_jobs` (script in gn-libs; scheduling in gn-machines). It only cares about expired jobs, not users. - 💭 Consider preventing a return to / resumption of earlier jobs once a later stage has started (see design notes). - 💭 Decide whether genotype upload stays "one script == one job" or becomes one job with multiple, gated tasks. **Not decided; treat as a separate feature.** See `gn-libs/WIP/background-jobs-design-notes.md` and the migration inventory in `gn-libs/WIP/background-jobs-tasks-and-todos.md`. - 💭 (Lower priority, only if the multi-task redesign is chosen) a way to re-run/"fix" failed tasks. ## Error handling - ✅ Collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so malformed files stop the chain before Stage 02. - ✅ Register `error_handler`s for the collector and QC jobs (other stages later). - ✅ Display structured collector/QC errors on the genotype upload error page; other failures fall back to stdout/stderr with recovery guidance. ## Testing / fixtures - ✅ Add small sample genotype files (`tests/uploader/genotypes/test_data/`): normal/transposed, clean, collector errors (column mismatches, empty values, invalid headings/marker/metadata/ position values), unknown value errors, ragged rows, and comments/directives. - ✅ Tests for `job_errors.errors_from_job` using the fixtures. ## Housekeeping - ✅ Track the new templates `genotype-encoding-elements.html` and `study-design-elements.html`; they are wired into Stage 02.