# Genotype uploads — design notes - **Feature:** genotypes upload - **Last updated:** 2026-11-04 - **Related:** [implementation notes](geno-uploads-implementation-notes.md) · [tasks & todos](geno-uploads-tasks-and-todos.md) · background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`) This file captures the *intended* design of the genotype upload feature: the pipeline stages, the data that flows between them, the behaviours each stage must honour, and the open design questions. It is expected to change as implementation realities surface (recorded in the implementation notes). ## Goal Allow a user to upload a genotype CSV file for a dataset, validate/characterise it, let the user confirm the study design and genotype encoding, then convert and ingest it (eventually to an R/qtl2 bundle and the `.geno`/database representation), without losing information or silently mis-reading the file. ## Pipeline stages The stage breakdown currently lives as a comment block in `uploader/genotypes/views.py` (`add_genotype_records`). Restated here as the design source of truth: 1. **Stage 01 — Collect info** - Detect custom `.geno` format lines (lines beginning with `@`). - Collect the list of samples/cases/individuals. - Collect the list of markers. - Identify which fields are the markers/chromosome (and cM/Mb) fields. - Collect the set of unique genotype symbols. - Collect comment lines into a human-readable description: strip the leading `#`, trim surrounding whitespace from each line, and preserve each line separately (see “Comment → description” below; currently *not* implemented). 2. **Stage 02 — Establish basic study information** - What is the study design? - What is the genotype encoding scheme? (Seed this from `@` lines when present.) - Which value(s) mean "missing"? 3. **Stage 03 — Final QC** - Value-level QC, parameterised by the results of Stages 01–02. 4. **Stage 04 — Convert to an R/qtl2 bundle.** 5. **Stage 05 — Ingest the R/qtl2 bundle.** 6. **Stage 06 — Generate the `.geno` file** (and other artefacts). 7. **Stage 07 — Pre-compute the mapping(s).** ## Cross-stage rules ### Stage 01 output drives Stage 02 and Stage 03 - The **set of genotype symbols** found in the file determines what the QC in Stage 03 checks: only symbols that actually occur in the file should be presented/validated. Do not offer or require symbols that are absent from the file. - The **custom `.geno` lines**, when present, predefine labels for symbols, e.g. - `mat` — maternal - `pat` — paternal - `het` — heterozygous - `unk` — unknown - **Missing `@` lines are not an error.** The absence of all or some of these custom lines must not fail Stage 02; the user simply supplies the encoding. - **Presence of `@` lines is not endorsement of their validity.** They are a starting point: the user may change the labels. We must not treat a supplied encoding as "correct" merely because it came from the file. ### Stage gating and job chaining (decided) Stages run in order as a chain of background jobs: Stage 01 (collect info), then Stage 02 (study design/encoding), then Stage 03 (QC), and so on. - If a stage fails, the chain stops there and the user is told, with as much diagnostic information as possible (the job's stdout/stderr) to help them resolve and recover where possible. - If a stage succeeds, the chain proceeds to the next stage. - Stage 02 is interactive: it presents the collected comment lines (with the comment character stripped) as the study/dataset description for the user to review and edit before continuing. Realisation: each stage job carries a `success_handler` (set in its job metadata) that proceeds to the next stage, following the phenotypes pattern (`uploader.phenotypes.views.load_data_to_database` / `load_phenotypes_success_handler`). Failures are reported by the jobs system's error path (stdout/stderr on the job page). How best to bind the stages together (one job with gated tasks vs a chain of separate jobs) is still open; see the background-jobs design notes in `gn-libs/WIP/background-jobs-design-notes.md`. ### Job status pages and stage transitions must be idempotent Job status URLs are resumable: opening `/background-jobs/status/` for a completed job calls its `success_handler` again. Some stages are not idempotent (for example Stage 02 writes to the database), so a success handler must not re-run the next stage merely because the status URL was revisited. - **Completed jobs are immutable.** Never update a job's metadata once it has completed (or errored). State the next stage needs is copied into the *next* job's metadata at launch. - Identify an existing next-stage job with `jobs_by_metadata` using a workflow-namespaced key (e.g. `genotype_collector_job_id`) plus `job-type`; redirect to it instead of scheduling a duplicate. - On a revisit, present the information already entered (read it from the next-stage job) instead of repeating the transition. - A unique claim / serialised transition to make this race-free is deferred until non-idempotent stages need it. - Consider preventing a return to (or resumption of) earlier jobs once a later stage has started; the exact mechanism is still to be decided. ### Stage error handling Each node in the chain registers an `error_handler` (alongside its `success_handler`) that informs the user and stops the chain. - **`FileStats.errors`** (malformed data found during collection): display them like the phenotypes job status page (`uploader.phenotypes.views.job_status`, `uploader/templates/phenotypes/job-status.html`). - **Any other failure** (script crash, IO/database/token issues): tell the user what failed and give reasonable recovery steps, including stdout/stderr where useful. ### Job cleanup Add a daemon/service that periodically calls `delete_expired_jobs` on the jobs database (script in `gn-libs`; scheduling in `gn-machines`). It only cares about expired jobs, not users. Note that deleting a job also removes its metadata and outputs, so state that must outlive the jobs has to be copied forward. ### Stage 02 state storage (interim) The confirmed description, genotype encoding and missing-value symbol(s) are stored in the collector job's metadata (key `stage02`) for now: - the database has nowhere suitable to store descriptions; - `GenoCode` only stores integer values, while encodings may be floating point, so it is not used at this stage. Revisiting Stage 02 reads these values back and presents them for editing. ### Comment → description (decided) Comment lines become a user-editable description that will later go into the R/qtl2 bundle's control file (and possibly the database). For each comment line: - strip the leading `#` character; - strip leading/trailing whitespace; - keep the lines separate (one line each) and present them to the user in a textbox as-is; - let the user edit the final description before it is used. Only comments **before the data matrix starts** feed the description: - untransposed: before the header row; - transposed: before the first data row (the markers row). Later comments are counted (`comment_lines`) but not used; they relate to adjacent rows, not the whole file, and untransposing would take them out of context. Stage 02 presents this as the study/dataset description for review and editing. Open UI question: how to visually demarcate consecutive lines in the textbox. ## Stage 01 data contract (target) The stats collector should produce enough information for the later stages: - counts: total/data/comment/blank/dot-geno lines; - `dot_geno_fields` parsed from `@` lines (`name`, `mat`, `pat`, `het`, `type`, `unk` where present); - `geno_symbols` (the distinct symbols in the data); - `samples_list`; - `markers_list`; - parse errors encountered (column mismatches; empty or missing-value headings, marker values, Chr/cM/Mb values, or sample values; negative or non-numeric cM/Mb values); - **a free-text description built from comment lines** (desired; not yet implemented). Each comment line is cleaned (leading `#` and surrounding whitespace removed) and kept as its own line. The current implementation of Stage 01 is described in the implementation notes. ## QC / file-generation concerns (running list) This is a running list; more QC concerns will be added as they surface. - **Unknown (`unk`) symbols are missing values.** Seed `dot_geno_fields.unk` into the Stage 02 missing-value symbols. - **Heading values.** Values that label cells must never be empty or be missing-value indicators: the header row in untransposed files, and the first column plus the marker headings in transposed files. Marker and Chromosome values must likewise not be empty or missing-value indicators, and cM/Mb values must be finite, non-negative numbers (non-negative integers are accepted). - **Allowed genotype values.** Every sample value must be one of the confirmed maternal/paternal/heterozygous symbols or a missing-value symbol. Empty sample values are not allowed—missing values must use the configured `na_strings`. Stage 01 catches these obvious data errors early; Stage 03 checks the values against the confirmed encoding. Matching is case-sensitive. Symbols may be longer than three characters and need not be single characters, so the Stage 02 UI's three-character limit should be revisited (low priority). - **Marker uniqueness (hard error).** Within a single chromosome, marker names must be unique. The same marker name may appear on different chromosomes. Duplicate marker names within one chromosome are errors and stop the chain. - **Non-fatal warnings (future).** Consider reporting (without blocking): high missingness/call rate (e.g. more than 10% missing calls), monomorphic markers (every sample has the same genotype), and male heterozygosity on non-pseudoautosomal X/Y regions when sample sex is available. Warnings are shown to the user but do not stop the chain; only errors do. - **Sample/individual/case order matters.** The order of samples in the generated `.geno` file must be preserved consistently through the pipeline. See the ordering rules below. Ordering is a late Stage 03 step. - **Markers/records ordering (placeholder).** For now, sort records by chromosome (ascending), then Mb (ascending), then cM (ascending). The sorting helper emits a warning because the appropriate ordering has not been established; replace this placeholder and remove the warning once known. TODO. ### Sample/individual/case ordering rules 1. If the population already has an order, use it, then append any missing samples (sorted alphabetically). 2. If there is no existing order, sort the samples alphabetically. 3. Present the resulting order to the user for review and reordering (one sample per line in an editable textbox). 4. Save the accepted order in the **next stage job's metadata** at launch (completed jobs are immutable). ## Untransposing transposed files (decided) Transposed files are untransposed as the **first step of Stage 03**, before any QC. Stage 02 only consumes metadata from `FileStats`, so there is no reason to normalise a file that might be abandoned at review; Stage 03 is the first stage that needs the data. - Keep the original upload immutable and write a canonical `__work` next to it. The path is deterministic, so it need not be recorded in job metadata. - The canonical file is a **pure data matrix**: comments and `@` directives are omitted because Stage 01 already captured them (`description`, `dot_geno_fields`). - Skip `#`/`@` lines when transposing and validate ragged rows. The existing `r_qtl.r_qtl2.transpose_csv_with_rename` is unsuitable (renames in place, skips `#`, treats `@` as data, truncates ragged rows). - Do not untranspose if Stage 01 reported errors; the collector now exits with `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so the chain stops before Stage 02. - After normalisation, downstream (QC and later) handles only the normal orientation (`transposed=False`), simplifying QC and ordering code. ## Open design questions - **Description storage.** The description feeds the R/qtl2 control file; it is undecided whether it is also persisted to the database. - **Encoding representation.** How do we represent a user-confirmed encoding so that it is stable for QC and conversion (symbol → label/missing marker)? - **Validation vs. confirmation.** Where exactly is the boundary between "what the file says" and "what the user confirms"? - **Size/scale.** Stage 01 must cope with large files; the collector already has a single-threaded and a multi-processed path. ## Out of scope here - Reworking the background-jobs system to support multi-task jobs. That is a separate, cross-cutting feature; see the background-jobs design notes in `gn-libs/WIP/background-jobs-design-notes.md`.