diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-design-notes.md | 67 |
1 files changed, 61 insertions, 6 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md index 48cb95f..d193f04 100644 --- a/docs/WIP/geno-uploads-design-notes.md +++ b/docs/WIP/geno-uploads-design-notes.md @@ -1,7 +1,7 @@ # Genotype uploads — design notes - **Feature:** genotypes upload -- **Last updated:** 2026-10-22 +- **Last updated:** 2026-11-04 - **Related:** [implementation notes](geno-uploads-implementation-notes.md) · [tasks & todos](geno-uploads-tasks-and-todos.md) · background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`) @@ -71,12 +71,53 @@ design source of truth: starting point: the user may change the labels. We must not treat a supplied encoding as "correct" merely because it came from the file. -### Stage gating +### Stage gating and job chaining (decided) -Each stage consumes the previous stage's output. A later stage must not run on a -file whose earlier stage failed or is incomplete. (How this gating is realised -in the background-job system is an open design question — see the background-jobs -design notes in `gn-libs/WIP/background-jobs-design-notes.md`.) +Stages run in order as a chain of background jobs: Stage 01 (collect info), +then Stage 02 (study design/encoding), then Stage 03 (QC), and so on. + +- If a stage fails, the chain stops there and the user is told, with as much + diagnostic information as possible (the job's stdout/stderr) to help them + resolve and recover where possible. +- If a stage succeeds, the chain proceeds to the next stage. +- Stage 02 is interactive: it presents the collected comment lines (with the + comment character stripped) as the study/dataset description for the user to + review and edit before continuing. + +Realisation: each stage job carries a `success_handler` (set in its job +metadata) that proceeds to the next stage, following the phenotypes pattern +(`uploader.phenotypes.views.load_data_to_database` / +`load_phenotypes_success_handler`). Failures are reported by the jobs system's +error path (stdout/stderr on the job page). + +How best to bind the stages together (one job with gated tasks vs a chain of +separate jobs) is still open; see the background-jobs design notes in +`gn-libs/WIP/background-jobs-design-notes.md`. + +### Job status pages and stage transitions must be idempotent + +Job status URLs are resumable: opening `/background-jobs/status/<job-id>` for a +completed job calls its `success_handler` again. Some stages are not idempotent +(for example Stage 02 writes to the database), so a success handler must not +re-run the next stage merely because the status URL was revisited. + +- Record that the next stage has started (for example store the next job id or + stage state in the job metadata) and check before starting anything. +- On a revisit, present the information the user already entered instead of + repeating the transition. +- Consider preventing a return to (or resumption of) earlier jobs once a later + stage has started; the exact mechanism is still to be decided. + +### Stage 02 state storage (interim) + +The confirmed description and genotype encoding are stored in the collector +job's metadata (key `stage02`) for now: + +- the database has nowhere suitable to store descriptions; +- `GenoCode` only stores integer values, while encodings may be floating point, + so it is not used at this stage. + +Revisiting Stage 02 reads these values back and presents them for editing. ### Comment → description (decided) @@ -90,6 +131,7 @@ line: textbox as-is; - let the user edit the final description before it is used. +Stage 02 presents this as the study/dataset description for review and editing. Open UI question: how to visually demarcate consecutive lines in the textbox. ## Stage 01 data contract (target) @@ -110,6 +152,19 @@ The stats collector should produce enough information for the later stages: The current implementation of Stage 01 is described in the implementation notes. +## QC / file-generation concerns (running list) + +This is a running list; more QC concerns will be added as they surface. + +- **Sample/individual/case order matters.** The order of samples in the + generated `.geno` file must be preserved consistently through the pipeline. +- **Markers/records must be ordered by chromosome and position.** In the + generated file(s) (e.g. the `.geno` file), records should be ordered by + chromosome (ascending) and then by position (ascending) within each + chromosome. This could be a Stage 03 QC step. How to detect and handle it is + still unclear: markers can overlap in position, so a strict "strictly + increasing position" check may not be valid. + ## Open design questions - **Description storage.** The description feeds the R/qtl2 control file; it is |
