about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-design-notes.md67
1 files changed, 61 insertions, 6 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md
index 48cb95f..d193f04 100644
--- a/docs/WIP/geno-uploads-design-notes.md
+++ b/docs/WIP/geno-uploads-design-notes.md
@@ -1,7 +1,7 @@
 # Genotype uploads — design notes
 
 - **Feature:** genotypes upload
-- **Last updated:** 2026-10-22
+- **Last updated:** 2026-11-04
 - **Related:** [implementation notes](geno-uploads-implementation-notes.md) ·
   [tasks & todos](geno-uploads-tasks-and-todos.md) ·
   background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)
@@ -71,12 +71,53 @@ design source of truth:
   starting point: the user may change the labels. We must not treat a supplied
   encoding as "correct" merely because it came from the file.
 
-### Stage gating
+### Stage gating and job chaining (decided)
 
-Each stage consumes the previous stage's output. A later stage must not run on a
-file whose earlier stage failed or is incomplete. (How this gating is realised
-in the background-job system is an open design question — see the background-jobs
-design notes in `gn-libs/WIP/background-jobs-design-notes.md`.)
+Stages run in order as a chain of background jobs: Stage 01 (collect info),
+then Stage 02 (study design/encoding), then Stage 03 (QC), and so on.
+
+- If a stage fails, the chain stops there and the user is told, with as much
+  diagnostic information as possible (the job's stdout/stderr) to help them
+  resolve and recover where possible.
+- If a stage succeeds, the chain proceeds to the next stage.
+- Stage 02 is interactive: it presents the collected comment lines (with the
+  comment character stripped) as the study/dataset description for the user to
+  review and edit before continuing.
+
+Realisation: each stage job carries a `success_handler` (set in its job
+metadata) that proceeds to the next stage, following the phenotypes pattern
+(`uploader.phenotypes.views.load_data_to_database` /
+`load_phenotypes_success_handler`). Failures are reported by the jobs system's
+error path (stdout/stderr on the job page).
+
+How best to bind the stages together (one job with gated tasks vs a chain of
+separate jobs) is still open; see the background-jobs design notes in
+`gn-libs/WIP/background-jobs-design-notes.md`.
+
+### Job status pages and stage transitions must be idempotent
+
+Job status URLs are resumable: opening `/background-jobs/status/<job-id>` for a
+completed job calls its `success_handler` again. Some stages are not idempotent
+(for example Stage 02 writes to the database), so a success handler must not
+re-run the next stage merely because the status URL was revisited.
+
+- Record that the next stage has started (for example store the next job id or
+  stage state in the job metadata) and check before starting anything.
+- On a revisit, present the information the user already entered instead of
+  repeating the transition.
+- Consider preventing a return to (or resumption of) earlier jobs once a later
+  stage has started; the exact mechanism is still to be decided.
+
+### Stage 02 state storage (interim)
+
+The confirmed description and genotype encoding are stored in the collector
+job's metadata (key `stage02`) for now:
+
+- the database has nowhere suitable to store descriptions;
+- `GenoCode` only stores integer values, while encodings may be floating point,
+  so it is not used at this stage.
+
+Revisiting Stage 02 reads these values back and presents them for editing.
 
 ### Comment → description (decided)
 
@@ -90,6 +131,7 @@ line:
   textbox as-is;
 - let the user edit the final description before it is used.
 
+Stage 02 presents this as the study/dataset description for review and editing.
 Open UI question: how to visually demarcate consecutive lines in the textbox.
 
 ## Stage 01 data contract (target)
@@ -110,6 +152,19 @@ The stats collector should produce enough information for the later stages:
 The current implementation of Stage 01 is described in the implementation
 notes.
 
+## QC / file-generation concerns (running list)
+
+This is a running list; more QC concerns will be added as they surface.
+
+- **Sample/individual/case order matters.** The order of samples in the
+  generated `.geno` file must be preserved consistently through the pipeline.
+- **Markers/records must be ordered by chromosome and position.** In the
+  generated file(s) (e.g. the `.geno` file), records should be ordered by
+  chromosome (ascending) and then by position (ascending) within each
+  chromosome. This could be a Stage 03 QC step. How to detect and handle it is
+  still unclear: markers can overlap in position, so a strict "strictly
+  increasing position" check may not be valid.
+
 ## Open design questions
 
 - **Description storage.** The description feeds the R/qtl2 control file; it is