about summary refs log tree commit diff
path: root/docs/WIP
diff options
context:
space:
mode:
authorpi coding agent <pi@localhost>2026-11-04 18:17:28 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 14:57:32 -0600
commit5d88dd6028b7d8244f0bfeda0acd01ac6b6b8213 (patch)
tree7668cd8b9aa22b2a77b02fd4be28a42aaf4a21b4 /docs/WIP
parent94ed0ab8f0e19d72a998b99a21e04159a3f69246 (diff)
downloadgn-uploader-5d88dd6028b7d8244f0bfeda0acd01ac6b6b8213.tar.gz
docs: capture a number of design-related concerns/requirements
- the stage-job chain and Stage 02 description behaviour
- the job-status idempotency requirement
- Stage 02 state storage and the no-going-back consideration
- start a running QC/file-generation concerns list
Diffstat (limited to 'docs/WIP')
-rw-r--r--docs/WIP/geno-uploads-design-notes.md67
1 files changed, 61 insertions, 6 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md
index 48cb95f..d193f04 100644
--- a/docs/WIP/geno-uploads-design-notes.md
+++ b/docs/WIP/geno-uploads-design-notes.md
@@ -1,7 +1,7 @@
 # Genotype uploads — design notes
 
 - **Feature:** genotypes upload
-- **Last updated:** 2026-10-22
+- **Last updated:** 2026-11-04
 - **Related:** [implementation notes](geno-uploads-implementation-notes.md) ·
   [tasks & todos](geno-uploads-tasks-and-todos.md) ·
   background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)
@@ -71,12 +71,53 @@ design source of truth:
   starting point: the user may change the labels. We must not treat a supplied
   encoding as "correct" merely because it came from the file.
 
-### Stage gating
+### Stage gating and job chaining (decided)
 
-Each stage consumes the previous stage's output. A later stage must not run on a
-file whose earlier stage failed or is incomplete. (How this gating is realised
-in the background-job system is an open design question — see the background-jobs
-design notes in `gn-libs/WIP/background-jobs-design-notes.md`.)
+Stages run in order as a chain of background jobs: Stage 01 (collect info),
+then Stage 02 (study design/encoding), then Stage 03 (QC), and so on.
+
+- If a stage fails, the chain stops there and the user is told, with as much
+  diagnostic information as possible (the job's stdout/stderr) to help them
+  resolve and recover where possible.
+- If a stage succeeds, the chain proceeds to the next stage.
+- Stage 02 is interactive: it presents the collected comment lines (with the
+  comment character stripped) as the study/dataset description for the user to
+  review and edit before continuing.
+
+Realisation: each stage job carries a `success_handler` (set in its job
+metadata) that proceeds to the next stage, following the phenotypes pattern
+(`uploader.phenotypes.views.load_data_to_database` /
+`load_phenotypes_success_handler`). Failures are reported by the jobs system's
+error path (stdout/stderr on the job page).
+
+How best to bind the stages together (one job with gated tasks vs a chain of
+separate jobs) is still open; see the background-jobs design notes in
+`gn-libs/WIP/background-jobs-design-notes.md`.
+
+### Job status pages and stage transitions must be idempotent
+
+Job status URLs are resumable: opening `/background-jobs/status/<job-id>` for a
+completed job calls its `success_handler` again. Some stages are not idempotent
+(for example Stage 02 writes to the database), so a success handler must not
+re-run the next stage merely because the status URL was revisited.
+
+- Record that the next stage has started (for example store the next job id or
+  stage state in the job metadata) and check before starting anything.
+- On a revisit, present the information the user already entered instead of
+  repeating the transition.
+- Consider preventing a return to (or resumption of) earlier jobs once a later
+  stage has started; the exact mechanism is still to be decided.
+
+### Stage 02 state storage (interim)
+
+The confirmed description and genotype encoding are stored in the collector
+job's metadata (key `stage02`) for now:
+
+- the database has nowhere suitable to store descriptions;
+- `GenoCode` only stores integer values, while encodings may be floating point,
+  so it is not used at this stage.
+
+Revisiting Stage 02 reads these values back and presents them for editing.
 
 ### Comment → description (decided)
 
@@ -90,6 +131,7 @@ line:
   textbox as-is;
 - let the user edit the final description before it is used.
 
+Stage 02 presents this as the study/dataset description for review and editing.
 Open UI question: how to visually demarcate consecutive lines in the textbox.
 
 ## Stage 01 data contract (target)
@@ -110,6 +152,19 @@ The stats collector should produce enough information for the later stages:
 The current implementation of Stage 01 is described in the implementation
 notes.
 
+## QC / file-generation concerns (running list)
+
+This is a running list; more QC concerns will be added as they surface.
+
+- **Sample/individual/case order matters.** The order of samples in the
+  generated `.geno` file must be preserved consistently through the pipeline.
+- **Markers/records must be ordered by chromosome and position.** In the
+  generated file(s) (e.g. the `.geno` file), records should be ordered by
+  chromosome (ascending) and then by position (ascending) within each
+  chromosome. This could be a Stage 03 QC step. How to detect and handle it is
+  still unclear: markers can overlap in position, so a strict "strictly
+  increasing position" check may not be valid.
+
 ## Open design questions
 
 - **Description storage.** The description feeds the R/qtl2 control file; it is