about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-06 20:53:35 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-06 14:57:39 -0600
commit5874283dcef47b25e2bbacf77260ebfb142edd15 (patch)
tree14b72b2a571b40c52fc3df2cf7456c4d7dd57f1a /docs
parent966f48fd5529082d2bdeecd6c8fe8e7a48b0c885 (diff)
downloadgn-uploader-5874283dcef47b25e2bbacf77260ebfb142edd15.tar.gz
docs: capture untranspose placement, comment scoping and error handling
Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-design-notes.md67
1 files changed, 42 insertions, 25 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md
index 820e5f3..2098b64 100644
--- a/docs/WIP/geno-uploads-design-notes.md
+++ b/docs/WIP/geno-uploads-design-notes.md
@@ -108,6 +108,19 @@ re-run the next stage merely because the status URL was revisited.
 - Consider preventing a return to (or resumption of) earlier jobs once a later
   stage has started; the exact mechanism is still to be decided.
 
+### Stage error handling
+
+Each node in the chain registers an `error_handler` (alongside its
+`success_handler`) that informs the user and stops the chain.
+
+- **`FileStats.errors`** (malformed data found during collection): display them
+  like the phenotypes job status page
+  (`uploader.phenotypes.views.job_status`,
+  `uploader/templates/phenotypes/job-status.html`).
+- **Any other failure** (script crash, IO/database/token issues): tell the user
+  what failed and give reasonable recovery steps, including stdout/stderr where
+  useful.
+
 ### Stage 02 state storage (interim)
 
 The confirmed description, genotype encoding and missing-value symbol(s) are
@@ -131,6 +144,15 @@ line:
   textbox as-is;
 - let the user edit the final description before it is used.
 
+Only comments **before the data matrix starts** feed the description:
+
+- untransposed: before the header row;
+- transposed: before the first data row (the markers row).
+
+Later comments are counted (`comment_lines`) but not used; they relate to
+adjacent rows, not the whole file, and untransposing would take them out of
+context.
+
 Stage 02 presents this as the study/dataset description for review and editing.
 Open UI question: how to visually demarcate consecutive lines in the textbox.
 
@@ -180,31 +202,26 @@ This is a running list; more QC concerns will be added as they surface.
 3. Present the resulting order to the user for review and reordering.
 4. Save the accepted order.
 
-## Untransposing transposed files (to decide)
-
-We may want to untranspose transposed genotype files so downstream processing
-only ever deals with a single orientation.
-
-Existing helper: `r_qtl.r_qtl2.transpose_csv` /
-`transpose_csv_with_rename` (used by `undo_transpose` in
-`uploader.scripts.rqtl2.phenotypes_qc`). It is not directly suitable for
-genotype files yet: it skips `#` comment lines, does not special-case `@`
-`.geno` directive lines, uses `zip(*...)` (truncates ragged rows), and renames
-the original file in place.
-
-Open questions:
-
-- **Where in the chain?** Options: before Stage 01 (normalise on upload); as
-  part of Stage 01 (after collecting headers/metadata); or just before Stage 04
-  (bundle conversion). Downstream QC currently handles both orientations.
-- **Preserve the original?** Prefer writing a canonical untransposed artifact
-  and keeping the original upload immutable (the existing helper renames the
-  original in place).
-- **Comments and `@` directives:** comments (the study description) and custom
-  `.geno` directives must survive untransposition.
-- **Ragged rows / validation:** ensure column consistency during untranspose.
-- **Storage/naming:** where the canonical file lives and how its path is
-  recorded in the job metadata.
+## Untransposing transposed files (decided)
+
+Transposed files are untransposed as the **first step of Stage 03**, before any
+QC. Stage 02 only consumes metadata from `FileStats`, so there is no reason to
+normalise a file that might be abandoned at review; Stage 03 is the first stage
+that needs the data.
+
+- Keep the original upload immutable and write a separate canonical `work_file`;
+  record it (and the original) in the collector job metadata.
+- The canonical file is a **pure data matrix**: comments and `@` directives are
+  omitted because Stage 01 already captured them (`description`,
+  `dot_geno_fields`).
+- Skip `#`/`@` lines when transposing and validate ragged rows. The existing
+  `r_qtl.r_qtl2.transpose_csv_with_rename` is unsuitable (renames in place,
+  skips `#`, treats `@` as data, truncates ragged rows).
+- Do not untranspose if Stage 01 reported errors; the collector now exits with
+  `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so the chain
+  stops before Stage 02.
+- After normalisation, downstream (QC and later) handles only the normal
+  orientation (`transposed=False`), simplifying QC and ordering code.
 
 ## Open design questions