diff options
| author | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-06 20:53:35 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-06 14:57:39 -0600 |
| commit | 5874283dcef47b25e2bbacf77260ebfb142edd15 (patch) | |
| tree | 14b72b2a571b40c52fc3df2cf7456c4d7dd57f1a /docs | |
| parent | 966f48fd5529082d2bdeecd6c8fe8e7a48b0c885 (diff) | |
| download | gn-uploader-5874283dcef47b25e2bbacf77260ebfb142edd15.tar.gz | |
docs: capture untranspose placement, comment scoping and error handling
Co-authored-by: pi coding agent <pi@pi.dev>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-design-notes.md | 67 |
1 files changed, 42 insertions, 25 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md index 820e5f3..2098b64 100644 --- a/docs/WIP/geno-uploads-design-notes.md +++ b/docs/WIP/geno-uploads-design-notes.md @@ -108,6 +108,19 @@ re-run the next stage merely because the status URL was revisited. - Consider preventing a return to (or resumption of) earlier jobs once a later stage has started; the exact mechanism is still to be decided. +### Stage error handling + +Each node in the chain registers an `error_handler` (alongside its +`success_handler`) that informs the user and stops the chain. + +- **`FileStats.errors`** (malformed data found during collection): display them + like the phenotypes job status page + (`uploader.phenotypes.views.job_status`, + `uploader/templates/phenotypes/job-status.html`). +- **Any other failure** (script crash, IO/database/token issues): tell the user + what failed and give reasonable recovery steps, including stdout/stderr where + useful. + ### Stage 02 state storage (interim) The confirmed description, genotype encoding and missing-value symbol(s) are @@ -131,6 +144,15 @@ line: textbox as-is; - let the user edit the final description before it is used. +Only comments **before the data matrix starts** feed the description: + +- untransposed: before the header row; +- transposed: before the first data row (the markers row). + +Later comments are counted (`comment_lines`) but not used; they relate to +adjacent rows, not the whole file, and untransposing would take them out of +context. + Stage 02 presents this as the study/dataset description for review and editing. Open UI question: how to visually demarcate consecutive lines in the textbox. @@ -180,31 +202,26 @@ This is a running list; more QC concerns will be added as they surface. 3. Present the resulting order to the user for review and reordering. 4. Save the accepted order. -## Untransposing transposed files (to decide) - -We may want to untranspose transposed genotype files so downstream processing -only ever deals with a single orientation. - -Existing helper: `r_qtl.r_qtl2.transpose_csv` / -`transpose_csv_with_rename` (used by `undo_transpose` in -`uploader.scripts.rqtl2.phenotypes_qc`). It is not directly suitable for -genotype files yet: it skips `#` comment lines, does not special-case `@` -`.geno` directive lines, uses `zip(*...)` (truncates ragged rows), and renames -the original file in place. - -Open questions: - -- **Where in the chain?** Options: before Stage 01 (normalise on upload); as - part of Stage 01 (after collecting headers/metadata); or just before Stage 04 - (bundle conversion). Downstream QC currently handles both orientations. -- **Preserve the original?** Prefer writing a canonical untransposed artifact - and keeping the original upload immutable (the existing helper renames the - original in place). -- **Comments and `@` directives:** comments (the study description) and custom - `.geno` directives must survive untransposition. -- **Ragged rows / validation:** ensure column consistency during untranspose. -- **Storage/naming:** where the canonical file lives and how its path is - recorded in the job metadata. +## Untransposing transposed files (decided) + +Transposed files are untransposed as the **first step of Stage 03**, before any +QC. Stage 02 only consumes metadata from `FileStats`, so there is no reason to +normalise a file that might be abandoned at review; Stage 03 is the first stage +that needs the data. + +- Keep the original upload immutable and write a separate canonical `work_file`; + record it (and the original) in the collector job metadata. +- The canonical file is a **pure data matrix**: comments and `@` directives are + omitted because Stage 01 already captured them (`description`, + `dot_geno_fields`). +- Skip `#`/`@` lines when transposing and validate ragged rows. The existing + `r_qtl.r_qtl2.transpose_csv_with_rename` is unsuitable (renames in place, + skips `#`, treats `@` as data, truncates ragged rows). +- Do not untranspose if Stage 01 reported errors; the collector now exits with + `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so the chain + stops before Stage 02. +- After normalisation, downstream (QC and later) handles only the normal + orientation (`transposed=False`), simplifying QC and ordering code. ## Open design questions |
