about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorpi coding agent <pi@localhost>2026-11-05 19:05:23 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-06 07:06:01 -0600
commit9cea8b483f61be415a431e19b508ea390cf10d87 (patch)
tree03dcd993fd166ecc77fb36e79aa62ee7f59de428 /docs
parent1f40a2114848aa35ed37ca26ed48ed26694214d0 (diff)
downloadgn-uploader-9cea8b483f61be415a431e19b508ea390cf10d87.tar.gz
docs: Update design docs
- include missing-value symbols in the Stage 02 state
- capture QC value and ordering decisions
- capture the untranspose-transposed-files idea
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-design-notes.md57
1 files changed, 49 insertions, 8 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md
index d193f04..820e5f3 100644
--- a/docs/WIP/geno-uploads-design-notes.md
+++ b/docs/WIP/geno-uploads-design-notes.md
@@ -110,8 +110,8 @@ re-run the next stage merely because the status URL was revisited.
 
 ### Stage 02 state storage (interim)
 
-The confirmed description and genotype encoding are stored in the collector
-job's metadata (key `stage02`) for now:
+The confirmed description, genotype encoding and missing-value symbol(s) are
+stored in the collector job's metadata (key `stage02`) for now:
 
 - the database has nowhere suitable to store descriptions;
 - `GenoCode` only stores integer values, while encodings may be floating point,
@@ -156,14 +156,55 @@ notes.
 
 This is a running list; more QC concerns will be added as they surface.
 
+- **Unknown (`unk`) symbols are missing values.** Seed `dot_geno_fields.unk`
+  into the Stage 02 missing-value symbols.
+- **Allowed genotype values.** Every sample value must be one of the confirmed
+  maternal/paternal/heterozygous symbols or a missing-value symbol. Empty
+  values are not allowed. Matching is case-sensitive. Symbols may be longer
+  than three characters and need not be single characters, so the Stage 02
+  UI's three-character limit should be revisited (low priority).
 - **Sample/individual/case order matters.** The order of samples in the
   generated `.geno` file must be preserved consistently through the pipeline.
-- **Markers/records must be ordered by chromosome and position.** In the
-  generated file(s) (e.g. the `.geno` file), records should be ordered by
-  chromosome (ascending) and then by position (ascending) within each
-  chromosome. This could be a Stage 03 QC step. How to detect and handle it is
-  still unclear: markers can overlap in position, so a strict "strictly
-  increasing position" check may not be valid.
+  See the ordering rules below.
+- **Markers/records ordering (placeholder).** For now, sort records by
+  chromosome (ascending), then cM (ascending), then Mb (ascending). The
+  sorting helper emits a warning because the appropriate ordering has not been
+  established; replace this placeholder and remove the warning once known.
+  TODO.
+
+### Sample/individual/case ordering rules
+
+1. If the population already has an order, use it, then append any missing
+   samples (sorted alphabetically).
+2. If there is no existing order, sort the samples alphabetically.
+3. Present the resulting order to the user for review and reordering.
+4. Save the accepted order.
+
+## Untransposing transposed files (to decide)
+
+We may want to untranspose transposed genotype files so downstream processing
+only ever deals with a single orientation.
+
+Existing helper: `r_qtl.r_qtl2.transpose_csv` /
+`transpose_csv_with_rename` (used by `undo_transpose` in
+`uploader.scripts.rqtl2.phenotypes_qc`). It is not directly suitable for
+genotype files yet: it skips `#` comment lines, does not special-case `@`
+`.geno` directive lines, uses `zip(*...)` (truncates ragged rows), and renames
+the original file in place.
+
+Open questions:
+
+- **Where in the chain?** Options: before Stage 01 (normalise on upload); as
+  part of Stage 01 (after collecting headers/metadata); or just before Stage 04
+  (bundle conversion). Downstream QC currently handles both orientations.
+- **Preserve the original?** Prefer writing a canonical untransposed artifact
+  and keeping the original upload immutable (the existing helper renames the
+  original in place).
+- **Comments and `@` directives:** comments (the study description) and custom
+  `.geno` directives must survive untransposition.
+- **Ragged rows / validation:** ensure column consistency during untranspose.
+- **Storage/naming:** where the canonical file lives and how its path is
+  recorded in the job metadata.
 
 ## Open design questions