diff options
| author | pi coding agent <pi@localhost> | 2026-11-05 19:05:23 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-06 07:06:01 -0600 |
| commit | 9cea8b483f61be415a431e19b508ea390cf10d87 (patch) | |
| tree | 03dcd993fd166ecc77fb36e79aa62ee7f59de428 /docs | |
| parent | 1f40a2114848aa35ed37ca26ed48ed26694214d0 (diff) | |
| download | gn-uploader-9cea8b483f61be415a431e19b508ea390cf10d87.tar.gz | |
docs: Update design docs
- include missing-value symbols in the Stage 02 state - capture QC value and ordering decisions - capture the untranspose-transposed-files idea
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-design-notes.md | 57 |
1 files changed, 49 insertions, 8 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md index d193f04..820e5f3 100644 --- a/docs/WIP/geno-uploads-design-notes.md +++ b/docs/WIP/geno-uploads-design-notes.md @@ -110,8 +110,8 @@ re-run the next stage merely because the status URL was revisited. ### Stage 02 state storage (interim) -The confirmed description and genotype encoding are stored in the collector -job's metadata (key `stage02`) for now: +The confirmed description, genotype encoding and missing-value symbol(s) are +stored in the collector job's metadata (key `stage02`) for now: - the database has nowhere suitable to store descriptions; - `GenoCode` only stores integer values, while encodings may be floating point, @@ -156,14 +156,55 @@ notes. This is a running list; more QC concerns will be added as they surface. +- **Unknown (`unk`) symbols are missing values.** Seed `dot_geno_fields.unk` + into the Stage 02 missing-value symbols. +- **Allowed genotype values.** Every sample value must be one of the confirmed + maternal/paternal/heterozygous symbols or a missing-value symbol. Empty + values are not allowed. Matching is case-sensitive. Symbols may be longer + than three characters and need not be single characters, so the Stage 02 + UI's three-character limit should be revisited (low priority). - **Sample/individual/case order matters.** The order of samples in the generated `.geno` file must be preserved consistently through the pipeline. -- **Markers/records must be ordered by chromosome and position.** In the - generated file(s) (e.g. the `.geno` file), records should be ordered by - chromosome (ascending) and then by position (ascending) within each - chromosome. This could be a Stage 03 QC step. How to detect and handle it is - still unclear: markers can overlap in position, so a strict "strictly - increasing position" check may not be valid. + See the ordering rules below. +- **Markers/records ordering (placeholder).** For now, sort records by + chromosome (ascending), then cM (ascending), then Mb (ascending). The + sorting helper emits a warning because the appropriate ordering has not been + established; replace this placeholder and remove the warning once known. + TODO. + +### Sample/individual/case ordering rules + +1. If the population already has an order, use it, then append any missing + samples (sorted alphabetically). +2. If there is no existing order, sort the samples alphabetically. +3. Present the resulting order to the user for review and reordering. +4. Save the accepted order. + +## Untransposing transposed files (to decide) + +We may want to untranspose transposed genotype files so downstream processing +only ever deals with a single orientation. + +Existing helper: `r_qtl.r_qtl2.transpose_csv` / +`transpose_csv_with_rename` (used by `undo_transpose` in +`uploader.scripts.rqtl2.phenotypes_qc`). It is not directly suitable for +genotype files yet: it skips `#` comment lines, does not special-case `@` +`.geno` directive lines, uses `zip(*...)` (truncates ragged rows), and renames +the original file in place. + +Open questions: + +- **Where in the chain?** Options: before Stage 01 (normalise on upload); as + part of Stage 01 (after collecting headers/metadata); or just before Stage 04 + (bundle conversion). Downstream QC currently handles both orientations. +- **Preserve the original?** Prefer writing a canonical untransposed artifact + and keeping the original upload immutable (the existing helper renames the + original in place). +- **Comments and `@` directives:** comments (the study description) and custom + `.geno` directives must survive untransposition. +- **Ragged rows / validation:** ensure column consistency during untranspose. +- **Storage/naming:** where the canonical file lives and how its path is + recorded in the job metadata. ## Open design questions |
