about summary refs log tree commit diff
path: root/docs/WIP
diff options
context:
space:
mode:
authorFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 10:42:14 -0600
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 10:42:14 -0600
commit863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c (patch)
treef2f46e6a8c0e79af70ffdbcad61695385401f2d3 /docs/WIP
parent2bc7e2c194dbb3f22c6e44202abc9ea2846d3504 (diff)
downloadgn-uploader-863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c.tar.gz
Move documentation under the `docs/` directory.
Diffstat (limited to 'docs/WIP')
-rw-r--r--docs/WIP/geno-uploads-design-notes.md128
-rw-r--r--docs/WIP/geno-uploads-implementation-notes.md151
-rw-r--r--docs/WIP/geno-uploads-tasks-and-todos.md84
3 files changed, 363 insertions, 0 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md
new file mode 100644
index 0000000..48cb95f
--- /dev/null
+++ b/docs/WIP/geno-uploads-design-notes.md
@@ -0,0 +1,128 @@
+# Genotype uploads — design notes
+
+- **Feature:** genotypes upload
+- **Last updated:** 2026-10-22
+- **Related:** [implementation notes](geno-uploads-implementation-notes.md) ·
+  [tasks & todos](geno-uploads-tasks-and-todos.md) ·
+  background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)
+
+This file captures the *intended* design of the genotype upload feature: the
+pipeline stages, the data that flows between them, the behaviours each stage
+must honour, and the open design questions. It is expected to change as
+implementation realities surface (recorded in the implementation notes).
+
+## Goal
+
+Allow a user to upload a genotype CSV file for a dataset, validate/characterise
+it, let the user confirm the study design and genotype encoding, then convert
+and ingest it (eventually to an R/qtl2 bundle and the `.geno`/database
+representation), without losing information or silently mis-reading the file.
+
+## Pipeline stages
+
+The stage breakdown currently lives as a comment block in
+`uploader/genotypes/views.py` (`add_genotype_records`). Restated here as the
+design source of truth:
+
+1. **Stage 01 — Collect info**
+   - Detect custom `.geno` format lines (lines beginning with `@`).
+   - Collect the list of samples/cases/individuals.
+   - Collect the list of markers.
+   - Identify which fields are the markers/chromosome (and cM/Mb) fields.
+   - Collect the set of unique genotype symbols.
+   - Collect comment lines into a human-readable description: strip the leading
+     `#`, trim surrounding whitespace from each line, and preserve each line
+     separately (see “Comment → description” below; currently *not*
+     implemented).
+
+2. **Stage 02 — Establish basic study information**
+   - What is the study design?
+   - What is the genotype encoding scheme? (Seed this from `@` lines when
+     present.)
+   - Which value(s) mean "missing"?
+
+3. **Stage 03 — Final QC**
+   - Value-level QC, parameterised by the results of Stages 01–02.
+
+4. **Stage 04 — Convert to an R/qtl2 bundle.**
+
+5. **Stage 05 — Ingest the R/qtl2 bundle.**
+
+6. **Stage 06 — Generate the `.geno` file** (and other artefacts).
+
+7. **Stage 07 — Pre-compute the mapping(s).**
+
+## Cross-stage rules
+
+### Stage 01 output drives Stage 02 and Stage 03
+
+- The **set of genotype symbols** found in the file determines what the QC in
+  Stage 03 checks: only symbols that actually occur in the file should be
+  presented/validated. Do not offer or require symbols that are absent from the
+  file.
+- The **custom `.geno` lines**, when present, predefine labels for symbols, e.g.
+  - `mat` — maternal
+  - `pat` — paternal
+  - `het` — heterozygous
+  - `unk` — unknown
+- **Missing `@` lines are not an error.** The absence of all or some of these
+  custom lines must not fail Stage 02; the user simply supplies the encoding.
+- **Presence of `@` lines is not endorsement of their validity.** They are a
+  starting point: the user may change the labels. We must not treat a supplied
+  encoding as "correct" merely because it came from the file.
+
+### Stage gating
+
+Each stage consumes the previous stage's output. A later stage must not run on a
+file whose earlier stage failed or is incomplete. (How this gating is realised
+in the background-job system is an open design question — see the background-jobs
+design notes in `gn-libs/WIP/background-jobs-design-notes.md`.)
+
+### Comment → description (decided)
+
+Comment lines become a user-editable description that will later go into the
+R/qtl2 bundle's control file (and possibly the database). For each comment
+line:
+
+- strip the leading `#` character;
+- strip leading/trailing whitespace;
+- keep the lines separate (one line each) and present them to the user in a
+  textbox as-is;
+- let the user edit the final description before it is used.
+
+Open UI question: how to visually demarcate consecutive lines in the textbox.
+
+## Stage 01 data contract (target)
+
+The stats collector should produce enough information for the later stages:
+
+- counts: total/data/comment/blank/dot-geno lines;
+- `dot_geno_fields` parsed from `@` lines (`name`, `mat`, `pat`, `het`,
+  `type`, `unk` where present);
+- `geno_symbols` (the distinct symbols in the data);
+- `samples_list`;
+- `markers_list`;
+- parse errors encountered (column mismatches, etc.);
+- **a free-text description built from comment lines** (desired; not yet
+  implemented). Each comment line is cleaned (leading `#` and surrounding
+  whitespace removed) and kept as its own line.
+
+The current implementation of Stage 01 is described in the implementation
+notes.
+
+## Open design questions
+
+- **Description storage.** The description feeds the R/qtl2 control file; it is
+  undecided whether it is also persisted to the database.
+- **Encoding representation.** How do we represent a user-confirmed encoding so
+  that it is stable for QC and conversion (symbol → label/missing marker)?
+- **Validation vs. confirmation.** Where exactly is the boundary between "what
+  the file says" and "what the user confirms"?
+- **Size/scale.** Stage 01 must cope with large files; the collector already
+  has a single-threaded and a multi-processed path.
+
+## Out of scope here
+
+- Reworking the background-jobs system to support multi-task jobs. That is a
+  separate, cross-cutting feature; see the background-jobs design notes in
+  `gn-libs/WIP/background-jobs-design-notes.md`.
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md
new file mode 100644
index 0000000..e4baf3c
--- /dev/null
+++ b/docs/WIP/geno-uploads-implementation-notes.md
@@ -0,0 +1,151 @@
+# Genotype uploads — implementation notes & concerns
+
+- **Feature:** genotypes upload
+- **Last updated:** 2026-10-22
+- **Related:** [design notes](geno-uploads-design-notes.md) ·
+  [tasks & todos](geno-uploads-tasks-and-todos.md)
+
+This file records what is actually implemented, how it maps to the design, and
+the concerns/deviations discovered while building it. It supersedes the
+2026-10-15 implementation summary.
+
+## Status at a glance
+
+- **Stage 01 engine: implemented and tested.** The statistics collector exists
+  at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`.
+- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is
+  resolved for both the resumable and direct upload paths and handed to the
+  collector stub; mapping the form metadata to the collector config, running
+  the collector and persisting its results are still pending.
+- **Stages 02–07: not implemented** (still design/TODO comment blocks in
+  `uploader/genotypes/views.py`).
+
+## Stage 01 implementation
+
+### Location
+
+`uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`).
+Runnable as a CLI; emits `FileStats` as JSON.
+
+### What it collects
+
+`FileStats` (`uploader/scripts/genotypes/types.py`) currently contains:
+
+- `total_lines`, `data_lines`, `comment_lines`, `blank_lines`,
+  `dot_geno_lines`;
+- `dot_geno_fields` (parsed from `@` lines, when present);
+- `geno_symbols` (distinct symbols in the data, sorted for determinism);
+- `samples_list`, `markers_list`;
+- `errors` (header validation errors + per-line errors).
+
+**Not collected:** the text of comment lines. Only a count is kept, so the
+"description string" part of Stage 01 is still missing.
+
+### Configuration
+
+`compute_statistics(filepath, fileconfigs)` expects a config dict with:
+
+```
+markers, chromosome, cm, mb, separator,
+comment.char, na_strings, transposed
+```
+
+### Processing design
+
+- Per-line processing is independent, so it was extracted into `_accumulate`,
+  which returns a `_ChunkStats` partial.
+- `file_stats_singlethread` = one `_accumulate` pass over the whole file.
+- `file_stats_multiprocessing` = stream the file into bounded chunks
+  (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a
+  worker process (`Pool.imap`), and merge in file order
+  (`_merge_chunk_stats` / `_merge_all_chunk_stats`).
+- The worker is module-level and takes only picklable data (headers, config,
+  chunk), with the split/skip predicates rebuilt inside the worker, so it works
+  under `fork`, `spawn` and `forkserver`.
+- `compute_statistics` chooses the single-threaded or multi-processed path
+  based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes.
+
+### Merge semantics (why the two paths agree)
+
+- counters: summed;
+- `geno_symbols`: unioned (then sorted on output);
+- `errors`: concatenated in file order;
+- `markers_list`: concatenated in order (untransposed), or last markers line
+  wins (transposed);
+- `dot_geno_fields`: combined, later values override earlier ones.
+
+### Defects fixed while implementing
+
+- Size threshold compared megabytes to bytes (branch was effectively
+  unreachable). Now bytes vs bytes.
+- Header-skip compared a `str` to a `tuple`, so header rows were counted as
+  data. Now compares the split line to the headers.
+- `FileStats.blank_lines` was declared but never populated. Now populated.
+- `geno_symbols` output order was set-iteration-dependent (unstable across
+  runs/processes). Now sorted.
+
+### Tests
+
+`tests/genotypes/` (pytest): accumulation, single-threaded stats, merge
+semantics, and single- vs multi-processed equivalence across chunk sizes,
+including transposed files and files with errors.
+
+`tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and
+resumable uploads, hashed/unhashed, error cases). Together with
+`tests/genotypes/` this is **31 passing**.
+
+## Integration concerns (upload flow)
+
+### Uploaded-file resolution (done)
+
+`uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname,
+hashed=True)` resolves either a chunked/resumable upload (the JSON blob
+returned by the resumable upload endpoint) or a direct multipart upload,
+returning `Right((path, original_name))` or `Left(message)`. It lives in
+`uploader.files` because it is generic — the only per-caller input is the field
+name — and it is the consumer counterpart of the resumable-upload contract that
+package owns.
+
+- `add_genotype_records` uses it with the `genotypes-records-file` field.
+- `process_phenotypes_individual_files` was refactored onto it, fixing:
+  directly-uploaded optional files being skipped, a `BadRequestKeyError` for
+  missing direct files, unhandled resumable-data errors, and a fragile
+  double-prefixed path.
+
+### The view does not run the collector
+
+In `uploader/genotypes/views.py`:
+
+- `_collect_geno_csv_stats_()` is a stub that just returns a dict
+  (`{"file": ..., "metadata": ...}`).
+- `add_genotype_records()` (POST) builds that dict and renders a placeholder
+  template; it never invokes `preprocess_csv_collect_info`/`compute_statistics`
+  and never schedules a job.
+
+### Config-key mismatch to bridge
+
+The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the
+collector expects `separator`, `comment.char`, `na_strings` plus `markers`,
+`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char`
+already matches).
+
+### Stage 01 form inputs exist; the view must read them
+
+The upload form
+(`uploader/templates/genotypes/add-genotypes-records-csv.html`) already
+provides `markers_field`, `chromosomes_field`, `centimorgans_field`,
+`megabases_field`, `transposed`, plus the shared `file-separator`,
+`file-comment-char` and `file-na` inputs. The work is to read and map them to
+the collector's config (see above), not to add new inputs.
+
+### Comment description missing
+
+As noted above, comment text is discarded (only counted). The desired
+multiline description output is not implemented.
+
+## Relationship to the background-jobs system
+
+The current job model is effectively "one script == one job". The genotype
+upload is one logical operation spanning several stages, which stresses that
+model. The options (and why a redesign is a separate feature) are recorded in
+`gn-libs/WIP/background-jobs-design-notes.md`.
diff --git a/docs/WIP/geno-uploads-tasks-and-todos.md b/docs/WIP/geno-uploads-tasks-and-todos.md
new file mode 100644
index 0000000..cf52c44
--- /dev/null
+++ b/docs/WIP/geno-uploads-tasks-and-todos.md
@@ -0,0 +1,84 @@
+# Genotype uploads — tasks, todos & progress
+
+- **Feature:** genotypes upload
+- **Last updated:** 2026-10-22
+- **Related:** [design notes](geno-uploads-design-notes.md) ·
+  [implementation notes](geno-uploads-implementation-notes.md) ·
+  background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)
+
+Legend: ✅ done · 🚧 in progress · ⬜ todo · 💭 consideration (not committed)
+
+This is the single place to avoid tasks falling through the cracks. Design
+rationale lives in the design notes; what actually got built lives in the
+implementation notes.
+
+## Stage 01 — Collect info
+
+- ✅ Statistics collector implemented (`uploader/scripts/genotypes/preprocess_csv_collect_info.py`).
+- ✅ Single- and multi-processed paths produce identical `FileStats`.
+- ✅ Unit tests for accumulation, merge and equivalence (`tests/genotypes/`).
+- ✅ Fix threshold / header-skip / `blank_lines` defects.
+- ⬜ **Collect comment lines into a multiline description string** (currently
+  only counted, text discarded). Format decided: strip the leading `#`, trim
+  surrounding whitespace, keep one line per comment, present in an editable
+  textbox for the user; the result feeds the R/qtl2 control file (see design
+  notes). Still to do: implement it and settle the textbox styling.
+- 💭 Decide whether the final description is also persisted to the database, or
+  only written into the R/qtl2 control file.
+- 🚧 **Wire the collector into the upload view** (`add_genotype_records`):
+  - ✅ Resolve the uploaded file for both the resumable and direct upload paths,
+    via `uploader.files.resolve_uploaded_file`.
+  - ⬜ Read the form's metadata (`markers_field`, `chromosomes_field`,
+    `centimorgans_field`, `megabases_field`, `transposed`, `file-separator`,
+    `file-comment-char`, `file-na`) and map it to the collector's config keys
+    (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`,
+    `comment.char`, `na_strings`). The inputs already exist in the form.
+  - ⬜ Run the collector (or schedule it) instead of returning the stub
+    `_collect_geno_csv_stats_` dict.
+  - ⬜ Persist/return the resulting `FileStats` for the next stage.
+
+## Stage 02 — Study design & genotype encoding
+
+- ⬜ Present the genotype symbols discovered in Stage 01 for encoding.
+  - ⬜ Only symbols that actually occur in the file should be offered.
+- ⬜ Seed labels from `@` lines when present (`mat`, `pat`, `het`, `unk`).
+  - ⬜ Missing `@` lines (all or some) must **not** fail the stage.
+  - ⬜ Treat `@`-derived labels as a starting point only; allow the user to
+    change them. Presence is not validation.
+- ⬜ Allow the user to specify missing-value symbol(s).
+- ⬜ Capture the confirmed encoding/study design for downstream stages.
+
+## Stage 03 — Final QC
+
+- ⬜ Perform value-level QC parameterised by Stage 01/02 results.
+- ⬜ Restrict checks to the symbols present in the file.
+
+## Stages 04–07 — Conversion & ingest
+
+- ⬜ Stage 04: convert to R/qtl2 bundle.
+- ⬜ Stage 05: ingest the R/qtl2 bundle.
+- ⬜ Stage 06: generate `.geno` (and other artefacts).
+- ⬜ Stage 07: pre-compute mapping(s).
+
+## Refactoring / technical debt
+
+- 💭 **Deferred:** once `add_genotype_records` grows to cover the later stages,
+  refactor it into a monadic pipeline rather than a series of guard clauses —
+  resolve the uploaded file, then chain each stage with `.then(...)`, ending
+  with `.either(_handle_error_path, _handle_success_path)`. Deferred until the
+  view is large enough for the split to pay off.
+
+## Background jobs
+
+- 💭 Decide whether genotype upload stays "one script == one job" or becomes
+  one job with multiple, gated tasks. **Not decided; treat as a separate
+  feature.** See `gn-libs/WIP/background-jobs-design-notes.md` and the
+  migration inventory in `gn-libs/WIP/background-jobs-tasks-and-todos.md`.
+- 💭 (Lower priority, only if the multi-task redesign is chosen) a way to
+  re-run/"fix" failed tasks.
+
+## Housekeeping
+
+- ⬜ Track the new templates `genotype-encoding-elements.html` and
+  `study-design-elements.html` (currently untracked) once they are wired into
+  Stage 02.