diff options
| author | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-04 10:42:14 -0600 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-04 10:42:14 -0600 |
| commit | 863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c (patch) | |
| tree | f2f46e6a8c0e79af70ffdbcad61695385401f2d3 /docs | |
| parent | 2bc7e2c194dbb3f22c6e44202abc9ea2846d3504 (diff) | |
| download | gn-uploader-863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c.tar.gz | |
Move documentation under the `docs/` directory.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-design-notes.md | 128 | ||||
| -rw-r--r-- | docs/WIP/geno-uploads-implementation-notes.md | 151 | ||||
| -rw-r--r-- | docs/WIP/geno-uploads-tasks-and-todos.md | 84 |
3 files changed, 363 insertions, 0 deletions
diff --git a/docs/WIP/geno-uploads-design-notes.md b/docs/WIP/geno-uploads-design-notes.md new file mode 100644 index 0000000..48cb95f --- /dev/null +++ b/docs/WIP/geno-uploads-design-notes.md @@ -0,0 +1,128 @@ +# Genotype uploads — design notes + +- **Feature:** genotypes upload +- **Last updated:** 2026-10-22 +- **Related:** [implementation notes](geno-uploads-implementation-notes.md) · + [tasks & todos](geno-uploads-tasks-and-todos.md) · + background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`) + +This file captures the *intended* design of the genotype upload feature: the +pipeline stages, the data that flows between them, the behaviours each stage +must honour, and the open design questions. It is expected to change as +implementation realities surface (recorded in the implementation notes). + +## Goal + +Allow a user to upload a genotype CSV file for a dataset, validate/characterise +it, let the user confirm the study design and genotype encoding, then convert +and ingest it (eventually to an R/qtl2 bundle and the `.geno`/database +representation), without losing information or silently mis-reading the file. + +## Pipeline stages + +The stage breakdown currently lives as a comment block in +`uploader/genotypes/views.py` (`add_genotype_records`). Restated here as the +design source of truth: + +1. **Stage 01 — Collect info** + - Detect custom `.geno` format lines (lines beginning with `@`). + - Collect the list of samples/cases/individuals. + - Collect the list of markers. + - Identify which fields are the markers/chromosome (and cM/Mb) fields. + - Collect the set of unique genotype symbols. + - Collect comment lines into a human-readable description: strip the leading + `#`, trim surrounding whitespace from each line, and preserve each line + separately (see “Comment → description” below; currently *not* + implemented). + +2. **Stage 02 — Establish basic study information** + - What is the study design? + - What is the genotype encoding scheme? (Seed this from `@` lines when + present.) + - Which value(s) mean "missing"? + +3. **Stage 03 — Final QC** + - Value-level QC, parameterised by the results of Stages 01–02. + +4. **Stage 04 — Convert to an R/qtl2 bundle.** + +5. **Stage 05 — Ingest the R/qtl2 bundle.** + +6. **Stage 06 — Generate the `.geno` file** (and other artefacts). + +7. **Stage 07 — Pre-compute the mapping(s).** + +## Cross-stage rules + +### Stage 01 output drives Stage 02 and Stage 03 + +- The **set of genotype symbols** found in the file determines what the QC in + Stage 03 checks: only symbols that actually occur in the file should be + presented/validated. Do not offer or require symbols that are absent from the + file. +- The **custom `.geno` lines**, when present, predefine labels for symbols, e.g. + - `mat` — maternal + - `pat` — paternal + - `het` — heterozygous + - `unk` — unknown +- **Missing `@` lines are not an error.** The absence of all or some of these + custom lines must not fail Stage 02; the user simply supplies the encoding. +- **Presence of `@` lines is not endorsement of their validity.** They are a + starting point: the user may change the labels. We must not treat a supplied + encoding as "correct" merely because it came from the file. + +### Stage gating + +Each stage consumes the previous stage's output. A later stage must not run on a +file whose earlier stage failed or is incomplete. (How this gating is realised +in the background-job system is an open design question — see the background-jobs +design notes in `gn-libs/WIP/background-jobs-design-notes.md`.) + +### Comment → description (decided) + +Comment lines become a user-editable description that will later go into the +R/qtl2 bundle's control file (and possibly the database). For each comment +line: + +- strip the leading `#` character; +- strip leading/trailing whitespace; +- keep the lines separate (one line each) and present them to the user in a + textbox as-is; +- let the user edit the final description before it is used. + +Open UI question: how to visually demarcate consecutive lines in the textbox. + +## Stage 01 data contract (target) + +The stats collector should produce enough information for the later stages: + +- counts: total/data/comment/blank/dot-geno lines; +- `dot_geno_fields` parsed from `@` lines (`name`, `mat`, `pat`, `het`, + `type`, `unk` where present); +- `geno_symbols` (the distinct symbols in the data); +- `samples_list`; +- `markers_list`; +- parse errors encountered (column mismatches, etc.); +- **a free-text description built from comment lines** (desired; not yet + implemented). Each comment line is cleaned (leading `#` and surrounding + whitespace removed) and kept as its own line. + +The current implementation of Stage 01 is described in the implementation +notes. + +## Open design questions + +- **Description storage.** The description feeds the R/qtl2 control file; it is + undecided whether it is also persisted to the database. +- **Encoding representation.** How do we represent a user-confirmed encoding so + that it is stable for QC and conversion (symbol → label/missing marker)? +- **Validation vs. confirmation.** Where exactly is the boundary between "what + the file says" and "what the user confirms"? +- **Size/scale.** Stage 01 must cope with large files; the collector already + has a single-threaded and a multi-processed path. + +## Out of scope here + +- Reworking the background-jobs system to support multi-task jobs. That is a + separate, cross-cutting feature; see the background-jobs design notes in + `gn-libs/WIP/background-jobs-design-notes.md`. diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md new file mode 100644 index 0000000..e4baf3c --- /dev/null +++ b/docs/WIP/geno-uploads-implementation-notes.md @@ -0,0 +1,151 @@ +# Genotype uploads — implementation notes & concerns + +- **Feature:** genotypes upload +- **Last updated:** 2026-10-22 +- **Related:** [design notes](geno-uploads-design-notes.md) · + [tasks & todos](geno-uploads-tasks-and-todos.md) + +This file records what is actually implemented, how it maps to the design, and +the concerns/deviations discovered while building it. It supersedes the +2026-10-15 implementation summary. + +## Status at a glance + +- **Stage 01 engine: implemented and tested.** The statistics collector exists + at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`. +- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is + resolved for both the resumable and direct upload paths and handed to the + collector stub; mapping the form metadata to the collector config, running + the collector and persisting its results are still pending. +- **Stages 02–07: not implemented** (still design/TODO comment blocks in + `uploader/genotypes/views.py`). + +## Stage 01 implementation + +### Location + +`uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`). +Runnable as a CLI; emits `FileStats` as JSON. + +### What it collects + +`FileStats` (`uploader/scripts/genotypes/types.py`) currently contains: + +- `total_lines`, `data_lines`, `comment_lines`, `blank_lines`, + `dot_geno_lines`; +- `dot_geno_fields` (parsed from `@` lines, when present); +- `geno_symbols` (distinct symbols in the data, sorted for determinism); +- `samples_list`, `markers_list`; +- `errors` (header validation errors + per-line errors). + +**Not collected:** the text of comment lines. Only a count is kept, so the +"description string" part of Stage 01 is still missing. + +### Configuration + +`compute_statistics(filepath, fileconfigs)` expects a config dict with: + +``` +markers, chromosome, cm, mb, separator, +comment.char, na_strings, transposed +``` + +### Processing design + +- Per-line processing is independent, so it was extracted into `_accumulate`, + which returns a `_ChunkStats` partial. +- `file_stats_singlethread` = one `_accumulate` pass over the whole file. +- `file_stats_multiprocessing` = stream the file into bounded chunks + (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a + worker process (`Pool.imap`), and merge in file order + (`_merge_chunk_stats` / `_merge_all_chunk_stats`). +- The worker is module-level and takes only picklable data (headers, config, + chunk), with the split/skip predicates rebuilt inside the worker, so it works + under `fork`, `spawn` and `forkserver`. +- `compute_statistics` chooses the single-threaded or multi-processed path + based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes. + +### Merge semantics (why the two paths agree) + +- counters: summed; +- `geno_symbols`: unioned (then sorted on output); +- `errors`: concatenated in file order; +- `markers_list`: concatenated in order (untransposed), or last markers line + wins (transposed); +- `dot_geno_fields`: combined, later values override earlier ones. + +### Defects fixed while implementing + +- Size threshold compared megabytes to bytes (branch was effectively + unreachable). Now bytes vs bytes. +- Header-skip compared a `str` to a `tuple`, so header rows were counted as + data. Now compares the split line to the headers. +- `FileStats.blank_lines` was declared but never populated. Now populated. +- `geno_symbols` output order was set-iteration-dependent (unstable across + runs/processes). Now sorted. + +### Tests + +`tests/genotypes/` (pytest): accumulation, single-threaded stats, merge +semantics, and single- vs multi-processed equivalence across chunk sizes, +including transposed files and files with errors. + +`tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and +resumable uploads, hashed/unhashed, error cases). Together with +`tests/genotypes/` this is **31 passing**. + +## Integration concerns (upload flow) + +### Uploaded-file resolution (done) + +`uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname, +hashed=True)` resolves either a chunked/resumable upload (the JSON blob +returned by the resumable upload endpoint) or a direct multipart upload, +returning `Right((path, original_name))` or `Left(message)`. It lives in +`uploader.files` because it is generic — the only per-caller input is the field +name — and it is the consumer counterpart of the resumable-upload contract that +package owns. + +- `add_genotype_records` uses it with the `genotypes-records-file` field. +- `process_phenotypes_individual_files` was refactored onto it, fixing: + directly-uploaded optional files being skipped, a `BadRequestKeyError` for + missing direct files, unhandled resumable-data errors, and a fragile + double-prefixed path. + +### The view does not run the collector + +In `uploader/genotypes/views.py`: + +- `_collect_geno_csv_stats_()` is a stub that just returns a dict + (`{"file": ..., "metadata": ...}`). +- `add_genotype_records()` (POST) builds that dict and renders a placeholder + template; it never invokes `preprocess_csv_collect_info`/`compute_statistics` + and never schedules a job. + +### Config-key mismatch to bridge + +The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the +collector expects `separator`, `comment.char`, `na_strings` plus `markers`, +`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char` +already matches). + +### Stage 01 form inputs exist; the view must read them + +The upload form +(`uploader/templates/genotypes/add-genotypes-records-csv.html`) already +provides `markers_field`, `chromosomes_field`, `centimorgans_field`, +`megabases_field`, `transposed`, plus the shared `file-separator`, +`file-comment-char` and `file-na` inputs. The work is to read and map them to +the collector's config (see above), not to add new inputs. + +### Comment description missing + +As noted above, comment text is discarded (only counted). The desired +multiline description output is not implemented. + +## Relationship to the background-jobs system + +The current job model is effectively "one script == one job". The genotype +upload is one logical operation spanning several stages, which stresses that +model. The options (and why a redesign is a separate feature) are recorded in +`gn-libs/WIP/background-jobs-design-notes.md`. diff --git a/docs/WIP/geno-uploads-tasks-and-todos.md b/docs/WIP/geno-uploads-tasks-and-todos.md new file mode 100644 index 0000000..cf52c44 --- /dev/null +++ b/docs/WIP/geno-uploads-tasks-and-todos.md @@ -0,0 +1,84 @@ +# Genotype uploads — tasks, todos & progress + +- **Feature:** genotypes upload +- **Last updated:** 2026-10-22 +- **Related:** [design notes](geno-uploads-design-notes.md) · + [implementation notes](geno-uploads-implementation-notes.md) · + background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`) + +Legend: ✅ done · 🚧 in progress · ⬜ todo · 💭 consideration (not committed) + +This is the single place to avoid tasks falling through the cracks. Design +rationale lives in the design notes; what actually got built lives in the +implementation notes. + +## Stage 01 — Collect info + +- ✅ Statistics collector implemented (`uploader/scripts/genotypes/preprocess_csv_collect_info.py`). +- ✅ Single- and multi-processed paths produce identical `FileStats`. +- ✅ Unit tests for accumulation, merge and equivalence (`tests/genotypes/`). +- ✅ Fix threshold / header-skip / `blank_lines` defects. +- ⬜ **Collect comment lines into a multiline description string** (currently + only counted, text discarded). Format decided: strip the leading `#`, trim + surrounding whitespace, keep one line per comment, present in an editable + textbox for the user; the result feeds the R/qtl2 control file (see design + notes). Still to do: implement it and settle the textbox styling. +- 💭 Decide whether the final description is also persisted to the database, or + only written into the R/qtl2 control file. +- 🚧 **Wire the collector into the upload view** (`add_genotype_records`): + - ✅ Resolve the uploaded file for both the resumable and direct upload paths, + via `uploader.files.resolve_uploaded_file`. + - ⬜ Read the form's metadata (`markers_field`, `chromosomes_field`, + `centimorgans_field`, `megabases_field`, `transposed`, `file-separator`, + `file-comment-char`, `file-na`) and map it to the collector's config keys + (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`, + `comment.char`, `na_strings`). The inputs already exist in the form. + - ⬜ Run the collector (or schedule it) instead of returning the stub + `_collect_geno_csv_stats_` dict. + - ⬜ Persist/return the resulting `FileStats` for the next stage. + +## Stage 02 — Study design & genotype encoding + +- ⬜ Present the genotype symbols discovered in Stage 01 for encoding. + - ⬜ Only symbols that actually occur in the file should be offered. +- ⬜ Seed labels from `@` lines when present (`mat`, `pat`, `het`, `unk`). + - ⬜ Missing `@` lines (all or some) must **not** fail the stage. + - ⬜ Treat `@`-derived labels as a starting point only; allow the user to + change them. Presence is not validation. +- ⬜ Allow the user to specify missing-value symbol(s). +- ⬜ Capture the confirmed encoding/study design for downstream stages. + +## Stage 03 — Final QC + +- ⬜ Perform value-level QC parameterised by Stage 01/02 results. +- ⬜ Restrict checks to the symbols present in the file. + +## Stages 04–07 — Conversion & ingest + +- ⬜ Stage 04: convert to R/qtl2 bundle. +- ⬜ Stage 05: ingest the R/qtl2 bundle. +- ⬜ Stage 06: generate `.geno` (and other artefacts). +- ⬜ Stage 07: pre-compute mapping(s). + +## Refactoring / technical debt + +- 💭 **Deferred:** once `add_genotype_records` grows to cover the later stages, + refactor it into a monadic pipeline rather than a series of guard clauses — + resolve the uploaded file, then chain each stage with `.then(...)`, ending + with `.either(_handle_error_path, _handle_success_path)`. Deferred until the + view is large enough for the split to pay off. + +## Background jobs + +- 💭 Decide whether genotype upload stays "one script == one job" or becomes + one job with multiple, gated tasks. **Not decided; treat as a separate + feature.** See `gn-libs/WIP/background-jobs-design-notes.md` and the + migration inventory in `gn-libs/WIP/background-jobs-tasks-and-todos.md`. +- 💭 (Lower priority, only if the multi-task redesign is chosen) a way to + re-run/"fix" failed tasks. + +## Housekeeping + +- ⬜ Track the new templates `genotype-encoding-elements.html` and + `study-design-elements.html` (currently untracked) once they are wired into + Stage 02. |
