about summary refs log tree commit diff
path: root/docs/WIP/geno-uploads-tasks-and-todos.md
blob: 353e26787446aea497201dae880340175e0b9355 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
# Genotype uploads — tasks, todos & progress

- **Feature:** genotypes upload
- **Last updated:** 2026-11-04
- **Related:** [design notes](geno-uploads-design-notes.md) ·
  [implementation notes](geno-uploads-implementation-notes.md) ·
  background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)

Legend: ✅ done · 🚧 in progress · ⬜ todo · 💭 consideration (not committed)

This is the single place to avoid tasks falling through the cracks. Design
rationale lives in the design notes; what actually got built lives in the
implementation notes.

## Stage 01 — Collect info

- ✅ Statistics collector implemented (`uploader/scripts/genotypes/preprocess_csv_collect_info.py`).
- ✅ Single- and multi-processed paths produce identical `FileStats`.
- ✅ Unit tests for accumulation, merge and equivalence (`tests/genotypes/`).
- ✅ Fix threshold / header-skip / `blank_lines` defects.
- ✅ **Collect comment lines into a multiline description string**: strip the
  configured comment character, trim surrounding whitespace, keep one line per
  comment and join with newlines into `FileStats.description`. Still to do:
  settle the Stage 02 textbox styling.
- ✅ Only feed comments **before the data matrix starts** into the description
  (before the header row; before the first data row when transposed). Later
  comments are counted but not used. A `matrix_started` flag per chunk, plus
  merge handling, keeps multiprocessing consistent with single-threaded.
- ✅ Flag empty/missing-value headings (header row, transposed first column and
  marker headings), empty/missing-value marker and Chr/cM/Mb values, empty
  sample values, and negative/non-numeric cM/Mb values early (both
  orientations); exclude `""` from `geno_symbols`.
- 💭 Decide whether the final description is also persisted to the database, or
  only written into the R/qtl2 control file.
- 🚧 **Wire the collector into the upload view** (`add_genotype_records`):
  - ✅ Resolve the uploaded file for both the resumable and direct upload paths,
    via `uploader.files.resolve_uploaded_file`.
  - ✅ Read the form's metadata (`markers_field`, `chromosomes_field`,
    `centimorgans_field`, `megabases_field`, `transposed`, `file-separator`,
    `file-comment-char`, `file-na`) and map it to the collector's config keys
    (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`,
    `comment.char`, `na_strings`) via `uploader.genotypes.collector_config`.
  - ✅ Run the collector as a `gn_libs.jobs` background job (job type
    `genotype-file-stats`); the job status/summary pages show its output and
    errors.
  - ✅ Persist/return the resulting `FileStats` for the next stage (Stage 02
    reads it from the collector job).

## Stage 02 — Study design & genotype encoding

- ✅ Present the collected comment lines (comment character stripped) as the
  study/dataset description for review and editing.
- ✅ Present the genotype symbols discovered in Stage 01 for encoding.
  - ✅ Only symbols that actually occur in the file are offered.
- ✅ Seed labels from `@` lines when present (`mat`, `pat`, `het`, `unk`).
  - ✅ Missing `@` lines (all or some) do **not** fail the stage (empty
    defaults).
  - ✅ `@`-derived labels are a starting point only; the user can change them.
    Presence is not validation.
- ✅ Allow the user to specify missing-value symbol(s).
- ✅ Seed `dot_geno_fields.unk` into the missing-value symbols.
- ⬜ Revisit the Stage 02 UI's three-character symbol limit: symbols may be
  longer and need not be single characters (low priority).
- ✅ Capture the confirmed encoding/study design for downstream stages (carried
  in the QC job metadata at launch, key `stage02`; the collector job is left
  untouched).

## Stage 03 — Final QC

- ✅ Untranspose transposed files as the first action of Stage 03: write a
  canonical `<stem>__work<suffix>`, keep the original immutable, skip `#`/`@`,
  validate ragged rows (see "Untransposing transposed files").
- ✅ Perform value-level QC parameterised by Stage 01/02 results: every sample
  value must be an encoding or missing-value symbol; unknown symbols are errors
  and matching is case-sensitive. (Empty values are rejected earlier, in
  Stage 01.)
- ✅ Restrict checks to the symbols present in the file (checks use the
  confirmed encoding/missing symbols; absent symbols are not required).
- ✅ **Marker uniqueness (hard error):** no two markers on the same chromosome
  may share a name; names may repeat across chromosomes. Implemented in
  `uploader.genotypes.qc`.
- ✅ Apply the sample/individual/case ordering rules and present the order for
  user review and reordering (one of the last Stage 03 steps; see design notes).
  The accepted order is carried into the next stage job's metadata; a
  placeholder Stage 04 job is the current hand-off point.
- ⬜ Sort/verify records by chromosome (ascending), then Mb (ascending), then cM
  (ascending) (one of the last Stage 03 steps) — placeholder ordering with a
  warning/TODO until the appropriate ordering is known.
- 💭 **Warnings (future):** report high missingness/call rate, monomorphic
  markers, and male heterozygosity on non-pseudoautosomal X/Y; warnings do not
  stop the chain.

## Stages 04–07 — Conversion & ingest

- ⬜ Stage 04: convert to R/qtl2 bundle.
- ⬜ Stage 05: ingest the R/qtl2 bundle.
- ⬜ Stage 06: generate `.geno` (and other artefacts), preserving sample order
  and ordering records by chromosome then position, both ascending (see
  Stage 03).
- ⬜ Stage 07: pre-compute mapping(s).

## Untransposing transposed files

- ✅ Decide placement: first step of Stage 03, before QC.
- ✅ Implement normalisation at the start of the Stage 03 job: write a canonical
  `<stem>__work<suffix>`, keep the original immutable, skip `#`/`@`, validate
  ragged rows. The path is deterministic, so it is not recorded.
- ✅ Run Stage 03 QC against the work file with `transposed=False`; QC handles
  only the normal orientation.

## Refactoring / technical debt

- 💭 **Deferred:** once `add_genotype_records` grows to cover the later stages,
  refactor it into a monadic pipeline rather than a series of guard clauses —
  resolve the uploaded file, then chain each stage with `.then(...)`, ending
  with `.either(_handle_error_path, _handle_success_path)`. Deferred until the
  view is large enough for the split to pay off.

## Background jobs

- ⬜ Implement the stage-job chain using each job's `success_handler` (as in
  the phenotypes `load_data_to_database`/`load_phenotypes_success_handler`
  pattern): proceed on success, and stop/report diagnostics on failure (see
  design notes).
- ✅ Completed jobs are immutable: Stage 02 state is carried in the QC job
  metadata at launch, not written back to the collector job.
- ✅ Reuse an existing next-stage job via `jobs_by_metadata` with namespaced
  keys (`genotype_collector_job_id` + `job-type`).
- ⬜ Make stage transitions idempotent: revisiting a completed job's status URL
  must not re-run a non-idempotent stage; present the previously entered
  information instead (see design notes).
- ⬜ Unique claim / serialised transition for non-idempotent stages (deferred).
- ⬜ Add a cleanup daemon/service that periodically calls
  `delete_expired_jobs` (script in gn-libs; scheduling in gn-machines). It only
  cares about expired jobs, not users.
- 💭 Consider preventing a return to / resumption of earlier jobs once a later
  stage has started (see design notes).
- 💭 Decide whether genotype upload stays "one script == one job" or becomes
  one job with multiple, gated tasks. **Not decided; treat as a separate
  feature.** See `gn-libs/WIP/background-jobs-design-notes.md` and the
  migration inventory in `gn-libs/WIP/background-jobs-tasks-and-todos.md`.
- 💭 (Lower priority, only if the multi-task redesign is chosen) a way to
  re-run/"fix" failed tasks.

## Error handling

- ✅ Collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is
  non-empty, so malformed files stop the chain before Stage 02.
- ✅ Register `error_handler`s for the collector and QC jobs (other stages
  later).
- ✅ Display structured collector/QC errors on the genotype upload error page;
  other failures fall back to stdout/stderr with recovery guidance.

## Testing / fixtures

- ✅ Add small sample genotype files
  (`tests/uploader/genotypes/test_data/`): normal/transposed, clean, collector
  errors (column mismatches, empty values, invalid headings/marker/metadata/
  position values), unknown value errors, ragged rows, and comments/directives.
- ✅ Tests for `job_errors.errors_from_job` using the fixtures.

## Housekeeping

- ✅ Track the new templates `genotype-encoding-elements.html` and
  `study-design-elements.html`; they are wired into Stage 02.