about summary refs log tree commit diff
path: root/docs/WIP/geno-uploads-design-notes.md
blob: 3bdd2bf6d970be59f0fae8925dfac39cd0e32fac (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
# Genotype uploads — design notes

- **Feature:** genotypes upload
- **Last updated:** 2026-11-04
- **Related:** [implementation notes](geno-uploads-implementation-notes.md) ·
  [tasks & todos](geno-uploads-tasks-and-todos.md) ·
  background jobs (in `gn-libs/WIP/`: `background-jobs-design-notes.md`)

This file captures the *intended* design of the genotype upload feature: the
pipeline stages, the data that flows between them, the behaviours each stage
must honour, and the open design questions. It is expected to change as
implementation realities surface (recorded in the implementation notes).

## Goal

Allow a user to upload a genotype CSV file for a dataset, validate/characterise
it, let the user confirm the study design and genotype encoding, then convert
and ingest it (eventually to an R/qtl2 bundle and the `.geno`/database
representation), without losing information or silently mis-reading the file.

## Pipeline stages

The stage breakdown currently lives as a comment block in
`uploader/genotypes/views.py` (`add_genotype_records`). Restated here as the
design source of truth:

1. **Stage 01 — Collect info**
   - Detect custom `.geno` format lines (lines beginning with `@`).
   - Collect the list of samples/cases/individuals.
   - Collect the list of markers.
   - Identify which fields are the markers/chromosome (and cM/Mb) fields.
   - Collect the set of unique genotype symbols.
   - Collect comment lines into a human-readable description: strip the leading
     `#`, trim surrounding whitespace from each line, and preserve each line
     separately (see “Comment → description” below; currently *not*
     implemented).

2. **Stage 02 — Establish basic study information**
   - What is the study design?
   - What is the genotype encoding scheme? (Seed this from `@` lines when
     present.)
   - Which value(s) mean "missing"?

3. **Stage 03 — Final QC**
   - Value-level QC, parameterised by the results of Stages 01–02.

4. **Stage 04 — Convert to an R/qtl2 bundle.**

5. **Stage 05 — Ingest the R/qtl2 bundle.**

6. **Stage 06 — Generate the `.geno` file** (and other artefacts).

7. **Stage 07 — Pre-compute the mapping(s).**

## Cross-stage rules

### Stage 01 output drives Stage 02 and Stage 03

- The **set of genotype symbols** found in the file determines what the QC in
  Stage 03 checks: only symbols that actually occur in the file should be
  presented/validated. Do not offer or require symbols that are absent from the
  file.
- The **custom `.geno` lines**, when present, predefine labels for symbols, e.g.
  - `mat` — maternal
  - `pat` — paternal
  - `het` — heterozygous
  - `unk` — unknown
- **Missing `@` lines are not an error.** The absence of all or some of these
  custom lines must not fail Stage 02; the user simply supplies the encoding.
- **Presence of `@` lines is not endorsement of their validity.** They are a
  starting point: the user may change the labels. We must not treat a supplied
  encoding as "correct" merely because it came from the file.

### Stage gating and job chaining (decided)

Stages run in order as a chain of background jobs: Stage 01 (collect info),
then Stage 02 (study design/encoding), then Stage 03 (QC), and so on.

- If a stage fails, the chain stops there and the user is told, with as much
  diagnostic information as possible (the job's stdout/stderr) to help them
  resolve and recover where possible.
- If a stage succeeds, the chain proceeds to the next stage.
- Stage 02 is interactive: it presents the collected comment lines (with the
  comment character stripped) as the study/dataset description for the user to
  review and edit before continuing.

Realisation: each stage job carries a `success_handler` (set in its job
metadata) that proceeds to the next stage, following the phenotypes pattern
(`uploader.phenotypes.views.load_data_to_database` /
`load_phenotypes_success_handler`). Failures are reported by the jobs system's
error path (stdout/stderr on the job page).

How best to bind the stages together (one job with gated tasks vs a chain of
separate jobs) is still open; see the background-jobs design notes in
`gn-libs/WIP/background-jobs-design-notes.md`.

### Job status pages and stage transitions must be idempotent

Job status URLs are resumable: opening `/background-jobs/status/<job-id>` for a
completed job calls its `success_handler` again. Some stages are not idempotent
(for example Stage 02 writes to the database), so a success handler must not
re-run the next stage merely because the status URL was revisited.

- **Completed jobs are immutable.** Never update a job's metadata once it has
  completed (or errored). State the next stage needs is copied into the *next*
  job's metadata at launch.
- Identify an existing next-stage job with `jobs_by_metadata` using a
  workflow-namespaced key (e.g. `genotype_collector_job_id`) plus `job-type`;
  redirect to it instead of scheduling a duplicate.
- On a revisit, present the information already entered (read it from the
  next-stage job) instead of repeating the transition.
- A unique claim / serialised transition to make this race-free is deferred
  until non-idempotent stages need it.
- Consider preventing a return to (or resumption of) earlier jobs once a later
  stage has started; the exact mechanism is still to be decided.

### Stage error handling

Each node in the chain registers an `error_handler` (alongside its
`success_handler`) that informs the user and stops the chain.

- **`FileStats.errors`** (malformed data found during collection): display them
  like the phenotypes job status page
  (`uploader.phenotypes.views.job_status`,
  `uploader/templates/phenotypes/job-status.html`).
- **Any other failure** (script crash, IO/database/token issues): tell the user
  what failed and give reasonable recovery steps, including stdout/stderr where
  useful.

### Job cleanup

Add a daemon/service that periodically calls `delete_expired_jobs` on the jobs
database (script in `gn-libs`; scheduling in `gn-machines`). It only cares about
expired jobs, not users. Note that deleting a job also removes its metadata and
outputs, so state that must outlive the jobs has to be copied forward.

### Stage 02 state storage (interim)

The confirmed description, genotype encoding and missing-value symbol(s) are
stored in the collector job's metadata (key `stage02`) for now:

- the database has nowhere suitable to store descriptions;
- `GenoCode` only stores integer values, while encodings may be floating point,
  so it is not used at this stage.

Revisiting Stage 02 reads these values back and presents them for editing.

### Comment → description (decided)

Comment lines become a user-editable description that will later go into the
R/qtl2 bundle's control file (and possibly the database). For each comment
line:

- strip the leading `#` character;
- strip leading/trailing whitespace;
- keep the lines separate (one line each) and present them to the user in a
  textbox as-is;
- let the user edit the final description before it is used.

Only comments **before the data matrix starts** feed the description:

- untransposed: before the header row;
- transposed: before the first data row (the markers row).

Later comments are counted (`comment_lines`) but not used; they relate to
adjacent rows, not the whole file, and untransposing would take them out of
context.

Stage 02 presents this as the study/dataset description for review and editing.
Open UI question: how to visually demarcate consecutive lines in the textbox.

## Stage 01 data contract (target)

The stats collector should produce enough information for the later stages:

- counts: total/data/comment/blank/dot-geno lines;
- `dot_geno_fields` parsed from `@` lines (`name`, `mat`, `pat`, `het`,
  `type`, `unk` where present);
- `geno_symbols` (the distinct symbols in the data);
- `samples_list`;
- `markers_list`;
- parse errors encountered (column mismatches; empty or missing-value
  headings, marker values, Chr/cM/Mb values, or sample values; negative or
  non-numeric cM/Mb values);
- **a free-text description built from comment lines** (desired; not yet
  implemented). Each comment line is cleaned (leading `#` and surrounding
  whitespace removed) and kept as its own line.

The current implementation of Stage 01 is described in the implementation
notes.

## QC / file-generation concerns (running list)

This is a running list; more QC concerns will be added as they surface.

- **Unknown (`unk`) symbols are missing values.** Seed `dot_geno_fields.unk`
  into the Stage 02 missing-value symbols.
- **Heading values.** Values that label cells must never be empty or be
  missing-value indicators: the header row in untransposed files, and the first
  column plus the marker headings in transposed files. Marker and Chromosome
  values must likewise not be empty or missing-value indicators, and cM/Mb
  values must be finite, non-negative numbers (non-negative integers are
  accepted).
- **Allowed genotype values.** Every sample value must be one of the confirmed
  maternal/paternal/heterozygous symbols or a missing-value symbol. Empty
  sample values are not allowed—missing values must use the configured
  `na_strings`. Stage 01 catches these obvious data errors early; Stage 03
  checks the values against the confirmed encoding. Matching is
  case-sensitive. Symbols may be longer than three characters and need not be
  single characters, so the Stage 02 UI's three-character limit should be
  revisited (low priority).
- **Marker uniqueness (hard error).** Within a single chromosome, marker names
  must be unique. The same marker name may appear on different chromosomes.
  Duplicate marker names within one chromosome are errors and stop the chain.
- **Non-fatal warnings (future).** Consider reporting (without blocking):
  high missingness/call rate (e.g. more than 10% missing calls), monomorphic
  markers (every sample has the same genotype), and male heterozygosity on
  non-pseudoautosomal X/Y regions when sample sex is available. Warnings are
  shown to the user but do not stop the chain; only errors do.
- **Sample/individual/case order matters.** The order of samples in the
  generated `.geno` file must be preserved consistently through the pipeline.
  See the ordering rules below. Ordering is a late Stage 03 step.
- **Markers/records ordering (placeholder).** For now, sort records by
  chromosome (ascending), then Mb (ascending), then cM (ascending). The sorting
  helper emits a warning because the appropriate ordering has not been
  established; replace this placeholder and remove the warning once known.
  TODO.

### Sample/individual/case ordering rules

1. If the population already has an order, use it, then append any missing
   samples (sorted alphabetically).
2. If there is no existing order, sort the samples alphabetically.
3. Present the resulting order to the user for review and reordering (one
   sample per line in an editable textbox).
4. Save the accepted order in the **next stage job's metadata** at launch
   (completed jobs are immutable).

## Untransposing transposed files (decided)

Transposed files are untransposed as the **first step of Stage 03**, before any
QC. Stage 02 only consumes metadata from `FileStats`, so there is no reason to
normalise a file that might be abandoned at review; Stage 03 is the first stage
that needs the data.

- Keep the original upload immutable and write a canonical
  `<stem>__work<suffix>` next to it. The path is deterministic, so it need not
  be recorded in job metadata.
- The canonical file is a **pure data matrix**: comments and `@` directives are
  omitted because Stage 01 already captured them (`description`,
  `dot_geno_fields`).
- Skip `#`/`@` lines when transposing and validate ragged rows. The existing
  `r_qtl.r_qtl2.transpose_csv_with_rename` is unsuitable (renames in place,
  skips `#`, treats `@` as data, truncates ragged rows).
- Do not untranspose if Stage 01 reported errors; the collector now exits with
  `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so the chain
  stops before Stage 02.
- After normalisation, downstream (QC and later) handles only the normal
  orientation (`transposed=False`), simplifying QC and ordering code.

## Open design questions

- **Description storage.** The description feeds the R/qtl2 control file; it is
  undecided whether it is also persisted to the database.
- **Encoding representation.** How do we represent a user-confirmed encoding so
  that it is stable for QC and conversion (symbol → label/missing marker)?
- **Validation vs. confirmation.** Where exactly is the boundary between "what
  the file says" and "what the user confirms"?
- **Size/scale.** Stage 01 must cope with large files; the collector already
  has a single-threaded and a multi-processed path.

## Out of scope here

- Reworking the background-jobs system to support multi-task jobs. That is a
  separate, cross-cutting feature; see the background-jobs design notes in
  `gn-libs/WIP/background-jobs-design-notes.md`.