about summary refs log tree commit diff
path: root/docs/WIP/geno-uploads-implementation-notes.md
blob: ab6761f3f89d635227dee8ec80fa50c7d1bbd831 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
# Genotype uploads — implementation notes & concerns

- **Feature:** genotypes upload
- **Last updated:** 2026-11-04
- **Related:** [design notes](geno-uploads-design-notes.md) ·
  [tasks & todos](geno-uploads-tasks-and-todos.md)

This file records what is actually implemented, how it maps to the design, and
the concerns/deviations discovered while building it. It supersedes the
2026-10-15 implementation summary.

## Status at a glance

- **Stage 01 engine: implemented and tested.** The statistics collector exists
  at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`.
- **Stage 01 wiring into the upload flow: done for the Stage 02 hand-off.** The
  uploaded file is resolved, the form metadata is mapped to the collector
  config, the collector runs as a `gn_libs.jobs` background job
  (`genotype-file-stats`), and Stage 02 reads its `FileStats` from the job.
- **Stage 02: review page implemented.** Description, genotype symbols, study
  design and encoding are presented; the confirmed values are stored in the
  collector job metadata.
- **Stage 03: value QC implemented.** The genotype file is re-read and every
  sample value is checked against the Stage 02 encoding/missing symbols; the
  QC job is scheduled from Stage 02. Ordering checks remain pending.
- **Stages 04–07: not implemented** (still design/TODO comment blocks in
  `uploader/genotypes/views.py`).

## Stage 01 implementation

### Location

`uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`).
Runnable as a CLI; emits `FileStats` as JSON.

### What it collects

`FileStats` (`uploader/scripts/genotypes/types.py`) currently contains:

- `total_lines`, `data_lines`, `comment_lines`, `blank_lines`,
  `dot_geno_lines`;
- `dot_geno_fields` (parsed from `@` lines, when present);
- `geno_symbols` (distinct symbols in the data, sorted for determinism);
- `samples_list`, `markers_list`;
- `errors` (header validation errors + per-line errors);
- `description` (cleaned comment lines, joined with newlines).

Comment lines are cleaned by stripping the configured comment character and
surrounding whitespace, then joined with `\n`; comment order is preserved when
merging chunks.

### Early data validation (done)

Stage 01 flags empty or missing-value headings (the untransposed header row,
the transposed first column and transposed marker headings), empty or
missing-value marker/Chr/cM/Mb values, empty sample values, and negative or
non-numeric cM/Mb values. cM/Mb values must be finite, non-negative numbers
(integers are accepted). Empty strings are excluded from `geno_symbols`.
Genuine missing genotype values must use the configured `na_strings`.

### Configuration

`compute_statistics(filepath, fileconfigs)` expects a config dict with:

```
markers, chromosome, cm, mb, separator,
comment.char, na_strings, transposed
```

### Processing design

- Per-line processing is independent, so it was extracted into `_accumulate`,
  which returns a `_ChunkStats` partial.
- `file_stats_singlethread` = one `_accumulate` pass over the whole file.
- `file_stats_multiprocessing` = stream the file into bounded chunks
  (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a
  worker process (`Pool.imap`), and merge in file order
  (`_merge_chunk_stats` / `_merge_all_chunk_stats`).
- The worker is module-level and takes only picklable data (headers, config,
  chunk), with the split/skip predicates rebuilt inside the worker, so it works
  under `fork`, `spawn` and `forkserver`.
- `compute_statistics` chooses the single-threaded or multi-processed path
  based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes.

### Merge semantics (why the two paths agree)

- counters: summed;
- `geno_symbols`: unioned (then sorted on output);
- `errors`: concatenated in file order;
- `markers_list`: concatenated in order (untransposed), or last markers line
  wins (transposed);
- `dot_geno_fields`: combined, later values override earlier ones.

### Defects fixed while implementing

- Size threshold compared megabytes to bytes (branch was effectively
  unreachable). Now bytes vs bytes.
- Header-skip compared a `str` to a `tuple`, so header rows were counted as
  data. Now compares the split line to the headers.
- `FileStats.blank_lines` was declared but never populated. Now populated.
- `geno_symbols` output order was set-iteration-dependent (unstable across
  runs/processes). Now sorted.

### Tests

`tests/genotypes/` (pytest): accumulation, single-threaded stats, merge
semantics, and single- vs multi-processed equivalence across chunk sizes,
including transposed files and files with errors.

`tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and
resumable uploads, hashed/unhashed, error cases). Together with
`tests/genotypes/` this is **31 passing**.

## Integration concerns (upload flow)

### Uploaded-file resolution (done)

`uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname,
hashed=True)` resolves either a chunked/resumable upload (the JSON blob
returned by the resumable upload endpoint) or a direct multipart upload,
returning `Right((path, original_name))` or `Left(message)`. It lives in
`uploader.files` because it is generic — the only per-caller input is the field
name — and it is the consumer counterpart of the resumable-upload contract that
package owns.

- `add_genotype_records` uses it with the `genotypes-records-file` field.
- `process_phenotypes_individual_files` was refactored onto it, fixing:
  directly-uploaded optional files being skipped, a `BadRequestKeyError` for
  missing direct files, unhandled resumable-data errors, and a fragile
  double-prefixed path.

### The view schedules the collector as a background job

`add_genotype_records()` (POST) builds the collector config from the form and
schedules `uploader.scripts.genotypes.preprocess_csv_collect_info` via
`gn_libs.jobs` (job type `genotype-file-stats`). The collector's stdout (the
`FileStats` JSON) and stderr are available on the standard job status/summary
pages, and Stage 02 reads the stats back from the job.

### Collector config mapping (done)

`uploader.genotypes.collector_config.from_form(form)` maps the upload form's
metadata fields (`markers_field`, `chromosomes_field`, `centimorgans_field`,
`megabases_field`, `transposed`, `file-separator`, `file-comment-char`,
`file-na`) to the collector's config keys (`markers`, `chromosome`, `cm`,
`mb`, `transposed`, `separator`, `comment.char`, `na_strings`), falling back
to the collector's defaults and normalising the missing-value strings.
`add_genotype_records` now passes this config to the collector stub.

### Stage 01 form inputs

The upload form
(`uploader/templates/genotypes/add-genotypes-records-csv.html`) already
provides `markers_field`, `chromosomes_field`, `centimorgans_field`,
`megabases_field`, `transposed`, plus the shared `file-separator`,
`file-comment-char` and `file-na` inputs. No new inputs are needed; they are
mapped to the collector config as described above.

### Comment description (done)

Comment text is collected into `FileStats.description`: one cleaned comment
line per line, joined with newlines. Only comments **before the data matrix
starts** (before the header row; before the first data row when transposed)
feed the description; later comments are counted but not used. This is what
Stage 02 presents as the editable study/dataset description.

### Data types moved

`FileStats` and the other genotype data types moved from
`uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py`, so
application code does not import from `uploader.scripts`. The collector script
and tests import them from the new location.

### Stage 02 review page

`uploader.genotypes.collector_job.file_stats(job, species_id, population_id,
dataset_id)` validates a completed `genotype-file-stats` job for the dataset
and parses the `FileStats` from its recorded stdout. The collector job's
`success_handler` redirects to the `study_design` view with the job id; that
view renders the collected description, the genotype symbols, the
study-design/encoding inputs and the missing-value symbol(s). Saving puts the
confirmed values (including `missing_symbols`) in the **QC job's** metadata at
launch; the completed collector job is left untouched. Revisits read them back
from the existing QC job (found via `jobs_by_metadata`).

### Stage 03 normalisation

`uploader.genotypes.normalise.normalise_file(original, config)` writes a
canonical untransposed matrix (`<stem>__work<suffix>`) next to the original,
dropping comments and `@` directives (already captured in `FileStats`), leaving
the original untouched, and rejecting ragged or empty matrices.
`qc.check_file` now handles only the normal orientation.

### Sample ordering (done)

`uploader.genotypes.ordering` provides `proposed_order` (retain the population's
existing order, append missing samples alphabetically, otherwise sort
alphabetically), `population_sample_order` (reads `StrainXRef.OrderId`) and
`is_valid_order` (validates a submitted order as a permutation of the file's
samples). The QC success handler redirects to a sample-order review page; on
acceptance it launches the next-stage job (type `genotype-stage04`) carrying
`sample_order` and the preceding stage state in its metadata. A minimal Stage 04
placeholder worker is the current hand-off point.

### Marker uniqueness (done)

Stage 03 QC tracks `(chromosome, marker)` pairs and reports an error when a
marker name occurs more than once within the same chromosome. The same marker
name may appear on different chromosomes.

### Stage 03 value QC

Stage 02 launches the QC job with `stage02`, `genotype_collector_job_id`,
`file` and `config` in the QC job's metadata. The worker loads its own job,
normalises the file, then runs `uploader.genotypes.qc.check_file` on the work
file with `transposed=False`, exiting non-zero on errors. The completed
collector job is never modified; an existing QC job is reused via
`jobs_by_metadata`.

The collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is
non-empty, so malformed files stop the chain before Stage 02.

`uploader.genotypes.qc.sort_markers` is a placeholder ordering helper
(chromosome, Mb, cM) that emits a warning until the appropriate ordering is
established.

### Pending: error handlers

Each stage will register an `error_handler` (FileStats errors shown like the
phenotypes job status page; other failures reported with recovery guidance).

## Relationship to the background-jobs system

The current job model is effectively "one script == one job". The genotype
upload is one logical operation spanning several stages, which stresses that
model. The options (and why a redesign is a separate feature) are recorded in
`gn-libs/WIP/background-jobs-design-notes.md`.