diff options
| author | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-04 10:42:14 -0600 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-04 10:42:14 -0600 |
| commit | 863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c (patch) | |
| tree | f2f46e6a8c0e79af70ffdbcad61695385401f2d3 /docs/WIP/geno-uploads-implementation-notes.md | |
| parent | 2bc7e2c194dbb3f22c6e44202abc9ea2846d3504 (diff) | |
| download | gn-uploader-863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c.tar.gz | |
Move documentation under the `docs/` directory.
Diffstat (limited to 'docs/WIP/geno-uploads-implementation-notes.md')
| -rw-r--r-- | docs/WIP/geno-uploads-implementation-notes.md | 151 |
1 files changed, 151 insertions, 0 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md new file mode 100644 index 0000000..e4baf3c --- /dev/null +++ b/docs/WIP/geno-uploads-implementation-notes.md @@ -0,0 +1,151 @@ +# Genotype uploads — implementation notes & concerns + +- **Feature:** genotypes upload +- **Last updated:** 2026-10-22 +- **Related:** [design notes](geno-uploads-design-notes.md) · + [tasks & todos](geno-uploads-tasks-and-todos.md) + +This file records what is actually implemented, how it maps to the design, and +the concerns/deviations discovered while building it. It supersedes the +2026-10-15 implementation summary. + +## Status at a glance + +- **Stage 01 engine: implemented and tested.** The statistics collector exists + at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`. +- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is + resolved for both the resumable and direct upload paths and handed to the + collector stub; mapping the form metadata to the collector config, running + the collector and persisting its results are still pending. +- **Stages 02–07: not implemented** (still design/TODO comment blocks in + `uploader/genotypes/views.py`). + +## Stage 01 implementation + +### Location + +`uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`). +Runnable as a CLI; emits `FileStats` as JSON. + +### What it collects + +`FileStats` (`uploader/scripts/genotypes/types.py`) currently contains: + +- `total_lines`, `data_lines`, `comment_lines`, `blank_lines`, + `dot_geno_lines`; +- `dot_geno_fields` (parsed from `@` lines, when present); +- `geno_symbols` (distinct symbols in the data, sorted for determinism); +- `samples_list`, `markers_list`; +- `errors` (header validation errors + per-line errors). + +**Not collected:** the text of comment lines. Only a count is kept, so the +"description string" part of Stage 01 is still missing. + +### Configuration + +`compute_statistics(filepath, fileconfigs)` expects a config dict with: + +``` +markers, chromosome, cm, mb, separator, +comment.char, na_strings, transposed +``` + +### Processing design + +- Per-line processing is independent, so it was extracted into `_accumulate`, + which returns a `_ChunkStats` partial. +- `file_stats_singlethread` = one `_accumulate` pass over the whole file. +- `file_stats_multiprocessing` = stream the file into bounded chunks + (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a + worker process (`Pool.imap`), and merge in file order + (`_merge_chunk_stats` / `_merge_all_chunk_stats`). +- The worker is module-level and takes only picklable data (headers, config, + chunk), with the split/skip predicates rebuilt inside the worker, so it works + under `fork`, `spawn` and `forkserver`. +- `compute_statistics` chooses the single-threaded or multi-processed path + based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes. + +### Merge semantics (why the two paths agree) + +- counters: summed; +- `geno_symbols`: unioned (then sorted on output); +- `errors`: concatenated in file order; +- `markers_list`: concatenated in order (untransposed), or last markers line + wins (transposed); +- `dot_geno_fields`: combined, later values override earlier ones. + +### Defects fixed while implementing + +- Size threshold compared megabytes to bytes (branch was effectively + unreachable). Now bytes vs bytes. +- Header-skip compared a `str` to a `tuple`, so header rows were counted as + data. Now compares the split line to the headers. +- `FileStats.blank_lines` was declared but never populated. Now populated. +- `geno_symbols` output order was set-iteration-dependent (unstable across + runs/processes). Now sorted. + +### Tests + +`tests/genotypes/` (pytest): accumulation, single-threaded stats, merge +semantics, and single- vs multi-processed equivalence across chunk sizes, +including transposed files and files with errors. + +`tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and +resumable uploads, hashed/unhashed, error cases). Together with +`tests/genotypes/` this is **31 passing**. + +## Integration concerns (upload flow) + +### Uploaded-file resolution (done) + +`uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname, +hashed=True)` resolves either a chunked/resumable upload (the JSON blob +returned by the resumable upload endpoint) or a direct multipart upload, +returning `Right((path, original_name))` or `Left(message)`. It lives in +`uploader.files` because it is generic — the only per-caller input is the field +name — and it is the consumer counterpart of the resumable-upload contract that +package owns. + +- `add_genotype_records` uses it with the `genotypes-records-file` field. +- `process_phenotypes_individual_files` was refactored onto it, fixing: + directly-uploaded optional files being skipped, a `BadRequestKeyError` for + missing direct files, unhandled resumable-data errors, and a fragile + double-prefixed path. + +### The view does not run the collector + +In `uploader/genotypes/views.py`: + +- `_collect_geno_csv_stats_()` is a stub that just returns a dict + (`{"file": ..., "metadata": ...}`). +- `add_genotype_records()` (POST) builds that dict and renders a placeholder + template; it never invokes `preprocess_csv_collect_info`/`compute_statistics` + and never schedules a job. + +### Config-key mismatch to bridge + +The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the +collector expects `separator`, `comment.char`, `na_strings` plus `markers`, +`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char` +already matches). + +### Stage 01 form inputs exist; the view must read them + +The upload form +(`uploader/templates/genotypes/add-genotypes-records-csv.html`) already +provides `markers_field`, `chromosomes_field`, `centimorgans_field`, +`megabases_field`, `transposed`, plus the shared `file-separator`, +`file-comment-char` and `file-na` inputs. The work is to read and map them to +the collector's config (see above), not to add new inputs. + +### Comment description missing + +As noted above, comment text is discarded (only counted). The desired +multiline description output is not implemented. + +## Relationship to the background-jobs system + +The current job model is effectively "one script == one job". The genotype +upload is one logical operation spanning several stages, which stresses that +model. The options (and why a redesign is a separate feature) are recorded in +`gn-libs/WIP/background-jobs-design-notes.md`. |
