about summary refs log tree commit diff
path: root/docs/WIP/geno-uploads-implementation-notes.md
diff options
context:
space:
mode:
authorFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 10:42:14 -0600
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 10:42:14 -0600
commit863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c (patch)
treef2f46e6a8c0e79af70ffdbcad61695385401f2d3 /docs/WIP/geno-uploads-implementation-notes.md
parent2bc7e2c194dbb3f22c6e44202abc9ea2846d3504 (diff)
downloadgn-uploader-863946f5ee15ccf0a8587dac757cb2dbcbe3dc5c.tar.gz
Move documentation under the `docs/` directory.
Diffstat (limited to 'docs/WIP/geno-uploads-implementation-notes.md')
-rw-r--r--docs/WIP/geno-uploads-implementation-notes.md151
1 files changed, 151 insertions, 0 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md
new file mode 100644
index 0000000..e4baf3c
--- /dev/null
+++ b/docs/WIP/geno-uploads-implementation-notes.md
@@ -0,0 +1,151 @@
+# Genotype uploads — implementation notes & concerns
+
+- **Feature:** genotypes upload
+- **Last updated:** 2026-10-22
+- **Related:** [design notes](geno-uploads-design-notes.md) ·
+  [tasks & todos](geno-uploads-tasks-and-todos.md)
+
+This file records what is actually implemented, how it maps to the design, and
+the concerns/deviations discovered while building it. It supersedes the
+2026-10-15 implementation summary.
+
+## Status at a glance
+
+- **Stage 01 engine: implemented and tested.** The statistics collector exists
+  at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`.
+- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is
+  resolved for both the resumable and direct upload paths and handed to the
+  collector stub; mapping the form metadata to the collector config, running
+  the collector and persisting its results are still pending.
+- **Stages 02–07: not implemented** (still design/TODO comment blocks in
+  `uploader/genotypes/views.py`).
+
+## Stage 01 implementation
+
+### Location
+
+`uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`).
+Runnable as a CLI; emits `FileStats` as JSON.
+
+### What it collects
+
+`FileStats` (`uploader/scripts/genotypes/types.py`) currently contains:
+
+- `total_lines`, `data_lines`, `comment_lines`, `blank_lines`,
+  `dot_geno_lines`;
+- `dot_geno_fields` (parsed from `@` lines, when present);
+- `geno_symbols` (distinct symbols in the data, sorted for determinism);
+- `samples_list`, `markers_list`;
+- `errors` (header validation errors + per-line errors).
+
+**Not collected:** the text of comment lines. Only a count is kept, so the
+"description string" part of Stage 01 is still missing.
+
+### Configuration
+
+`compute_statistics(filepath, fileconfigs)` expects a config dict with:
+
+```
+markers, chromosome, cm, mb, separator,
+comment.char, na_strings, transposed
+```
+
+### Processing design
+
+- Per-line processing is independent, so it was extracted into `_accumulate`,
+  which returns a `_ChunkStats` partial.
+- `file_stats_singlethread` = one `_accumulate` pass over the whole file.
+- `file_stats_multiprocessing` = stream the file into bounded chunks
+  (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a
+  worker process (`Pool.imap`), and merge in file order
+  (`_merge_chunk_stats` / `_merge_all_chunk_stats`).
+- The worker is module-level and takes only picklable data (headers, config,
+  chunk), with the split/skip predicates rebuilt inside the worker, so it works
+  under `fork`, `spawn` and `forkserver`.
+- `compute_statistics` chooses the single-threaded or multi-processed path
+  based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes.
+
+### Merge semantics (why the two paths agree)
+
+- counters: summed;
+- `geno_symbols`: unioned (then sorted on output);
+- `errors`: concatenated in file order;
+- `markers_list`: concatenated in order (untransposed), or last markers line
+  wins (transposed);
+- `dot_geno_fields`: combined, later values override earlier ones.
+
+### Defects fixed while implementing
+
+- Size threshold compared megabytes to bytes (branch was effectively
+  unreachable). Now bytes vs bytes.
+- Header-skip compared a `str` to a `tuple`, so header rows were counted as
+  data. Now compares the split line to the headers.
+- `FileStats.blank_lines` was declared but never populated. Now populated.
+- `geno_symbols` output order was set-iteration-dependent (unstable across
+  runs/processes). Now sorted.
+
+### Tests
+
+`tests/genotypes/` (pytest): accumulation, single-threaded stats, merge
+semantics, and single- vs multi-processed equivalence across chunk sizes,
+including transposed files and files with errors.
+
+`tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and
+resumable uploads, hashed/unhashed, error cases). Together with
+`tests/genotypes/` this is **31 passing**.
+
+## Integration concerns (upload flow)
+
+### Uploaded-file resolution (done)
+
+`uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname,
+hashed=True)` resolves either a chunked/resumable upload (the JSON blob
+returned by the resumable upload endpoint) or a direct multipart upload,
+returning `Right((path, original_name))` or `Left(message)`. It lives in
+`uploader.files` because it is generic — the only per-caller input is the field
+name — and it is the consumer counterpart of the resumable-upload contract that
+package owns.
+
+- `add_genotype_records` uses it with the `genotypes-records-file` field.
+- `process_phenotypes_individual_files` was refactored onto it, fixing:
+  directly-uploaded optional files being skipped, a `BadRequestKeyError` for
+  missing direct files, unhandled resumable-data errors, and a fragile
+  double-prefixed path.
+
+### The view does not run the collector
+
+In `uploader/genotypes/views.py`:
+
+- `_collect_geno_csv_stats_()` is a stub that just returns a dict
+  (`{"file": ..., "metadata": ...}`).
+- `add_genotype_records()` (POST) builds that dict and renders a placeholder
+  template; it never invokes `preprocess_csv_collect_info`/`compute_statistics`
+  and never schedules a job.
+
+### Config-key mismatch to bridge
+
+The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the
+collector expects `separator`, `comment.char`, `na_strings` plus `markers`,
+`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char`
+already matches).
+
+### Stage 01 form inputs exist; the view must read them
+
+The upload form
+(`uploader/templates/genotypes/add-genotypes-records-csv.html`) already
+provides `markers_field`, `chromosomes_field`, `centimorgans_field`,
+`megabases_field`, `transposed`, plus the shared `file-separator`,
+`file-comment-char` and `file-na` inputs. The work is to read and map them to
+the collector's config (see above), not to add new inputs.
+
+### Comment description missing
+
+As noted above, comment text is discarded (only counted). The desired
+multiline description output is not implemented.
+
+## Relationship to the background-jobs system
+
+The current job model is effectively "one script == one job". The genotype
+upload is one logical operation spanning several stages, which stresses that
+model. The options (and why a redesign is a separate feature) are recorded in
+`gn-libs/WIP/background-jobs-design-notes.md`.