# Genotype uploads — implementation notes & concerns - **Feature:** genotypes upload - **Last updated:** 2026-11-04 - **Related:** [design notes](geno-uploads-design-notes.md) · [tasks & todos](geno-uploads-tasks-and-todos.md) This file records what is actually implemented, how it maps to the design, and the concerns/deviations discovered while building it. It supersedes the 2026-10-15 implementation summary. ## Status at a glance - **Stage 01 engine: implemented and tested.** The statistics collector exists at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`. - **Stage 01 wiring into the upload flow: done for the Stage 02 hand-off.** The uploaded file is resolved, the form metadata is mapped to the collector config, the collector runs as a `gn_libs.jobs` background job (`genotype-file-stats`), and Stage 02 reads its `FileStats` from the job. - **Stage 02: review page implemented.** Description, genotype symbols, study design and encoding are presented; the confirmed values are stored in the collector job metadata. - **Stage 03: value QC implemented.** The genotype file is re-read and every sample value is checked against the Stage 02 encoding/missing symbols; the QC job is scheduled from Stage 02. Ordering checks remain pending. - **Stages 04–07: not implemented** (still design/TODO comment blocks in `uploader/genotypes/views.py`). ## Stage 01 implementation ### Location `uploader/scripts/genotypes/preprocess_csv_collect_info.py` (+ `uploader/scripts/genotypes/types.py`). Runnable as a CLI; emits `FileStats` as JSON. ### What it collects `FileStats` (`uploader/scripts/genotypes/types.py`) currently contains: - `total_lines`, `data_lines`, `comment_lines`, `blank_lines`, `dot_geno_lines`; - `dot_geno_fields` (parsed from `@` lines, when present); - `geno_symbols` (distinct symbols in the data, sorted for determinism); - `samples_list`, `markers_list`; - `errors` (header validation errors + per-line errors); - `description` (cleaned comment lines, joined with newlines). Comment lines are cleaned by stripping the configured comment character and surrounding whitespace, then joined with `\n`; comment order is preserved when merging chunks. ### Early data validation (done) Stage 01 flags empty or missing-value headings (the untransposed header row, the transposed first column and transposed marker headings), empty or missing-value marker/Chr/cM/Mb values, empty sample values, and negative or non-numeric cM/Mb values. cM/Mb values must be finite, non-negative numbers (integers are accepted). Empty strings are excluded from `geno_symbols`. Genuine missing genotype values must use the configured `na_strings`. ### Configuration `compute_statistics(filepath, fileconfigs)` expects a config dict with: ``` markers, chromosome, cm, mb, separator, comment.char, na_strings, transposed ``` ### Processing design - Per-line processing is independent, so it was extracted into `_accumulate`, which returns a `_ChunkStats` partial. - `file_stats_singlethread` = one `_accumulate` pass over the whole file. - `file_stats_multiprocessing` = stream the file into bounded chunks (`MULTIPROCESSING_CHUNK_SIZE`, default 10 000 lines), process each chunk in a worker process (`Pool.imap`), and merge in file order (`_merge_chunk_stats` / `_merge_all_chunk_stats`). - The worker is module-level and takes only picklable data (headers, config, chunk), with the split/skip predicates rebuilt inside the worker, so it works under `fork`, `spawn` and `forkserver`. - `compute_statistics` chooses the single-threaded or multi-processed path based on `FILE_SIZE_THRESHOLD` (100 MiB), comparing bytes to bytes. ### Merge semantics (why the two paths agree) - counters: summed; - `geno_symbols`: unioned (then sorted on output); - `errors`: concatenated in file order; - `markers_list`: concatenated in order (untransposed), or last markers line wins (transposed); - `dot_geno_fields`: combined, later values override earlier ones. ### Defects fixed while implementing - Size threshold compared megabytes to bytes (branch was effectively unreachable). Now bytes vs bytes. - Header-skip compared a `str` to a `tuple`, so header rows were counted as data. Now compares the split line to the headers. - `FileStats.blank_lines` was declared but never populated. Now populated. - `geno_symbols` output order was set-iteration-dependent (unstable across runs/processes). Now sorted. ### Tests `tests/genotypes/` (pytest): accumulation, single-threaded stats, merge semantics, and single- vs multi-processed equivalence across chunk sizes, including transposed files and files with errors. `tests/uploader/test_files.py`: the generic uploaded-file resolver (direct and resumable uploads, hashed/unhashed, error cases). Together with `tests/genotypes/` this is **31 passing**. ## Integration concerns (upload flow) ### Uploaded-file resolution (done) `uploader.files.resolve_uploaded_file(form, files, upload_dir, fieldname, hashed=True)` resolves either a chunked/resumable upload (the JSON blob returned by the resumable upload endpoint) or a direct multipart upload, returning `Right((path, original_name))` or `Left(message)`. It lives in `uploader.files` because it is generic — the only per-caller input is the field name — and it is the consumer counterpart of the resumable-upload contract that package owns. - `add_genotype_records` uses it with the `genotypes-records-file` field. - `process_phenotypes_individual_files` was refactored onto it, fixing: directly-uploaded optional files being skipped, a `BadRequestKeyError` for missing direct files, unhandled resumable-data errors, and a fragile double-prefixed path. ### The view schedules the collector as a background job `add_genotype_records()` (POST) builds the collector config from the form and schedules `uploader.scripts.genotypes.preprocess_csv_collect_info` via `gn_libs.jobs` (job type `genotype-file-stats`). The collector's stdout (the `FileStats` JSON) and stderr are available on the standard job status/summary pages, and Stage 02 reads the stats back from the job. ### Collector config mapping (done) `uploader.genotypes.collector_config.from_form(form)` maps the upload form's metadata fields (`markers_field`, `chromosomes_field`, `centimorgans_field`, `megabases_field`, `transposed`, `file-separator`, `file-comment-char`, `file-na`) to the collector's config keys (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`, `comment.char`, `na_strings`), falling back to the collector's defaults and normalising the missing-value strings. `add_genotype_records` now passes this config to the collector stub. ### Stage 01 form inputs The upload form (`uploader/templates/genotypes/add-genotypes-records-csv.html`) already provides `markers_field`, `chromosomes_field`, `centimorgans_field`, `megabases_field`, `transposed`, plus the shared `file-separator`, `file-comment-char` and `file-na` inputs. No new inputs are needed; they are mapped to the collector config as described above. ### Comment description (done) Comment text is collected into `FileStats.description`: one cleaned comment line per line, joined with newlines. Only comments **before the data matrix starts** (before the header row; before the first data row when transposed) feed the description; later comments are counted but not used. This is what Stage 02 presents as the editable study/dataset description. ### Data types moved `FileStats` and the other genotype data types moved from `uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py`, so application code does not import from `uploader.scripts`. The collector script and tests import them from the new location. ### Stage 02 review page `uploader.genotypes.collector_job.file_stats(job, species_id, population_id, dataset_id)` validates a completed `genotype-file-stats` job for the dataset and parses the `FileStats` from its recorded stdout. The collector job's `success_handler` redirects to the `study_design` view with the job id; that view renders the collected description, the genotype symbols, the study-design/encoding inputs and the missing-value symbol(s). Saving puts the confirmed values (including `missing_symbols`) in the **QC job's** metadata at launch; the completed collector job is left untouched. Revisits read them back from the existing QC job (found via `jobs_by_metadata`). ### Stage 03 normalisation `uploader.genotypes.normalise.normalise_file(original, config)` writes a canonical untransposed matrix (`__work`) next to the original, dropping comments and `@` directives (already captured in `FileStats`), leaving the original untouched, and rejecting ragged or empty matrices. `qc.check_file` now handles only the normal orientation. ### Sample ordering (done) `uploader.genotypes.ordering` provides `proposed_order` (retain the population's existing order, append missing samples alphabetically, otherwise sort alphabetically), `population_sample_order` (reads `StrainXRef.OrderId`) and `is_valid_order` (validates a submitted order as a permutation of the file's samples). The QC success handler redirects to a sample-order review page; on acceptance it launches the next-stage job (type `genotype-stage04`) carrying `sample_order` and the preceding stage state in its metadata. A minimal Stage 04 placeholder worker is the current hand-off point. ### Marker uniqueness (done) Stage 03 QC tracks `(chromosome, marker)` pairs and reports an error when a marker name occurs more than once within the same chromosome. The same marker name may appear on different chromosomes. ### Stage 03 value QC Stage 02 launches the QC job with `stage02`, `genotype_collector_job_id`, `file` and `config` in the QC job's metadata. The worker loads its own job, normalises the file, then runs `uploader.genotypes.qc.check_file` on the work file with `transposed=False`, exiting non-zero on errors. The completed collector job is never modified; an existing QC job is reused via `jobs_by_metadata`. The collector exits with `SysExit.GENERALERROR` when `FileStats.errors` is non-empty, so malformed files stop the chain before Stage 02. `uploader.genotypes.qc.sort_markers` is a placeholder ordering helper (chromosome, Mb, cM) that emits a warning until the appropriate ordering is established. ### Pending: error handlers Each stage will register an `error_handler` (FileStats errors shown like the phenotypes job status page; other failures reported with recovery guidance). ## Relationship to the background-jobs system The current job model is effectively "one script == one job". The genotype upload is one logical operation spanning several stages, which stresses that model. The options (and why a redesign is a separate feature) are recorded in `gn-libs/WIP/background-jobs-design-notes.md`.