diff options
| author | pi coding agent <pi@localhost> | 2026-11-04 18:17:28 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-11-04 15:01:10 -0600 |
| commit | f5fd6d3eae46d138a8901b87711b6fd60a36d0a5 (patch) | |
| tree | 1bf02f93d311f3a7a5c45619cf664f4e377a5b2d /docs | |
| parent | 5d88dd6028b7d8244f0bfeda0acd01ac6b6b8213 (diff) | |
| download | gn-uploader-f5fd6d3eae46d138a8901b87711b6fd60a36d0a5.tar.gz | |
docs: record more implementation concerns
- Stage 01 wiring progress in the implementation notes - comment collection in the implementation notes - Stage 02 review page and the type move
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/WIP/geno-uploads-implementation-notes.md | 80 |
1 files changed, 52 insertions, 28 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md index e4baf3c..e8ce6db 100644 --- a/docs/WIP/geno-uploads-implementation-notes.md +++ b/docs/WIP/geno-uploads-implementation-notes.md @@ -1,7 +1,7 @@ # Genotype uploads โ implementation notes & concerns - **Feature:** genotypes upload -- **Last updated:** 2026-10-22 +- **Last updated:** 2026-11-04 - **Related:** [design notes](geno-uploads-design-notes.md) ยท [tasks & todos](geno-uploads-tasks-and-todos.md) @@ -13,11 +13,14 @@ the concerns/deviations discovered while building it. It supersedes the - **Stage 01 engine: implemented and tested.** The statistics collector exists at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`. -- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is - resolved for both the resumable and direct upload paths and handed to the - collector stub; mapping the form metadata to the collector config, running - the collector and persisting its results are still pending. -- **Stages 02โ07: not implemented** (still design/TODO comment blocks in +- **Stage 01 wiring into the upload flow: done for the Stage 02 hand-off.** The + uploaded file is resolved, the form metadata is mapped to the collector + config, the collector runs as a `gn_libs.jobs` background job + (`genotype-file-stats`), and Stage 02 reads its `FileStats` from the job. +- **Stage 02: review page implemented.** Description, genotype symbols, study + design and encoding are presented; the confirmed values are stored in the + collector job metadata. +- **Stages 03โ07: not implemented** (still design/TODO comment blocks in `uploader/genotypes/views.py`). ## Stage 01 implementation @@ -36,10 +39,12 @@ Runnable as a CLI; emits `FileStats` as JSON. - `dot_geno_fields` (parsed from `@` lines, when present); - `geno_symbols` (distinct symbols in the data, sorted for determinism); - `samples_list`, `markers_list`; -- `errors` (header validation errors + per-line errors). +- `errors` (header validation errors + per-line errors); +- `description` (cleaned comment lines, joined with newlines). -**Not collected:** the text of comment lines. Only a count is kept, so the -"description string" part of Stage 01 is still missing. +Comment lines are cleaned by stripping the configured comment character and +surrounding whitespace, then joined with `\n`; comment order is preserved when +merging chunks. ### Configuration @@ -112,36 +117,55 @@ package owns. missing direct files, unhandled resumable-data errors, and a fragile double-prefixed path. -### The view does not run the collector +### The view schedules the collector as a background job -In `uploader/genotypes/views.py`: +`add_genotype_records()` (POST) builds the collector config from the form and +schedules `uploader.scripts.genotypes.preprocess_csv_collect_info` via +`gn_libs.jobs` (job type `genotype-file-stats`). The collector's stdout (the +`FileStats` JSON) and stderr are available on the standard job status/summary +pages, and Stage 02 reads the stats back from the job. -- `_collect_geno_csv_stats_()` is a stub that just returns a dict - (`{"file": ..., "metadata": ...}`). -- `add_genotype_records()` (POST) builds that dict and renders a placeholder - template; it never invokes `preprocess_csv_collect_info`/`compute_statistics` - and never schedules a job. +### Collector config mapping (done) -### Config-key mismatch to bridge +`uploader.genotypes.collector_config.from_form(form)` maps the upload form's +metadata fields (`markers_field`, `chromosomes_field`, `centimorgans_field`, +`megabases_field`, `transposed`, `file-separator`, `file-comment-char`, +`file-na`) to the collector's config keys (`markers`, `chromosome`, `cm`, +`mb`, `transposed`, `separator`, `comment.char`, `na_strings`), falling back +to the collector's defaults and normalising the missing-value strings. +`add_genotype_records` now passes this config to the collector stub. -The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the -collector expects `separator`, `comment.char`, `na_strings` plus `markers`, -`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char` -already matches). - -### Stage 01 form inputs exist; the view must read them +### Stage 01 form inputs The upload form (`uploader/templates/genotypes/add-genotypes-records-csv.html`) already provides `markers_field`, `chromosomes_field`, `centimorgans_field`, `megabases_field`, `transposed`, plus the shared `file-separator`, -`file-comment-char` and `file-na` inputs. The work is to read and map them to -the collector's config (see above), not to add new inputs. +`file-comment-char` and `file-na` inputs. No new inputs are needed; they are +mapped to the collector config as described above. + +### Comment description (done) + +Comment text is collected into `FileStats.description`: one cleaned comment +line per line, joined with newlines. This is what Stage 02 will present as the +editable study/dataset description. + +### Data types moved + +`FileStats` and the other genotype data types moved from +`uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py`, so +application code does not import from `uploader.scripts`. The collector script +and tests import them from the new location. -### Comment description missing +### Stage 02 review page -As noted above, comment text is discarded (only counted). The desired -multiline description output is not implemented. +`uploader.genotypes.collector_job.file_stats(job, species_id, population_id, +dataset_id)` validates a completed `genotype-file-stats` job for the dataset +and parses the `FileStats` from its recorded stdout. The collector job's +`success_handler` redirects to the `study_design` view with the job id; that +view renders the collected description, the genotype symbols, and the +study-design/encoding inputs. Saving stores the confirmed values in the job +metadata (`stage02`) and redirects back (post/redirect/get). ## Relationship to the background-jobs system |
