about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorpi coding agent <pi@localhost>2026-11-04 18:17:28 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-11-04 15:01:10 -0600
commitf5fd6d3eae46d138a8901b87711b6fd60a36d0a5 (patch)
tree1bf02f93d311f3a7a5c45619cf664f4e377a5b2d /docs
parent5d88dd6028b7d8244f0bfeda0acd01ac6b6b8213 (diff)
downloadgn-uploader-f5fd6d3eae46d138a8901b87711b6fd60a36d0a5.tar.gz
docs: record more implementation concerns
- Stage 01 wiring progress in the implementation notes
- comment collection in the implementation notes
- Stage 02 review page and the type move
Diffstat (limited to 'docs')
-rw-r--r--docs/WIP/geno-uploads-implementation-notes.md80
1 files changed, 52 insertions, 28 deletions
diff --git a/docs/WIP/geno-uploads-implementation-notes.md b/docs/WIP/geno-uploads-implementation-notes.md
index e4baf3c..e8ce6db 100644
--- a/docs/WIP/geno-uploads-implementation-notes.md
+++ b/docs/WIP/geno-uploads-implementation-notes.md
@@ -1,7 +1,7 @@
 # Genotype uploads โ€” implementation notes & concerns
 
 - **Feature:** genotypes upload
-- **Last updated:** 2026-10-22
+- **Last updated:** 2026-11-04
 - **Related:** [design notes](geno-uploads-design-notes.md) ยท
   [tasks & todos](geno-uploads-tasks-and-todos.md)
 
@@ -13,11 +13,14 @@ the concerns/deviations discovered while building it. It supersedes the
 
 - **Stage 01 engine: implemented and tested.** The statistics collector exists
   at `uploader/scripts/genotypes/preprocess_csv_collect_info.py`.
-- **Stage 01 wiring into the upload flow: in progress.** The uploaded file is
-  resolved for both the resumable and direct upload paths and handed to the
-  collector stub; mapping the form metadata to the collector config, running
-  the collector and persisting its results are still pending.
-- **Stages 02โ€“07: not implemented** (still design/TODO comment blocks in
+- **Stage 01 wiring into the upload flow: done for the Stage 02 hand-off.** The
+  uploaded file is resolved, the form metadata is mapped to the collector
+  config, the collector runs as a `gn_libs.jobs` background job
+  (`genotype-file-stats`), and Stage 02 reads its `FileStats` from the job.
+- **Stage 02: review page implemented.** Description, genotype symbols, study
+  design and encoding are presented; the confirmed values are stored in the
+  collector job metadata.
+- **Stages 03โ€“07: not implemented** (still design/TODO comment blocks in
   `uploader/genotypes/views.py`).
 
 ## Stage 01 implementation
@@ -36,10 +39,12 @@ Runnable as a CLI; emits `FileStats` as JSON.
 - `dot_geno_fields` (parsed from `@` lines, when present);
 - `geno_symbols` (distinct symbols in the data, sorted for determinism);
 - `samples_list`, `markers_list`;
-- `errors` (header validation errors + per-line errors).
+- `errors` (header validation errors + per-line errors);
+- `description` (cleaned comment lines, joined with newlines).
 
-**Not collected:** the text of comment lines. Only a count is kept, so the
-"description string" part of Stage 01 is still missing.
+Comment lines are cleaned by stripping the configured comment character and
+surrounding whitespace, then joined with `\n`; comment order is preserved when
+merging chunks.
 
 ### Configuration
 
@@ -112,36 +117,55 @@ package owns.
   missing direct files, unhandled resumable-data errors, and a fragile
   double-prefixed path.
 
-### The view does not run the collector
+### The view schedules the collector as a background job
 
-In `uploader/genotypes/views.py`:
+`add_genotype_records()` (POST) builds the collector config from the form and
+schedules `uploader.scripts.genotypes.preprocess_csv_collect_info` via
+`gn_libs.jobs` (job type `genotype-file-stats`). The collector's stdout (the
+`FileStats` JSON) and stderr are available on the standard job status/summary
+pages, and Stage 02 reads the stats back from the job.
 
-- `_collect_geno_csv_stats_()` is a stub that just returns a dict
-  (`{"file": ..., "metadata": ...}`).
-- `add_genotype_records()` (POST) builds that dict and renders a placeholder
-  template; it never invokes `preprocess_csv_collect_info`/`compute_statistics`
-  and never schedules a job.
+### Collector config mapping (done)
 
-### Config-key mismatch to bridge
+`uploader.genotypes.collector_config.from_form(form)` maps the upload form's
+metadata fields (`markers_field`, `chromosomes_field`, `centimorgans_field`,
+`megabases_field`, `transposed`, `file-separator`, `file-comment-char`,
+`file-na`) to the collector's config keys (`markers`, `chromosome`, `cm`,
+`mb`, `transposed`, `separator`, `comment.char`, `na_strings`), falling back
+to the collector's defaults and normalising the missing-value strings.
+`add_genotype_records` now passes this config to the collector stub.
 
-The view's `csv_meta` uses `sep`, `comment.char`, `na.strings`, while the
-collector expects `separator`, `comment.char`, `na_strings` plus `markers`,
-`chromosome`, `cm`, `mb`, `transposed`. An adapter is needed (`comment.char`
-already matches).
-
-### Stage 01 form inputs exist; the view must read them
+### Stage 01 form inputs
 
 The upload form
 (`uploader/templates/genotypes/add-genotypes-records-csv.html`) already
 provides `markers_field`, `chromosomes_field`, `centimorgans_field`,
 `megabases_field`, `transposed`, plus the shared `file-separator`,
-`file-comment-char` and `file-na` inputs. The work is to read and map them to
-the collector's config (see above), not to add new inputs.
+`file-comment-char` and `file-na` inputs. No new inputs are needed; they are
+mapped to the collector config as described above.
+
+### Comment description (done)
+
+Comment text is collected into `FileStats.description`: one cleaned comment
+line per line, joined with newlines. This is what Stage 02 will present as the
+editable study/dataset description.
+
+### Data types moved
+
+`FileStats` and the other genotype data types moved from
+`uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py`, so
+application code does not import from `uploader.scripts`. The collector script
+and tests import them from the new location.
 
-### Comment description missing
+### Stage 02 review page
 
-As noted above, comment text is discarded (only counted). The desired
-multiline description output is not implemented.
+`uploader.genotypes.collector_job.file_stats(job, species_id, population_id,
+dataset_id)` validates a completed `genotype-file-stats` job for the dataset
+and parses the `FileStats` from its recorded stdout. The collector job's
+`success_handler` redirects to the `study_design` view with the job id; that
+view renders the collected description, the genotype symbols, and the
+study-design/encoding inputs. Saving stores the confirmed values in the job
+metadata (`stage02`) and redirects back (post/redirect/get).
 
 ## Relationship to the background-jobs system