about summary refs log tree commit diff
path: root/uploader/scripts
AgeCommit message (Collapse)Author
5 daysAdd the Stage 03 sample-order review and hand-offFrederick Muriuki Muriithi
Carry the file's sample list in the QC job metadata, redirect the QC success handler to a sample-order review page, and validate the submitted order as a permutation of the file's samples. On acceptance, launch the next-stage job (type genotype-stage04) with the accepted order and the preceding stage state in its metadata; re-submitting reuses an existing Stage 04 job. Add a minimal Stage 04 placeholder worker for the hand-off point. Co-authored-by: pi coding agent <pi@pi.dev>
5 daysValidate non-negative numeric positions during collectionFrederick Muriuki Muriithi
Require cM/Mb values to be finite, non-negative numbers (integers are accepted since they convert to floats without loss) in both orientations, alongside the existing empty/missing-value checks. Co-authored-by: pi coding agent <pi@pi.dev>
5 daysValidate headings and data values early during collectionFrederick Muriuki Muriithi
Flag empty or missing-value headings (untransposed header row, transposed row labels and marker headings), empty or missing-value marker/Chr/cM/Mb values, and empty sample values during Stage 01 in both orientations. Empty strings are excluded from `geno_symbols`. Genuine missing genotype values must use the configured `na_strings`. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysNormalise genotype files and move Stage 02 state into the QC jobFrederick Muriuki Muriithi
Add `uploader.genotypes.normalise`, which writes a canonical untransposed matrix (`<stem>__work<suffix>`) next to the original, dropping comments and `@` directives and leaving the original untouched. Ragged or empty matrices raise `NormalisationError`, and `qc.check_file` now handles only the normal orientation. Stage 02 now puts `stage02`, `genotype_collector_job_id`, `file` and `config` in the QC job's metadata at launch, and uses `jobs_by_metadata` to reuse an existing QC job. The completed collector job's metadata is no longer modified. The Stage 03 worker normalises and QCs the working file from its own metadata. Change the placeholder marker ordering to chromosome, then Mb, then cM. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysScope the genotype description to pre-matrix commentsFrederick Muriuki Muriithi
Only comments before the data matrix start feed `FileStats.description`: before the header row for untransposed files, and before the first data (markers) row for transposed files. Later comments are still counted but not used, since they relate to adjacent rows and would be taken out of context by untransposing. Track `matrix_started` per chunk and drop later chunks' comments once the matrix has started, so the multi-processed path matches the single-threaded one. Co-authored-by: pi coding agent <pi@pi.dev>
11 daysCheck for errors before exit.Frederick Muriuki Muriithi
If errors exist, exit with "GENERALERROR" category rather than with the "OK" category.
12 daysAdd Stage 03 genotype value QC and schedule it from Stage 02pi coding agent
- Store the genotype file path and collector config in the collector job metadata so later stages can re-read the file. - Seed `dot_geno_fields.unk` into the Stage 02 missing-value symbols default. - Add `uploader.genotypes.qc` with value checks (empty and unknown symbols) and a placeholder marker-ordering helper (chromosome, cM, Mb) that warns. - Add the `uploader.scripts.genotypes.qc` worker that reads the collector job, runs the checks and exits non-zero when errors are found. - On Stage 02 save, schedule the QC job and record its id for idempotency.
13 daysMove genotype data types into the uploader.genotypes packagepi coding agent
Move `uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py` so application code can use `FileStats` and friends without importing from `uploader.scripts`. Update the collector script and tests to the new location.
13 daysCollect comment lines into the file descriptionpi coding agent
Add a `description` field to `FileStats`, built by cleaning comment lines (strip the configured comment character and surrounding whitespace) and joining them with newlines. Detect comments using the configured comment character rather than a hard-coded '#', and preserve comment order when merging chunks.
13 daysFix minor linting errors.Frederick Muriuki Muriithi
2026-10-22Update references to the moved scripts packagepi coding agent
The `scripts` package now lives under `uploader`, so update every reference from `scripts.<...>` to `uploader.scripts.<...>`: Python imports, `-m` command module strings, logging module names, test imports, CLI/gunicorn entry points, lint/guix paths and the WIP documentation. Note: the untracked `run-dev-flask.sh` and `run-dev-gunicorn.sh` helper scripts still need their entry-point module updated manually.
2026-10-22Move scripts into the uploader packagepi coding agent
Move the top-level `scripts` package to `uploader/scripts` so that it becomes the `uploader.scripts` package. All files are moved unchanged; references are updated in a follow-up commit.