about summary refs log tree commit diff
path: root/uploader
AgeCommit message (Collapse)Author
6 daysAdd the sample-order review pageFrederick Muriuki Muriithi
Present the proposed sample order in an editable textarea (one sample per line) with instructions, and submit it to the sample-order view. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysAdd the Stage 03 sample-order review and hand-offFrederick Muriuki Muriithi
Carry the file's sample list in the QC job metadata, redirect the QC success handler to a sample-order review page, and validate the submitted order as a permutation of the file's samples. On acceptance, launch the next-stage job (type genotype-stage04) with the accepted order and the preceding stage state in its metadata; re-submitting reuses an existing Stage 04 job. Add a minimal Stage 04 placeholder worker for the hand-off point. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysValidate submitted sample ordersFrederick Muriuki Muriithi
Add `is_valid_order`, which checks that a submitted order is a permutation of the file's samples (catching missing, extra and duplicated samples), with tests. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysAdd genotype sample order proposal helpersFrederick Muriuki Muriithi
Add `proposed_order`, which keeps an existing population order and appends missing samples alphabetically (or sorts alphabetically without an existing order), and `population_sample_order`, which reads a population's order from `StrainXRef.OrderId`. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysReject duplicate marker names within a chromosomeFrederick Muriuki Muriithi
Track marker names per chromosome during Stage 03 QC and report an error for duplicates within the same chromosome. Marker names may still be reused on different chromosomes. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysValidate non-negative numeric positions during collectionFrederick Muriuki Muriithi
Require cM/Mb values to be finite, non-negative numbers (integers are accepted since they convert to floats without loss) in both orientations, alongside the existing empty/missing-value checks. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysValidate headings and data values early during collectionFrederick Muriuki Muriithi
Flag empty or missing-value headings (untransposed header row, transposed row labels and marker headings), empty or missing-value marker/Chr/cM/Mb values, and empty sample values during Stage 01 in both orientations. Empty strings are excluded from `geno_symbols`. Genuine missing genotype values must use the configured `na_strings`. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysRegister genotype job error handlersFrederick Muriuki Muriithi
Set `error_handler` on the genotype statistics collection and QC jobs so failures render the genotype upload error page. Structured file errors are listed; other failures fall back to stdout/stderr and recovery guidance. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysAdd the genotype upload error pageFrederick Muriuki Muriithi
Add `genotypes/job-errors.html`, which lists structured file errors (row, column, value, line, message) when present, and falls back to recovery guidance with the job's stdout/stderr otherwise. Co-authored-by: pi coding agent <pi@pi.dev>
6 daysAdd helpers and handlers for genotype job errorsFrederick Muriuki Muriithi
Add `job_errors.errors_from_job`, which normalises the structured errors a genotype job records in its stdout (collector `FileStats.errors` and QC result errors), and collector/QC error handler functions that render them. The HTML template and job-metadata registration follow in separate commits. Co-authored-by: pi coding agent <pi@pi.dev>
7 daysUpdate genotype/phenotype/sample job call sites for keyword metadataFrederick Muriuki Muriithi
`initialise_job` now takes metadata as keyword arguments, so replace the `extra_meta={...}` arguments with `**{...}` at the gn-libs job call sites. Co-authored-by: pi coding agent <pi@pi.dev>
7 daysNormalise genotype files and move Stage 02 state into the QC jobFrederick Muriuki Muriithi
Add `uploader.genotypes.normalise`, which writes a canonical untransposed matrix (`<stem>__work<suffix>`) next to the original, dropping comments and `@` directives and leaving the original untouched. Ragged or empty matrices raise `NormalisationError`, and `qc.check_file` now handles only the normal orientation. Stage 02 now puts `stage02`, `genotype_collector_job_id`, `file` and `config` in the QC job's metadata at launch, and uses `jobs_by_metadata` to reuse an existing QC job. The completed collector job's metadata is no longer modified. The Stage 03 worker normalises and QCs the working file from its own metadata. Change the placeholder marker ordering to chromosome, then Mb, then cM. Co-authored-by: pi coding agent <pi@pi.dev>
7 daysScope the genotype description to pre-matrix commentsFrederick Muriuki Muriithi
Only comments before the data matrix start feed `FileStats.description`: before the header row for untransposed files, and before the first data (markers) row for transposed files. Later comments are still counted but not used, since they relate to adjacent rows and would be taken out of context by untransposing. Track `matrix_started` per chunk and drop later chunks' comments once the matrix has started, so the multi-processed path matches the single-threaded one. Co-authored-by: pi coding agent <pi@pi.dev>
12 daysCheck for errors before exit.Frederick Muriuki Muriithi
If errors exist, exit with "GENERALERROR" category rather than with the "OK" category.
13 daysAdd Stage 03 genotype value QC and schedule it from Stage 02pi coding agent
- Store the genotype file path and collector config in the collector job metadata so later stages can re-read the file. - Seed `dot_geno_fields.unk` into the Stage 02 missing-value symbols default. - Add `uploader.genotypes.qc` with value checks (empty and unknown symbols) and a placeholder marker-ordering helper (chromosome, cM, Mb) that warns. - Add the `uploader.scripts.genotypes.qc` worker that reads the collector job, runs the checks and exits non-zero when errors are found. - On Stage 02 save, schedule the QC job and record its id for idempotency.
13 daysShare missing-value symbol parsing in the browserpi coding agent
Add `uploader/static/js/data_utils.js` with `parseSymbols`, the JS counterpart of `r_qtl.utils.parse_missing_values_symbols`, and use it in the CSV previews (files.js and the phenotypes raw-files template) instead of their inline space-splitting. Load data_utils.js before files.js.
13 daysUnify missing-value symbol parsing in r_qtl.utilspi coding agent
Move `parse_missing_values_symbols` from the genotypes views into `r_qtl.utils` and use it in the genotypes Stage 01 collector config and the phenotypes individual-file uploads, replacing their divergent inline splits with the more robust whitespace/comma parsing. Move the tests to `tests/r_qtl/test_utils.py`.
13 daysAdd missing-value symbols to Stage 02pi coding agent
Add a missing-value symbol(s) input to the Stage 02 encoding form and a `parse_missing_values_symbols` helper in the genotypes views. Parse the input (whitespace/comma-separated, unique symbols) and store it in the collector job's `stage02` metadata alongside the description and encoding; prefill the input when the page is revisited.
14 daysImprove the UI copy.Frederick Muriuki Muriithi
14 daysAdd the Stage 02 study design/encoding review pagepi coding agent
Redirect the collector job's success handler to the new Stage 02 route, carrying the collector job id. The `study_design` view loads the job, validates it and parses the FileStats via `collector_job.file_stats`, then renders the collected description, the genotype symbols, and the study-design/encoding inputs. POST stores the confirmed description and encoding in the collector job's metadata (`stage02`) and redirects back to the review page (post/redirect/get), which makes revisiting and re-submitting idempotent. Bind the study-design and encoding fragments to the saved values.
14 daysAdd a helper to read FileStats from a collector jobpi coding agent
Validate that a job is a completed genotype-stats job for the requested species/population/dataset and parse its FileStats from the job's recorded stdout.
14 daysMove genotype data types into the uploader.genotypes packagepi coding agent
Move `uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py` so application code can use `FileStats` and friends without importing from `uploader.scripts`. Update the collector script and tests to the new location.
14 daysAdd fragment for study design elements.Frederick Muriuki Muriithi
14 daysAdd fragment for genotype encoding elements.Frederick Muriuki Muriithi
14 daysCollect comment lines into the file descriptionpi coding agent
Add a `description` field to `FileStats`, built by cleaning comment lines (strip the configured comment character and surrounding whitespace) and joining them with newlines. Detect comments using the configured comment character rather than a hard-coded '#', and preserve comment order when merging chunks.
14 daysFix minor linting errors.Frederick Muriuki Muriithi
14 daysSchedule the genotype statistics collector as a background jobpi coding agent
Add `collector_config.command` to build the collector CLI command from the collector config. Replace the `_collect_geno_csv_stats_` stub in `add_genotype_records` with a `gn_libs.jobs` background job that runs the collector. The standard background-job status/summary pages surface the collector's stdout and stderr, so failures show the available diagnostics. JSON requests receive a `redirect-to` for the job status page; HTML requests are redirected there too. Set the collector job's `success_handler` and add `collect_genotype_stats_success_handler`, which currently redirects to the job summary page (dumping the job details). It will be replaced by the Stage 02 study-design/encoding flow.
14 daysBuild the collector config from the upload formpi coding agent
Add `uploader.genotypes.collector_config.from_form` to map the upload form's metadata fields to the statistics collector's config keys (`markers`, `chromosome`, `cm`, `mb`, `transposed`, `separator`, `comment.char`, `na_strings`), falling back to the collector's defaults and normalising the missing-value strings. Use it in `add_genotype_records` instead of the previous `sep`/`na.strings` stub metadata.
2026-10-22Reuse the shared resolver for phenotype uploadspi coding agent
Replace the hand-rolled resumable/direct branching in `process_phenotypes_individual_files` with `resolve_uploaded_file`, fixing several bugs in the direct-upload path: - optional files uploaded directly were skipped because presence was only checked in `request.form`, never in `request.files`; - `request.files[formkey]` raised `BadRequestKeyError` for a missing file, making the friendly error branch unreachable; - malformed/missing resumable data raised unhandled exceptions; - the absolute path from `save_file` was unnecessarily re-prefixed with the uploads directory. Use the returned original name (basename) as the zip arcname and in the control data, so both stay consistent.
2026-10-22Use the shared resolver for genotype uploadspi coding agent
Wire `add_genotype_records` to `uploader.files.resolve_uploaded_file` for both the resumable and direct upload paths, replacing the incorrect `form["uploaded-file"]` lookup. On failure, return a JSON error for JSON requests, or flash and redirect back to the add-records page.
2026-10-22Add a generic uploaded-file resolver to uploader.filespi coding agent
Add `resolve_uploaded_file(form, files, upload_dir, fieldname, hashed=True)` to `uploader.files`. It resolves either a chunked/resumable upload (the JSON blob produced by the resumable upload endpoint) or a direct multipart upload, returning `Right((path, original_name))` or `Left(message)`. This is generic: the only per-caller input is the form field name. It is the consumer counterpart of the resumable upload contract already implemented in `uploader.files.views`.
2026-10-22Update references to the moved scripts packagepi coding agent
The `scripts` package now lives under `uploader`, so update every reference from `scripts.<...>` to `uploader.scripts.<...>`: Python imports, `-m` command module strings, logging module names, test imports, CLI/gunicorn entry points, lint/guix paths and the WIP documentation. Note: the untracked `run-dev-flask.sh` and `run-dev-gunicorn.sh` helper scripts still need their entry-point module updated manually.
2026-10-22Move scripts into the uploader packagepi coding agent
Move the top-level `scripts` package to `uploader/scripts` so that it becomes the `uploader.scripts` package. All files are moved unchanged; references are updated in a follow-up commit.
2026-10-02Move CSV-file specific processing code to new module.Frederick Muriuki Muriithi
2026-10-02Initialise module for common CSV processing utilities.Frederick Muriuki Muriithi
2026-10-01Fix type errors.Frederick Muriuki Muriithi
2026-10-01Improve function documentation.Frederick Muriuki Muriithi
Use python's official documentation format, specified by PEP 287 for documenting the function's parameters, return and exceptions.
2026-10-01Fetch genotype dataset by species and population only.Frederick Muriuki Muriithi
2026-10-01TooManyResultsError: Add new error type.Frederick Muriuki Muriithi
2026-10-01Have consistent interface for "dataset_by_id" functions.Frederick Muriuki Muriithi
2026-10-01Fix pylint issues to get a clean working state.Frederick Muriuki Muriithi
2026-09-28BugFix: Myriad of minor bugfixes.Frederick Muriuki Muriithi
2026-09-24Fix indentation.Frederick Muriuki Muriithi
2026-09-24Collect metadata on the fileFrederick Muriuki Muriithi
Collect details on which headers/headings in the file represent the chromosomes, the markers, and the genetic and physical distances.
2026-09-24Allow transposed filesFrederick Muriuki Muriithi
Allow files that have transposed the expected rows and columns to be uploaded and processed too.
2026-09-24Add border around CSV fields in macroFrederick Muriuki Muriithi
Add the border around the CSV fields in the macro file itself so that we have a consistent UI.
2026-09-17Add outline of conversion of file to R/qtl2 format.Frederick Muriuki Muriithi
2026-09-17Use 'Accept' header in ajax requests for content negotiation.Frederick Muriuki Muriithi
The endpoint can return either HTML or JSON depending on the 'Accept' header.
2026-09-16Enable form submission.Frederick Muriuki Muriithi
2026-09-16BugFix: Fix bugs in the common code.Frederick Muriuki Muriithi