| Age | Commit message (Collapse) | Author |
|
Present the proposed sample order in an editable textarea (one sample per line)
with instructions, and submit it to the sample-order view.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Carry the file's sample list in the QC job metadata, redirect the QC success
handler to a sample-order review page, and validate the submitted order as a
permutation of the file's samples. On acceptance, launch the next-stage job
(type genotype-stage04) with the accepted order and the preceding stage state in
its metadata; re-submitting reuses an existing Stage 04 job.
Add a minimal Stage 04 placeholder worker for the hand-off point.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Add `is_valid_order`, which checks that a submitted order is a permutation of
the file's samples (catching missing, extra and duplicated samples), with tests.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Add `proposed_order`, which keeps an existing population order and appends
missing samples alphabetically (or sorts alphabetically without an existing
order), and `population_sample_order`, which reads a population's order from
`StrainXRef.OrderId`.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Track marker names per chromosome during Stage 03 QC and report an error for
duplicates within the same chromosome. Marker names may still be reused on
different chromosomes.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Require cM/Mb values to be finite, non-negative numbers (integers are accepted
since they convert to floats without loss) in both orientations, alongside the
existing empty/missing-value checks.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Flag empty or missing-value headings (untransposed header row, transposed row
labels and marker headings), empty or missing-value marker/Chr/cM/Mb values, and
empty sample values during Stage 01 in both orientations. Empty strings are
excluded from `geno_symbols`. Genuine missing genotype values must use the
configured `na_strings`.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Set `error_handler` on the genotype statistics collection and QC jobs so
failures render the genotype upload error page. Structured file errors are
listed; other failures fall back to stdout/stderr and recovery guidance.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Add `genotypes/job-errors.html`, which lists structured file errors (row,
column, value, line, message) when present, and falls back to recovery
guidance with the job's stdout/stderr otherwise.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Add `job_errors.errors_from_job`, which normalises the structured errors a
genotype job records in its stdout (collector `FileStats.errors` and QC result
errors), and collector/QC error handler functions that render them. The HTML
template and job-metadata registration follow in separate commits.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
`initialise_job` now takes metadata as keyword arguments, so replace the
`extra_meta={...}` arguments with `**{...}` at the gn-libs job call sites.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Add `uploader.genotypes.normalise`, which writes a canonical untransposed
matrix (`<stem>__work<suffix>`) next to the original, dropping comments and
`@` directives and leaving the original untouched. Ragged or empty matrices
raise `NormalisationError`, and `qc.check_file` now handles only the normal
orientation.
Stage 02 now puts `stage02`, `genotype_collector_job_id`, `file` and `config`
in the QC job's metadata at launch, and uses `jobs_by_metadata` to reuse an
existing QC job. The completed collector job's metadata is no longer modified.
The Stage 03 worker normalises and QCs the working file from its own metadata.
Change the placeholder marker ordering to chromosome, then Mb, then cM.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
Only comments before the data matrix start feed `FileStats.description`:
before the header row for untransposed files, and before the first data
(markers) row for transposed files. Later comments are still counted but not
used, since they relate to adjacent rows and would be taken out of context by
untransposing.
Track `matrix_started` per chunk and drop later chunks' comments once the
matrix has started, so the multi-processed path matches the single-threaded
one.
Co-authored-by: pi coding agent <pi@pi.dev>
|
|
If errors exist, exit with "GENERALERROR" category rather than with
the "OK" category.
|
|
- Store the genotype file path and collector config in the collector job
metadata so later stages can re-read the file.
- Seed `dot_geno_fields.unk` into the Stage 02 missing-value symbols default.
- Add `uploader.genotypes.qc` with value checks (empty and unknown symbols)
and a placeholder marker-ordering helper (chromosome, cM, Mb) that warns.
- Add the `uploader.scripts.genotypes.qc` worker that reads the collector job,
runs the checks and exits non-zero when errors are found.
- On Stage 02 save, schedule the QC job and record its id for idempotency.
|
|
Add `uploader/static/js/data_utils.js` with `parseSymbols`, the JS counterpart
of `r_qtl.utils.parse_missing_values_symbols`, and use it in the CSV previews
(files.js and the phenotypes raw-files template) instead of their inline
space-splitting. Load data_utils.js before files.js.
|
|
Move `parse_missing_values_symbols` from the genotypes views into
`r_qtl.utils` and use it in the genotypes Stage 01 collector config and the
phenotypes individual-file uploads, replacing their divergent inline splits
with the more robust whitespace/comma parsing. Move the tests to
`tests/r_qtl/test_utils.py`.
|
|
Add a missing-value symbol(s) input to the Stage 02 encoding form and a
`parse_missing_values_symbols` helper in the genotypes views. Parse the input
(whitespace/comma-separated, unique symbols) and store it in the collector
job's `stage02` metadata alongside the description and encoding; prefill the
input when the page is revisited.
|
|
|
|
Redirect the collector job's success handler to the new Stage 02 route,
carrying the collector job id. The `study_design` view loads the job, validates
it and parses the FileStats via `collector_job.file_stats`, then renders the
collected description, the genotype symbols, and the study-design/encoding
inputs.
POST stores the confirmed description and encoding in the collector job's
metadata (`stage02`) and redirects back to the review page (post/redirect/get),
which makes revisiting and re-submitting idempotent. Bind the study-design and
encoding fragments to the saved values.
|
|
Validate that a job is a completed genotype-stats job for the requested
species/population/dataset and parse its FileStats from the job's recorded
stdout.
|
|
Move `uploader/scripts/genotypes/types.py` to `uploader/genotypes/types.py` so
application code can use `FileStats` and friends without importing from
`uploader.scripts`. Update the collector script and tests to the new location.
|
|
|
|
|
|
Add a `description` field to `FileStats`, built by cleaning comment lines
(strip the configured comment character and surrounding whitespace) and joining
them with newlines. Detect comments using the configured comment character
rather than a hard-coded '#', and preserve comment order when merging chunks.
|
|
|
|
Add `collector_config.command` to build the collector CLI command from the
collector config. Replace the `_collect_geno_csv_stats_` stub in
`add_genotype_records` with a `gn_libs.jobs` background job that runs the
collector.
The standard background-job status/summary pages surface the collector's
stdout and stderr, so failures show the available diagnostics. JSON requests
receive a `redirect-to` for the job status page; HTML requests are redirected
there too.
Set the collector job's `success_handler` and add
`collect_genotype_stats_success_handler`, which currently redirects to the job
summary page (dumping the job details). It will be replaced by the Stage 02
study-design/encoding flow.
|
|
Add `uploader.genotypes.collector_config.from_form` to map the upload form's
metadata fields to the statistics collector's config keys (`markers`,
`chromosome`, `cm`, `mb`, `transposed`, `separator`, `comment.char`,
`na_strings`), falling back to the collector's defaults and normalising the
missing-value strings.
Use it in `add_genotype_records` instead of the previous `sep`/`na.strings`
stub metadata.
|
|
Replace the hand-rolled resumable/direct branching in
`process_phenotypes_individual_files` with `resolve_uploaded_file`, fixing
several bugs in the direct-upload path:
- optional files uploaded directly were skipped because presence was only
checked in `request.form`, never in `request.files`;
- `request.files[formkey]` raised `BadRequestKeyError` for a missing file,
making the friendly error branch unreachable;
- malformed/missing resumable data raised unhandled exceptions;
- the absolute path from `save_file` was unnecessarily re-prefixed with the
uploads directory.
Use the returned original name (basename) as the zip arcname and in the
control data, so both stay consistent.
|
|
Wire `add_genotype_records` to `uploader.files.resolve_uploaded_file` for both
the resumable and direct upload paths, replacing the incorrect
`form["uploaded-file"]` lookup. On failure, return a JSON error for JSON
requests, or flash and redirect back to the add-records page.
|
|
Add `resolve_uploaded_file(form, files, upload_dir, fieldname, hashed=True)`
to `uploader.files`. It resolves either a chunked/resumable upload (the JSON
blob produced by the resumable upload endpoint) or a direct multipart upload,
returning `Right((path, original_name))` or `Left(message)`.
This is generic: the only per-caller input is the form field name. It is the
consumer counterpart of the resumable upload contract already implemented in
`uploader.files.views`.
|
|
The `scripts` package now lives under `uploader`, so update every reference
from `scripts.<...>` to `uploader.scripts.<...>`: Python imports, `-m` command
module strings, logging module names, test imports, CLI/gunicorn entry points,
lint/guix paths and the WIP documentation.
Note: the untracked `run-dev-flask.sh` and `run-dev-gunicorn.sh` helper scripts
still need their entry-point module updated manually.
|
|
Move the top-level `scripts` package to `uploader/scripts` so that it becomes
the `uploader.scripts` package. All files are moved unchanged; references are
updated in a follow-up commit.
|
|
|
|
|
|
|
|
Use python's official documentation format, specified by PEP 287 for
documenting the function's parameters, return and exceptions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Collect details on which headers/headings in the file represent the
chromosomes, the markers, and the genetic and physical distances.
|
|
Allow files that have transposed the expected rows and columns to be
uploaded and processed too.
|
|
Add the border around the CSV fields in the macro file itself so that
we have a consistent UI.
|
|
|
|
The endpoint can return either HTML or JSON depending on the 'Accept'
header.
|
|
|
|
|