about summary refs log tree commit diff
path: root/scripts/rqtl_wrapper.R
AgeCommit message (Collapse)Author
9 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
9 daysrqtl: map the cM-only genotype filesPjotr
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and none of them could be scanned. GeneNetwork3's wrapper (copied verbatim here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts the samples at a hardcoded column 5. On a cM-only file that is one column too far, so the genotype block loses its first individual and the whole scan dies in `do.call(rbind, ...)` with "number of columns of result is not a multiple of vector length" -- a message that says nothing about the cause. The wrapper now picks the map column from whichever of the requested scale and the file's own columns exists (falling back, and saying so), and starts the samples right after the position columns; the sex default and the sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same assumption and got the same fix. This is the second deliberate deviation from GeneNetwork3's script, after the sample-count guard, and both are commented as such at the code. * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the first sample column are computed, not assumed. * tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb` dropped -- same panel, cM only. * tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six markers come back (the failure mode was five-vs-six columns, not an error message that pointed at the genotype file). * src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards `--scale`; it does, and now falls back, so the request field says what `pos` is in. Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499 markers) and BXD.4 (3 811) are unchanged. 307 tests pass; the same four as before do not.
11 daysrqtl: refuse a phenotype whose sample count is not the cross'sPjotr
Asked to map a 9-strain trait, the wrapper scanned it against the 6-individual fixture cross and returned six markers with LOD scores and `state: complete` -- a plausible answer belonging to nobody. R recycled and truncated the values silently, and nothing downstream could tell. GeneNetwork3's copy of this script has the same hole. The cross-file format carries no per-sample names, so the count is the only check available; order remains the caller's responsibility (and the reason values and genotype file must come from the same panel). With the check, the same request is a 502 rqtl_failed quoting both counts, and the correctly-shaped request still returns the documented LODs. * scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of the cross, and stop. * README.md (/compute/rqtl): the second fix relative to gn3, and what stays the caller's job. * docs/www/ai-instruction.md (4a): values and genotypes must describe the same individuals, one for one, in the same order. * docs/www/operator-checklist.md (Compute): what genotype files actually exist on this box (test fixtures, 198 BXD individuals) and why a database trait, with its own strain subset, cannot be mapped against them.
2026-10-14gemma/rqtl: three fixes from the endpoint reviewAlexander_Kabui
- rqtl: per-call unique output filename. The content hash alone was deterministic across processes, so two concurrent identical requests shared one output file; R truncates on open, so one request could read the other's half-written results with no error. The per-call unique (pid + nanos, already computed for the pheno file) now names the output file too. - streaming/gemma/rqtl: append a failure marker to the run log on every error path, mirroring the completion marker. A status poller can now tell a failed run from a still-running one. - rqtl: --scale is now honored. The wrapper selects the geno file's cM/Mb map column from it, and the route rejects values other than mb/cm with 422 instead of silently ignoring them. Both new behaviors are unit-tested; the live Rscript tests pass.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>