about summary refs log tree commit diff
path: root/src/rqtl.rs
AgeCommit message (Collapse)Author
4 daysqtl2: permutations for the LOD thresholdsPjotr
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply gains meta.permutation {n, seed, significant, suggestive, values} -- the genome-wide maximum of every permutation, so a histogram needs no second call. `data` stays the positional rows array on purpose: /plot/rqtl posts it straight back and the monitor counts its rows, so the null went beside the rows rather than around them (a generic Envelope::with_meta/Meta::extra, skip_serializing_if, so no other endpoint's response changes). Thresholds come from R/qtl2's scan1perm over the same genotype probabilities, kinship and covariates as the scan, with chr_lengths from the map -- without those it derives chromosome lengths from marker positions and the numbers stop being comparable with R/qtl's. The permutation CSV is written in R/qtl's two-column shape so one parser and one percentile routine serve both engines: PermutationResult::from_values is now shared with /compute/rqtl instead of duplicated. Semantics, each with the note that says it: * scan1perm shuffles phenotypes across individuals -- right for an outbred cross, only approximate for a selfing RIL panel, where qtl2perm is the correct package (not installed here, and not packaged); * perm_strata takes a case attribute name (sex, Epoch, ...) and permutes within it, because shuffling across sexes inflates the null; * fewer than 100 permutations gets a note that the 0.05 quantile is a quantile of too few values; more than 10 000 is refused, since each permutation is a whole genome scan; * a seed is reported always -- without one, two identical requests would give different thresholds. Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s, 1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at 1000 on the raw probe) and are reproducible: the same seed gives the same numbers on 1 and 4 cores. Four things that only surfaced by running it, all commented where they bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so opt$perm_out was always NULL and the file landed somewhere the caller did not look); perm_strata must be a named character vector, not a factor; the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the permutation index first, and a first-field parser reads that as the LOD); and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not 0.67. bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging meta.permutation) so a deploy that loses the R path shows up as a red check rather than a surprise.
10 daysrqtl: map the cM-only genotype filesPjotr
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and none of them could be scanned. GeneNetwork3's wrapper (copied verbatim here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts the samples at a hardcoded column 5. On a cM-only file that is one column too far, so the genotype block loses its first individual and the whole scan dies in `do.call(rbind, ...)` with "number of columns of result is not a multiple of vector length" -- a message that says nothing about the cause. The wrapper now picks the map column from whichever of the requested scale and the file's own columns exists (falling back, and saying so), and starts the samples right after the position columns; the sex default and the sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same assumption and got the same fix. This is the second deliberate deviation from GeneNetwork3's script, after the sample-count guard, and both are commented as such at the code. * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the first sample column are computed, not assumed. * tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb` dropped -- same panel, cM only. * tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six markers come back (the failure mode was five-vs-six columns, not an error message that pointed at the genotype file). * src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards `--scale`; it does, and now falls back, so the request field says what `pos` is in. Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499 markers) and BXD.4 (3 811) are unchanged. 307 tests pass; the same four as before do not.
11 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
11 dayscompute: fit covariates in GEMMA, and share them with R/qtlPjotr
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit.
12 dayscompute/rqtl: fit covariates, by case-attribute name or as numbersPjotr
R/qtl has always been able to fit covariates; the API now passes them through. `covariates` takes one or more specs, each either a case attribute of the group by name -- which the server reads, aligns onto the panel and codes -- or explicit per-sample numbers: {"case_attribute": "Sex"} {"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}} {"name": "age_days", "kind": "numerical", "values": [...]} The coding is the server's job because the phenotype file is numeric: text with 2-20 levels is coded 0..k-1 in sorted order and declared `categorical` (the wrapper dummy-expands more than two levels), numeric text goes through as `numerical`, and one level, more than 20 levels, or an explicit map that misses a value is a 422 asking for a map rather than an invented ordering. An attribute the group does not define is a 422 that lists the ones it has. Missing values become `x`/NA and R/qtl drops those individuals. Everything that decides the model goes into the cache key -- covariate names *and* kinds, since a categorical covariate's dummy expansion is not the same model as a numeric one -- and the response says what happened: Covariate Sex: categorical coded F=0, M=1, 3281 of 3281 samples have a value, 3281 with both the trait and the covariate Measured on BXD-Longevity AgeAtDeath (3281 animals, WGS panel, 21056 markers): LOD 23.100 without the covariate, 23.522 with Sex, same peak (rs30847416, chr 1, 5.0 Mb), ~15s more for the covariate run. * src/rqtl.rs (Covariate, CovariateKind, code_case_attribute, write_covarstruct_file): the coding rules, the extra pheno columns, the covarstruct file, `--addcovar --covarstruct`, and the model's cache key; five unit tests. * src/routes/rqtl.rs (RqtlCovariate): the request shape, resolution through the group's case attributes, duplicate/limit/ambiguity checks (8 covariates at most), and the per-covariate note. * tests/rqtl.rs: a real covariate run on UM-HET3 (894 markers, so the test stays a test) asserting the coding note and the `--addcovar` line, plus the unknown-attribute 422. * README.md, docs/www/ai-instruction.md (4a): how to pass them, and that the note's coding and complete-case count are the things to read.
12 dayscompute: suggest a PheWAS at the peak, and the plot that followsPjotr
A mapping's job is not finished when the rows come back: the next two questions are "what else peaks here?" and "show me". So a group-scoped mapping response now ends with both -- - `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA, last column). A GET that resolves 200, which is what `next` promises; measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's rsm10000005700 -> 12. - a note saying which two columns to plot: chr against LOD (R/qtl) or against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns naming them, so plotting needs no reshaping -- the API does not render images, and saying exactly what to plot is more useful than pretending to. Path-based requests get neither: without a `group` there is no group to PheWAS over, and the note alone would be noise. * src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest p_lrt, with unit tests for both shapes and the empty cases. * src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the request named a group. * docs/www/ai-instruction.md: 4b rewritten around what a reader does with a PheWAS -- where the position comes from (a mapping's `next`, a trait's `locus`, or chr+mb), the parameter table, what a hit contains, and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`, peak-only and published-only limits, several traits on one marker), with the chr 9 example; 4a points at the new forward.
2026-10-19Merge branch 'gemma-cwl-parallel'Pjotr Prins
# Conflicts: # src/gemma.rs # src/rqtl.rs # src/streaming.rs
2026-10-15rqtl: return marker rows positionally, columns in meta.columnsPjotr
Same space-saving shape as the gemma commit: /compute/rqtl's markers are now positional arrays (name, chr, pos, lod, plus any 4-way-cross extra effect columns trailing) instead of one keyed object per row, with the fixed column names in meta.columns. Also fixes tests/gemma.rs, which still asserted the old keyed shape from the gemma change -- missed in that commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15CWL used to run gemmaPjotr Prins
2026-10-14gemma/rqtl: three fixes from the endpoint reviewAlexander_Kabui
- rqtl: per-call unique output filename. The content hash alone was deterministic across processes, so two concurrent identical requests shared one output file; R truncates on open, so one request could read the other's half-written results with no error. The per-call unique (pid + nanos, already computed for the pheno file) now names the output file too. - streaming/gemma/rqtl: append a failure marker to the run log on every error path, mirroring the completion marker. A status poller can now tell a failed run from a still-running one. - rqtl: --scale is now honored. The wrapper selects the geno file's cM/Mb map column from it, and the route rejects values other than mb/cm with 422 instead of silently ignoring them. Both new behaviors are unit-tested; the live Rscript tests pass.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>