| Age | Commit message (Collapse) | Author |
|
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply
gains meta.permutation {n, seed, significant, suggestive, values} -- the
genome-wide maximum of every permutation, so a histogram needs no second
call. `data` stays the positional rows array on purpose: /plot/rqtl posts
it straight back and the monitor counts its rows, so the null went beside
the rows rather than around them (a generic Envelope::with_meta/Meta::extra,
skip_serializing_if, so no other endpoint's response changes).
Thresholds come from R/qtl2's scan1perm over the same genotype
probabilities, kinship and covariates as the scan, with chr_lengths from
the map -- without those it derives chromosome lengths from marker
positions and the numbers stop being comparable with R/qtl's. The
permutation CSV is written in R/qtl's two-column shape so one parser and
one percentile routine serve both engines: PermutationResult::from_values
is now shared with /compute/rqtl instead of duplicated.
Semantics, each with the note that says it:
* scan1perm shuffles phenotypes across individuals -- right for an
outbred cross, only approximate for a selfing RIL panel, where qtl2perm
is the correct package (not installed here, and not packaged);
* perm_strata takes a case attribute name (sex, Epoch, ...) and permutes
within it, because shuffling across sexes inflates the null;
* fewer than 100 permutations gets a note that the 0.05 quantile is a
quantile of too few values; more than 10 000 is refused, since each
permutation is a whole genome scan;
* a seed is reported always -- without one, two identical requests would
give different thresholds.
Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO
kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s,
1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds
converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at
1000 on the raw probe) and are reproducible: the same seed gives the same
numbers on 1 and 4 cores.
Four things that only surfaced by running it, all commented where they
bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so
opt$perm_out was always NULL and the file landed somewhere the caller did
not look); perm_strata must be a named character vector, not a factor;
the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the
permutation index first, and a first-field parser reads that as the LOD);
and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not
0.67.
bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging
meta.permutation) so a deploy that loses the R path shows up as a red
check rather than a surprise.
|
|
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and
none of them could be scanned. GeneNetwork3's wrapper (copied verbatim
here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts
the samples at a hardcoded column 5. On a cM-only file that is one column
too far, so the genotype block loses its first individual and the whole scan
dies in `do.call(rbind, ...)` with "number of columns of result is not a
multiple of vector length" -- a message that says nothing about the cause.
The wrapper now picks the map column from whichever of the requested scale
and the file's own columns exists (falling back, and saying so), and starts
the samples right after the position columns; the sex default and the
sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same
assumption and got the same fix.
This is the second deliberate deviation from GeneNetwork3's script, after
the sample-count guard, and both are commented as such at the code.
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the
first sample column are computed, not assumed.
* tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb`
dropped -- same panel, cM only.
* tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six
markers come back (the failure mode was five-vs-six columns, not an
error message that pointed at the genotype file).
* src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards
`--scale`; it does, and now falls back, so the request field says what
`pos` is in.
Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499
markers) and BXD.4 (3 811) are unchanged.
307 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
|
|
R/qtl has always been able to fit covariates; the API now passes them
through. `covariates` takes one or more specs, each either a case
attribute of the group by name -- which the server reads, aligns onto the
panel and codes -- or explicit per-sample numbers:
{"case_attribute": "Sex"}
{"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}}
{"name": "age_days", "kind": "numerical", "values": [...]}
The coding is the server's job because the phenotype file is numeric:
text with 2-20 levels is coded 0..k-1 in sorted order and declared
`categorical` (the wrapper dummy-expands more than two levels), numeric
text goes through as `numerical`, and one level, more than 20 levels, or
an explicit map that misses a value is a 422 asking for a map rather than
an invented ordering. An attribute the group does not define is a 422
that lists the ones it has. Missing values become `x`/NA and R/qtl drops
those individuals.
Everything that decides the model goes into the cache key -- covariate
names *and* kinds, since a categorical covariate's dummy expansion is not
the same model as a numeric one -- and the response says what happened:
Covariate Sex: categorical coded F=0, M=1, 3281 of 3281 samples have a
value, 3281 with both the trait and the covariate
Measured on BXD-Longevity AgeAtDeath (3281 animals, WGS panel, 21056
markers): LOD 23.100 without the covariate, 23.522 with Sex, same peak
(rs30847416, chr 1, 5.0 Mb), ~15s more for the covariate run.
* src/rqtl.rs (Covariate, CovariateKind, code_case_attribute,
write_covarstruct_file): the coding rules, the extra pheno columns, the
covarstruct file, `--addcovar --covarstruct`, and the model's cache key;
five unit tests.
* src/routes/rqtl.rs (RqtlCovariate): the request shape, resolution
through the group's case attributes, duplicate/limit/ambiguity checks
(8 covariates at most), and the per-covariate note.
* tests/rqtl.rs: a real covariate run on UM-HET3 (894 markers, so the
test stays a test) asserting the coding note and the `--addcovar` line,
plus the unknown-attribute 422.
* README.md, docs/www/ai-instruction.md (4a): how to pass them, and that
the note's coding and complete-case count are the things to read.
|
|
A mapping's job is not finished when the rows come back: the next two
questions are "what else peaks here?" and "show me". So a group-scoped
mapping response now ends with both --
- `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak
being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA,
last column). A GET that resolves 200, which is what `next` promises;
measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's
rsm10000005700 -> 12.
- a note saying which two columns to plot: chr against LOD (R/qtl) or
against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns
naming them, so plotting needs no reshaping -- the API does not render
images, and saying exactly what to plot is more useful than pretending
to.
Path-based requests get neither: without a `group` there is no group to
PheWAS over, and the note alone would be noise.
* src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest
p_lrt, with unit tests for both shapes and the empty cases.
* src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the
request named a group.
* docs/www/ai-instruction.md: 4b rewritten around what a reader does with
a PheWAS -- where the position comes from (a mapping's `next`, a
trait's `locus`, or chr+mb), the parameter table, what a hit contains,
and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`,
peak-only and published-only limits, several traits on one marker),
with the chr 9 example; 4a points at the new forward.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
Same space-saving shape as the gemma commit: /compute/rqtl's markers
are now positional arrays (name, chr, pos, lod, plus any 4-way-cross
extra effect columns trailing) instead of one keyed object per row,
with the fixed column names in meta.columns.
Also fixes tests/gemma.rs, which still asserted the old keyed shape
from the gemma change -- missed in that commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
- rqtl: per-call unique output filename. The content hash alone was
deterministic across processes, so two concurrent identical
requests shared one output file; R truncates on open, so one
request could read the other's half-written results with no error.
The per-call unique (pid + nanos, already computed for the pheno
file) now names the output file too.
- streaming/gemma/rqtl: append a failure marker to the run log on
every error path, mirroring the completion marker. A status
poller can now tell a failed run from a still-running one.
- rqtl: --scale is now honored. The wrapper selects the geno file's
cM/Mb map column from it, and the route rejects values other
than mb/cm with 422 instead of silently ignoring them.
Both new behaviors are unit-tested; the live Rscript tests pass.
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|