| Age | Commit message (Collapse) | Author |
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
cannot do
Two gaps, both invisible until a real request ran against a cluster.
The endpoint read only genotype_file, snps_file and trait_values from the
request; group, genotype_version and samples were accepted by the shared
request type and silently dropped. A /compute/gemma-shaped request
(group + samples) therefore arrived as empty paths and empty values and
died with "I/O error: No such file or directory (os error 2)" from
reading "". Running it exposed the second gap in the same breath: the
gate sized the request from trait_values.len(), so a samples-only request
was sized 0 and took no concurrency slot at all.
resolve_inputs is now one function in routes/gemma.rs, called by both
handlers: upload materialised into the pair the engine reads, basket
resolved to named values, group prepared with the parent/F1 and
empty-genotype rules, named samples aligned by name onto whatever
individuals the file has, positional values checked against the panel
size. That block was ~250 lines of duplicated rules inside compute_gemma,
which is how it drifted in the first place; the direct runner now calls
the same function and keeps only its covariates, normalization and gate.
The ravanan reply also carries the input notes now (which genotype
version, how many values aligned, what was dropped), so a run says what it
mapped as well as where the peak is.
Refused rather than ignored, because the CWL workflow has no step for
either: covariates (422 covariates_not_supported) and normalize (422
normalization_not_supported), each naming /compute/gemma as the endpoint
that can do it. A new test asserts both refusals happen before any job is
submitted, so it needs no ravanan, ccwl or cluster.
Verified against the Slurm container from gn-machines: group: BXD became
40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's
chromosomes), all completed, 21 055 rows in ~3.5 min. README and
docs/ravanan-slurm.md record the four requirements that run turned up,
three of them not obvious: ccwl on PATH, a ravanan build with the
--slurm-api-endpoint fix, the *server's* HOME writable by the job user
inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and
GUIX_PACKAGE_PATH for the manifest.
|
|
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the
gate gives GEMMA one slot -- but "not supported here" is not the same as
"not available". The refusal now says both:
1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come
from rotations that keep the kinship (R/qtl's nperm does not, and is
anti-conservative on a structured panel);
2. run GEMMA's own permutation locally. The reply's notes already name the
bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or
`gemma -lmm 9 --permutate` on those inputs reproduces exactly the null
GEMMA would have used -- hours instead of seconds, which is the honest
price of that particular null.
docs/bulklmm.md gains the table this belongs in: three ways to get a
threshold, ordered by how honest the null is for a structured panel
(rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's
nperm as a sanity check only). SKILL.md says the same in two lines.
tests/gemma.rs now asserts both halves of the refusal, and the earlier
assertion on /compute/rqtl -- which stopped being true when the refusal was
rewritten -- is replaced rather than left passing by accident.
|
|
Asking whether the singular matrices could be fixed turned out to have two
answers, one of them a real bug on our side.
The bug: a panel can name individuals whose genotype row is empty, and an
empty row makes the kinship matrix singular. GEMMA then reports
"eigenvalues close to zero" and dies in LAPACK with
`LUDecomp ... input domain error` -- a message about linear algebra for a
problem that is a missing genotype. Measured on this deployment's own files:
BXD-AE.4 136 of 344 individuals have no genotypes
MDP 93 of 178
BXD-Heart-Metals.4 200 of 481
HET3-ITP.3 2867 of 2867 (the whole file is empty)
Dropping them makes those panels map: BXD-AE.4's GWA step goes from a
failure to 3.2 s, and MDP's kinship step from a failure to success. The
individuals contribute nothing to a scan either way, so nothing is lost;
`notes` counts them, and the "Genotype:" line no longer blames parents/F1s
for them (`prepared.excluded` is "panel minus kept", which now includes
both, so the two counts are separated).
Not the cause, but worth naming: `AXB` fails earlier still, on its
annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ...
ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a
pre-flight check of the annotation (report the offending marker) rather
than a GEMMA failure, and is not in this commit.
Considered and rejected: a ridge on K (`K + epsilon*I`) would work
mathematically but GEMMA has no option for it and the wrapper caches K by
content hash, so it would mean patching the wrapper's cached matrix; `-gk 2`
and dropping markers do not add rank; and the non-LOCO path is the one that
does not run at all here.
* src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam
file marking the individuals with any genotype at all.
* src/routes/gemma.rs: filter `keep` by it before alignment (so the trait
values drop with them), and say so in `notes`.
* tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4
maps, the note names the drop and the reason, and the genotype line no
longer claims 344 individuals.
* README.md: the behaviour and why it exists, under /compute/gemma.
307 tests pass; the same four as before do not.
|
|
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a
message about LU decomposition: its values are depths (`40`, `41`) with a
few two-library entries (`36+39L`), so the attribute was coded as **twenty
categories** and GEMMA died on the resulting dummies. The failure named
GEMMA, which is the wrong place to look.
`code_case_attribute` now refuses that shape at the covariate step, naming
the values and what to send instead:
422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes
numbers with number-like text (14 of 19 values look numeric, but e.g.
'36+30L' does not), so coding it as categories would fit a quantity as a
factor -- and GEMMA fails on the result. Pass an explicit `map` giving
each of those a number (there are 5 such value(s): ...), or send the
numbers inline.
The rule is deliberately narrow: an odd level with *no* digits (`1, 2,
unknown`) is still a level and still coded, so nothing that worked stops
working; only dirty numbers are caught.
Also, the skill's covariate coverage -- which the review above found thin:
* when a covariate is worth trying (metadata that could confound: sex, age,
epoch, batch, sequencing depth), and the covariate step in the session
outline;
* read `.data.descriptions` before choosing one -- they are the only place
that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup --
and the endpoint table now says so;
* read the covariate line in `notes` for the coding and the complete-case
n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being
two levels in BXD;
* the categorical-vs-numeric rule measured rather than asserted: `RRID`
refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value
trap written up with the `map` workaround;
* GEMMA's dummy columns, the column it drops when a level has no fitted
individual, and that R/qtl's note has no such line; missing values
dropping individuals; the 8-covariate cap and the `group` requirement.
* src/covariates.rs: the refusal, with a unit test for both sides of it;
* tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure;
* docs/www/SKILL.md: the covariate section, the session line, and the
case-attributes rows in the endpoint table.
300 tests pass; the same four as before do not.
|
|
A GEMMA row now carries the effect beside the p-value it belongs to:
`… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is
what GeneNetwork's own tables and CSV call the additive effect
(gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half
the difference between the two homozygotes is the usual additive effect;
negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta
is per *maternal* copy while GeneNetwork's published sign is anchored on the
other allele. Two facts that make this worth doing carefully:
* `beta` and `se` are left exactly where GEMMA put them. `se` is what makes
the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both
are GEMMA's own numbers, so replacing them with a derived one would lose
information;
* the derived column goes *before* the p-value, not after it. `p_lrt` must
stay the last column: `gemma::peak_marker` reads `row.last()` for the peak
(and with it the PheWAS and plot suggestions), and the plotters and the
docs all say "the last column is the p-value". Appending would have
silently pointed every peak at an effect size.
The response also says what the three columns mean, and that they describe
*this run* on these samples and this genotype version -- not the `additive`
trait info reports, which came from the published mapping (trait 20010:
this run -0.54, published +2.118, same peak marker).
Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns
are then exactly GEMMA's.
* src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so
/compute/gemma and /compute/gemma/ravanan both get it.
* src/routes/gemma.rs: the note.
* tests: four unit tests (placement, sign and halving, `peak_marker` still
finds the peak from the last column, a table without beta untouched, a
null effect for a row with `NA` beta) and one end-to-end test asserting
the column sits immediately before `p_lrt`, every value is `-beta/2`,
`p_lrt` stays a probability, and the note is there.
* README.md (the response example and what the column means),
docs/www/SKILL.md (how to read and quote it).
286 tests pass; the same four as before do not.
|
|
Two GEMMA runs over one BXD panel drew genuinely different pictures --
traces differ on all 20 chromosomes, peaks -log10 p 5.48 at chr8 97.0 Mb
against 5.43 at chr1 175.6 Mb -- and both came out headed
**"trait gemma: GEMMA association along the genome"**, because GEMMA's
request had no trait label at all: `/compute/gemma` fell back to
`<group>::gemma`, and `/plot/gemma` fell back to "gemma". The plot of each
trait was told apart by its annotation and nothing else.
`trait_name` is now an optional field on `GemmaRequest`, used for the
`Trait: ...` note (first in `notes`, as on `/compute/rqtl`), for the PheWAS
and R/qtl suggestions, and for the `/plot/gemma?trait=` suggestion -- so a
posted response titles its own plot, and `?trait=` still overrides.
Without it nothing changes: the label stays `<group>::gemma`.
* src/routes/gemma.rs: the field, the label resolution, the note, the
plot href.
* tests/gemma.rs: the covariate run names its trait and asserts the note is
first; tests/plot.rs: a GEMMA plot titles itself from the note, and an
explicit `?trait=` wins over it.
* README.md (`/compute/gemma`, and the plot section), docs/www/SKILL.md
(the GEMMA options list, and what the plot title comes from).
281 tests pass; the same four as before do not.
|
|
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:
- every GEMMA run goes through **one shared cache directory**
(`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
kinship archives of hundreds of megabytes by content hash. Two runs at
once are two writers over the same archives, and a half-extracted kinship
another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
individuals write 207 MB of kinship per chromosome.
So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.
Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.
* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
(50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
the test that pinned one number now pins both and that each endpoint is
wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
the two thresholds and the cache-directory reason for the lower one.
277 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
The shared covariate note promised "5 dummy column(s) named Status.<level>"
on both compute endpoints, but only one of them gets dummies: GEMMA is
handed k-1 columns and R/qtl's wrapper is handed the single level column
and expands it itself. On `/compute/rqtl` the sentence described something
that never appears -- caught by reading a live response on production,
where the note said "5 dummy column(s) named Status." one line above the
Rscript command line that passes no such thing.
The note is now engine-neutral -- `... 90 of 93 samples have a value, 34
with both the trait and the covariate; 6 levels` -- and GEMMA says what it
was handed in its own note, which is the part a reader comparing two runs
wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC,
Status.UTHSC2, ...`. The unit test asserts the level count, and the
end-to-end test asserts the columns note names `Status.` columns twice or
more, so a covariate silently fitted as one ordinal number fails it.
* src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs,
docs/www/ai-instruction.md (the example note).
|
|
`gemma-wrapper --permutate n` repeats the *whole* LOCO association scan
per permutation -- it shuffles the phenotype file and starts over -- so a
thousand permutations is thousands of full scans: hours to days on a real
panel, against the single slot `/compute/gemma` has. That is not a test
this deployment carries, and an assistant asked for significance
thresholds will ask for it, so the answer has to be an explicit one.
Until now the request type simply had no such field and serde ignores
unknown keys, which is the worst of the three outcomes: the caller gets a
200, no thresholds, and no indication that what it asked for was dropped.
`nperm`, `permutations` and `permutate` are now named fields and refused
with `422 permutations_not_supported`, naming the cost, pointing at
`POST /compute/rqtl` with `nperm` (the endpoint that does have the test,
over the same panel and values) and at the `actions` member every
successful GEMMA response already carries, pre-filled with exactly that
call. Zero keeps its usual meaning -- off -- so a client sharing one body
shape across the two endpoints still works.
* src/routes/gemma.rs: the three fields, the refusal before anything
expensive runs, and the response note now saying permutations are
refused rather than merely absent.
* tests/gemma.rs: `compute_gemma_refuses_permutations` pins all three
spellings, the code, and that `nperm: 0` is not a request for any. It
takes the compute lock: the gate answers before the handler does, so a
concurrent job in the same test binary came back 409 instead.
* README.md, docs/www/ai-instruction.md: what to do instead -- judge a
peak against the smallest `p_lrt`, and get thresholds from R/qtl.
273 tests pass; the same four as before do not.
|
|
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.
Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
|
|
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper
invocation covering all chromosomes, relying on gemma-wrapper's own
internal GNU-parallel fan-out) with one gemma-wrapper invocation PER
CHROMOSOME, restricted via its existing --chromosomes/--no-parallel
flags and scattered by ravanan itself (ScatterFeatureRequirement,
scatterMethod: dotproduct). gemma-wrapper still does the correct
per-chromosome LOCO math (exclude this chromosome from kinship, test
only its markers) against the full genotype/SNP files -- only which
chromosome(s) it loops over changes. compute_ravanan now computes the
chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own
get_chromosomes) and passes it as the workflow's scatter input, and
collates the resulting per-chromosome gwa_json/archive array outputs.
Measured 16.7s wall-clock end-to-end against the BXD fixtures (20
chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for
the previous sequential ravanan workflow -- confirmed byte-for-byte
identical output (rs31443144's exact values, 7320 total rows).
Also fixes three real, previously-unexercised bugs in compute_ravanan
found getting this to run at all (POST /compute/gemma/ravanan had no
automated test until this commit):
- missing --guix-manifest (ravanan requires it; added
gemma_ravanan_manifest config key and cwl/gemma/manifest.scm)
- missing --store files/logs/data subdirectories (ravanan does not
create them itself)
- --output-directory dropped entirely: every chromosome's gwa archive
shares gemma-wrapper's own content-hash basename (the hash doesn't
factor in --chromosomes), so ravanan's optional flat-symlink step
collides across them; unneeded anyway since results are read
straight from ravanan's --store via its stdout JSON record.
Also found and worked around a ravanan scheduler bug: a step scattered
over exactly one input needs an explicit scatterMethod even though CWL
only requires one for multiple scattered inputs -- ravanan has no
fallback case for a missing one and silently misbehaves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
So a caller of /compute/gemma(/ravanan) or /compute/rqtl doesn't need
to count data's rows or columns.len() itself. with_columns now takes
the row count too and derives num_cols from the column list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Same space-saving shape as the gemma commit: /compute/rqtl's markers
are now positional arrays (name, chr, pos, lod, plus any 4-way-cross
extra effect columns trailing) instead of one keyed object per row,
with the fixed column names in meta.columns.
Also fixes tests/gemma.rs, which still asserted the old keyed shape
from the gemma change -- missed in that commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
peak -> peek: the status endpoint's byte-offset query parameter and
everywhere it's documented (code comments, the GemmaStatusParams
field, tests, README). GeneNetwork3's own equivalent
(gn3/api/streaming.py) is spelled "peak"; noted where this now
deliberately differs, not touching the unrelated "peak marker" wording
elsewhere in the README.
Run id uniqueness: the log file is now created with
OpenOptions::create_new(true) instead of a plain truncating write, so
it fails atomically (no separate check-then-write race) if a log
already exists for that id -- another run still in progress, or a
finished one never polled/cleaned up. Added GemmaError::RunIdInUse,
mapped to 409 Conflict in the route, caught before any subprocess
runs. Previously a reused id would have silently truncated whatever
was already there.
Verified: the new compute_gemma_rejects_a_run_id_already_in_use test
pre-seeds a log file for an id and confirms the POST is refused (this
needs no gemma-wrapper, since the check runs before any subprocess
call). cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
/compute/gemma can genuinely take minutes (see the README's ~100K
marker caveat), and until now gave no visibility into a run already
in flight. Same shape GeneNetwork2/3 use for R/qtl
(gn3/computations/streaming.py, gn3/api/streaming.py): the caller
picks a run id, passes it as POST /compute/gemma?id=<id>, and polls
GET /compute/gemma/status/{id}?peak=<offset> on a second connection
while the first is still blocked.
What's actually tailed is gemma-wrapper's stderr, not stdout: under
--json, all of its human-readable progress (banner, chromosome list,
"Executing: ...", per-step info) goes to stderr, and stdout carries
only the one final JSON record, printed once at the very end -- so
there is nothing to stream on stdout. run_streaming() replaces the
old buffered Command::output() call with one that reads stderr line
by line as the process runs, appending each line to a log file if one
was given, across both the kinship and GWA steps into a single
continuous log; stdout is still read in full (unchanged, one JSON
blob). A completion marker ("GN4: computation complete") is appended
once the whole request finishes, the same convention GeneNetwork3
uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this
project's own name.
run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever
joined into a path, on both the POST's ?id= and the status endpoint's
path segment -- GeneNetwork3's own /api/stream/{run_id} does no such
check on its equivalent id, which is not knowingly replicated here.
Verified end to end, not just unit tested: the integration test spawns
the POST on its own task and polls the status endpoint from this one
while the ~19s real computation is still running, asserting it
actually observed non-empty progress lines mid-run (not merely after
the fact), then checks the finished log for both real gemma-wrapper
progress text and the completion marker. cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
commands in the response notes
README: added /compute/gemma to the endpoint table and a dedicated
section with a real example response, and the explicit caveat the
user asked for -- not feasible past roughly 100,000 markers at this
point (synchronous, sequential, no parallel LOCO jobs; the BXD
fixtures, 7,320 markers, already take ~19s, and real datasets go up to
~1M markers).
notes: the response now carries the exact gemma-wrapper invocations
used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk",
"GWA: gemma-wrapper ... -lmm 9"), not just a generic description.
kinship_args/gwa_args build the argument list once and feed both the
actual subprocess call and the note text, so the two can't drift
apart. Added unit tests asserting -gk and -lmm never appear together,
and extended the real end-to-end test to check both notes' shape.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run, with the notes field inspected directly against the actual
command lines used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile
(gn/runner/gemma.scm) both always use -lmm 9 in their working --loco
invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the
resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other
than stock GEMMA's 1-4 scale, not independently verified here, but
consistent with what both of those codebases rely on in production.
Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4,
9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so
they're still rejected rather than guessed at.
Updated the real end-to-end test to assert on p_lrt (what lmm=9
actually produces) instead of p_wald, and moved the
out-of-range-lmm test from 9 (now valid) to 5.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run against the BXD fixtures with the new default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|