| Age | Commit message (Collapse) | Author |
|
Ravanan's reuse key is (script, inputs) and both live in its --store under
RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the
container's. Measured: 13.6 s cold, then 3.6 s for the identical request
after restarting the container, with the store unchanged at 14 files --
Slurm's own spool state is lost and irrelevant to reuse.
The default cache dir is under /tmp, which is neither durable nor visible
inside a container that bind-mounts one host path, so a job cannot read
the interned inputs. The reply now carries a note when the cache dir is
under /tmp, naming the fix, rather than leaving it to be discovered as a
slow or failing run.
docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan
store, scratch, token registry, gemma-wrapper cache, Slurm state) and a
path checklist; the README points at it.
|
|
stale tests
`ccwl compile` emits CWL as YAML (its only other target is dot), so the
new input check must read that shape -- it was parsing JSON and would
have turned a working request into a 502. A hand-written scan of the
`inputs:` block, deliberately not a YAML dependency: ccwl's output is
machine-generated and unvaried, and adding serde_yaml would mean
regenerating the crate list in guix.scm for one small check.
Two tests were stale in ways only the package's own check phase shows:
- tests/auth.rs built AuthToken literals without the new `scheme`/`id`
fields, so the crate's integration tests did not compile;
- test/python/test_envelope.py asserted that /version carries no `next`
links, which stopped being true when every read endpoint gained them
("always offer the next step"). The deployed build fails that test
too, so it is not a regression from this work -- but it did block
`guix build -f guix.scm` at the python-rest-test phase.
Verified with the packaged binary (guix build -f guix.scm, all phases
including both test suites): POST /compute/gemma/ravanan through ravanan
and the Slurm container returned 200 with 636 rows in 13.6 s, on a
request authenticated with a gn4ai_ short token -- the log line naming
user zachs, scheme opaque and token smoke-2.
|
|
cwl/gemma/gemma-loco.scm is read from the deployment's checkout at
request time (GEMMA_RAVANAN_WORKFLOW), not compiled into the binary, so
a checkout and a binary from different commits can meet. When they do,
cwltool answers "Type mismatch for input `kinship_pheno_file'; expected
`File' but got `null'" -- which says nothing about the cause. Found by
pointing the deployed f64a708 binary at the current workflow.
The compiled CWL now has its declared inputs compared with what this
build is about to send, before anything is staged for ravanan: a missing
input or an undeclared one is refused with both lists named, plus the
instruction to take the .scm and the binary from the same commit.
|
|
gemma -gk reads the phenotype file only to learn which individuals are
analysed -- the matrix comes from the genotypes -- so the workflow now
gives the kinship step a constant-valued pheno file carrying just the
trait's missing-data pattern, and gives gwa the real values. Verified
byte-for-byte: -gk on a real trait and on its constant-valued twin
produce identical .cXX.txt files, and trait 2's assoc checksums are
identical whether K came from the constant pheno or the real one.
Consequence, measured on the BXD fixture: a second trait over the same
animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of
+13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices
are identical inputs for both traits. Three keys now: workflow (the
ccwl source), kinship (genotype + snps + lmm + the NA pattern) and
trait (kinship key + the values), so the compiled CWL and the kinship
pheno file are shared and only the trait's pheno file and inputs YAML
are per-trait.
This is also stricter than gemma-wrapper's own key, which ignores the
phenotype entirely and would reuse a K computed for a different set of
animals.
|
|
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and
-gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on
every call and a repeat of the identical request recomputed all 40
steps. Naming them by a hash of the request's inputs (genotype path and
size, snps contents, trait values, lmm) makes an identical request hit
the store instead.
Measured on the BXD fixture, full 20-chromosome LOCO mapping,
single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums.
For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s
warm -- its own cache did not hit on a repeat here -- so the ravanan
path is now the cheaper one for any repeated request and the slightly
dearer one for a first request.
The run log stays per-call: it is what a caller streams by run_id. The
three shared files are written aside and renamed, so two concurrent
identical requests cannot read a half-written file.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
The ravanan path asks for the same variance components as the direct one:
gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes
pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them
into the same note (every chromosome, plus the extremes with vg/ve) and
the same unrounded "variance:" lines in the ?id= run log.
It reads them from the extracted GWA archive rather than from the copy
gemma-wrapper keeps beside it, and that is the point of doing it here:
the copy beside the archive is a 5eb53fa feature, while a ravanan job
runs whatever gemma-wrapper its own Guix manifest resolves, which need
not be that new. The .log.txt has always been inside the archive, and
the archive is unpacked anyway for the chromosome's .assoc.txt -- so
read_variance_components_from_logs labels each log by the "-loco <chr>"
in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt
file name, and sorts chr1..chrX.
Verification, precisely: the CWL compiles with -pve (ccwl compile, checked
by hand), the archive reader is unit-tested (labelling from the command
line, from the file name, numeric ordering, a log without the block
dropped), and a real archive from this deployment does carry the block.
An end-to-end ravanan request could NOT be completed on the dev box, for
reasons that have nothing to do with -pve and are now written down in
docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest
fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set,
since the production process does not set GUIX_PACKAGE_PATH at all, so
this endpoint cannot resolve gemma-wrapper there), its intern of inputs
hard-links and so needs store and inputs on one filesystem ("Invalid
cross-device link"), and the one job that did start sat "Waiting for
lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
|
|
The note reported the range and named the two chromosomes that bounded it,
which is not enough to answer the question the numbers exist for. Under
--loco each pve belongs to a *different* kinship matrix, and GEMMA's
per-chromosome .log.txt files live on the server -- so a caller told "0.31
to 0.95" could not ask which chromosome was which, or compare a single
chromosome against BulkLMM.
The note now lists all twenty, compactly, with the range and the two
extremes keeping their vg/ve:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).
and streaming::append_lines puts the unrounded values in the run log as
variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612
so a ?id= run's /compute/gemma/status stream carries them while the run is
still in flight, rather than only in a reply someone has to keep. Both
measured on BXD::10001 through the dev server before committing.
The values are printed with four significant figures (scientific when the
magnitude needs it), unchanged: rounding variance components to four
decimals was the earlier bug, and these are the same numbers a reader
compares with BulkLMM's h2.
Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote
per-chromosome values, not an average: they are twenty different kinship
matrices") and docs/bulklmm.md's example.
|
|
The GWA step now passes -pve, and the reply gains a note with the
variance components GEMMA computed for the null model. Under --loco
there is one per chromosome, so the note gives their range and names
the two that bound it:
GEMMA -pve, null model, per chromosome (LOCO): pve 0.3118 to 0.9549
over 20 chromosomes -- lowest 8 pve 0.3118 +/- 0.5009 (vg 23.3659, ve
12.3389), highest 7 pve 0.9549 +/- 0.3570 (vg 76.0933, ve 0.8661).
That is BXD::10001 measured, not assumed. pve is the proportion of
variance the kinship explains -- the same quantity as /compute/bulklmm's
h2, per chromosome rather than genome-wide -- which is what the
16627/16769 comparison against BulkLMM needed.
Reading the numbers took three decisions worth stating:
- Four decimals is wrong for variance components. The first live run
reported 'vg 0.0000, ve 0.0000' for a BXD::16769 null fit whose real
components are 3.411e-5 and 3.411e-10 -- both rounded to zero, which
reads as 'no variance' when the split is 1.0. Components are printed
with four significant figures, scientific when the magnitude needs it,
the way GEMMA's own logs do with %g.
- se(pve) is -nan on a boundary, and Rust parses -nan as a NaN f64, so
the numeric parse alone let it through as NaN. Non-finite values are
now 'not available': a boundary estimate has no standard error.
- Twenty chromosomes of values bury the other notes, and every value is
in the .log.txt beside the archive anyway, so the note names the ends.
Requires gemma-wrapper 5eb53fa, which keeps each run's .log.txt beside
its archive instead of only inside the .tar.xz.
Docs: README's /compute/gemma section, SKILL's gemma bullet (compare pve
with h2, never vg with sigma2_e -- pve is scale-free), and
docs/bulklmm.md's 'what would settle it', which now says the plumbing is
in rather than pending.
|
|
Every GEMMA failure the sweep produced read the same way -- "gemma-wrapper
bailed out with pid ... exit 1", sometimes with the Ruby backtrace -- which
says nothing about whether the API or the panel is at fault. The line that
does is GEMMA's, printed in the middle of the wrapper's per-chromosome
chatter:
ERROR: Enforce failed for N>0 in src/fastblas.cpp at line 194 in fast_cblas_dgemm
so `CommandFailed` now carries the last GEMMA diagnostic (an `ERROR`,
`error!`, `****` warning, `Enforce failed` or "not enough" line) rather than
the raw stderr, falling back to the wrapper's own tail when GEMMA said
nothing at all. The wrapper's banner and `.rb:` backtrace lines are skipped:
they are the noise this exists to get past.
Two tests: the complaint is GEMMA's line, not the banner; and a silent
failure still names something.
|
|
A GEMMA row now carries the effect beside the p-value it belongs to:
`… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is
what GeneNetwork's own tables and CSV call the additive effect
(gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half
the difference between the two homozygotes is the usual additive effect;
negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta
is per *maternal* copy while GeneNetwork's published sign is anchored on the
other allele. Two facts that make this worth doing carefully:
* `beta` and `se` are left exactly where GEMMA put them. `se` is what makes
the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both
are GEMMA's own numbers, so replacing them with a derived one would lose
information;
* the derived column goes *before* the p-value, not after it. `p_lrt` must
stay the last column: `gemma::peak_marker` reads `row.last()` for the peak
(and with it the PheWAS and plot suggestions), and the plotters and the
docs all say "the last column is the p-value". Appending would have
silently pointed every peak at an effect size.
The response also says what the three columns mean, and that they describe
*this run* on these samples and this genotype version -- not the `additive`
trait info reports, which came from the published mapping (trait 20010:
this run -0.54, published +2.118, same peak marker).
Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns
are then exactly GEMMA's.
* src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so
/compute/gemma and /compute/gemma/ravanan both get it.
* src/routes/gemma.rs: the note.
* tests: four unit tests (placement, sign and halving, `peak_marker` still
finds the peak from the last column, a table without beta untouched, a
null effect for a row with `NA` beta) and one end-to-end test asserting
the column sits immediately before `p_lrt`, every value is `-beta/2`,
`p_lrt` stays a probability, and the note is there.
* README.md (the response example and what the column means),
docs/www/SKILL.md (how to read and quote it).
286 tests pass; the same four as before do not.
|
|
The shared covariate note promised "5 dummy column(s) named Status.<level>"
on both compute endpoints, but only one of them gets dummies: GEMMA is
handed k-1 columns and R/qtl's wrapper is handed the single level column
and expands it itself. On `/compute/rqtl` the sentence described something
that never appears -- caught by reading a live response on production,
where the note said "5 dummy column(s) named Status." one line above the
Rscript command line that passes no such thing.
The note is now engine-neutral -- `... 90 of 93 samples have a value, 34
with both the trait and the covariate; 6 levels` -- and GEMMA says what it
was handed in its own note, which is the part a reader comparing two runs
wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC,
Status.UTHSC2, ...`. The unit test asserts the level count, and the
end-to-end test asserts the columns note names `Status.` columns twice or
more, so a covariate silently fitted as one ordinal number fails it.
* src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs,
docs/www/ai-instruction.md (the example note).
|
|
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
|
|
`POST /compute/gemma/ravanan` is now gated like the other two: one
workflow at a time, one per caller, `409 compute_busy` naming the run and
linking `/compute/gemma/status/{id}` (ravanan has no status route of its
own). One slot because ravanan already fans its per-chromosome steps out
-- over Slurm, when that backend is selected -- so a second request would
only queue behind the same scratch directory. Its gate is separate from
`/compute/gemma`'s: different runners, different cache directories, so one
of each at once is the overlap `/compute/rqtl` is already allowed.
Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the
slot for the full 5 s, the same caller's second request was refused at
once, and another caller's waited for the slot and then ran.
While testing that, `GemmaError::Io` turned out to print "failed to run
gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced
"failed to run gemma-wrapper: No such file or directory", which sent me
looking at the wrong binary. Spawn failures are now their own variant
naming the command, and plain file I/O says "I/O error: …".
* src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()`
exposes the slot count, and a test pins all three endpoints' counts
(gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT).
* src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with
`status = "/compute/gemma/status"`.
* src/gemma.rs (Spawn): names the command; the four run_streaming call
sites use it; unit test for both messages.
* README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the
slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
|
|
A mapping's job is not finished when the rows come back: the next two
questions are "what else peaks here?" and "show me". So a group-scoped
mapping response now ends with both --
- `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak
being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA,
last column). A GET that resolves 200, which is what `next` promises;
measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's
rsm10000005700 -> 12.
- a note saying which two columns to plot: chr against LOD (R/qtl) or
against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns
naming them, so plotting needs no reshaping -- the API does not render
images, and saying exactly what to plot is more useful than pretending
to.
Path-based requests get neither: without a `group` there is no group to
PheWAS over, and the note alone would be noise.
* src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest
p_lrt, with unit tests for both shapes and the empty cases.
* src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the
request named a group.
* docs/www/ai-instruction.md: 4b rewritten around what a reader does with
a PheWAS -- where the position comes from (a mapping's `next`, a
trait's `locus`, or chr+mb), the parameter table, what a hit contains,
and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`,
peak-only and published-only limits, several traits on one marker),
with the chr 9 example; 4a points at the new forward.
|
|
The cache dir was keyed by chromosome only and shared across requests,
so a request with different inputs (another trait, or another snps or
genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir.
gemma-wrapper names those archives by a content hash, so the workflow's
single-File glob output binding then matched more than one path and the
step failed -- silently, since ravanan's user-error only logs when
%log-level is set, which never happens inside a generated job script.
Key the cache root by a hash of the input set (genotype path and size,
snps contents, trait values, lmm) as well as by chromosome, so repeated
identical requests still hit gemma-wrapper's cache while different
input sets cannot see each other's archives. chromosomes_in reads the
snps file, so it now takes the already-read contents (chromosomes_from)
to avoid reading it twice.
cwl/README.md: describe the two-part keying, and note that a node needs
a writable HOME -- ravanan forwards the submitting process's HOME into
the job (as sbatch would) and gemma-wrapper writes its lock file there.
|
|
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system. Add a
RavananBatch setting:
- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
--slurm-jwt (a *file*, which is what ravanan reads) and, when set,
--slurm-partition/--slurm-nice; a slurm selection without an endpoint
or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
the job scratch space live on the same shared filesystem (with slurm
that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.
The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config. docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.
Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
|
|
Per Arun's request, express the --loco kinship/GWA workflow as a
ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm,
compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`.
ccwl inlines the two command definitions (kinship, gwa) directly into
the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer
needed as separate files.
The two scatter steps (kinship over [chr, cache_dir], gwa over [chr,
cache_dir, kinship_json], both dotproduct) and the gwa_json/archive
outputs are unchanged in substance -- verified by compiling and
running the generated gemma-loco.cwl through ravanan against the BXD
fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm
(~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144:
chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it
replaces.
|
|
--cache-dir "." pointed every scattered job at its own hermetic CWL
working directory (fresh under ravanan's --store every run), so
gemma-wrapper's cache was never actually reused across requests.
Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>,
one persistent directory per chromosome -- not a single shared
directory, because gemma-wrapper's cache-hit hash does not vary with
--chromosomes, so sharing one directory would let one chromosome's
job silently reuse another's cached result.
Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since
kinship and gwa for the same chromosome now point at the same
persistent cache_dir, gemma-wrapper finds the kinship step's archive
there directly, no staging needed.
Confirmed via a real integration run (BXD fixtures) that gemma-wrapper
itself unconditionally re-executes GEMMA regardless of whether a
matching cached archive already exists -- --cache-dir is only used to
hand results between gemma-wrapper's own pipeline stages within one
request, not to skip recomputation across requests. So this fixes a
real latent bug (archive-glob collisions once a chromosome's cache
directory holds more than one request's output) but does not, on its
own, speed up repeated identical requests; verified the direct
/compute/gemma path shows the same lack of cross-request speedup.
|
|
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper
invocation covering all chromosomes, relying on gemma-wrapper's own
internal GNU-parallel fan-out) with one gemma-wrapper invocation PER
CHROMOSOME, restricted via its existing --chromosomes/--no-parallel
flags and scattered by ravanan itself (ScatterFeatureRequirement,
scatterMethod: dotproduct). gemma-wrapper still does the correct
per-chromosome LOCO math (exclude this chromosome from kinship, test
only its markers) against the full genotype/SNP files -- only which
chromosome(s) it loops over changes. compute_ravanan now computes the
chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own
get_chromosomes) and passes it as the workflow's scatter input, and
collates the resulting per-chromosome gwa_json/archive array outputs.
Measured 16.7s wall-clock end-to-end against the BXD fixtures (20
chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for
the previous sequential ravanan workflow -- confirmed byte-for-byte
identical output (rs31443144's exact values, 7320 total rows).
Also fixes three real, previously-unexercised bugs in compute_ravanan
found getting this to run at all (POST /compute/gemma/ravanan had no
automated test until this commit):
- missing --guix-manifest (ravanan requires it; added
gemma_ravanan_manifest config key and cwl/gemma/manifest.scm)
- missing --store files/logs/data subdirectories (ravanan does not
create them itself)
- --output-directory dropped entirely: every chromosome's gwa archive
shares gemma-wrapper's own content-hash basename (the hash doesn't
factor in --chromosomes), so ravanan's optional flat-symlink step
collides across them; unneeded anyway since results are read
straight from ravanan's --store via its stdout JSON record.
Also found and worked around a ravanan scheduler bug: a step scattered
over exactly one input needs an explicit scatterMethod even though CWL
only requires one for multiple scattered inputs -- ravanan has no
fallback case for a missing one and silently misbehaves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Cuts repeated key names out of large /compute/gemma(/ravanan) responses:
data is now a list of positional row arrays instead of one keyed object
per row, with the column names moved to a new meta.columns field shared
with any other endpoint that wants the same shape.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
- rqtl: per-call unique output filename. The content hash alone was
deterministic across processes, so two concurrent identical
requests shared one output file; R truncates on open, so one
request could read the other's half-written results with no error.
The per-call unique (pid + nanos, already computed for the pheno
file) now names the output file too.
- streaming/gemma/rqtl: append a failure marker to the run log on
every error path, mirroring the completion marker. A status
poller can now tell a failed run from a still-running one.
- rqtl: --scale is now honored. The wrapper selects the geno file's
cM/Mb map column from it, and the route rejects values other
than mb/cm with 422 instead of silently ignoring them.
Both new behaviors are unit-tested; the live Rscript tests pass.
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
peak -> peek: the status endpoint's byte-offset query parameter and
everywhere it's documented (code comments, the GemmaStatusParams
field, tests, README). GeneNetwork3's own equivalent
(gn3/api/streaming.py) is spelled "peak"; noted where this now
deliberately differs, not touching the unrelated "peak marker" wording
elsewhere in the README.
Run id uniqueness: the log file is now created with
OpenOptions::create_new(true) instead of a plain truncating write, so
it fails atomically (no separate check-then-write race) if a log
already exists for that id -- another run still in progress, or a
finished one never polled/cleaned up. Added GemmaError::RunIdInUse,
mapped to 409 Conflict in the route, caught before any subprocess
runs. Previously a reused id would have silently truncated whatever
was already there.
Verified: the new compute_gemma_rejects_a_run_id_already_in_use test
pre-seeds a log file for an id and confirms the POST is refused (this
needs no gemma-wrapper, since the check runs before any subprocess
call). cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
/compute/gemma can genuinely take minutes (see the README's ~100K
marker caveat), and until now gave no visibility into a run already
in flight. Same shape GeneNetwork2/3 use for R/qtl
(gn3/computations/streaming.py, gn3/api/streaming.py): the caller
picks a run id, passes it as POST /compute/gemma?id=<id>, and polls
GET /compute/gemma/status/{id}?peak=<offset> on a second connection
while the first is still blocked.
What's actually tailed is gemma-wrapper's stderr, not stdout: under
--json, all of its human-readable progress (banner, chromosome list,
"Executing: ...", per-step info) goes to stderr, and stdout carries
only the one final JSON record, printed once at the very end -- so
there is nothing to stream on stdout. run_streaming() replaces the
old buffered Command::output() call with one that reads stderr line
by line as the process runs, appending each line to a log file if one
was given, across both the kinship and GWA steps into a single
continuous log; stdout is still read in full (unchanged, one JSON
blob). A completion marker ("GN4: computation complete") is appended
once the whole request finishes, the same convention GeneNetwork3
uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this
project's own name.
run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever
joined into a path, on both the POST's ?id= and the status endpoint's
path segment -- GeneNetwork3's own /api/stream/{run_id} does no such
check on its equivalent id, which is not knowingly replicated here.
Verified end to end, not just unit tested: the integration test spawns
the POST on its own task and polls the status endpoint from this one
while the ~19s real computation is still running, asserting it
actually observed non-empty progress lines mid-run (not merely after
the fact), then checks the finished log for both real gemma-wrapper
progress text and the completion marker. cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
commands in the response notes
README: added /compute/gemma to the endpoint table and a dedicated
section with a real example response, and the explicit caveat the
user asked for -- not feasible past roughly 100,000 markers at this
point (synchronous, sequential, no parallel LOCO jobs; the BXD
fixtures, 7,320 markers, already take ~19s, and real datasets go up to
~1M markers).
notes: the response now carries the exact gemma-wrapper invocations
used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk",
"GWA: gemma-wrapper ... -lmm 9"), not just a generic description.
kinship_args/gwa_args build the argument list once and feed both the
actual subprocess call and the note text, so the two can't drift
apart. Added unit tests asserting -gk and -lmm never appear together,
and extended the real end-to-end test to check both notes' shape.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run, with the notes field inspected directly against the actual
command lines used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile
(gn/runner/gemma.scm) both always use -lmm 9 in their working --loco
invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the
resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other
than stock GEMMA's 1-4 scale, not independently verified here, but
consistent with what both of those codebases rely on in production.
Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4,
9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so
they're still rejected rather than guessed at.
Updated the real end-to-end test to assert on p_lrt (what lmm=9
actually produces) instead of p_wald, and moved the
out-of-range-lmm test from 9 (now valid) to 5.
Verified: cargo test passes in full, including the real gemma-wrapper/
GEMMA run against the BXD fixtures with the new default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|