about summary refs log tree commit diff
path: root/src/gemma.rs
AgeCommit message (Collapse)Author
4 dayscache: say when ravanan's cache cannot survive a restart, and document what doesPjotr Prins
Ravanan's reuse key is (script, inputs) and both live in its --store under RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the container's. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, with the store unchanged at 14 files -- Slurm's own spool state is lost and irrelevant to reuse. The default cache dir is under /tmp, which is neither durable nor visible inside a container that bind-mounts one host path, so a job cannot read the interned inputs. The reply now carries a note when the cache dir is under /tmp, naming the fix, rather than leaving it to be discovered as a slow or failing run. docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan store, scratch, token registry, gemma-wrapper cache, Slurm state) and a path checklist; the README points at it.
4 daysgemma: read ccwl's YAML when checking the workflow's inputs, and fix two ↵Pjotr Prins
stale tests `ccwl compile` emits CWL as YAML (its only other target is dot), so the new input check must read that shape -- it was parsing JSON and would have turned a working request into a 502. A hand-written scan of the `inputs:` block, deliberately not a YAML dependency: ccwl's output is machine-generated and unvaried, and adding serde_yaml would mean regenerating the crate list in guix.scm for one small check. Two tests were stale in ways only the package's own check phase shows: - tests/auth.rs built AuthToken literals without the new `scheme`/`id` fields, so the crate's integration tests did not compile; - test/python/test_envelope.py asserted that /version carries no `next` links, which stopped being true when every read endpoint gained them ("always offer the next step"). The deployed build fails that test too, so it is not a regression from this work -- but it did block `guix build -f guix.scm` at the python-rest-test phase. Verified with the packaged binary (guix build -f guix.scm, all phases including both test suites): POST /compute/gemma/ravanan through ravanan and the Slurm container returned 200 with 636 rows in 13.6 s, on a request authenticated with a gn4ai_ short token -- the log line naming user zachs, scheme opaque and token smoke-2.
4 daysgemma: refuse a workflow from another commit, by namePjotr Prins
cwl/gemma/gemma-loco.scm is read from the deployment's checkout at request time (GEMMA_RAVANAN_WORKFLOW), not compiled into the binary, so a checkout and a binary from different commits can meet. When they do, cwltool answers "Type mismatch for input `kinship_pheno_file'; expected `File' but got `null'" -- which says nothing about the cause. Found by pointing the deployed f64a708 binary at the current workflow. The compiled CWL now has its declared inputs compared with what this build is about to send, before anything is staged for ravanan: a missing input or an undeclared one is refused with both lists named, plus the instruction to take the .scm and the binary from the same commit.
4 daysravanan: key the kinship step on the animal set, so K is shared across traitsPjotr Prins
gemma -gk reads the phenotype file only to learn which individuals are analysed -- the matrix comes from the genotypes -- so the workflow now gives the kinship step a constant-valued pheno file carrying just the trait's missing-data pattern, and gives gwa the real values. Verified byte-for-byte: -gk on a real trait and on its constant-valued twin produce identical .cXX.txt files, and trait 2's assoc checksums are identical whether K came from the constant pheno or the real one. Consequence, measured on the BXD fixture: a second trait over the same animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices are identical inputs for both traits. Three keys now: workflow (the ccwl source), kinship (genotype + snps + lmm + the NA pattern) and trait (kinship key + the values), so the compiled CWL and the kinship pheno file are shared and only the trait's pheno file and inputs YAML are per-trait. This is also stricter than gemma-wrapper's own key, which ignores the phenotype entirely and would reuse a K computed for a different set of animals.
4 daysravanan: name the workflow inputs by content, so a repeat request is warmPjotr Prins
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and -gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on every call and a repeat of the identical request recomputed all 40 steps. Naming them by a hash of the request's inputs (genotype path and size, snps contents, trait values, lmm) makes an identical request hit the store instead. Measured on the BXD fixture, full 20-chromosome LOCO mapping, single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums. For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s warm -- its own cache did not hit on a repeat here -- so the ravanan path is now the cheaper one for any repeated request and the slightly dearer one for a first request. The run log stays per-call: it is what a caller streams by run_id. The three shared files are written aside and renamed, so two concurrent identical requests cannot read a half-written file.
4 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
7 daysgemma: pve on the ravanan/slurm path, read out of each archivePjotr
The ravanan path asks for the same variance components as the direct one: gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them into the same note (every chromosome, plus the extremes with vg/ve) and the same unrounded "variance:" lines in the ?id= run log. It reads them from the extracted GWA archive rather than from the copy gemma-wrapper keeps beside it, and that is the point of doing it here: the copy beside the archive is a 5eb53fa feature, while a ravanan job runs whatever gemma-wrapper its own Guix manifest resolves, which need not be that new. The .log.txt has always been inside the archive, and the archive is unpacked anyway for the chromosome's .assoc.txt -- so read_variance_components_from_logs labels each log by the "-loco <chr>" in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt file name, and sorts chr1..chrX. Verification, precisely: the CWL compiles with -pve (ccwl compile, checked by hand), the archive reader is unit-tested (labelling from the command line, from the file name, numeric ordering, a log without the block dropped), and a real archive from this deployment does carry the block. An end-to-end ravanan request could NOT be completed on the dev box, for reasons that have nothing to do with -pve and are now written down in docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set, since the production process does not set GUIX_PACKAGE_PATH at all, so this endpoint cannot resolve gemma-wrapper there), its intern of inputs hard-links and so needs store and inputs on one filesystem ("Invalid cross-device link"), and the one job that did start sat "Waiting for lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
7 daysgemma: the pve note carries every chromosome, and the log carries the numbersPjotr
The note reported the range and named the two chromosomes that bounded it, which is not enough to answer the question the numbers exist for. Under --loco each pve belongs to a *different* kinship matrix, and GEMMA's per-chromosome .log.txt files live on the server -- so a caller told "0.31 to 0.95" could not ask which chromosome was which, or compare a single chromosome against BulkLMM. The note now lists all twenty, compactly, with the range and the two extremes keeping their vg/ve: GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451, chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661). and streaming::append_lines puts the unrounded values in the run log as variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612 so a ?id= run's /compute/gemma/status stream carries them while the run is still in flight, rather than only in a reply someone has to keep. Both measured on BXD::10001 through the dev server before committing. The values are printed with four significant figures (scientific when the magnitude needs it), unchanged: rounding variance components to four decimals was the earlier bug, and these are the same numbers a reader compares with BulkLMM's h2. Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote per-chromosome values, not an average: they are twenty different kinship matrices") and docs/bulklmm.md's example.
7 daysgemma: report GEMMA's own pve for the null modelPjotr
The GWA step now passes -pve, and the reply gains a note with the variance components GEMMA computed for the null model. Under --loco there is one per chromosome, so the note gives their range and names the two that bound it: GEMMA -pve, null model, per chromosome (LOCO): pve 0.3118 to 0.9549 over 20 chromosomes -- lowest 8 pve 0.3118 +/- 0.5009 (vg 23.3659, ve 12.3389), highest 7 pve 0.9549 +/- 0.3570 (vg 76.0933, ve 0.8661). That is BXD::10001 measured, not assumed. pve is the proportion of variance the kinship explains -- the same quantity as /compute/bulklmm's h2, per chromosome rather than genome-wide -- which is what the 16627/16769 comparison against BulkLMM needed. Reading the numbers took three decisions worth stating: - Four decimals is wrong for variance components. The first live run reported 'vg 0.0000, ve 0.0000' for a BXD::16769 null fit whose real components are 3.411e-5 and 3.411e-10 -- both rounded to zero, which reads as 'no variance' when the split is 1.0. Components are printed with four significant figures, scientific when the magnitude needs it, the way GEMMA's own logs do with %g. - se(pve) is -nan on a boundary, and Rust parses -nan as a NaN f64, so the numeric parse alone let it through as NaN. Non-finite values are now 'not available': a boundary estimate has no standard error. - Twenty chromosomes of values bury the other notes, and every value is in the .log.txt beside the archive anyway, so the note names the ends. Requires gemma-wrapper 5eb53fa, which keeps each run's .log.txt beside its archive instead of only inside the .tar.xz. Docs: README's /compute/gemma section, SKILL's gemma bullet (compare pve with h2, never vg with sigma2_e -- pve is scale-free), and docs/bulklmm.md's 'what would settle it', which now says the plumbing is in rather than pending.
9 daysgemma: quote GEMMA's own error, not the wrapper's bannerPjotr
Every GEMMA failure the sweep produced read the same way -- "gemma-wrapper bailed out with pid ... exit 1", sometimes with the Ruby backtrace -- which says nothing about whether the API or the panel is at fault. The line that does is GEMMA's, printed in the middle of the wrapper's per-chromosome chatter: ERROR: Enforce failed for N>0 in src/fastblas.cpp at line 194 in fast_cblas_dgemm so `CommandFailed` now carries the last GEMMA diagnostic (an `ERROR`, `error!`, `****` warning, `Enforce failed` or "not enough" line) rather than the raw stderr, falling back to the wrapper's own tail when GEMMA said nothing at all. The wrapper's banner and `.rb:` backtrace lines are skipped: they are the noise this exists to get past. Two tests: the complaint is GEMMA's line, not the banner; and a silent failure still names something.
9 dayscompute: report GEMMA's effect size next to the scorePjotr
A GEMMA row now carries the effect beside the p-value it belongs to: `… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is what GeneNetwork's own tables and CSV call the additive effect (gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half the difference between the two homozygotes is the usual additive effect; negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta is per *maternal* copy while GeneNetwork's published sign is anchored on the other allele. Two facts that make this worth doing carefully: * `beta` and `se` are left exactly where GEMMA put them. `se` is what makes the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both are GEMMA's own numbers, so replacing them with a derived one would lose information; * the derived column goes *before* the p-value, not after it. `p_lrt` must stay the last column: `gemma::peak_marker` reads `row.last()` for the peak (and with it the PheWAS and plot suggestions), and the plotters and the docs all say "the last column is the p-value". Appending would have silently pointed every peak at an effect size. The response also says what the three columns mean, and that they describe *this run* on these samples and this genotype version -- not the `additive` trait info reports, which came from the published mapping (trait 20010: this run -0.54, published +2.118, same peak marker). Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns are then exactly GEMMA's. * src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so /compute/gemma and /compute/gemma/ravanan both get it. * src/routes/gemma.rs: the note. * tests: four unit tests (placement, sign and halving, `peak_marker` still finds the peak from the last column, a table without beta untouched, a null effect for a row with `NA` beta) and one end-to-end test asserting the column sits immediately before `p_lrt`, every value is `-beta/2`, `p_lrt` stays a probability, and the note is there. * README.md (the response example and what the column means), docs/www/SKILL.md (how to read and quote it). 286 tests pass; the same four as before do not.
10 dayscompute: say what each covariate note is actually aboutPjotr
The shared covariate note promised "5 dummy column(s) named Status.<level>" on both compute endpoints, but only one of them gets dummies: GEMMA is handed k-1 columns and R/qtl's wrapper is handed the single level column and expands it itself. On `/compute/rqtl` the sentence described something that never appears -- caught by reading a live response on production, where the note said "5 dummy column(s) named Status." one line above the Rscript command line that passes no such thing. The note is now engine-neutral -- `... 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels` -- and GEMMA says what it was handed in its own note, which is the part a reader comparing two runs wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC, Status.UTHSC2, ...`. The unit test asserts the level count, and the end-to-end test asserts the columns note names `Status.` columns twice or more, so a covariate silently fitted as one ordinal number fails it. * src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs, docs/www/ai-instruction.md (the example note).
10 dayscompute: fit covariates in GEMMA, and share them with R/qtlPjotr
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit.
10 dayscompute: gate ravanan at one job, and name the binary that failedPjotr
`POST /compute/gemma/ravanan` is now gated like the other two: one workflow at a time, one per caller, `409 compute_busy` naming the run and linking `/compute/gemma/status/{id}` (ravanan has no status route of its own). One slot because ravanan already fans its per-chromosome steps out -- over Slurm, when that backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from `/compute/gemma`'s: different runners, different cache directories, so one of each at once is the overlap `/compute/rqtl` is already allowed. Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the slot for the full 5 s, the same caller's second request was refused at once, and another caller's waited for the slot and then ran. While testing that, `GemmaError::Io` turned out to print "failed to run gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced "failed to run gemma-wrapper: No such file or directory", which sent me looking at the wrong binary. Spawn failures are now their own variant naming the command, and plain file I/O says "I/O error: …". * src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()` exposes the slot count, and a test pins all three endpoints' counts (gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT). * src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with `status = "/compute/gemma/status"`. * src/gemma.rs (Spawn): names the command; the four run_streaming call sites use it; unit test for both messages. * README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
11 dayscompute: suggest a PheWAS at the peak, and the plot that followsPjotr
A mapping's job is not finished when the rows come back: the next two questions are "what else peaks here?" and "show me". So a group-scoped mapping response now ends with both -- - `next`: `/groups/{group}/phewas?locus=<peak>&window=10000`, the peak being the strongest LOD (R/qtl, column 4) or the smallest p_lrt (GEMMA, last column). A GET that resolves 200, which is what `next` promises; measured: R/qtl's peak rs48756159 -> 3 traits, GEMMA's rsm10000005700 -> 12. - a note saying which two columns to plot: chr against LOD (R/qtl) or against -log10 of p_lrt (GEMMA). Rows are positional with meta.columns naming them, so plotting needs no reshaping -- the API does not render images, and saying exactly what to plot is more useful than pretending to. Path-based requests get neither: without a `group` there is no group to PheWAS over, and the note alone would be noise. * src/rqtl.rs, src/gemma.rs (peak_marker): strongest LOD / smallest p_lrt, with unit tests for both shapes and the empty cases. * src/routes/{rqtl,gemma}.rs: the `next` link and the plot note, when the request named a group. * docs/www/ai-instruction.md: 4b rewritten around what a reader does with a PheWAS -- where the position comes from (a mapping's `next`, a trait's `locus`, or chr+mb), the parameter table, what a hit contains, and how to read it (the 460 LRS cap, `traits: []` versus `markers: []`, peak-only and published-only limits, several traits on one marker), with the chr 9 example; 4a points at the new forward.
13 daysgemma: key the gemma-wrapper cache dir by input setPjotr
The cache dir was keyed by chromosome only and shared across requests, so a request with different inputs (another trait, or another snps or genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir. gemma-wrapper names those archives by a content hash, so the workflow's single-File glob output binding then matched more than one path and the step failed -- silently, since ravanan's user-error only logs when %log-level is set, which never happens inside a generated job script. Key the cache root by a hash of the input set (genotype path and size, snps contents, trait values, lmm) as well as by chromosome, so repeated identical requests still hit gemma-wrapper's cache while different input sets cannot see each other's archives. chromosomes_in reads the snps file, so it now takes the already-read contents (chromosomes_from) to avoid reading it twice. cwl/README.md: describe the two-part keying, and note that a node needs a writable HOME -- ravanan forwards the submitting process's HOME into the job (as sbatch would) and gemma-wrapper writes its lock file there.
2026-10-24gemma: Make ravanan's batch system configurable (optional Slurm).Pjotr
POST /compute/gemma/ravanan used to run every CWL step locally, because compute_ravanan never passed ravanan a batch system. Add a RavananBatch setting: - RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing closed on anything else, exactly like GN4_DB_MODE; - slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint, --slurm-jwt (a *file*, which is what ravanan reads) and, when set, --slurm-partition/--slurm-nice; a slurm selection without an endpoint or a JWT file refuses to start rather than 502-ing every request; - --scratch defaults to ravanan_cache_dir for slurm, so the store and the job scratch space live on the same shared filesystem (with slurm that directory has to be visible to the compute nodes); - the response's ravanan note records which backend ran. The switch building is RavananBatch::ravanan_args so it is unit testable; compute_ravanan takes the backend and the scratch directory, and the route passes them from config. docs/ravanan-slurm.md documents the keys, the shared-directory requirement, and the slurm-side auth wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in slurmrestd's environment) that makes REST-submitted jobs actually run.
2026-10-19Merge branch 'gemma-cwl-parallel'Pjotr Prins
# Conflicts: # src/gemma.rs # src/rqtl.rs # src/streaming.rs
2026-10-19gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallelPjotr Prins
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
2026-10-18gemma: write the ravanan workflow in ccwl, not raw CWLPjotr Prins
Per Arun's request, express the --loco kinship/GWA workflow as a ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm, compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`. ccwl inlines the two command definitions (kinship, gwa) directly into the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer needed as separate files. The two scatter steps (kinship over [chr, cache_dir], gwa over [chr, cache_dir, kinship_json], both dotproduct) and the gwa_json/archive outputs are unchanged in substance -- verified by compiling and running the generated gemma-loco.cwl through ravanan against the BXD fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm (~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144: chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it replaces.
2026-10-15gemma: give each scattered chromosome its own persistent cache dirPjotr
--cache-dir "." pointed every scattered job at its own hermetic CWL working directory (fresh under ravanan's --store every run), so gemma-wrapper's cache was never actually reused across requests. Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>, one persistent directory per chromosome -- not a single shared directory, because gemma-wrapper's cache-hit hash does not vary with --chromosomes, so sharing one directory would let one chromosome's job silently reuse another's cached result. Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since kinship and gwa for the same chromosome now point at the same persistent cache_dir, gemma-wrapper finds the kinship step's archive there directly, no staging needed. Confirmed via a real integration run (BXD fixtures) that gemma-wrapper itself unconditionally re-executes GEMMA regardless of whether a matching cached archive already exists -- --cache-dir is only used to hand results between gemma-wrapper's own pipeline stages within one request, not to skip recomputation across requests. So this fixes a real latent bug (archive-glob collisions once a chromosome's cache directory holds more than one request's output) but does not, on its own, speed up repeated identical requests; verified the direct /compute/gemma path shows the same lack of cross-request speedup.
2026-10-15gemma: scatter ravanan's --loco workflow by chromosome via CWL scatterPjotr
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper invocation covering all chromosomes, relying on gemma-wrapper's own internal GNU-parallel fan-out) with one gemma-wrapper invocation PER CHROMOSOME, restricted via its existing --chromosomes/--no-parallel flags and scattered by ravanan itself (ScatterFeatureRequirement, scatterMethod: dotproduct). gemma-wrapper still does the correct per-chromosome LOCO math (exclude this chromosome from kinship, test only its markers) against the full genotype/SNP files -- only which chromosome(s) it loops over changes. compute_ravanan now computes the chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own get_chromosomes) and passes it as the workflow's scatter input, and collates the resulting per-chromosome gwa_json/archive array outputs. Measured 16.7s wall-clock end-to-end against the BXD fixtures (20 chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for the previous sequential ravanan workflow -- confirmed byte-for-byte identical output (rs31443144's exact values, 7320 total rows). Also fixes three real, previously-unexercised bugs in compute_ravanan found getting this to run at all (POST /compute/gemma/ravanan had no automated test until this commit): - missing --guix-manifest (ravanan requires it; added gemma_ravanan_manifest config key and cwl/gemma/manifest.scm) - missing --store files/logs/data subdirectories (ravanan does not create them itself) - --output-directory dropped entirely: every chromosome's gwa archive shares gemma-wrapper's own content-hash basename (the hash doesn't factor in --chromosomes), so ravanan's optional flat-symlink step collides across them; unneeded anyway since results are read straight from ravanan's --store via its stdout JSON record. Also found and worked around a ravanan scheduler bug: a step scattered over exactly one input needs an explicit scatterMethod even though CWL only requires one for multiple scattered inputs -- ravanan has no fallback case for a missing one and silently misbehaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15gemma: return .assoc.txt rows positionally, columns in meta.columnsPjotr
Cuts repeated key names out of large /compute/gemma(/ravanan) responses: data is now a list of positional row arrays instead of one keyed object per row, with the column names moved to a new meta.columns field shared with any other endpoint that wants the same shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15CWL used to run gemmaPjotr Prins
2026-10-14gemma/rqtl: three fixes from the endpoint reviewAlexander_Kabui
- rqtl: per-call unique output filename. The content hash alone was deterministic across processes, so two concurrent identical requests shared one output file; R truncates on open, so one request could read the other's half-written results with no error. The per-call unique (pid + nanos, already computed for the pheno file) now names the output file too. - streaming/gemma/rqtl: append a failure marker to the run log on every error path, mirroring the completion marker. A status poller can now tell a failed run from a still-running one. - rqtl: --scale is now honored. The wrapper selects the geno file's cM/Mb map column from it, and the route rejects values other than mb/cm with 422 instead of silently ignoring them. Both new behaviors are unit-tested; the live Rscript tests pass.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: rename peak to peek; reject a reused run id with 409Pjotr
peak -> peek: the status endpoint's byte-offset query parameter and everywhere it's documented (code comments, the GemmaStatusParams field, tests, README). GeneNetwork3's own equivalent (gn3/api/streaming.py) is spelled "peak"; noted where this now deliberately differs, not touching the unrelated "peak marker" wording elsewhere in the README. Run id uniqueness: the log file is now created with OpenOptions::create_new(true) instead of a plain truncating write, so it fails atomically (no separate check-then-write race) if a log already exists for that id -- another run still in progress, or a finished one never polled/cleaned up. Added GemmaError::RunIdInUse, mapped to 409 Conflict in the route, caught before any subprocess runs. Previously a reused id would have silently truncated whatever was already there. Verified: the new compute_gemma_rejects_a_run_id_already_in_use test pre-seeds a log file for an id and confirms the POST is refused (this needs no gemma-wrapper, since the check runs before any subprocess call). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: stream gemma-wrapper's progress; add GET /compute/gemma/statusPjotr
/compute/gemma can genuinely take minutes (see the README's ~100K marker caveat), and until now gave no visibility into a run already in flight. Same shape GeneNetwork2/3 use for R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the caller picks a run id, passes it as POST /compute/gemma?id=<id>, and polls GET /compute/gemma/status/{id}?peak=<offset> on a second connection while the first is still blocked. What's actually tailed is gemma-wrapper's stderr, not stdout: under --json, all of its human-readable progress (banner, chromosome list, "Executing: ...", per-step info) goes to stderr, and stdout carries only the one final JSON record, printed once at the very end -- so there is nothing to stream on stdout. run_streaming() replaces the old buffered Command::output() call with one that reads stderr line by line as the process runs, appending each line to a log file if one was given, across both the kinship and GWA steps into a single continuous log; stdout is still read in full (unchanged, one JSON blob). A completion marker ("GN4: computation complete") is appended once the whole request finishes, the same convention GeneNetwork3 uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this project's own name. run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever joined into a path, on both the POST's ?id= and the status endpoint's path segment -- GeneNetwork3's own /api/stream/{run_id} does no such check on its equivalent id, which is not knowingly replicated here. Verified end to end, not just unit tested: the integration test spawns the POST on its own task and polls the status endpoint from this one while the ~19s real computation is still running, asserting it actually observed non-empty progress lines mid-run (not merely after the fact), then checks the finished log for both real gemma-wrapper progress text and the completion marker. cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: document the endpoint in README; put the actual gemma-wrapperPjotr
commands in the response notes README: added /compute/gemma to the endpoint table and a dedicated section with a real example response, and the explicit caveat the user asked for -- not feasible past roughly 100,000 markers at this point (synchronous, sequential, no parallel LOCO jobs; the BXD fixtures, 7,320 markers, already take ~19s, and real datasets go up to ~1M markers). notes: the response now carries the exact gemma-wrapper invocations used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk", "GWA: gemma-wrapper ... -lmm 9"), not just a generic description. kinship_args/gwa_args build the argument list once and feed both the actual subprocess call and the note text, so the two can't drift apart. Added unit tests asserting -gk and -lmm never appear together, and extended the real end-to-end test to check both notes' shape. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run, with the notes field inspected directly against the actual command lines used. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: default lmm to 9, matching GeneNetwork's own conventionPjotr
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile (gn/runner/gemma.scm) both always use -lmm 9 in their working --loco invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other than stock GEMMA's 1-4 scale, not independently verified here, but consistent with what both of those codebases rely on in production. Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4, 9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so they're still rejected rather than guessed at. Updated the real end-to-end test to assert on p_lrt (what lmm=9 actually produces) instead of p_wald, and moved the out-of-range-lmm test from 9 (now valid) to 5. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run against the BXD fixtures with the new default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>