about summary refs log tree commit diff
path: root/tests/gemma.rs
AgeCommit message (Collapse)Author
5 daysindividuals: select who enters a mapping, by namePjotr
Parents and F1s are excluded by default (as in GeneNetwork2/3), with include_parents_f1 to keep them all. That boolean was not enough for what was asked -- "include these, drop that substrain" -- and the exclusion list itself is data (parents_and_f1s.json, seven groups) that no request can reach. F1 names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH, C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong tool. Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes "individuals": {"include": [...], "exclude": [...]} applied after the parent/F1 rule, exclude winning over include. An unknown name is a 422 invalid_individuals whose message lists the panel: a silently dropped animal changes a kinship matrix without anyone noticing, so the failure has to be loud. genotypes::select_panel is the one implementation, and it returns what it did (dropped parents/F1s, re-included by name, excluded by name) so the Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded by name". The note now also spells out the distinction that makes this option treacherous: the kinship uses *every* individual in the genotype file, the association uses only those with a trait value -- GEMMA reads no phenotypes for the kinship. So "include the F1s" changes the relatedness structure without putting the F1s in the fit, and a request can legitimately have 234 individuals in the genotype file and 34 in the association. Verified on BXD.8: excluding BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1 re-included and a substrain excluded gives 234 / 34. A basket supplies values by name, so a selection applies to a basket-fed mapping the same way: the individuals excluded are simply not in the panel any more, and their basket values are ignored. GenotypeError gains Invalid for this (422) so a caller's typo is not reported as a missing genotype file; the read-only endpoints that can never see one map it the same way for exhaustiveness.
7 daysgemma/ravanan: resolve the same inputs as the direct runner; refuse what it ↵Pjotr
cannot do Two gaps, both invisible until a real request ran against a cluster. The endpoint read only genotype_file, snps_file and trait_values from the request; group, genotype_version and samples were accepted by the shared request type and silently dropped. A /compute/gemma-shaped request (group + samples) therefore arrived as empty paths and empty values and died with "I/O error: No such file or directory (os error 2)" from reading "". Running it exposed the second gap in the same breath: the gate sized the request from trait_values.len(), so a samples-only request was sized 0 and took no concurrency slot at all. resolve_inputs is now one function in routes/gemma.rs, called by both handlers: upload materialised into the pair the engine reads, basket resolved to named values, group prepared with the parent/F1 and empty-genotype rules, named samples aligned by name onto whatever individuals the file has, positional values checked against the panel size. That block was ~250 lines of duplicated rules inside compute_gemma, which is how it drifted in the first place; the direct runner now calls the same function and keeps only its covariates, normalization and gate. The ravanan reply also carries the input notes now (which genotype version, how many values aligned, what was dropped), so a run says what it mapped as well as where the peak is. Refused rather than ignored, because the CWL workflow has no step for either: covariates (422 covariates_not_supported) and normalize (422 normalization_not_supported), each naming /compute/gemma as the endpoint that can do it. A new test asserts both refusals happen before any job is submitted, so it needs no ravanan, ccwl or cluster. Verified against the Slurm container from gn-machines: group: BXD became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's chromosomes), all completed, 21 055 rows in ~3.5 min. README and docs/ravanan-slurm.md record the four requirements that run turned up, three of them not obvious: ccwl on PATH, a ravanan build with the --slurm-api-endpoint fix, the *server's* HOME writable by the job user inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and GUIX_PACKAGE_PATH for the manifest.
8 daysgemma: the permutation refusal names both routes, including running GEMMA itselfPjotr
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the gate gives GEMMA one slot -- but "not supported here" is not the same as "not available". The refusal now says both: 1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come from rotations that keep the kinship (R/qtl's nperm does not, and is anti-conservative on a structured panel); 2. run GEMMA's own permutation locally. The reply's notes already name the bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or `gemma -lmm 9 --permutate` on those inputs reproduces exactly the null GEMMA would have used -- hours instead of seconds, which is the honest price of that particular null. docs/bulklmm.md gains the table this belongs in: three ways to get a threshold, ordered by how honest the null is for a structured panel (rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's nperm as a sanity check only). SKILL.md says the same in two lines. tests/gemma.rs now asserts both halves of the refusal, and the earlier assertion on /compute/rqtl -- which stopped being true when the refusal was rewritten -- is replaced rather than left passing by accident.
10 daysgemma: leave out individuals with no genotypes, which make the kinship singularPjotr
Asking whether the singular matrices could be fixed turned out to have two answers, one of them a real bug on our side. The bug: a panel can name individuals whose genotype row is empty, and an empty row makes the kinship matrix singular. GEMMA then reports "eigenvalues close to zero" and dies in LAPACK with `LUDecomp ... input domain error` -- a message about linear algebra for a problem that is a missing genotype. Measured on this deployment's own files: BXD-AE.4 136 of 344 individuals have no genotypes MDP 93 of 178 BXD-Heart-Metals.4 200 of 481 HET3-ITP.3 2867 of 2867 (the whole file is empty) Dropping them makes those panels map: BXD-AE.4's GWA step goes from a failure to 3.2 s, and MDP's kinship step from a failure to success. The individuals contribute nothing to a scan either way, so nothing is lost; `notes` counts them, and the "Genotype:" line no longer blames parents/F1s for them (`prepared.excluded` is "panel minus kept", which now includes both, so the two counts are separated). Not the cause, but worth naming: `AXB` fails earlier still, on its annotation -- `ERROR: Enforce failed for D1Mit296 in src/gemma_io.cpp ... ReadFile_anno` -- which is a broken `.snps` file, not a matrix. That wants a pre-flight check of the annotation (report the offending marker) rather than a GEMMA failure, and is not in this commit. Considered and rejected: a ridge on K (`K + epsilon*I`) would work mathematically but GEMMA has no option for it and the wrapper caches K by content hash, so it would mean patching the wrapper's cached matrix; `-gk 2` and dropping markers do not add rank; and the non-LOCO path is the one that does not run at all here. * src/genotypes.rs: `individuals_with_genotypes`, one pass over the bimbam file marking the individuals with any genotype at all. * src/routes/gemma.rs: filter `keep` by it before alignment (so the trait values drop with them), and say so in `notes`. * tests/gemma.rs: `individuals_with_no_genotypes_are_left_out` -- BXD-AE.4 maps, the note names the drop and the reason, and the genotype line no longer claims 344 individuals. * README.md: the behaviour and why it exists, under /compute/gemma. 307 tests pass; the same four as before do not.
10 dayscovariates: refuse a mostly-numeric attribute, and teach the skill to use themPjotr
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a message about LU decomposition: its values are depths (`40`, `41`) with a few two-library entries (`36+39L`), so the attribute was coded as **twenty categories** and GEMMA died on the resulting dummies. The failure named GEMMA, which is the wrong place to look. `code_case_attribute` now refuses that shape at the covariate step, naming the values and what to send instead: 422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes numbers with number-like text (14 of 19 values look numeric, but e.g. '36+30L' does not), so coding it as categories would fit a quantity as a factor -- and GEMMA fails on the result. Pass an explicit `map` giving each of those a number (there are 5 such value(s): ...), or send the numbers inline. The rule is deliberately narrow: an odd level with *no* digits (`1, 2, unknown`) is still a level and still coded, so nothing that worked stops working; only dirty numbers are caught. Also, the skill's covariate coverage -- which the review above found thin: * when a covariate is worth trying (metadata that could confound: sex, age, epoch, batch, sequencing depth), and the covariate step in the session outline; * read `.data.descriptions` before choosing one -- they are the only place that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup -- and the endpoint table now says so; * read the covariate line in `notes` for the coding and the complete-case n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being two levels in BXD; * the categorical-vs-numeric rule measured rather than asserted: `RRID` refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value trap written up with the `map` workaround; * GEMMA's dummy columns, the column it drops when a level has no fitted individual, and that R/qtl's note has no such line; missing values dropping individuals; the 8-covariate cap and the `group` requirement. * src/covariates.rs: the refusal, with a unit test for both sides of it; * tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure; * docs/www/SKILL.md: the covariate section, the session line, and the case-attributes rows in the endpoint table. 300 tests pass; the same four as before do not.
10 dayscompute: report GEMMA's effect size next to the scorePjotr
A GEMMA row now carries the effect beside the p-value it belongs to: `… beta, se, l_mle, additive, p_lrt`. `additive` is `-beta/2`, which is what GeneNetwork's own tables and CSV call the additive effect (gn2/wqflask/marker_regression/gemma_mapping.py:240). Halved because half the difference between the two homozygotes is the usual additive effect; negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's beta is per *maternal* copy while GeneNetwork's published sign is anchored on the other allele. Two facts that make this worth doing carefully: * `beta` and `se` are left exactly where GEMMA put them. `se` is what makes the effect interpretable (`beta +/- 1.96*se` is a 95% interval) and both are GEMMA's own numbers, so replacing them with a derived one would lose information; * the derived column goes *before* the p-value, not after it. `p_lrt` must stay the last column: `gemma::peak_marker` reads `row.last()` for the peak (and with it the PheWAS and plot suggestions), and the plotters and the docs all say "the last column is the p-value". Appending would have silently pointed every peak at an effect size. The response also says what the three columns mean, and that they describe *this run* on these samples and this genotype version -- not the `additive` trait info reports, which came from the published mapping (trait 20010: this run -0.54, published +2.118, same peak marker). Nothing is added when GEMMA wrote no `beta` (a non-LMM run): the columns are then exactly GEMMA's. * src/gemma.rs: `with_effect_size`, applied by `parse_assoc_file`, so /compute/gemma and /compute/gemma/ravanan both get it. * src/routes/gemma.rs: the note. * tests: four unit tests (placement, sign and halving, `peak_marker` still finds the peak from the last column, a table without beta untouched, a null effect for a row with `NA` beta) and one end-to-end test asserting the column sits immediately before `p_lrt`, every value is `-beta/2`, `p_lrt` stays a probability, and the note is there. * README.md (the response example and what the column means), docs/www/SKILL.md (how to read and quote it). 286 tests pass; the same four as before do not.
10 dayscompute: let GEMMA name the trait, so its plot can be titledPjotr
Two GEMMA runs over one BXD panel drew genuinely different pictures -- traces differ on all 20 chromosomes, peaks -log10 p 5.48 at chr8 97.0 Mb against 5.43 at chr1 175.6 Mb -- and both came out headed **"trait gemma: GEMMA association along the genome"**, because GEMMA's request had no trait label at all: `/compute/gemma` fell back to `<group>::gemma`, and `/plot/gemma` fell back to "gemma". The plot of each trait was told apart by its annotation and nothing else. `trait_name` is now an optional field on `GemmaRequest`, used for the `Trait: ...` note (first in `notes`, as on `/compute/rqtl`), for the PheWAS and R/qtl suggestions, and for the `/plot/gemma?trait=` suggestion -- so a posted response titles its own plot, and `?trait=` still overrides. Without it nothing changes: the label stays `<group>::gemma`. * src/routes/gemma.rs: the field, the label resolution, the note, the plot href. * tests/gemma.rs: the covariate run names its trait and asserts the note is first; tests/plot.rs: a GEMMA plot titles itself from the note, and an explicit `?trait=` wins over it. * README.md (`/compute/gemma`, and the plot section), docs/www/SKILL.md (the GEMMA options list, and what the plot title comes from). 281 tests pass; the same four as before do not.
11 dayscompute: gate GEMMA from 50 individuals, R/qtl from 1 000Pjotr
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
11 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
11 dayscompute: say what each covariate note is actually aboutPjotr
The shared covariate note promised "5 dummy column(s) named Status.<level>" on both compute endpoints, but only one of them gets dummies: GEMMA is handed k-1 columns and R/qtl's wrapper is handed the single level column and expands it itself. On `/compute/rqtl` the sentence described something that never appears -- caught by reading a live response on production, where the note said "5 dummy column(s) named Status." one line above the Rscript command line that passes no such thing. The note is now engine-neutral -- `... 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels` -- and GEMMA says what it was handed in its own note, which is the part a reader comparing two runs wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC, Status.UTHSC2, ...`. The unit test asserts the level count, and the end-to-end test asserts the columns note names `Status.` columns twice or more, so a covariate silently fitted as one ordinal number fails it. * src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs, docs/www/ai-instruction.md (the example note).
11 dayscompute: refuse GEMMA permutations instead of ignoring themPjotr
`gemma-wrapper --permutate n` repeats the *whole* LOCO association scan per permutation -- it shuffles the phenotype file and starts over -- so a thousand permutations is thousands of full scans: hours to days on a real panel, against the single slot `/compute/gemma` has. That is not a test this deployment carries, and an assistant asked for significance thresholds will ask for it, so the answer has to be an explicit one. Until now the request type simply had no such field and serde ignores unknown keys, which is the worst of the three outcomes: the caller gets a 200, no thresholds, and no indication that what it asked for was dropped. `nperm`, `permutations` and `permutate` are now named fields and refused with `422 permutations_not_supported`, naming the cost, pointing at `POST /compute/rqtl` with `nperm` (the endpoint that does have the test, over the same panel and values) and at the `actions` member every successful GEMMA response already carries, pre-filled with exactly that call. Zero keeps its usual meaning -- off -- so a client sharing one body shape across the two endpoints still works. * src/routes/gemma.rs: the three fields, the refusal before anything expensive runs, and the response note now saying permutations are refused rather than merely absent. * tests/gemma.rs: `compute_gemma_refuses_permutations` pins all three spellings, the code, and that `nperm: 0` is not a request for any. It takes the compute lock: the gate answers before the handler does, so a concurrent job in the same test binary came back 409 instead. * README.md, docs/www/ai-instruction.md: what to do instead -- judge a peak against the smallest `p_lrt`, and get thresholds from R/qtl. 273 tests pass; the same four as before do not.
11 dayscompute: fit covariates in GEMMA, and share them with R/qtlPjotr
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit.
2026-10-19Merge branch 'gemma-cwl-parallel'Pjotr Prins
# Conflicts: # src/gemma.rs # src/rqtl.rs # src/streaming.rs
2026-10-19gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallelPjotr Prins
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
2026-10-15gemma: scatter ravanan's --loco workflow by chromosome via CWL scatterPjotr
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper invocation covering all chromosomes, relying on gemma-wrapper's own internal GNU-parallel fan-out) with one gemma-wrapper invocation PER CHROMOSOME, restricted via its existing --chromosomes/--no-parallel flags and scattered by ravanan itself (ScatterFeatureRequirement, scatterMethod: dotproduct). gemma-wrapper still does the correct per-chromosome LOCO math (exclude this chromosome from kinship, test only its markers) against the full genotype/SNP files -- only which chromosome(s) it loops over changes. compute_ravanan now computes the chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own get_chromosomes) and passes it as the workflow's scatter input, and collates the resulting per-chromosome gwa_json/archive array outputs. Measured 16.7s wall-clock end-to-end against the BXD fixtures (20 chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for the previous sequential ravanan workflow -- confirmed byte-for-byte identical output (rs31443144's exact values, 7320 total rows). Also fixes three real, previously-unexercised bugs in compute_ravanan found getting this to run at all (POST /compute/gemma/ravanan had no automated test until this commit): - missing --guix-manifest (ravanan requires it; added gemma_ravanan_manifest config key and cwl/gemma/manifest.scm) - missing --store files/logs/data subdirectories (ravanan does not create them itself) - --output-directory dropped entirely: every chromosome's gwa archive shares gemma-wrapper's own content-hash basename (the hash doesn't factor in --chromosomes), so ravanan's optional flat-symlink step collides across them; unneeded anyway since results are read straight from ravanan's --store via its stdout JSON record. Also found and worked around a ravanan scheduler bug: a step scattered over exactly one input needs an explicit scatterMethod even though CWL only requires one for multiple scattered inputs -- ravanan has no fallback case for a missing one and silently misbehaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15response: add meta.num_rows/num_cols alongside meta.columnsPjotr
So a caller of /compute/gemma(/ravanan) or /compute/rqtl doesn't need to count data's rows or columns.len() itself. with_columns now takes the row count too and derives num_cols from the column list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15rqtl: return marker rows positionally, columns in meta.columnsPjotr
Same space-saving shape as the gemma commit: /compute/rqtl's markers are now positional arrays (name, chr, pos, lod, plus any 4-way-cross extra effect columns trailing) instead of one keyed object per row, with the fixed column names in meta.columns. Also fixes tests/gemma.rs, which still asserted the old keyed shape from the gemma change -- missed in that commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-12gemma: rename peak to peek; reject a reused run id with 409Pjotr
peak -> peek: the status endpoint's byte-offset query parameter and everywhere it's documented (code comments, the GemmaStatusParams field, tests, README). GeneNetwork3's own equivalent (gn3/api/streaming.py) is spelled "peak"; noted where this now deliberately differs, not touching the unrelated "peak marker" wording elsewhere in the README. Run id uniqueness: the log file is now created with OpenOptions::create_new(true) instead of a plain truncating write, so it fails atomically (no separate check-then-write race) if a log already exists for that id -- another run still in progress, or a finished one never polled/cleaned up. Added GemmaError::RunIdInUse, mapped to 409 Conflict in the route, caught before any subprocess runs. Previously a reused id would have silently truncated whatever was already there. Verified: the new compute_gemma_rejects_a_run_id_already_in_use test pre-seeds a log file for an id and confirms the POST is refused (this needs no gemma-wrapper, since the check runs before any subprocess call). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: stream gemma-wrapper's progress; add GET /compute/gemma/statusPjotr
/compute/gemma can genuinely take minutes (see the README's ~100K marker caveat), and until now gave no visibility into a run already in flight. Same shape GeneNetwork2/3 use for R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the caller picks a run id, passes it as POST /compute/gemma?id=<id>, and polls GET /compute/gemma/status/{id}?peak=<offset> on a second connection while the first is still blocked. What's actually tailed is gemma-wrapper's stderr, not stdout: under --json, all of its human-readable progress (banner, chromosome list, "Executing: ...", per-step info) goes to stderr, and stdout carries only the one final JSON record, printed once at the very end -- so there is nothing to stream on stdout. run_streaming() replaces the old buffered Command::output() call with one that reads stderr line by line as the process runs, appending each line to a log file if one was given, across both the kinship and GWA steps into a single continuous log; stdout is still read in full (unchanged, one JSON blob). A completion marker ("GN4: computation complete") is appended once the whole request finishes, the same convention GeneNetwork3 uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this project's own name. run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever joined into a path, on both the POST's ?id= and the status endpoint's path segment -- GeneNetwork3's own /api/stream/{run_id} does no such check on its equivalent id, which is not knowingly replicated here. Verified end to end, not just unit tested: the integration test spawns the POST on its own task and polls the status endpoint from this one while the ~19s real computation is still running, asserting it actually observed non-empty progress lines mid-run (not merely after the fact), then checks the finished log for both real gemma-wrapper progress text and the completion marker. cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: document the endpoint in README; put the actual gemma-wrapperPjotr
commands in the response notes README: added /compute/gemma to the endpoint table and a dedicated section with a real example response, and the explicit caveat the user asked for -- not feasible past roughly 100,000 markers at this point (synchronous, sequential, no parallel LOCO jobs; the BXD fixtures, 7,320 markers, already take ~19s, and real datasets go up to ~1M markers). notes: the response now carries the exact gemma-wrapper invocations used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk", "GWA: gemma-wrapper ... -lmm 9"), not just a generic description. kinship_args/gwa_args build the argument list once and feed both the actual subprocess call and the note text, so the two can't drift apart. Added unit tests asserting -gk and -lmm never appear together, and extended the real end-to-end test to check both notes' shape. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run, with the notes field inspected directly against the actual command lines used. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: default lmm to 9, matching GeneNetwork's own conventionPjotr
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile (gn/runner/gemma.scm) both always use -lmm 9 in their working --loco invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other than stock GEMMA's 1-4 scale, not independently verified here, but consistent with what both of those codebases rely on in production. Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4, 9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so they're still rejected rather than guessed at. Updated the real end-to-end test to assert on p_lrt (what lmm=9 actually produces) instead of p_wald, and moved the out-of-range-lmm test from 9 (now valid) to 5. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run against the BXD fixtures with the new default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>