about summary refs log tree commit diff
path: root/cwl/gemma
AgeCommit message (Collapse)Author
2 dayscwl: make the ravanan path use the gemma build the wrapper path usesPjotr Prins
`(specifications->manifest (list "gemma"))` resolved to Guix's `gemma`, which is 0.98.5 (banner 2021-08-25); gemma-wrapper pins `gemma-gn2`, which is 0.98.6 (banner 2022-08-05). Same source, different build, and the two paths disagreed in beta by ~3e-3 because of it. Now both name `gemma-gn2`. Verified on the job side: the latest runs' own logs report GEMMA 0.98.6 and OpenBLAS 0.3.31, the same pair the wrapper profile resolves to. The remaining difference is smaller and has a different cause, measured rather than guessed: same path, two cold runs: worst |Δbeta| 5.1e-6 (1034/21055 bit-identical) wrapper vs ravanan: worst |Δbeta| 3.1e-3 their kinship matrices: differ by ~1e-5 cell-wise, same 198 individuals So GEMMA is not reproducible run to run under multithreaded BLAS (both contexts report 96 threads), and the kinship matrix amplifies that by roughly two orders of magnitude into beta. Pinning threads on both paths is what would make the parity gate exact; until then 0.005 is the honest tolerance, and scripts/gemma-compare.py now defaults to exact and says which one it is enforcing. Also notes in the comment that a manifest change alone does not take effect: ravanan keys a step on (script, inputs), so it kept reusing step scripts built against the old manifest until the workflow source changed.
2 dayscwl: name the caching model, so nobody hunts a gemma-wrapper cache that is ↵Pjotr Prins
not there ravanan's store keys a step on (script, inputs) and there is no wrapper cache in this path. Recorded next to the 'No gemma-wrapper' rationale because the detour was expensive: a /gnu/store script that did mention --cache-dir /tmp/gn4-gemma-ravanan looked current only because /gnu/store's epoch mtimes defeat ls -t.
4 daysravanan: key the kinship step on the animal set, so K is shared across traitsPjotr Prins
gemma -gk reads the phenotype file only to learn which individuals are analysed -- the matrix comes from the genotypes -- so the workflow now gives the kinship step a constant-valued pheno file carrying just the trait's missing-data pattern, and gives gwa the real values. Verified byte-for-byte: -gk on a real trait and on its constant-valued twin produce identical .cXX.txt files, and trait 2's assoc checksums are identical whether K came from the constant pheno or the real one. Consequence, measured on the BXD fixture: a second trait over the same animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices are identical inputs for both traits. Three keys now: workflow (the ccwl source), kinship (genotype + snps + lmm + the NA pattern) and trait (kinship key + the values), so the compiled CWL and the kinship pheno file are shared and only the trait's pheno file and inputs YAML are per-trait. This is also stricter than gemma-wrapper's own key, which ignores the phenotype entirely and would reuse a K computed for a different set of animals.
5 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
7 daysgemma: pve on the ravanan/slurm path, read out of each archivePjotr
The ravanan path asks for the same variance components as the direct one: gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them into the same note (every chromosome, plus the extremes with vg/ve) and the same unrounded "variance:" lines in the ?id= run log. It reads them from the extracted GWA archive rather than from the copy gemma-wrapper keeps beside it, and that is the point of doing it here: the copy beside the archive is a 5eb53fa feature, while a ravanan job runs whatever gemma-wrapper its own Guix manifest resolves, which need not be that new. The .log.txt has always been inside the archive, and the archive is unpacked anyway for the chromosome's .assoc.txt -- so read_variance_components_from_logs labels each log by the "-loco <chr>" in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt file name, and sorts chr1..chrX. Verification, precisely: the CWL compiles with -pve (ccwl compile, checked by hand), the archive reader is unit-tested (labelling from the command line, from the file name, numeric ordering, a log without the block dropped), and a real archive from this deployment does carry the block. An end-to-end ravanan request could NOT be completed on the dev box, for reasons that have nothing to do with -pve and are now written down in docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set, since the production process does not set GUIX_PACKAGE_PATH at all, so this endpoint cannot resolve gemma-wrapper there), its intern of inputs hard-links and so needs store and inputs on one filesystem ("Invalid cross-device link"), and the one job that did start sat "Waiting for lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
2026-10-19gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallelPjotr Prins
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
2026-10-18gemma: write the ravanan workflow in ccwl, not raw CWLPjotr Prins
Per Arun's request, express the --loco kinship/GWA workflow as a ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm, compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`. ccwl inlines the two command definitions (kinship, gwa) directly into the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer needed as separate files. The two scatter steps (kinship over [chr, cache_dir], gwa over [chr, cache_dir, kinship_json], both dotproduct) and the gwa_json/archive outputs are unchanged in substance -- verified by compiling and running the generated gemma-loco.cwl through ravanan against the BXD fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm (~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144: chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it replaces.
2026-10-15gemma: give each scattered chromosome its own persistent cache dirPjotr
--cache-dir "." pointed every scattered job at its own hermetic CWL working directory (fresh under ravanan's --store every run), so gemma-wrapper's cache was never actually reused across requests. Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>, one persistent directory per chromosome -- not a single shared directory, because gemma-wrapper's cache-hit hash does not vary with --chromosomes, so sharing one directory would let one chromosome's job silently reuse another's cached result. Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since kinship and gwa for the same chromosome now point at the same persistent cache_dir, gemma-wrapper finds the kinship step's archive there directly, no staging needed. Confirmed via a real integration run (BXD fixtures) that gemma-wrapper itself unconditionally re-executes GEMMA regardless of whether a matching cached archive already exists -- --cache-dir is only used to hand results between gemma-wrapper's own pipeline stages within one request, not to skip recomputation across requests. So this fixes a real latent bug (archive-glob collisions once a chromosome's cache directory holds more than one request's output) but does not, on its own, speed up repeated identical requests; verified the direct /compute/gemma path shows the same lack of cross-request speedup.
2026-10-15gemma: scatter ravanan's --loco workflow by chromosome via CWL scatterPjotr
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper invocation covering all chromosomes, relying on gemma-wrapper's own internal GNU-parallel fan-out) with one gemma-wrapper invocation PER CHROMOSOME, restricted via its existing --chromosomes/--no-parallel flags and scattered by ravanan itself (ScatterFeatureRequirement, scatterMethod: dotproduct). gemma-wrapper still does the correct per-chromosome LOCO math (exclude this chromosome from kinship, test only its markers) against the full genotype/SNP files -- only which chromosome(s) it loops over changes. compute_ravanan now computes the chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own get_chromosomes) and passes it as the workflow's scatter input, and collates the resulting per-chromosome gwa_json/archive array outputs. Measured 16.7s wall-clock end-to-end against the BXD fixtures (20 chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for the previous sequential ravanan workflow -- confirmed byte-for-byte identical output (rs31443144's exact values, 7320 total rows). Also fixes three real, previously-unexercised bugs in compute_ravanan found getting this to run at all (POST /compute/gemma/ravanan had no automated test until this commit): - missing --guix-manifest (ravanan requires it; added gemma_ravanan_manifest config key and cwl/gemma/manifest.scm) - missing --store files/logs/data subdirectories (ravanan does not create them itself) - --output-directory dropped entirely: every chromosome's gwa archive shares gemma-wrapper's own content-hash basename (the hash doesn't factor in --chromosomes), so ravanan's optional flat-symlink step collides across them; unneeded anyway since results are read straight from ravanan's --store via its stdout JSON record. Also found and worked around a ravanan scheduler bug: a step scattered over exactly one input needs an explicit scatterMethod even though CWL only requires one for multiple scattered inputs -- ravanan has no fallback case for a missing one and silently misbehaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15CWL used to run gemmaPjotr Prins