| Age | Commit message (Collapse) | Author |
|
`(specifications->manifest (list "gemma"))` resolved to Guix's `gemma`,
which is 0.98.5 (banner 2021-08-25); gemma-wrapper pins `gemma-gn2`,
which is 0.98.6 (banner 2022-08-05). Same source, different build, and the
two paths disagreed in beta by ~3e-3 because of it.
Now both name `gemma-gn2`. Verified on the job side: the latest runs'
own logs report GEMMA 0.98.6 and OpenBLAS 0.3.31, the same pair the
wrapper profile resolves to. The remaining difference is smaller and has
a different cause, measured rather than guessed:
same path, two cold runs: worst |Δbeta| 5.1e-6 (1034/21055 bit-identical)
wrapper vs ravanan: worst |Δbeta| 3.1e-3
their kinship matrices: differ by ~1e-5 cell-wise, same 198 individuals
So GEMMA is not reproducible run to run under multithreaded BLAS (both
contexts report 96 threads), and the kinship matrix amplifies that by
roughly two orders of magnitude into beta. Pinning threads on both paths
is what would make the parity gate exact; until then 0.005 is the honest
tolerance, and scripts/gemma-compare.py now defaults to exact and says
which one it is enforcing.
Also notes in the comment that a manifest change alone does not take
effect: ravanan keys a step on (script, inputs), so it kept reusing step
scripts built against the old manifest until the workflow source changed.
|
|
not there
ravanan's store keys a step on (script, inputs) and there is no wrapper
cache in this path. Recorded next to the 'No gemma-wrapper' rationale
because the detour was expensive: a /gnu/store script that did mention
--cache-dir /tmp/gn4-gemma-ravanan looked current only because
/gnu/store's epoch mtimes defeat ls -t.
|
|
gemma -gk reads the phenotype file only to learn which individuals are
analysed -- the matrix comes from the genotypes -- so the workflow now
gives the kinship step a constant-valued pheno file carrying just the
trait's missing-data pattern, and gives gwa the real values. Verified
byte-for-byte: -gk on a real trait and on its constant-valued twin
produce identical .cXX.txt files, and trait 2's assoc checksums are
identical whether K came from the constant pheno or the real one.
Consequence, measured on the BXD fixture: a second trait over the same
animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of
+13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices
are identical inputs for both traits. Three keys now: workflow (the
ccwl source), kinship (genotype + snps + lmm + the NA pattern) and
trait (kinship key + the values), so the compiled CWL and the kinship
pheno file are shared and only the trait's pheno file and inputs YAML
are per-trait.
This is also stricter than gemma-wrapper's own key, which ignores the
phenotype entirely and would reuse a K computed for a different set of
animals.
|
|
Full 20-chromosome LOCO mapping on the BXD fixture: 14.3 s cold and
1.6 s warm single-machine, 275.6 s cold and 1.6 s warm through the Slurm
test container -- 40 jobs, 7320 rows, identical .assoc.txt checksums in
every case. Slurm costs ~6.9 s per job here against ~0.36 s
single-machine, which is container and scheduling overhead rather than
gemma: the first request is cheaper locally, and only a repeated request
(answered from ravanan's store) is fast either way.
|
|
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and
-gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on
every call and a repeat of the identical request recomputed all 40
steps. Naming them by a hash of the request's inputs (genotype path and
size, snps contents, trait values, lmm) makes an identical request hit
the store instead.
Measured on the BXD fixture, full 20-chromosome LOCO mapping,
single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums.
For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s
warm -- its own cache did not hit on a repeat here -- so the ravanan
path is now the cheaper one for any repeated request and the slightly
dearer one for a first request.
The run log stays per-call: it is what a caller streams by run_id. The
three shared files are written aside and renamed, so two concurrent
identical requests cannot read a half-written file.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
The ravanan path asks for the same variance components as the direct one:
gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes
pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them
into the same note (every chromosome, plus the extremes with vg/ve) and
the same unrounded "variance:" lines in the ?id= run log.
It reads them from the extracted GWA archive rather than from the copy
gemma-wrapper keeps beside it, and that is the point of doing it here:
the copy beside the archive is a 5eb53fa feature, while a ravanan job
runs whatever gemma-wrapper its own Guix manifest resolves, which need
not be that new. The .log.txt has always been inside the archive, and
the archive is unpacked anyway for the chromosome's .assoc.txt -- so
read_variance_components_from_logs labels each log by the "-loco <chr>"
in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt
file name, and sorts chr1..chrX.
Verification, precisely: the CWL compiles with -pve (ccwl compile, checked
by hand), the archive reader is unit-tested (labelling from the command
line, from the file name, numeric ordering, a log without the block
dropped), and a real archive from this deployment does carry the block.
An end-to-end ravanan request could NOT be completed on the dev box, for
reasons that have nothing to do with -pve and are now written down in
docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest
fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set,
since the production process does not set GUIX_PACKAGE_PATH at all, so
this endpoint cannot resolve gemma-wrapper there), its intern of inputs
hard-links and so needs store and inputs on one filesystem ("Invalid
cross-device link"), and the one job that did start sat "Waiting for
lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
|
|
The cache dir was keyed by chromosome only and shared across requests,
so a request with different inputs (another trait, or another snps or
genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir.
gemma-wrapper names those archives by a content hash, so the workflow's
single-File glob output binding then matched more than one path and the
step failed -- silently, since ravanan's user-error only logs when
%log-level is set, which never happens inside a generated job script.
Key the cache root by a hash of the input set (genotype path and size,
snps contents, trait values, lmm) as well as by chromosome, so repeated
identical requests still hit gemma-wrapper's cache while different
input sets cannot see each other's archives. chromosomes_in reads the
snps file, so it now takes the already-read contents (chromosomes_from)
to avoid reading it twice.
cwl/README.md: describe the two-part keying, and note that a node needs
a writable HOME -- ravanan forwards the submitting process's HOME into
the job (as sbatch would) and gemma-wrapper writes its lock file there.
|
|
Spell out why the current design needs the store visible (ravanan
submits a store path and Slurm does not substitute, so an absent path is
an immediate ENOENT), then list the five ways out as a table -- shared
store, realise-then-run wrapper, per-workflow realise job, guix publish
plus wrapper, shipping closures -- with what runs, what each needs on
the nodes, and what it costs in caching.
Also records where the wrapper would live (build-gexp-script already
builds the derivation but returns only its output path; submit-job in
ravanan/slurm-api.scm assembles the script field), that nothing of it is
implemented here, and the four costs that decide the question: channel
skew, build time inside the job's time limit, caching on ephemeral node
stores, and guix publish as the sane companion.
Tables tightened so no line runs past ~170 characters.
|
|
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan ->
slurmrestd -> slurmd), where Guix sits (front end only: nodes just need
the same /gnu/store mounted, plus bash, a writable ravanan store for
data/logs, and scratch for the job's working directory), the two caches
(ravanan's work reuse, checked before submitting; gemma-wrapper's
per-chromosome --cache-dir, inside the job) and why the latter is what
actually saves work for this endpoint.
It also answers "can the nodes rebuild the tools instead of sharing a
store": yes in principle, since Guix is deterministic and a node
building the same derivations would materialise identical store paths,
but ravanan submits a store path and Slurm does not substitute
automatically -- so it needs a realise-then-run wrapper (ravanan's
slurm backend already holds the script's derivation), a per-workflow
realise job, or guix publish plus a node-side daemon, with the costs
and channel-pinning caveats listed. The timing table from the test
cluster (cached: identical; fresh: waves vs synchronous steps) is
included so expectations are explicit.
docs/ravanan-slurm.md and README.md link to it.
|
|
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.
Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
|
|
Per Arun's request, express the --loco kinship/GWA workflow as a
ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm,
compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`.
ccwl inlines the two command definitions (kinship, gwa) directly into
the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer
needed as separate files.
The two scatter steps (kinship over [chr, cache_dir], gwa over [chr,
cache_dir, kinship_json], both dotproduct) and the gwa_json/archive
outputs are unchanged in substance -- verified by compiling and
running the generated gemma-loco.cwl through ravanan against the BXD
fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm
(~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144:
chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it
replaces.
|
|
--cache-dir "." pointed every scattered job at its own hermetic CWL
working directory (fresh under ravanan's --store every run), so
gemma-wrapper's cache was never actually reused across requests.
Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>,
one persistent directory per chromosome -- not a single shared
directory, because gemma-wrapper's cache-hit hash does not vary with
--chromosomes, so sharing one directory would let one chromosome's
job silently reuse another's cached result.
Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since
kinship and gwa for the same chromosome now point at the same
persistent cache_dir, gemma-wrapper finds the kinship step's archive
there directly, no staging needed.
Confirmed via a real integration run (BXD fixtures) that gemma-wrapper
itself unconditionally re-executes GEMMA regardless of whether a
matching cached archive already exists -- --cache-dir is only used to
hand results between gemma-wrapper's own pipeline stages within one
request, not to skip recomputation across requests. So this fixes a
real latent bug (archive-glob collisions once a chromosome's cache
directory holds more than one request's output) but does not, on its
own, speed up repeated identical requests; verified the direct
/compute/gemma path shows the same lack of cross-request speedup.
|
|
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper
invocation covering all chromosomes, relying on gemma-wrapper's own
internal GNU-parallel fan-out) with one gemma-wrapper invocation PER
CHROMOSOME, restricted via its existing --chromosomes/--no-parallel
flags and scattered by ravanan itself (ScatterFeatureRequirement,
scatterMethod: dotproduct). gemma-wrapper still does the correct
per-chromosome LOCO math (exclude this chromosome from kinship, test
only its markers) against the full genotype/SNP files -- only which
chromosome(s) it loops over changes. compute_ravanan now computes the
chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own
get_chromosomes) and passes it as the workflow's scatter input, and
collates the resulting per-chromosome gwa_json/archive array outputs.
Measured 16.7s wall-clock end-to-end against the BXD fixtures (20
chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for
the previous sequential ravanan workflow -- confirmed byte-for-byte
identical output (rs31443144's exact values, 7320 total rows).
Also fixes three real, previously-unexercised bugs in compute_ravanan
found getting this to run at all (POST /compute/gemma/ravanan had no
automated test until this commit):
- missing --guix-manifest (ravanan requires it; added
gemma_ravanan_manifest config key and cwl/gemma/manifest.scm)
- missing --store files/logs/data subdirectories (ravanan does not
create them itself)
- --output-directory dropped entirely: every chromosome's gwa archive
shares gemma-wrapper's own content-hash basename (the hash doesn't
factor in --chromosomes), so ravanan's optional flat-symlink step
collides across them; unneeded anyway since results are read
straight from ravanan's --store via its stdout JSON record.
Also found and worked around a ravanan scheduler bug: a step scattered
over exactly one input needs an explicit scatterMethod even though CWL
only requires one for multiple scattered inputs -- ravanan has no
fallback case for a missing one and silently misbehaves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|