| Age | Commit message (Collapse) | Author |
|
gemma -gk reads the phenotype file only to learn which individuals are
analysed -- the matrix comes from the genotypes -- so the workflow now
gives the kinship step a constant-valued pheno file carrying just the
trait's missing-data pattern, and gives gwa the real values. Verified
byte-for-byte: -gk on a real trait and on its constant-valued twin
produce identical .cXX.txt files, and trait 2's assoc checksums are
identical whether K came from the constant pheno or the real one.
Consequence, measured on the BXD fixture: a second trait over the same
animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of
+13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices
are identical inputs for both traits. Three keys now: workflow (the
ccwl source), kinship (genotype + snps + lmm + the NA pattern) and
trait (kinship key + the values), so the compiled CWL and the kinship
pheno file are shared and only the trait's pheno file and inputs YAML
are per-trait.
This is also stricter than gemma-wrapper's own key, which ignores the
phenotype entirely and would reuse a K computed for a different set of
animals.
|
|
Full 20-chromosome LOCO mapping on the BXD fixture: 14.3 s cold and
1.6 s warm single-machine, 275.6 s cold and 1.6 s warm through the Slurm
test container -- 40 jobs, 7320 rows, identical .assoc.txt checksums in
every case. Slurm costs ~6.9 s per job here against ~0.36 s
single-machine, which is container and scheduling overhead rather than
gemma: the first request is cheaper locally, and only a repeated request
(answered from ravanan's store) is fast either way.
|
|
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and
-gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on
every call and a repeat of the identical request recomputed all 40
steps. Naming them by a hash of the request's inputs (genotype path and
size, snps contents, trait values, lmm) makes an identical request hit
the store instead.
Measured on the BXD fixture, full 20-chromosome LOCO mapping,
single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums.
For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s
warm -- its own cache did not hit on a repeat here -- so the ravanan
path is now the cheaper one for any repeated request and the slightly
dearer one for a first request.
The run log stays per-call: it is what a caller streams by run_id. The
three shared files are written aside and renamed, so two concurrent
identical requests cannot read a half-written file.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
The cache dir was keyed by chromosome only and shared across requests,
so a request with different inputs (another trait, or another snps or
genotype file) left a second *-gemma-cXX.tar.xz in the same chr-N dir.
gemma-wrapper names those archives by a content hash, so the workflow's
single-File glob output binding then matched more than one path and the
step failed -- silently, since ravanan's user-error only logs when
%log-level is set, which never happens inside a generated job script.
Key the cache root by a hash of the input set (genotype path and size,
snps contents, trait values, lmm) as well as by chromosome, so repeated
identical requests still hit gemma-wrapper's cache while different
input sets cannot see each other's archives. chromosomes_in reads the
snps file, so it now takes the already-read contents (chromosomes_from)
to avoid reading it twice.
cwl/README.md: describe the two-part keying, and note that a node needs
a writable HOME -- ravanan forwards the submitting process's HOME into
the job (as sbatch would) and gemma-wrapper writes its lock file there.
|
|
Spell out why the current design needs the store visible (ravanan
submits a store path and Slurm does not substitute, so an absent path is
an immediate ENOENT), then list the five ways out as a table -- shared
store, realise-then-run wrapper, per-workflow realise job, guix publish
plus wrapper, shipping closures -- with what runs, what each needs on
the nodes, and what it costs in caching.
Also records where the wrapper would live (build-gexp-script already
builds the derivation but returns only its output path; submit-job in
ravanan/slurm-api.scm assembles the script field), that nothing of it is
implemented here, and the four costs that decide the question: channel
skew, build time inside the job's time limit, caching on ephemeral node
stores, and guix publish as the sane companion.
Tables tightened so no line runs past ~170 characters.
|
|
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan ->
slurmrestd -> slurmd), where Guix sits (front end only: nodes just need
the same /gnu/store mounted, plus bash, a writable ravanan store for
data/logs, and scratch for the job's working directory), the two caches
(ravanan's work reuse, checked before submitting; gemma-wrapper's
per-chromosome --cache-dir, inside the job) and why the latter is what
actually saves work for this endpoint.
It also answers "can the nodes rebuild the tools instead of sharing a
store": yes in principle, since Guix is deterministic and a node
building the same derivations would materialise identical store paths,
but ravanan submits a store path and Slurm does not substitute
automatically -- so it needs a realise-then-run wrapper (ravanan's
slurm backend already holds the script's derivation), a per-workflow
realise job, or guix publish plus a node-side daemon, with the costs
and channel-pinning caveats listed. The timing table from the test
cluster (cached: identical; fresh: waves vs synchronous steps) is
included so expectations are explicit.
docs/ravanan-slurm.md and README.md link to it.
|