| Age | Commit message (Collapse) | Author |
|
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline
200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s;
container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no
partial results); the same request after Slurm returns -> 200 in 163.3 s,
i.e. the retry reuses the steps that had finished, because ravanan's store
lives in RAVANAN_CACHE_DIR and not in Slurm.
So: no automatic failover, no fallback to single-machine, nothing sticky,
and a retry that is cheaper than cold. The 502's detail is ravanan's Guile
backtrace, so the one useful line sits at its end; a pre-flight
/slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there
is no timeout on the ravanan child for a Slurm that accepts connections
and never runs the job -- both recorded as gaps.
|
|
Ravanan's reuse key is (script, inputs) and both live in its --store under
RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the
container's. Measured: 13.6 s cold, then 3.6 s for the identical request
after restarting the container, with the store unchanged at 14 files --
Slurm's own spool state is lost and irrelevant to reuse.
The default cache dir is under /tmp, which is neither durable nor visible
inside a container that bind-mounts one host path, so a job cannot read
the interned inputs. The reply now carries a note when the cache dir is
under /tmp, naming the fix, rather than leaving it to be discovered as a
slow or failing run.
docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan
store, scratch, token registry, gemma-wrapper cache, Slurm state) and a
path checklist; the README points at it.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
cannot do
Two gaps, both invisible until a real request ran against a cluster.
The endpoint read only genotype_file, snps_file and trait_values from the
request; group, genotype_version and samples were accepted by the shared
request type and silently dropped. A /compute/gemma-shaped request
(group + samples) therefore arrived as empty paths and empty values and
died with "I/O error: No such file or directory (os error 2)" from
reading "". Running it exposed the second gap in the same breath: the
gate sized the request from trait_values.len(), so a samples-only request
was sized 0 and took no concurrency slot at all.
resolve_inputs is now one function in routes/gemma.rs, called by both
handlers: upload materialised into the pair the engine reads, basket
resolved to named values, group prepared with the parent/F1 and
empty-genotype rules, named samples aligned by name onto whatever
individuals the file has, positional values checked against the panel
size. That block was ~250 lines of duplicated rules inside compute_gemma,
which is how it drifted in the first place; the direct runner now calls
the same function and keeps only its covariates, normalization and gate.
The ravanan reply also carries the input notes now (which genotype
version, how many values aligned, what was dropped), so a run says what it
mapped as well as where the peak is.
Refused rather than ignored, because the CWL workflow has no step for
either: covariates (422 covariates_not_supported) and normalize (422
normalization_not_supported), each naming /compute/gemma as the endpoint
that can do it. A new test asserts both refusals happen before any job is
submitted, so it needs no ravanan, ccwl or cluster.
Verified against the Slurm container from gn-machines: group: BXD became
40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's
chromosomes), all completed, 21 055 rows in ~3.5 min. README and
docs/ravanan-slurm.md record the four requirements that run turned up,
three of them not obvious: ccwl on PATH, a ravanan build with the
--slurm-api-endpoint fix, the *server's* HOME writable by the job user
inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and
GUIX_PACKAGE_PATH for the manifest.
|
|
The ravanan path asks for the same variance components as the direct one:
gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes
pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them
into the same note (every chromosome, plus the extremes with vg/ve) and
the same unrounded "variance:" lines in the ?id= run log.
It reads them from the extracted GWA archive rather than from the copy
gemma-wrapper keeps beside it, and that is the point of doing it here:
the copy beside the archive is a 5eb53fa feature, while a ravanan job
runs whatever gemma-wrapper its own Guix manifest resolves, which need
not be that new. The .log.txt has always been inside the archive, and
the archive is unpacked anyway for the chromosome's .assoc.txt -- so
read_variance_components_from_logs labels each log by the "-loco <chr>"
in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt
file name, and sorts chr1..chrX.
Verification, precisely: the CWL compiles with -pve (ccwl compile, checked
by hand), the archive reader is unit-tested (labelling from the command
line, from the file name, numeric ordering, a log without the block
dropped), and a real archive from this deployment does carry the block.
An end-to-end ravanan request could NOT be completed on the dev box, for
reasons that have nothing to do with -pve and are now written down in
docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest
fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set,
since the production process does not set GUIX_PACKAGE_PATH at all, so
this endpoint cannot resolve gemma-wrapper there), its intern of inputs
hard-links and so needs store and inputs on one filesystem ("Invalid
cross-device link"), and the one job that did start sat "Waiting for
lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
|
|
Measured the same request (BXD fixtures, three chromosomes) through
/compute/gemma, ravanan's single-machine backend and ravanan on Slurm,
cold and warm. The Slurm backend is dispatch-bound at this size -- one
scheduler tick per step, with each job itself running in under a second
-- so cold and warm are identical there, and the README's 16.7s figure
belongs to the single-machine backend.
Also document the Slurm-side requirement behind that measurement: a
completed job has to stay queryable for as long as the workflow runs.
Once slurmctld purges the record ravanan falls back to slurmdbd, and on
a cluster without accounting the workflow aborts on a 404 long after the
job succeeded.
|
|
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan ->
slurmrestd -> slurmd), where Guix sits (front end only: nodes just need
the same /gnu/store mounted, plus bash, a writable ravanan store for
data/logs, and scratch for the job's working directory), the two caches
(ravanan's work reuse, checked before submitting; gemma-wrapper's
per-chromosome --cache-dir, inside the job) and why the latter is what
actually saves work for this endpoint.
It also answers "can the nodes rebuild the tools instead of sharing a
store": yes in principle, since Guix is deterministic and a node
building the same derivations would materialise identical store paths,
but ravanan submits a store path and Slurm does not substitute
automatically -- so it needs a realise-then-run wrapper (ravanan's
slurm backend already holds the script's derivation), a per-workflow
realise job, or guix publish plus a node-side daemon, with the costs
and channel-pinning caveats listed. The timing table from the test
cluster (cached: identical; fresh: waves vs synchronous steps) is
included so expectations are explicit.
docs/ravanan-slurm.md and README.md link to it.
|
|
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system. Add a
RavananBatch setting:
- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
--slurm-jwt (a *file*, which is what ravanan reads) and, when set,
--slurm-partition/--slurm-nice; a slurm selection without an endpoint
or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
the job scratch space live on the same shared filesystem (with slurm
that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.
The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config. docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
|