about summary refs log tree commit diff
path: root/docs/ravanan-slurm.md
AgeCommit message (Collapse)Author
5 daysravanan-slurm: what happens when Slurm goes awayPjotr
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline 200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s; container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no partial results); the same request after Slurm returns -> 200 in 163.3 s, i.e. the retry reuses the steps that had finished, because ravanan's store lives in RAVANAN_CACHE_DIR and not in Slurm. So: no automatic failover, no fallback to single-machine, nothing sticky, and a retry that is cheaper than cold. The 502's detail is ravanan's Guile backtrace, so the one useful line sits at its end; a pre-flight /slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there is no timeout on the ravanan child for a Slurm that accepts connections and never runs the job -- both recorded as gaps.
6 dayscache: say when ravanan's cache cannot survive a restart, and document what doesPjotr Prins
Ravanan's reuse key is (script, inputs) and both live in its --store under RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the container's. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, with the store unchanged at 14 files -- Slurm's own spool state is lost and irrelevant to reuse. The default cache dir is under /tmp, which is neither durable nor visible inside a container that bind-mounts one host path, so a job cannot read the interned inputs. The reply now carries a note when the cache dir is under /tmp, naming the fix, rather than leaving it to be discovered as a slow or failing run. docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan store, scratch, token registry, gemma-wrapper cache, Slurm state) and a path checklist; the README points at it.
6 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
8 daysgemma/ravanan: resolve the same inputs as the direct runner; refuse what it ↵Pjotr
cannot do Two gaps, both invisible until a real request ran against a cluster. The endpoint read only genotype_file, snps_file and trait_values from the request; group, genotype_version and samples were accepted by the shared request type and silently dropped. A /compute/gemma-shaped request (group + samples) therefore arrived as empty paths and empty values and died with "I/O error: No such file or directory (os error 2)" from reading "". Running it exposed the second gap in the same breath: the gate sized the request from trait_values.len(), so a samples-only request was sized 0 and took no concurrency slot at all. resolve_inputs is now one function in routes/gemma.rs, called by both handlers: upload materialised into the pair the engine reads, basket resolved to named values, group prepared with the parent/F1 and empty-genotype rules, named samples aligned by name onto whatever individuals the file has, positional values checked against the panel size. That block was ~250 lines of duplicated rules inside compute_gemma, which is how it drifted in the first place; the direct runner now calls the same function and keeps only its covariates, normalization and gate. The ravanan reply also carries the input notes now (which genotype version, how many values aligned, what was dropped), so a run says what it mapped as well as where the peak is. Refused rather than ignored, because the CWL workflow has no step for either: covariates (422 covariates_not_supported) and normalize (422 normalization_not_supported), each naming /compute/gemma as the endpoint that can do it. A new test asserts both refusals happen before any job is submitted, so it needs no ravanan, ccwl or cluster. Verified against the Slurm container from gn-machines: group: BXD became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's chromosomes), all completed, 21 055 rows in ~3.5 min. README and docs/ravanan-slurm.md record the four requirements that run turned up, three of them not obvious: ccwl on PATH, a ravanan build with the --slurm-api-endpoint fix, the *server's* HOME writable by the job user inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and GUIX_PACKAGE_PATH for the manifest.
8 daysgemma: pve on the ravanan/slurm path, read out of each archivePjotr
The ravanan path asks for the same variance components as the direct one: gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them into the same note (every chromosome, plus the extremes with vg/ve) and the same unrounded "variance:" lines in the ?id= run log. It reads them from the extracted GWA archive rather than from the copy gemma-wrapper keeps beside it, and that is the point of doing it here: the copy beside the archive is a 5eb53fa feature, while a ravanan job runs whatever gemma-wrapper its own Guix manifest resolves, which need not be that new. The .log.txt has always been inside the archive, and the archive is unpacked anyway for the chromosome's .assoc.txt -- so read_variance_components_from_logs labels each log by the "-loco <chr>" in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt file name, and sorts chr1..chrX. Verification, precisely: the CWL compiles with -pve (ccwl compile, checked by hand), the archive reader is unit-tested (labelling from the command line, from the file name, numeric ordering, a log without the block dropped), and a real archive from this deployment does carry the block. An end-to-end ravanan request could NOT be completed on the dev box, for reasons that have nothing to do with -pve and are now written down in docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set, since the production process does not set GUIX_PACKAGE_PATH at all, so this endpoint cannot resolve gemma-wrapper there), its intern of inputs hard-links and so needs store and inputs on one filesystem ("Invalid cross-device link"), and the one job that did start sat "Waiting for lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
2026-11-05docs: what the slurm backend costs, and why job records must stayPjotr
Measured the same request (BXD fixtures, three chromosomes) through /compute/gemma, ravanan's single-machine backend and ravanan on Slurm, cold and warm. The Slurm backend is dispatch-bound at this size -- one scheduler tick per step, with each job itself running in under a second -- so cold and warm are identical there, and the README's 16.7s figure belongs to the single-machine backend. Also document the Slurm-side requirement behind that measurement: a completed job has to stay queryable for as long as the workflow runs. Once slurmctld purges the record ravanan falls back to slurmdbd, and on a cluster without accounting the workflow aborts on a 404 long after the job succeeded.
2026-10-24cwl: Document how a ravanan+Guix+Slurm request actually travels.Pjotr
cwl/README.md explains the whole path (API -> ccwl compile -> ravanan -> slurmrestd -> slurmd), where Guix sits (front end only: nodes just need the same /gnu/store mounted, plus bash, a writable ravanan store for data/logs, and scratch for the job's working directory), the two caches (ravanan's work reuse, checked before submitting; gemma-wrapper's per-chromosome --cache-dir, inside the job) and why the latter is what actually saves work for this endpoint. It also answers "can the nodes rebuild the tools instead of sharing a store": yes in principle, since Guix is deterministic and a node building the same derivations would materialise identical store paths, but ravanan submits a store path and Slurm does not substitute automatically -- so it needs a realise-then-run wrapper (ravanan's slurm backend already holds the script's derivation), a per-workflow realise job, or guix publish plus a node-side daemon, with the costs and channel-pinning caveats listed. The timing table from the test cluster (cached: identical; fresh: waves vs synchronous steps) is included so expectations are explicit. docs/ravanan-slurm.md and README.md link to it.
2026-10-24gemma: Make ravanan's batch system configurable (optional Slurm).Pjotr
POST /compute/gemma/ravanan used to run every CWL step locally, because compute_ravanan never passed ravanan a batch system. Add a RavananBatch setting: - RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing closed on anything else, exactly like GN4_DB_MODE; - slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint, --slurm-jwt (a *file*, which is what ravanan reads) and, when set, --slurm-partition/--slurm-nice; a slurm selection without an endpoint or a JWT file refuses to start rather than 502-ing every request; - --scratch defaults to ravanan_cache_dir for slurm, so the store and the job scratch space live on the same shared filesystem (with slurm that directory has to be visible to the compute nodes); - the response's ravanan note records which backend ran. The switch building is RavananBatch::ravanan_args so it is unit testable; compute_ravanan takes the backend and the scratch directory, and the route passes them from config. docs/ravanan-slurm.md documents the keys, the shared-directory requirement, and the slurm-side auth wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in slurmrestd's environment) that makes REST-submitted jobs actually run.