ravanan + Guix + Slurm: how a request travels
What is in this directory, how POST /compute/gemma/ravanan gets from an
HTTP request to jobs running on cluster nodes, and which part of the
stack has to exist where. Configuration keys are in
../docs/ravanan-slurm.md; the Slurm-side
authentication wiring is in gn-machines/slurm-session.md.
cwl/
gemma/gemma-loco.scm the workflow, in ccwl (not raw CWL)
gemma/manifest.scm the Guix manifest ravanan resolves job commands against
Both files are sources: compute_ravanan compiles the ccwl to a
throwaway .cwl under ravanan_cache_dir on every call (ccwl
compile, pure macro expansion, no Guix build), so no CWL is ever
checked in. To inspect it by hand:
ccwl compile --output=gemma-loco.cwl gemma/gemma-loco.scm
The sequence
POST /compute/gemma/ravanan
│
├─ compute_ravanan (Rust, API host)
│ writes <cache_dir>/<pid>-<ns>-pheno.txt and -ravanan-inputs.yaml
│ ccwl compile gemma-loco.scm -> <cache_dir>/<pid>-<ns>-gemma-loco.cwl
│ creates <cache_dir>/ravanan-store/{files,logs,data}
│ invokes: ravanan --store=<cache_dir>/ravanan-store
│ --guix-manifest=gemma/manifest.scm
│ [--batch-system=slurm-api --slurm-api-endpoint=…
│ --slurm-jwt=<file> [--slurm-partition=…] [--scratch=…]]
│ gemma-loco.cwl inputs.yaml
│
├─ ravanan (front end / API host; never on a node)
│ builds a Guix profile from the manifest (gemma)
│ per CWL step: gexp->script -> a script in /gnu/store that embeds
│ the profile's PATH, ravanan's own work library, and the command
│ slurm-api: POST /slurm/v0.0.41/job/submit with
│ script = "#!/bin/bash\n<script>"
│ environment = WORKFLOW_INPUTS, WORKFLOW_OUTPUT_DIRECTORY,
│ WORKFLOW_OUTPUT_DATA_FILE
│ working dir = "/"
│ stdout/stderr = <store>/logs/<hash>-<step>.{stdout,stderr}
│ polls GET /slurm/v0.0.41/job/{id} until COMPLETED, then reads
│ <store>/data/<hash>.json and <store>/files/<hash>-<step>/…
│
└─ slurmd on a node
bash -> the store script -> gemma -g … -p … -a … -loco <chr>
-gk|-lmm [-k <kinship from the previous step>] -o <chr>-kin|<chr>-gwa
The CWL scatter is over the chromosomes compute_ravanan extracted
from snps_file, so a 20-chromosome dataset is 20 kinship jobs and
20 gwa jobs, each restricted to its own chromosome.
Where Guix is, and what a node needs
Guix runs only on the front end: ravanan asks the local
guix-daemon for the manifest profile, and the only thing that reaches
a node is store paths baked into the script (#!/gnu/store/…-guile,
PATH=/gnu/store/…-profile/bin:…, the gemma path).
Consequently:
| on the front end | on the compute nodes |
|---|---|
genenetwork4, ravanan, ccwl, guix + guix-daemon, the channel/manifests, network |
no ravanan, no ccwl, no guix, no daemon |
A node does need:
| requirement | why |
|---|---|
/bin/bash |
ravanan submits #!/bin/bash + the store script; Slurm runs it via that shebang |
same /gnu/store mounted (read-only ok) |
the script, guile, gemma, coreutils are all store paths |
| read access to ravanan's store | input files are interned there and staged from there into the job's cwd |
write access to its data/, logs/ |
the job writes its outputs record and its stdout/stderr there |
| a writable scratch dir | the job's cwd is a temporary dir under --scratch: keep it node-local |
| permission as the job user | identity comes from the JWT (e.g. testuser); these paths must be writable by it |
a writable $HOME for the job user |
ravanan forwards the submitting process's HOME into the job (as sbatch would). The jobs now run gemma, which takes no lock and does not care; this row is kept because a step that does assume a writable $HOME would fail silently cheaply |
resolvable NodeAddr |
Slurm's launch path needs the node address, not just a name |
Practical split: RAVANAN_CACHE_DIR (ravanan's store) on a
shared filesystem, RAVANAN_SCRATCH
node-local (/scratch). The default, "scratch = cache dir", is
only meant for single-box setups like the test container.
Caches
| cache | lives in | scope |
|---|---|---|
| ravanan work reuse | <store>/data/<hash>.json, keyed by (script, inputs) |
front end; a hit never touches Slurm |
The kinship step is keyed on the animal set, not the trait. gemma -gk
reads the phenotype file only to learn which individuals are analysed; the
matrix comes from the genotypes. So the workflow gives the kinship step a
constant-valued pheno file with the trait's missing-data pattern
(kinship_pheno_file, written by write_constant_pheno_file), keyed by
kinship_input_key, and gives gwa the real values. Two traits over the same
animals then share all twenty K matrices: on the BXD fixture a second trait
costs +1.0 MB and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s
(20 kinship + 20 GWA jobs), and its rows are byte-identical to a run that
computed K from the real phenotype.
ravanan's work reuse used to never hit here, because
compute_ravanan named the pheno/inputs/CWL files after a pid and a
timestamp, so the (script, inputs) key was new on every call. Those
three files are now named by a hash of the request's inputs
(ravanan_input_key), so a repeat of the same request is answered
from the store: measured on the BXD fixture, the full 20-chromosome
LOCO mapping costs 14.3 s cold and 1.6 s warm single-machine, and
275.6 s cold and 1.6 s warm through Slurm in the test container
(40 jobs, 7 320 rows, identical per-chromosome .assoc.txt checksums
both ways). The Slurm cold run is ~6.9 s per job against ~0.36 s
single-machine: on this fixture gemma itself is well under a second
per chromosome, so the cluster figure is container and scheduling
overhead, and Slurm only pays off with real node parallelism. The run log stays per-call,
since it is what a caller streams by run_id.
There is deliberately no kinship cache. The workflow runs gemma
itself (see cwl/gemma/gemma-loco.scm), one invocation per chromosome,
and hands each chromosome's kinship matrix from the kinship step to
the gwa step as a CWL File output -- so within a request nothing is
computed twice, and a repeat request recomputes. That is the one thing
gemma-wrapper's content-hash --cache-dir used to buy, and it bought it
with a key that does not vary with --chromosomes (the reason the cache
directory had to be one per chromosome) and with .tar.xz archives
named by content hash (the reason it had to be one per input set). If
repeat-request latency ever matters, the honest replacement is our own
cache keyed by sha256(genotype, pheno, snps, lmm, chr) -- the
chromosome included by construction -- but measure first: gemma's own
kinship step on the BXD fixture is a couple of seconds.
Can the nodes rebuild the tools instead of sharing a store?
Short answer: yes, in principle, but nothing in the current setup triggers it, so today the store really does have to be visible to the nodes.
Why the current design needs the store visible:
- ravanan builds each CWL step into a job script on the front end
(
gexp->script) and submits that store path (#!/bin/bash+\n+/gnu/store/…-<step>). Everything the job executes — theguilein the shebang, the profilePATH,gemma, coreutils — is a store path baked in at that moment; - Slurm knows nothing about Guix, so there is no substitution: a
store path that is absent on the node just fails with
ENOENT, the job never starts, and ravanan reports the step as failed; - Guix is deterministic, so a node that builds the same derivations with the same channels would produce byte-identical paths and the script would then exist — which is what makes the options below possible at all.
| option | what runs | prerequisites | caching |
|---|---|---|---|
| 1. shared store | nodes read the front end's /gnu/store; nothing extra runs |
store mounted on the nodes | full (~0.9 s cached rerun) |
| 2. realise-then-run | guix build <script>.drv then exec <script> per job |
guix + daemon per node, identical channels | only if the node store persists |
| 3. per-workflow realise | one job builds/imports every step derivation first | as (2) plus job ordering | as (2) |
4. guix publish + (2) |
as (2), substituting from the front end | as (2) with --substitute-urls |
as (2), faster cold nodes |
| 5. ship closures | guix archive --export / guix copy before the run |
no guix on nodes; per-run shipping | within one run only |
The wrapper of option 2/4 is about ten lines of change on the ravanan
side: build-gexp-script in ravanan/command-line-tool.scm already
builds the derivation (gexp->script returns a <derivation>) but
returns only its output path, and submit-job in
ravanan/slurm-api.scm assembles the script field of the job spec —
so keeping the .drv path and submitting
#!/bin/bash\nguix build <drv> && exec <script> is the whole
mechanism. Nothing of that is implemented in this repo; the endpoint
uses ravanan as-is, and compute_ravanan cannot influence how a step is
launched beyond the switches it passes.
Costs to weigh before choosing it:
- channel skew breaks it. Every node and the front end must run the
same Guix commit and the same channels; a
guix pullon one side changes the hashes and nothing matches any more. Pinning that across a cluster is an operational commitment, not a config file. - build time lands inside the job. A cold gemma/libc closure is
minutes per node; the first job on each node pays it inside the Slurm
time limit, and
gemma's build is the same one the front end already did. - caching evaporates on ephemeral node stores. ravanan's free
"cached rerun" (below) only holds if the node's store survives between
jobs; with per-job containers or a wiped
/gnu/storeyou re-realise the closure every time. guix publishis the sane companion. Front end keeps building and serving (guix publish), nodes substitute with--substitute-urls=http://frontend:5000; that turns "rebuild on the node" into "download from the front end", which is what NFS would have given you anyway — minus the shared-filesystem dependency, plus per-node guix and a published store.
Recommendation: keep the shared (or NFS-exported) store as the default —
ravanan is designed around it and the timings below assume it. If the
store genuinely cannot be exported and the cluster already runs a Guix
daemon with pinned channels, implement options 2 + 4 upstream in
ravanan and expose it here as an opt-in config switch; until then,
RAVANAN_SCRATCH should stay node-local (/scratch) while
RAVANAN_CACHE_DIR stays shared.
What to expect, timing-wise
Measured on a single-box test cluster (one node, 2 CPUs); ravanan's
single-machine backend runs steps synchronously, so a scatter gets
no parallelism there, while Slurm runs as many as fit (2 here).
| workload | single-machine | slurm-api |
|---|---|---|
| cached rerun (warm store), 2-step workflow | 0.91 s | 0.92 s |
| 8 × 2 s steps | 16.0 s | 12.6 s |
| 20 × 2 s steps | 40.0 s | 30.0 s |
| 20 × 1 s steps | 20.1 s | 25.4 s |
| 20 × 0.2 s steps | 4.1 s | 27.6 s |
Cached runs are backend-independent (ravanan returns the stored result
before it ever talks to the batch system). Fresh runs trade roughly
ceil(jobs / cores) waves against a per-wave scheduling overhead of
~1–2.5 s, so Slurm only wins when there are at least two usable cores
and steps are longer than a second or two; with per-chromosome
gemma steps of a second or two on a 2-CPU box it can be slower than
local, and on a real cluster with ≥4 effective cores it wins
approximately linearly. The wrapper's Ruby startup used to be part of
every one of those steps; the workflow now invokes gemma directly,
which shortens each step by that much.
Slurm-side prerequisite (one-time)
The endpoint only speaks HTTP to slurmrestd, so the cluster must accept REST-submitted jobs and run them:
AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/path/to/jwt.key
slurmrestd -a rest_auth/jwt started with SLURM_JWT in its own
environment (that is what makes auth_g_init pick auth/jwt for the
REST daemon so rest_auth/jwt's per-request identity switch is
accepted), and the JWT minted for the user the jobs should run as.
Setting AuthType=auth/jwt cluster-wide instead records submissions but
rejects every launch with SLURM_AUTH_NOBODY (uid 99): slurmd never
loads the JWT key. gn-machines/slurm-session.md has the source-level
details and gn-machines/specials/slurm-container.scm a working test
cluster.
