about summary refs log tree commit diff

Running ravanan's jobs on Slurm

POST /compute/gemma/ravanan hands the whole per-chromosome LOCO workflow to ravanan, which runs each step itself. By default every step is a local child process (ravanan's single-machine batch system). Setting RAVANAN_BATCH_SYSTEM=slurm-api makes ravanan submit each step to a Slurm cluster through slurmrestd's REST API instead, so the CWL scatter really fans out over the cluster's nodes.

Nothing else about the endpoint changes: same request, same response, same rows. ../cwl/README.md walks through the whole path a request takes (API -> ccwl -> ravanan -> slurmrestd -> slurmd), exactly what a compute node needs, and why the Guix store has to be visible to the nodes. The slurm switches are appended to ravanan's command line by RavananBatch::ravanan_args (src/config.rs), and the note in the response's meta.notes says which backend ran (ravanan (…, slurm-api backend): …).

Configuration

key meaning
RAVANAN_BATCH_SYSTEM single-machine (default) or slurm-api; anything else refuses to start
RAVANAN_SLURM_API_ENDPOINT slurmrestd base URL, e.g. http://127.0.0.1:6820 (required for slurm-api)
RAVANAN_SLURM_JWT path to a file holding the JWT (ravanan's --slurm-jwt takes a file, not the token); required for slurm-api
RAVANAN_SLURM_PARTITION optional --slurm-partition, e.g. debug
RAVANAN_SLURM_NICE optional integer --slurm-nice
RAVANAN_SCRATCH optional --scratch; defaults to RAVANAN_CACHE_DIR when the backend is slurm
RAVANAN_CACHE_DIR where pheno files, logs, the compiled CWL and ravanan's --store live

Selecting slurm without an endpoint or a JWT file is a startup error (fail closed, like GN4_DB_MODE): ravanan would fail every step, and a refusal at boot beats a 502 on every request.

With slurm, RAVANAN_CACHE_DIR must be on a filesystem the compute nodes can see, because ravanan's --store and --scratch live there and the slurmd-side job reads and writes them. On a single box that is just any shared directory; the same applies to the genotype/SNP/pheno inputs, whose paths are passed to ravanan verbatim (relative paths are resolved relative to the generated inputs YAML, so absolute paths are the safe choice).

Two things the cluster needs that a plain dev box may not have:

  • gemma reachable in the profile ravanan builds from GEMMA_RAVANAN_MANIFEST (the workflow invokes gemma itself, not gemma-wrapper -- see cwl/gemma/gemma-loco.scm); ravanan resolves that manifest against the ambient Guix process, so GUIX_PACKAGE_PATH (or a pulled guix-bioinformatics channel) must be set in the server's environment, as it already must for the single-machine backend;
  • /bin/bash on the nodes: ravanan submits every step as #!/bin/bash + the store script.

Example

export RAVANAN_BATCH_SYSTEM=slurm-api
export RAVANAN_SLURM_API_ENDPOINT=http://127.0.0.1:6820
export RAVANAN_SLURM_JWT=/etc/gn4/slurm-jwt        # file with the token
export RAVANAN_SLURM_PARTITION=debug
export RAVANAN_CACHE_DIR=/srv/gn4-ravanan          # shared with the nodes
./genenetwork4

src/config.rs's tests cover the switch, the argument construction and the fail-closed cases (ravanan_batch_*).

What survives a restart

Ravanan's reuse key is (script, inputs), and both halves live in RAVANAN_CACHE_DIR (its --store). So the cache's lifetime is exactly the lifetime of that directory — not of the container and not of Slurm:

state where survives a container restart survives a host reboot must the nodes see it
ravanan store (--store: interned inputs, per-step outputs, reuse records) RAVANAN_CACHE_DIR yes, if it is under the bind-mounted share yes, if it is not under /tmp yes
ravanan scratch RAVANAN_SCRATCH, or the cache dir when unset as above as above yes
short-token registry GN4_API_TOKENS_DB (tokens.sqlite) yes, if not on an ephemeral path yes, if not under /tmp no
gemma-wrapper cache (direct runner only) GEMMA_CACHE_DIR yes yes, if not under /tmp no
Slurm controller/spool state (/var/spool/slurmctld, /var/spool/slurmd) inside the container no — rebuilt at every boot — —

Measured: with the cache at <shared mount>/gn4-ravanan-cache, a full 20-chromosome LOCO mapping cost 13.6 s cold; after restarting the container the identical request returned in 3.6 s with the store unchanged (14 files before and after). Slurm's lost state is irrelevant to that — the reuse record is in the store, not in slurmctld.

Two ways to get this wrong, both silent until a run fails or is slow:

  • the default RAVANAN_CACHE_DIR is under /tmp, which is neither durable nor visible inside a container that bind-mounts one host path, so jobs cannot read the interned inputs. The reply says so in a note when the cache dir is under /tmp.
  • GN4_API_TOKENS_DB defaults to ./tokens.sqlite in the working directory: a service whose WorkingDirectory is a checkout under /tmp, or a directory that is recreated, loses every short token on reboot.

Deployment checklist for the paths:

RAVANAN_CACHE_DIR=<shared mount>/gn4-ravanan-cache     # durable, nodes can see it
RAVANAN_SCRATCH=<shared mount>/gn4-ravanan-cache       # or node-local scratch
GN4_API_TOKENS_DB=<durable dir>/tokens.sqlite          # mode 0600, never in the repo
GEMMA_CACHE_DIR=<durable dir>/gn4-gemma                # direct runner only

Variance components (pve) on this path

gemma-loco.scm's gwa step passes -pve, as the direct path does, and compute_ravanan reads the components back out of each job's own .log.txt, which the workflow returns as a first-class CWL output (gwa_log). No archive is unpacked and no wrapper-named copy is involved: the workflow runs gemma itself, so the log is just a file it produced.

The note is the same shape as /compute/gemma's: every chromosome's pve and se(pve), the lowest and highest with their vg/ve. Each log is labelled by the -loco <chr> in its own command line, falling back to the <hash>.<chr>.assoc.txt.log.txt file name.

Not verified end to end on the dev box. The CWL compiles with -pve (check with ccwl compile --output=/tmp/w.cwl cwl/gemma/gemma-loco.scm), the archive reader is unit-tested, and a real archive from this deployment does carry the pve block -- but a full ravanan request could not be completed here: ravanan's own Guix evaluation of the manifest and its intern of the input files fail (see below). None of that is about -pve, and all of it would show up identically on a run without it.

Two environment facts found while trying, both worth fixing before this path is trusted:

  • GUIX_PACKAGE_PATH must be in the server's environment, since ravanan resolves --guix-manifest against the ambient Guix; the production process does not set it, so /compute/gemma/ravanan cannot resolve gemma at all there. Without it the manifest fails with a message about an unrelated package (rust-pyo3-0.26.0: unbound variable) from the user's channel config.
  • ravanan interns each input file into --store with a hard link, so the store and every input path must be on the same filesystem; /tmp and the deployment's data directories are, but a pheno file written into a different mount gives Invalid cross-device link.

Requirements found by running it (2026-11-12)

Verified against gn-machines/specials/slurm-container.scm started through run-guix-system-container (daemon on 127.0.0.1:8989, fingerprint 74c85932): POST /compute/gemma/ravanan with group: BXD produced 40 Slurm jobs -- 20 kinship + 20 gwa, one pair per chromosome -- all completed, 21 055 rows back, in about 3.5 minutes.

Four things had to be right, and three of them were not obvious:

  • ccwl and ravanan on the server's PATH. Without ccwl the endpoint answers 502 failed to run ccwl: No such file or directory before doing anything.
  • A ravanan with the --slurm-api-endpoint fix. The 0.2.0 build in the store predates ravanan: Parse --slurm-api-endpoint into a URI and dies with struct-vtable: Wrong type argument: "http://127.0.0.1:6820" (it uses the string where a uri record is expected); a non-default endpoint therefore needs that build. Using the built-in default (http://localhost:6820, i.e. omitting the switch) is the workaround on an older ravanan.
  • The server's HOME must be writable by the job user inside the cluster. ravanan forwards the submitting process's HOME into every job (as sbatch would) and jobs inherit it. This mattered when the jobs ran gemma-wrapper, which locks $HOME/.<hash>.lck: inside the container there is no /home/wrk, so every kinship job failed with can not release ... .lck because it is not owned by me / EACCES, and starting the API with HOME=<a directory on the shared mount> fixed it. The workflow now runs gemma itself, which takes no such lock -- the requirement is kept here because ravanan still forwards HOME and a future step could assume it. This is a server-side setting, not a node one.
  • GUIX_PACKAGE_PATH, as above: ravanan resolves the manifest against the ambient Guix, so without it the manifest cannot see (gn packages gemma) and the failure surfaces as an unrelated channel error (rust-pyo3-…: unbound variable).

Slurm-side requirements

The endpoint only speaks HTTP to slurmrestd, so the cluster has to be wired the way Slurm expects for REST-submitted jobs to actually run:

AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/path/to/jwt.key

with slurmrestd -a rest_auth/jwt started with SLURM_JWT in its environment, and the JSON Web Token minted for the user the jobs should run as. The token's identity is forwarded to slurmctld, which decodes it; slurmd keeps using munge, which is what it can verify. Setting AuthType=auth/jwt cluster-wide instead lets submissions be recorded but every launch RPC is rejected as SLURM_AUTH_NOBODY (uid 99), since slurmd never loads the JWT key. See gn-machines/slurm-session.md for the source-level details and a working test container (gn-machines/specials/slurm-container.scm).

Completed jobs have to stay queryable for as long as the workflow runs: ravanan polls each job with GET /slurm/v0.0.41/job/{id}, and when slurmctld has already purged the record it falls back to slurmdbd (/slurmdb/v0.0.41/job/{id}), which 404s on a cluster without accounting. The run then aborts with Slurm API request failed 404 GET "/slurmdb/...", long after the job itself succeeded — from the caller's side it looks like an ordinary step failure. Either keep the records around (MinJobAge=86400 or more, as the test container does) or run a real slurmdbd.

ravanan's own switches are --batch-system, --slurm-api-endpoint, --slurm-jwt, --slurm-partition, --slurm-nice and --scratch; note that --slurm-api-endpoint only accepts a URL on ravanan v0.2.0-92-era checkouts after commit "ravanan: Parse --slurm-api-endpoint into a URI" (before that the value was used as a uri record while being stored as a string, so only the built-in default http://localhost:6820 worked).

What the slurm backend costs

Ravanan's default single-machine backend runs each CWL step as a local child process. With slurm-api every step becomes a Slurm job, and each of those pays dispatch on top of the step's own runtime: submit over REST, wait for the scheduler's next evaluation, launch through slurmd, and then poll until the state says COMPLETED. On small inputs that dispatch is the whole runtime.

Measured on gn-machines/specials/slurm-container.scm (one node, two CPUs, SchedulerParameters=sched_interval=5,...), BXD fixtures, the same three chromosomes and trait through all three runners — identical 1650 result rows each time:

runner cold warm
/compute/gemma (direct) 2.29 s 0.71 s
/compute/gemma/ravanan, single-machine 4.70 s 3.45 s
/compute/gemma/ravanan, slurm-api (5 s scheduler interval) 32.6 s 32.6 s
/compute/gemma/ravanan, slurm-api (Slurm defaults) 157.9 s 172.9 s

Six Slurm jobs (three kinship, three gwa), each reporting under a second of runtime in Slurm's own records: the ~33 s is 6 × one scheduler tick, and the ~160-170 s is the same with Slurm's default tens-of-seconds intervals (start delays measured at 19/49/79 s). Cold and warm are identical there because the GEMMA work the cache saves is smaller than a single tick; the direct and single-machine rows show the cache clearly (3.2× and 1.4×).

So: the slurm backend is worth it when a step's own runtime is much larger than the scheduler interval (real genome scans with many markers) and when the cluster actually has several nodes to fan out to (ravanan scatters one job per chromosome; this test container has a single node, so the steps serialize). On a 20-chromosome BXD run against the same container with default intervals, 40 jobs took 1193 s end to end.

What happens when Slurm goes away

There is no automatic failover: nothing retries, nothing falls back to the single-machine backend, and a request in flight is not resumed. What there is, is a fast failure and a cheap retry. Measured on the same container (a 20-chromosome BXD run, 40 Slurm jobs, slurm-api backend, POST /compute/gemma/ravanan):

when Slurm dies response rows
never — baseline 200 21 055 205.5 s
before the request (slurmrestd unreachable) 502 gemma_failed 0 3.7 s
45 s into the request (container killed) 502 gemma_failed 0 50.2 s
after the abort, Slurm back, same request 200 21 051 163.3 s

Four things to know:

  • The failure is fast and identical either way: ravanan raises inside its Guile process — In procedure connect: Connection refused for POST /slurm/v0.0.41/job/submit. The 502's detail is ravanan's whole stderr, so the useful line is at the very end of a long backtrace. A pre-flight GET /slurm/v0.0.41/ping would turn the first row into a 503 slurm_unavailable naming the endpoint, before any staging; that is not implemented.
  • No partial results are returned, which is right — a half-mapped genome with 0 rows and a 502 is honest. ?id=/status tails ravanan's log, not a result stream, so there is nothing half-written to read.
  • Nothing is sticky. The same request after Slurm returns is 200; the failed attempt leaves no lock, no half-interned store entry, nothing to clean up.
  • The retry is cheaper than cold, because ravanan's store survives: 163.3 s against 205.5 s cold on the same 20 chromosomes. The steps that finished before the kill are reused — (script, inputs) pairs live in RAVANAN_CACHE_DIR, not in Slurm, and the kinship step is trait-independent by design (write_constant_pheno_file), so its 20 jobs also serve the next trait. Practical rule for a caller: treat a 502 from this endpoint as retryable, and retry the same request.

Two gaps this leaves, both worth closing before production trusts the path:

  • no pre-flight, so a dead cluster costs a ccwl compile, an input staging and a ravanan startup before the connection is even attempted;
  • no timeout on the ravanan child process. Nothing here exercises a Slurm that accepts connections and then never runs the job, and in that case the request waits as long as ravanan does. A wall-clock bound with a 504 would be the honest answer.