# Running ravanan's jobs on Slurm `POST /compute/gemma/ravanan` hands the whole per-chromosome LOCO workflow to [ravanan](https://git.systemreboot.net/ravanan), which runs each step itself. By default every step is a local child process (ravanan's `single-machine` batch system). Setting `RAVANAN_BATCH_SYSTEM=slurm-api` makes ravanan submit each step to a Slurm cluster through slurmrestd's REST API instead, so the CWL scatter really fans out over the cluster's nodes. Nothing else about the endpoint changes: same request, same response, same rows. [../cwl/README.md](../cwl/README.md) walks through the whole path a request takes (API -> ccwl -> ravanan -> slurmrestd -> slurmd), exactly what a compute node needs, and why the Guix store has to be visible to the nodes. The slurm switches are appended to ravanan's command line by `RavananBatch::ravanan_args` (`src/config.rs`), and the note in the response's `meta.notes` says which backend ran (`ravanan (…, slurm-api backend): …`). ## Configuration | key | meaning | |---|---| | `RAVANAN_BATCH_SYSTEM` | `single-machine` (default) or `slurm-api`; anything else refuses to start | | `RAVANAN_SLURM_API_ENDPOINT` | slurmrestd base URL, e.g. `http://127.0.0.1:6820` (required for slurm-api) | | `RAVANAN_SLURM_JWT` | path to a **file** holding the JWT (ravanan's `--slurm-jwt` takes a file, not the token); required for slurm-api | | `RAVANAN_SLURM_PARTITION` | optional `--slurm-partition`, e.g. `debug` | | `RAVANAN_SLURM_NICE` | optional integer `--slurm-nice` | | `RAVANAN_SCRATCH` | optional `--scratch`; defaults to `RAVANAN_CACHE_DIR` when the backend is slurm | | `RAVANAN_CACHE_DIR` | where pheno files, logs, the compiled CWL and ravanan's `--store` live | Selecting slurm without an endpoint or a JWT file is a startup error (fail closed, like `GN4_DB_MODE`): ravanan would fail every step, and a refusal at boot beats a 502 on every request. **With slurm, `RAVANAN_CACHE_DIR` must be on a filesystem the compute nodes can see**, because ravanan's `--store` and `--scratch` live there and the slurmd-side job reads and writes them. On a single box that is just any shared directory; the same applies to the genotype/SNP/pheno inputs, whose paths are passed to ravanan verbatim (relative paths are resolved relative to the generated inputs YAML, so absolute paths are the safe choice). Two things the cluster needs that a plain dev box may not have: - `gemma` reachable in the profile ravanan builds from `GEMMA_RAVANAN_MANIFEST` (the workflow invokes `gemma` itself, not `gemma-wrapper` -- see `cwl/gemma/gemma-loco.scm`); ravanan resolves that manifest against the *ambient* Guix process, so `GUIX_PACKAGE_PATH` (or a pulled `guix-bioinformatics` channel) must be set in the server's environment, as it already must for the single-machine backend; - `/bin/bash` on the nodes: ravanan submits every step as `#!/bin/bash` + the store script. ## Example ```sh export RAVANAN_BATCH_SYSTEM=slurm-api export RAVANAN_SLURM_API_ENDPOINT=http://127.0.0.1:6820 export RAVANAN_SLURM_JWT=/etc/gn4/slurm-jwt # file with the token export RAVANAN_SLURM_PARTITION=debug export RAVANAN_CACHE_DIR=/srv/gn4-ravanan # shared with the nodes ./genenetwork4 ``` `src/config.rs`'s tests cover the switch, the argument construction and the fail-closed cases (`ravanan_batch_*`). ## What survives a restart Ravanan's reuse key is `(script, inputs)`, and both halves live in `RAVANAN_CACHE_DIR` (its `--store`). So the cache's lifetime is exactly the lifetime of that directory — not of the container and not of Slurm: | state | where | survives a container restart | survives a host reboot | must the nodes see it | |---|---|---|---|---| | ravanan store (`--store`: interned inputs, per-step outputs, reuse records) | `RAVANAN_CACHE_DIR` | **yes, if it is under the bind-mounted share** | yes, if it is not under `/tmp` | **yes** | | ravanan scratch | `RAVANAN_SCRATCH`, or the cache dir when unset | as above | as above | yes | | short-token registry | `GN4_API_TOKENS_DB` (`tokens.sqlite`) | yes, if not on an ephemeral path | yes, if not under `/tmp` | no | | gemma-wrapper cache (direct runner only) | `GEMMA_CACHE_DIR` | yes | yes, if not under `/tmp` | no | | Slurm controller/spool state (`/var/spool/slurmctld`, `/var/spool/slurmd`) | inside the container | **no** — rebuilt at every boot | — | — | Measured: with the cache at `/gn4-ravanan-cache`, a full 20-chromosome LOCO mapping cost 13.6 s cold; after **restarting the container** the identical request returned in 3.6 s with the store unchanged (14 files before and after). Slurm's lost state is irrelevant to that — the reuse record is in the store, not in `slurmctld`. Two ways to get this wrong, both silent until a run fails or is slow: - the default `RAVANAN_CACHE_DIR` is under `/tmp`, which is neither durable nor visible inside a container that bind-mounts one host path, so jobs cannot read the interned inputs. The reply says so in a note when the cache dir is under `/tmp`. - `GN4_API_TOKENS_DB` defaults to `./tokens.sqlite` in the working directory: a service whose `WorkingDirectory` is a checkout under `/tmp`, or a directory that is recreated, loses every short token on reboot. Deployment checklist for the paths: ```sh RAVANAN_CACHE_DIR=/gn4-ravanan-cache # durable, nodes can see it RAVANAN_SCRATCH=/gn4-ravanan-cache # or node-local scratch GN4_API_TOKENS_DB=/tokens.sqlite # mode 0600, never in the repo GEMMA_CACHE_DIR=/gn4-gemma # direct runner only ``` ## Variance components (`pve`) on this path `gemma-loco.scm`'s `gwa` step passes `-pve`, as the direct path does, and `compute_ravanan` reads the components back **out of each job's own `.log.txt`**, which the workflow returns as a first-class CWL output (`gwa_log`). No archive is unpacked and no wrapper-named copy is involved: the workflow runs `gemma` itself, so the log is just a file it produced. The note is the same shape as `/compute/gemma`'s: every chromosome's `pve` and `se(pve)`, the lowest and highest with their `vg`/`ve`. Each log is labelled by the `-loco ` in its own command line, falling back to the `..assoc.txt.log.txt` file name. **Not verified end to end on the dev box.** The CWL compiles with `-pve` (check with `ccwl compile --output=/tmp/w.cwl cwl/gemma/gemma-loco.scm`), the archive reader is unit-tested, and a real archive from this deployment does carry the `pve` block -- but a full ravanan request could not be completed here: ravanan's own Guix evaluation of the manifest and its intern of the input files fail (see below). None of that is about `-pve`, and all of it would show up identically on a run without it. Two environment facts found while trying, both worth fixing before this path is trusted: - `GUIX_PACKAGE_PATH` must be in the **server's** environment, since ravanan resolves `--guix-manifest` against the ambient Guix; the production process does not set it, so `/compute/gemma/ravanan` cannot resolve `gemma` at all there. Without it the manifest fails with a message about an unrelated package (`rust-pyo3-0.26.0: unbound variable`) from the user's channel config. - ravanan interns each input file into `--store` with a hard link, so the store and every input path must be on the same filesystem; `/tmp` and the deployment's data directories are, but a pheno file written into a *different* mount gives `Invalid cross-device link`. ## Requirements found by running it (2026-11-12) Verified against `gn-machines/specials/slurm-container.scm` started through `run-guix-system-container` (daemon on `127.0.0.1:8989`, fingerprint `74c85932`): `POST /compute/gemma/ravanan` with `group: BXD` produced 40 Slurm jobs -- 20 `kinship` + 20 `gwa`, one pair per chromosome -- all completed, 21 055 rows back, in about 3.5 minutes. Four things had to be right, and three of them were not obvious: - **`ccwl` and `ravanan` on the server's `PATH`.** Without `ccwl` the endpoint answers `502 failed to run ccwl: No such file or directory` before doing anything. - **A `ravanan` with the `--slurm-api-endpoint` fix.** The 0.2.0 build in the store predates `ravanan: Parse --slurm-api-endpoint into a URI` and dies with `struct-vtable: Wrong type argument: "http://127.0.0.1:6820"` (it uses the string where a `uri` record is expected); a non-default endpoint therefore needs that build. Using the built-in default (`http://localhost:6820`, i.e. omitting the switch) is the workaround on an older ravanan. - **The server's `HOME` must be writable by the job user inside the cluster.** ravanan forwards the submitting process's `HOME` into every job (as `sbatch` would) and jobs inherit it. This mattered when the jobs ran `gemma-wrapper`, which locks `$HOME/..lck`: inside the container there is no `/home/wrk`, so every kinship job failed with `can not release ... .lck because it is not owned by me` / `EACCES`, and starting the API with `HOME=` fixed it. The workflow now runs `gemma` itself, which takes no such lock -- the requirement is kept here because ravanan still forwards `HOME` and a future step could assume it. This is a *server-side* setting, not a node one. - **`GUIX_PACKAGE_PATH`**, as above: ravanan resolves the manifest against the ambient Guix, so without it the manifest cannot see `(gn packages gemma)` and the failure surfaces as an unrelated channel error (`rust-pyo3-…: unbound variable`). ## Slurm-side requirements The endpoint only speaks HTTP to slurmrestd, so the cluster has to be wired the way Slurm expects for REST-submitted jobs to actually run: ``` AuthType=auth/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/path/to/jwt.key ``` with `slurmrestd -a rest_auth/jwt` started with `SLURM_JWT` in **its** environment, and the JSON Web Token minted for the user the jobs should run as. The token's identity is forwarded to slurmctld, which decodes it; slurmd keeps using munge, which is what it can verify. Setting `AuthType=auth/jwt` cluster-wide instead lets submissions be recorded but every launch RPC is rejected as `SLURM_AUTH_NOBODY` (uid 99), since slurmd never loads the JWT key. See `gn-machines/slurm-session.md` for the source-level details and a working test container (`gn-machines/specials/slurm-container.scm`). Completed jobs have to stay *queryable* for as long as the workflow runs: ravanan polls each job with `GET /slurm/v0.0.41/job/{id}`, and when slurmctld has already purged the record it falls back to `slurmdbd` (`/slurmdb/v0.0.41/job/{id}`), which 404s on a cluster without accounting. The run then aborts with `Slurm API request failed 404 GET "/slurmdb/..."`, long after the job itself succeeded — from the caller's side it looks like an ordinary step failure. Either keep the records around (`MinJobAge=86400` or more, as the test container does) or run a real `slurmdbd`. ravanan's own switches are `--batch-system`, `--slurm-api-endpoint`, `--slurm-jwt`, `--slurm-partition`, `--slurm-nice` and `--scratch`; note that `--slurm-api-endpoint` only accepts a URL on ravanan `v0.2.0-92`-era checkouts after commit "ravanan: Parse --slurm-api-endpoint into a URI" (before that the value was used as a `uri` record while being stored as a string, so only the built-in default `http://localhost:6820` worked). ## What the slurm backend costs Ravanan's default `single-machine` backend runs each CWL step as a local child process. With `slurm-api` every step becomes a Slurm job, and each of those pays dispatch on top of the step's own runtime: submit over REST, wait for the scheduler's next evaluation, launch through slurmd, and then poll until the state says COMPLETED. On small inputs that dispatch is the whole runtime. Measured on `gn-machines/specials/slurm-container.scm` (one node, two CPUs, `SchedulerParameters=sched_interval=5,...`), BXD fixtures, the same three chromosomes and trait through all three runners — identical 1650 result rows each time: | runner | cold | warm | |---------------------------------|----------|----------| | `/compute/gemma` (direct) | 2.29 s | 0.71 s | | `/compute/gemma/ravanan`, single-machine | 4.70 s | 3.45 s | | `/compute/gemma/ravanan`, slurm-api (5 s scheduler interval) | 32.6 s | 32.6 s | | `/compute/gemma/ravanan`, slurm-api (Slurm defaults) | 157.9 s | 172.9 s | Six Slurm jobs (three `kinship`, three `gwa`), each reporting under a second of runtime in Slurm's own records: the ~33 s is 6 × one scheduler tick, and the ~160-170 s is the same with Slurm's default tens-of-seconds intervals (start delays measured at 19/49/79 s). Cold and warm are identical there because the GEMMA work the cache saves is smaller than a single tick; the direct and single-machine rows show the cache clearly (3.2× and 1.4×). So: the slurm backend is worth it when a step's own runtime is much larger than the scheduler interval (real genome scans with many markers) and when the cluster actually has several nodes to fan out to (ravanan scatters one job per chromosome; this test container has a single node, so the steps serialize). On a 20-chromosome BXD run against the same container with default intervals, 40 jobs took 1193 s end to end. ## What happens when Slurm goes away There is **no automatic failover**: nothing retries, nothing falls back to the `single-machine` backend, and a request in flight is not resumed. What there is, is a fast failure and a cheap retry. Measured on the same container (a 20-chromosome BXD run, 40 Slurm jobs, `slurm-api` backend, `POST /compute/gemma/ravanan`): | when Slurm dies | response | rows | | |---|---|---|---| | never — baseline | `200` | 21 055 | 205.5 s | | before the request (`slurmrestd` unreachable) | `502 gemma_failed` | 0 | **3.7 s** | | 45 s into the request (container killed) | `502 gemma_failed` | 0 | **50.2 s** | | after the abort, Slurm back, same request | `200` | 21 051 | **163.3 s** | Four things to know: - **The failure is fast and identical either way**: ravanan raises inside its Guile process — `In procedure connect: Connection refused` for `POST /slurm/v0.0.41/job/submit`. The 502's `detail` is ravanan's whole stderr, so the useful line is at the very end of a long backtrace. A pre-flight `GET /slurm/v0.0.41/ping` would turn the first row into a `503 slurm_unavailable` naming the endpoint, before any staging; that is not implemented. - **No partial results are returned**, which is right — a half-mapped genome with 0 rows and a 502 is honest. `?id=/status` tails ravanan's log, not a result stream, so there is nothing half-written to read. - **Nothing is sticky.** The same request after Slurm returns is `200`; the failed attempt leaves no lock, no half-interned store entry, nothing to clean up. - **The retry is cheaper than cold, because ravanan's store survives**: 163.3 s against 205.5 s cold on the same 20 chromosomes. The steps that finished before the kill are reused — `(script, inputs)` pairs live in `RAVANAN_CACHE_DIR`, not in Slurm, and the kinship step is trait-independent by design (`write_constant_pheno_file`), so its 20 jobs also serve the next trait. Practical rule for a caller: treat a 502 from this endpoint as retryable, and retry the *same* request. Two gaps this leaves, both worth closing before production trusts the path: - **no pre-flight**, so a dead cluster costs a `ccwl compile`, an input staging and a ravanan startup before the connection is even attempted; - **no timeout on the ravanan child process.** Nothing here exercises a Slurm that *accepts* connections and then never runs the job, and in that case the request waits as long as ravanan does. A wall-clock bound with a `504` would be the honest answer.