1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
|
# Running ravanan's jobs on Slurm
`POST /compute/gemma/ravanan` hands the whole per-chromosome LOCO
workflow to [ravanan](https://git.systemreboot.net/ravanan), which runs
each step itself. By default every step is a local child process
(ravanan's `single-machine` batch system). Setting
`RAVANAN_BATCH_SYSTEM=slurm-api` makes ravanan submit each step to a
Slurm cluster through slurmrestd's REST API instead, so the CWL scatter
really fans out over the cluster's nodes.
Nothing else about the endpoint changes: same request, same response,
same rows. [../cwl/README.md](../cwl/README.md) walks through the whole
path a request takes (API -> ccwl -> ravanan -> slurmrestd -> slurmd),
exactly what a compute node needs, and why the Guix store has to be
visible to the nodes. The slurm switches are appended to ravanan's command line by
`RavananBatch::ravanan_args` (`src/config.rs`), and the note in the
response's `meta.notes` says which backend ran (`ravanan (…, slurm-api
backend): …`).
## Configuration
| key | meaning |
|---|---|
| `RAVANAN_BATCH_SYSTEM` | `single-machine` (default) or `slurm-api`; anything else refuses to start |
| `RAVANAN_SLURM_API_ENDPOINT` | slurmrestd base URL, e.g. `http://127.0.0.1:6820` (required for slurm-api) |
| `RAVANAN_SLURM_JWT` | path to a **file** holding the JWT (ravanan's `--slurm-jwt` takes a file, not the token); required for slurm-api |
| `RAVANAN_SLURM_PARTITION` | optional `--slurm-partition`, e.g. `debug` |
| `RAVANAN_SLURM_NICE` | optional integer `--slurm-nice` |
| `RAVANAN_SCRATCH` | optional `--scratch`; defaults to `RAVANAN_CACHE_DIR` when the backend is slurm |
| `RAVANAN_CACHE_DIR` | where pheno files, logs, the compiled CWL and ravanan's `--store` live |
Selecting slurm without an endpoint or a JWT file is a startup error
(fail closed, like `GN4_DB_MODE`): ravanan would fail every step, and a
refusal at boot beats a 502 on every request.
**With slurm, `RAVANAN_CACHE_DIR` must be on a filesystem the compute
nodes can see**, because ravanan's `--store` and `--scratch` live there
and the slurmd-side job reads and writes them. On a single box that is
just any shared directory; the same applies to the genotype/SNP/pheno
inputs, whose paths are passed to ravanan verbatim (relative paths are
resolved relative to the generated inputs YAML, so absolute paths are
the safe choice).
Two things the cluster needs that a plain dev box may not have:
- `gemma` reachable in the profile ravanan builds from
`GEMMA_RAVANAN_MANIFEST` (the workflow invokes `gemma` itself, not
`gemma-wrapper` -- see `cwl/gemma/gemma-loco.scm`); ravanan resolves that
manifest against the *ambient* Guix process, so `GUIX_PACKAGE_PATH`
(or a pulled `guix-bioinformatics` channel) must be set in the
server's environment, as it already must for the single-machine
backend;
- `/bin/bash` on the nodes: ravanan submits every step as
`#!/bin/bash` + the store script.
## Example
```sh
export RAVANAN_BATCH_SYSTEM=slurm-api
export RAVANAN_SLURM_API_ENDPOINT=http://127.0.0.1:6820
export RAVANAN_SLURM_JWT=/etc/gn4/slurm-jwt # file with the token
export RAVANAN_SLURM_PARTITION=debug
export RAVANAN_CACHE_DIR=/srv/gn4-ravanan # shared with the nodes
./genenetwork4
```
`src/config.rs`'s tests cover the switch, the argument construction and
the fail-closed cases (`ravanan_batch_*`).
## What survives a restart
Ravanan's reuse key is `(script, inputs)`, and both halves live in
`RAVANAN_CACHE_DIR` (its `--store`). So the cache's lifetime is exactly the
lifetime of that directory — not of the container and not of Slurm:
| state | where | survives a container restart | survives a host reboot | must the nodes see it |
|---|---|---|---|---|
| ravanan store (`--store`: interned inputs, per-step outputs, reuse records) | `RAVANAN_CACHE_DIR` | **yes, if it is under the bind-mounted share** | yes, if it is not under `/tmp` | **yes** |
| ravanan scratch | `RAVANAN_SCRATCH`, or the cache dir when unset | as above | as above | yes |
| short-token registry | `GN4_API_TOKENS_DB` (`tokens.sqlite`) | yes, if not on an ephemeral path | yes, if not under `/tmp` | no |
| gemma-wrapper cache (direct runner only) | `GEMMA_CACHE_DIR` | yes | yes, if not under `/tmp` | no |
| Slurm controller/spool state (`/var/spool/slurmctld`, `/var/spool/slurmd`) | inside the container | **no** — rebuilt at every boot | — | — |
Measured: with the cache at `<shared mount>/gn4-ravanan-cache`, a full
20-chromosome LOCO mapping cost 13.6 s cold; after **restarting the container**
the identical request returned in 3.6 s with the store unchanged (14 files before
and after). Slurm's lost state is irrelevant to that — the reuse record is in the
store, not in `slurmctld`.
Two ways to get this wrong, both silent until a run fails or is slow:
- the default `RAVANAN_CACHE_DIR` is under `/tmp`, which is neither durable nor
visible inside a container that bind-mounts one host path, so jobs cannot read
the interned inputs. The reply says so in a note when the cache dir is under
`/tmp`.
- `GN4_API_TOKENS_DB` defaults to `./tokens.sqlite` in the working directory: a
service whose `WorkingDirectory` is a checkout under `/tmp`, or a directory
that is recreated, loses every short token on reboot.
Deployment checklist for the paths:
```sh
RAVANAN_CACHE_DIR=<shared mount>/gn4-ravanan-cache # durable, nodes can see it
RAVANAN_SCRATCH=<shared mount>/gn4-ravanan-cache # or node-local scratch
GN4_API_TOKENS_DB=<durable dir>/tokens.sqlite # mode 0600, never in the repo
GEMMA_CACHE_DIR=<durable dir>/gn4-gemma # direct runner only
```
## Variance components (`pve`) on this path
`gemma-loco.scm`'s `gwa` step passes `-pve`, as the direct path does, and
`compute_ravanan` reads the components back **out of each job's own
`.log.txt`**, which the workflow returns as a first-class CWL output
(`gwa_log`). No archive is unpacked and no wrapper-named copy is involved:
the workflow runs `gemma` itself, so the log is just a file it produced.
The note is the same shape as `/compute/gemma`'s: every chromosome's `pve` and
`se(pve)`, the lowest and highest with their `vg`/`ve`. Each log is labelled by
the `-loco <chr>` in its own command line, falling back to the
`<hash>.<chr>.assoc.txt.log.txt` file name.
**Not verified end to end on the dev box.** The CWL compiles with `-pve` (check
with `ccwl compile --output=/tmp/w.cwl cwl/gemma/gemma-loco.scm`), the archive
reader is unit-tested, and a real archive from this deployment does carry the
`pve` block -- but a full ravanan request could not be completed here:
ravanan's own Guix evaluation of the manifest and its intern of the input files
fail (see below). None of that is about `-pve`, and all of it would show up
identically on a run without it.
Two environment facts found while trying, both worth fixing before this path is
trusted:
- `GUIX_PACKAGE_PATH` must be in the **server's** environment, since ravanan
resolves `--guix-manifest` against the ambient Guix; the production process
does not set it, so `/compute/gemma/ravanan` cannot resolve `gemma` at all
there. Without it the manifest fails with a message about an unrelated
package (`rust-pyo3-0.26.0: unbound variable`) from the user's channel config.
- ravanan interns each input file into `--store` with a hard link, so the store
and every input path must be on the same filesystem; `/tmp` and the
deployment's data directories are, but a pheno file written into a *different*
mount gives `Invalid cross-device link`.
## Requirements found by running it (2026-11-12)
Verified against `gn-machines/specials/slurm-container.scm` started through
`run-guix-system-container` (daemon on `127.0.0.1:8989`, fingerprint
`74c85932`): `POST /compute/gemma/ravanan` with `group: BXD` produced 40
Slurm jobs -- 20 `kinship` + 20 `gwa`, one pair per chromosome -- all
completed, 21 055 rows back, in about 3.5 minutes.
Four things had to be right, and three of them were not obvious:
- **`ccwl` and `ravanan` on the server's `PATH`.** Without `ccwl` the
endpoint answers `502 failed to run ccwl: No such file or directory`
before doing anything.
- **A `ravanan` with the `--slurm-api-endpoint` fix.** The 0.2.0 build in
the store predates `ravanan: Parse --slurm-api-endpoint into a URI`
and dies with `struct-vtable: Wrong type argument: "http://127.0.0.1:6820"`
(it uses the string where a `uri` record is expected); a non-default
endpoint therefore needs that build. Using the built-in default
(`http://localhost:6820`, i.e. omitting the switch) is the workaround on
an older ravanan.
- **The server's `HOME` must be writable by the job user inside the
cluster.** ravanan forwards the submitting process's `HOME` into every
job (as `sbatch` would) and jobs inherit it. This mattered when the jobs
ran `gemma-wrapper`, which locks `$HOME/.<hash>.lck`: inside the container
there is no `/home/wrk`, so every kinship job failed with `can not
release ... .lck because it is not owned by me` / `EACCES`, and starting
the API with `HOME=<a directory on the shared mount>` fixed it. The
workflow now runs `gemma` itself, which takes no such lock -- the
requirement is kept here because ravanan still forwards `HOME` and a
future step could assume it. This is a
*server-side* setting, not a node one.
- **`GUIX_PACKAGE_PATH`**, as above: ravanan resolves the manifest against
the ambient Guix, so without it the manifest cannot see
`(gn packages gemma)` and the failure surfaces as an unrelated channel
error (`rust-pyo3-…: unbound variable`).
## Slurm-side requirements
The endpoint only speaks HTTP to slurmrestd, so the cluster has to be
wired the way Slurm expects for REST-submitted jobs to actually run:
```
AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/path/to/jwt.key
```
with `slurmrestd -a rest_auth/jwt` started with `SLURM_JWT` in **its**
environment, and the JSON Web Token minted for the user the jobs should
run as. The token's identity is forwarded to slurmctld, which decodes
it; slurmd keeps using munge, which is what it can verify. Setting
`AuthType=auth/jwt` cluster-wide instead lets submissions be recorded
but every launch RPC is rejected as `SLURM_AUTH_NOBODY` (uid 99), since
slurmd never loads the JWT key. See `gn-machines/slurm-session.md` for
the source-level details and a working test container
(`gn-machines/specials/slurm-container.scm`).
Completed jobs have to stay *queryable* for as long as the workflow
runs: ravanan polls each job with `GET /slurm/v0.0.41/job/{id}`, and
when slurmctld has already purged the record it falls back to
`slurmdbd` (`/slurmdb/v0.0.41/job/{id}`), which 404s on a cluster
without accounting. The run then aborts with `Slurm API request failed
404 GET "/slurmdb/..."`, long after the job itself succeeded — from the
caller's side it looks like an ordinary step failure. Either keep the
records around (`MinJobAge=86400` or more, as the test container does)
or run a real `slurmdbd`.
ravanan's own switches are `--batch-system`, `--slurm-api-endpoint`,
`--slurm-jwt`, `--slurm-partition`, `--slurm-nice` and `--scratch`; note
that `--slurm-api-endpoint` only accepts a URL on ravanan
`v0.2.0-92`-era checkouts after commit "ravanan: Parse
--slurm-api-endpoint into a URI" (before that the value was used as a
`uri` record while being stored as a string, so only the built-in
default `http://localhost:6820` worked).
## What the slurm backend costs
Ravanan's default `single-machine` backend runs each CWL step as a local
child process. With `slurm-api` every step becomes a Slurm job, and each
of those pays dispatch on top of the step's own runtime: submit over
REST, wait for the scheduler's next evaluation, launch through slurmd,
and then poll until the state says COMPLETED. On small inputs that
dispatch is the whole runtime.
Measured on `gn-machines/specials/slurm-container.scm` (one node, two
CPUs, `SchedulerParameters=sched_interval=5,...`), BXD fixtures, the same
three chromosomes and trait through all three runners — identical 1650
result rows each time:
| runner | cold | warm |
|---------------------------------|----------|----------|
| `/compute/gemma` (direct) | 2.29 s | 0.71 s |
| `/compute/gemma/ravanan`, single-machine | 4.70 s | 3.45 s |
| `/compute/gemma/ravanan`, slurm-api (5 s scheduler interval) | 32.6 s | 32.6 s |
| `/compute/gemma/ravanan`, slurm-api (Slurm defaults) | 157.9 s | 172.9 s |
Six Slurm jobs (three `kinship`, three `gwa`), each reporting under a
second of runtime in Slurm's own records: the ~33 s is 6 × one scheduler
tick, and the ~160-170 s is the same with Slurm's default tens-of-seconds
intervals (start delays measured at 19/49/79 s). Cold and warm are
identical there because the GEMMA work the cache saves is smaller than a
single tick; the direct and single-machine rows show the cache clearly
(3.2× and 1.4×).
So: the slurm backend is worth it when a step's own runtime is much larger
than the scheduler interval (real genome scans with many markers) and when
the cluster actually has several nodes to fan out to (ravanan scatters one
job per chromosome; this test container has a single node, so the steps
serialize). On a 20-chromosome BXD run against the same container with
default intervals, 40 jobs took 1193 s end to end.
## What happens when Slurm goes away
There is **no automatic failover**: nothing retries, nothing falls back to
the `single-machine` backend, and a request in flight is not resumed. What
there is, is a fast failure and a cheap retry. Measured on the same
container (a 20-chromosome BXD run, 40 Slurm jobs, `slurm-api` backend,
`POST /compute/gemma/ravanan`):
| when Slurm dies | response | rows | |
|---|---|---|---|
| never — baseline | `200` | 21 055 | 205.5 s |
| before the request (`slurmrestd` unreachable) | `502 gemma_failed` | 0 | **3.7 s** |
| 45 s into the request (container killed) | `502 gemma_failed` | 0 | **50.2 s** |
| after the abort, Slurm back, same request | `200` | 21 051 | **163.3 s** |
Four things to know:
- **The failure is fast and identical either way**: ravanan raises inside
its Guile process — `In procedure connect: Connection refused` for
`POST /slurm/v0.0.41/job/submit`. The 502's `detail` is ravanan's whole
stderr, so the useful line is at the very end of a long backtrace. A
pre-flight `GET /slurm/v0.0.41/ping` would turn the first row into a
`503 slurm_unavailable` naming the endpoint, before any staging; that is
not implemented.
- **No partial results are returned**, which is right — a half-mapped
genome with 0 rows and a 502 is honest. `?id=/status` tails ravanan's
log, not a result stream, so there is nothing half-written to read.
- **Nothing is sticky.** The same request after Slurm returns is `200`;
the failed attempt leaves no lock, no half-interned store entry, nothing
to clean up.
- **The retry is cheaper than cold, because ravanan's store survives**:
163.3 s against 205.5 s cold on the same 20 chromosomes. The steps that
finished before the kill are reused — `(script, inputs)` pairs live in
`RAVANAN_CACHE_DIR`, not in Slurm, and the kinship step is
trait-independent by design (`write_constant_pheno_file`), so its 20
jobs also serve the next trait. Practical rule for a caller: treat a 502
from this endpoint as retryable, and retry the *same* request.
Two gaps this leaves, both worth closing before production trusts the path:
- **no pre-flight**, so a dead cluster costs a `ccwl compile`, an input
staging and a ravanan startup before the connection is even attempted;
- **no timeout on the ravanan child process.** Nothing here exercises a
Slurm that *accepts* connections and then never runs the job, and in that
case the request waits as long as ravanan does. A wall-clock bound with a
`504` would be the honest answer.
|