1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
|
# ravanan + Guix + Slurm: how a request travels
What is in this directory, how `POST /compute/gemma/ravanan` gets from an
HTTP request to jobs running on cluster nodes, and which part of the
stack has to exist where. Configuration keys are in
[../docs/ravanan-slurm.md](../docs/ravanan-slurm.md); the Slurm-side
authentication wiring is in `gn-machines/slurm-session.md`.
```
cwl/
gemma/gemma-loco.scm the workflow, in ccwl (not raw CWL)
gemma/manifest.scm the Guix manifest ravanan resolves job commands against
```
Both files are *sources*: `compute_ravanan` compiles the ccwl to a
throwaway `.cwl` under `ravanan_cache_dir` on every call (`ccwl
compile`, pure macro expansion, no Guix build), so no CWL is ever
checked in. To inspect it by hand:
```sh
ccwl compile --output=gemma-loco.cwl gemma/gemma-loco.scm
```
## The sequence
```
POST /compute/gemma/ravanan
│
├─ compute_ravanan (Rust, API host)
│ writes <cache_dir>/<pid>-<ns>-pheno.txt and -ravanan-inputs.yaml
│ ccwl compile gemma-loco.scm -> <cache_dir>/<pid>-<ns>-gemma-loco.cwl
│ creates <cache_dir>/ravanan-store/{files,logs,data}
│ invokes: ravanan --store=<cache_dir>/ravanan-store
│ --guix-manifest=gemma/manifest.scm
│ [--batch-system=slurm-api --slurm-api-endpoint=…
│ --slurm-jwt=<file> [--slurm-partition=…] [--scratch=…]]
│ gemma-loco.cwl inputs.yaml
│
├─ ravanan (front end / API host; never on a node)
│ builds a Guix profile from the manifest (gemma)
│ per CWL step: gexp->script -> a script in /gnu/store that embeds
│ the profile's PATH, ravanan's own work library, and the command
│ slurm-api: POST /slurm/v0.0.41/job/submit with
│ script = "#!/bin/bash\n<script>"
│ environment = WORKFLOW_INPUTS, WORKFLOW_OUTPUT_DIRECTORY,
│ WORKFLOW_OUTPUT_DATA_FILE
│ working dir = "/"
│ stdout/stderr = <store>/logs/<hash>-<step>.{stdout,stderr}
│ polls GET /slurm/v0.0.41/job/{id} until COMPLETED, then reads
│ <store>/data/<hash>.json and <store>/files/<hash>-<step>/…
│
└─ slurmd on a node
bash -> the store script -> gemma -g … -p … -a … -loco <chr>
-gk|-lmm [-k <kinship from the previous step>] -o <chr>-kin|<chr>-gwa
```
The CWL `scatter` is over the chromosomes `compute_ravanan` extracted
from `snps_file`, so a 20-chromosome dataset is 20 `kinship` jobs and
20 `gwa` jobs, each restricted to its own chromosome.
## Where Guix is, and what a node needs
Guix runs **only on the front end**: ravanan asks the local
`guix-daemon` for the manifest profile, and the only thing that reaches
a node is store paths baked into the script (`#!/gnu/store/…-guile`,
`PATH=/gnu/store/…-profile/bin:…`, the gemma path).
Consequently:
| on the front end | on the compute nodes |
|-------------------------------------------------------------------------------------------|----------------------|
| `genenetwork4`, `ravanan`, `ccwl`, `guix` + `guix-daemon`, the channel/manifests, network | **no ravanan, no ccwl, no guix, no daemon** |
A node does need:
| requirement | why |
|------------------------------------------|-----|
| `/bin/bash` | ravanan submits `#!/bin/bash` + the store script; Slurm runs it via that shebang |
| same `/gnu/store` mounted (read-only ok) | the script, `guile`, `gemma`, coreutils are all store paths |
| read access to ravanan's store | input files are interned there and staged from there into the job's cwd |
| write access to its `data/`, `logs/` | the job writes its outputs record and its stdout/stderr there |
| a writable **scratch** dir | the job's cwd is a temporary dir under `--scratch`: keep it node-local |
| permission as the **job user** | identity comes from the JWT (e.g. `testuser`); these paths must be writable by it |
| a **writable `$HOME`** for the job user | ravanan forwards the submitting process's `HOME` into the job (as `sbatch` would). The jobs now run `gemma`, which takes no lock and does not care; this row is kept because a step that does assume a writable `$HOME` would fail silently cheaply |
| resolvable `NodeAddr` | Slurm's launch path needs the node address, not just a name |
Practical split: `RAVANAN_CACHE_DIR` (ravanan's store) on a
**shared** filesystem, `RAVANAN_SCRATCH`
**node-local** (`/scratch`). The default, "scratch = cache dir", is
only meant for single-box setups like the test container.
## Caches
| cache | lives in | scope |
|-----------------------------|-------------------------------------------------------------------------|-------|
| ravanan work reuse | `<store>/data/<hash>.json`, keyed by (script, inputs) | front end; a hit never touches Slurm |
**The kinship step is keyed on the animal set, not the trait.** `gemma -gk`
reads the phenotype file only to learn which individuals are analysed; the
matrix comes from the genotypes. So the workflow gives the `kinship` step a
constant-valued pheno file with the trait's missing-data pattern
(`kinship_pheno_file`, written by `write_constant_pheno_file`), keyed by
`kinship_input_key`, and gives `gwa` the real values. Two traits over the same
animals then share all twenty K matrices: on the BXD fixture a second trait
costs **+1.0 MB and 7.5 s** (20 GWA jobs) instead of **+13 MB and 14.6 s**
(20 kinship + 20 GWA jobs), and its rows are byte-identical to a run that
computed K from the real phenotype.
ravanan's work reuse used to **never hit** here, because
`compute_ravanan` named the pheno/inputs/CWL files after a pid and a
timestamp, so the (script, inputs) key was new on every call. Those
three files are now named by a hash of the request's inputs
(`ravanan_input_key`), so a repeat of the *same* request is answered
from the store: measured on the BXD fixture, the full 20-chromosome
LOCO mapping costs 14.3 s cold and **1.6 s warm** single-machine, and
**275.6 s cold** and **1.6 s warm** through Slurm in the test container
(40 jobs, 7 320 rows, identical per-chromosome `.assoc.txt` checksums
both ways). The Slurm cold run is ~6.9 s per job against ~0.36 s
single-machine: on this fixture `gemma` itself is well under a second
per chromosome, so the cluster figure is container and scheduling
overhead, and Slurm only pays off with real node parallelism. The run log stays per-call,
since it is what a caller streams by `run_id`.
There is deliberately **no kinship cache**. The workflow runs `gemma`
itself (see `cwl/gemma/gemma-loco.scm`), one invocation per chromosome,
and hands each chromosome's kinship matrix from the `kinship` step to
the `gwa` step as a CWL `File` output -- so within a request nothing is
computed twice, and a repeat request recomputes. That is the one thing
gemma-wrapper's content-hash `--cache-dir` used to buy, and it bought it
with a key that does not vary with `--chromosomes` (the reason the cache
directory had to be one per chromosome) and with `.tar.xz` archives
named by content hash (the reason it had to be one per input set). If
repeat-request latency ever matters, the honest replacement is our own
cache keyed by `sha256(genotype, pheno, snps, lmm, chr)` -- the
chromosome included by construction -- but measure first: gemma's own
kinship step on the BXD fixture is a couple of seconds.
## Can the nodes rebuild the tools instead of sharing a store?
Short answer: **yes, in principle, but nothing in the current setup
triggers it**, so today the store really does have to be visible to the
nodes.
Why the current design needs the store visible:
- ravanan builds each CWL step into a *job script* on the front end
(`gexp->script`) and submits that **store path** (`#!/bin/bash` +
`\n` + `/gnu/store/…-<step>`). Everything the job executes — the
`guile` in the shebang, the profile `PATH`, `gemma`, coreutils — is a store path baked in at that moment;
- Slurm knows nothing about Guix, so there is **no substitution**: a
store path that is absent on the node just fails with `ENOENT`, the
job never starts, and ravanan reports the step as failed;
- Guix *is* deterministic, so a node that builds the same derivations
with the same channels would produce **byte-identical paths** and the
script would then exist — which is what makes the options below
possible at all.
| option | what runs | prerequisites | caching |
|-------------------------|-------------------------------------------------------------|--------------------------------------------|---------|
| 1. shared store | nodes read the front end's `/gnu/store`; nothing extra runs | store mounted on the nodes | full (~0.9 s cached rerun) |
| 2. realise-then-run | `guix build <script>.drv` then `exec <script>` per job | guix + daemon per node, identical channels | only if the node store persists |
| 3. per-workflow realise | one job builds/imports every step derivation first | as (2) plus job ordering | as (2) |
| 4. `guix publish` + (2) | as (2), substituting from the front end | as (2) with `--substitute-urls` | as (2), faster cold nodes |
| 5. ship closures | `guix archive --export` / `guix copy` before the run | no guix on nodes; per-run shipping | within one run only |
The wrapper of option 2/4 is about ten lines of change on the ravanan
side: `build-gexp-script` in `ravanan/command-line-tool.scm` already
builds the derivation (`gexp->script` returns a `<derivation>`) but
returns only its output path, and `submit-job` in
`ravanan/slurm-api.scm` assembles the `script` field of the job spec —
so keeping the `.drv` path and submitting
`#!/bin/bash\nguix build <drv> && exec <script>` is the whole
mechanism. Nothing of that is implemented in this repo; the endpoint
uses ravanan as-is, and `compute_ravanan` cannot influence how a step is
launched beyond the switches it passes.
Costs to weigh before choosing it:
- **channel skew breaks it.** Every node and the front end must run the
same Guix commit *and* the same channels; a `guix pull` on one side
changes the hashes and nothing matches any more. Pinning that across a
cluster is an operational commitment, not a config file.
- **build time lands inside the job.** A cold gemma/libc closure is
minutes per node; the first job on each node pays it inside the Slurm
time limit, and `gemma`'s build is the same one the front end already
did.
- **caching evaporates on ephemeral node stores.** ravanan's free
"cached rerun" (below) only holds if the node's store survives between
jobs; with per-job containers or a wiped `/gnu/store` you re-realise
the closure every time.
- **`guix publish` is the sane companion.** Front end keeps building
and serving (`guix publish`), nodes substitute with
`--substitute-urls=http://frontend:5000`; that turns "rebuild on the
node" into "download from the front end", which is what NFS would
have given you anyway — minus the shared-filesystem dependency, plus
per-node guix and a published store.
Recommendation: keep the shared (or NFS-exported) store as the default —
ravanan is designed around it and the timings below assume it. If the
store genuinely cannot be exported and the cluster already runs a Guix
daemon with pinned channels, implement options 2 + 4 upstream in
ravanan and expose it here as an opt-in config switch; until then,
`RAVANAN_SCRATCH` should stay node-local (`/scratch`) while
`RAVANAN_CACHE_DIR` stays shared.
## What to expect, timing-wise
Measured on a single-box test cluster (one node, 2 CPUs); ravanan's
`single-machine` backend runs steps **synchronously**, so a scatter gets
no parallelism there, while Slurm runs as many as fit (2 here).
| workload | single-machine | slurm-api |
|--------------------------------------------|----------------|-----------|
| cached rerun (warm store), 2-step workflow | 0.91 s | 0.92 s |
| 8 × 2 s steps | 16.0 s | 12.6 s |
| 20 × 2 s steps | 40.0 s | 30.0 s |
| 20 × 1 s steps | 20.1 s | 25.4 s |
| 20 × 0.2 s steps | 4.1 s | 27.6 s |
Cached runs are backend-independent (ravanan returns the stored result
before it ever talks to the batch system). Fresh runs trade roughly
`ceil(jobs / cores)` waves against a per-wave scheduling overhead of
~1–2.5 s, so Slurm only wins when there are at least two usable cores
and steps are longer than a second or two; with per-chromosome
`gemma` steps of a second or two on a 2-CPU box it can be *slower* than
local, and on a real cluster with ≥4 effective cores it wins
approximately linearly. The wrapper's Ruby startup used to be part of
every one of those steps; the workflow now invokes `gemma` directly,
which shortens each step by that much.
## Slurm-side prerequisite (one-time)
The endpoint only speaks HTTP to slurmrestd, so the cluster must accept
REST-submitted jobs *and* run them:
```
AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/path/to/jwt.key
```
`slurmrestd -a rest_auth/jwt` started with `SLURM_JWT` in **its** own
environment (that is what makes `auth_g_init` pick `auth/jwt` for the
REST daemon so `rest_auth/jwt`'s per-request identity switch is
accepted), and the JWT minted for the user the jobs should run as.
Setting `AuthType=auth/jwt` cluster-wide instead records submissions but
rejects every launch with `SLURM_AUTH_NOBODY` (uid 99): slurmd never
loads the JWT key. `gn-machines/slurm-session.md` has the source-level
details and `gn-machines/specials/slurm-container.scm` a working test
cluster.
|