about summary refs log tree commit diff

ravanan + Guix + Slurm: how a request travels

What is in this directory, how POST /compute/gemma/ravanan gets from an HTTP request to jobs running on cluster nodes, and which part of the stack has to exist where. Configuration keys are in ../docs/ravanan-slurm.md; the Slurm-side authentication wiring is in gn-machines/slurm-session.md.

cwl/
  gemma/gemma-loco.scm   the workflow, in ccwl (not raw CWL)
  gemma/manifest.scm     the Guix manifest ravanan resolves job commands against

Both files are sources: compute_ravanan compiles the ccwl to a throwaway .cwl under ravanan_cache_dir on every call (ccwl compile, pure macro expansion, no Guix build), so no CWL is ever checked in. To inspect it by hand:

ccwl compile --output=gemma-loco.cwl gemma/gemma-loco.scm

The sequence

POST /compute/gemma/ravanan
      │
      ├─ compute_ravanan (Rust, API host)
      │    writes <cache_dir>/<pid>-<ns>-pheno.txt and -ravanan-inputs.yaml
      │    ccwl compile gemma-loco.scm -> <cache_dir>/<pid>-<ns>-gemma-loco.cwl
      │    creates <cache_dir>/ravanan-store/{files,logs,data}
      │    invokes: ravanan --store=<cache_dir>/ravanan-store
      │              --guix-manifest=gemma/manifest.scm
      │              [--batch-system=slurm-api --slurm-api-endpoint=…
      │               --slurm-jwt=<file> [--slurm-partition=…] [--scratch=…]]
      │              gemma-loco.cwl inputs.yaml
      │
      ├─ ravanan (front end / API host; never on a node)
      │    builds a Guix profile from the manifest  (gemma)
      │    per CWL step: gexp->script -> a script in /gnu/store that embeds
      │      the profile's PATH, ravanan's own work library, and the command
      │    slurm-api: POST /slurm/v0.0.41/job/submit with
      │      script       = "#!/bin/bash\n<script>"
      │      environment  = WORKFLOW_INPUTS, WORKFLOW_OUTPUT_DIRECTORY,
      │                     WORKFLOW_OUTPUT_DATA_FILE
      │      working dir   = "/"
      │      stdout/stderr = <store>/logs/<hash>-<step>.{stdout,stderr}
      │    polls GET /slurm/v0.0.41/job/{id} until COMPLETED, then reads
      │    <store>/data/<hash>.json and <store>/files/<hash>-<step>/…
      │
      └─ slurmd on a node
           bash -> the store script -> gemma -g … -p … -a … -loco <chr>
           -gk|-lmm [-k <kinship from the previous step>] -o <chr>-kin|<chr>-gwa

The CWL scatter is over the chromosomes compute_ravanan extracted from snps_file, so a 20-chromosome dataset is 20 kinship jobs and 20 gwa jobs, each restricted to its own chromosome.

Where Guix is, and what a node needs

Guix runs only on the front end: ravanan asks the local guix-daemon for the manifest profile, and the only thing that reaches a node is store paths baked into the script (#!/gnu/store/…-guile, PATH=/gnu/store/…-profile/bin:…, the gemma path). Consequently:

on the front end on the compute nodes
genenetwork4, ravanan, ccwl, guix + guix-daemon, the channel/manifests, network no ravanan, no ccwl, no guix, no daemon

A node does need:

requirement why
/bin/bash ravanan submits #!/bin/bash + the store script; Slurm runs it via that shebang
same /gnu/store mounted (read-only ok) the script, guile, gemma, coreutils are all store paths
read access to ravanan's store input files are interned there and staged from there into the job's cwd
write access to its data/, logs/ the job writes its outputs record and its stdout/stderr there
a writable scratch dir the job's cwd is a temporary dir under --scratch: keep it node-local
permission as the job user identity comes from the JWT (e.g. testuser); these paths must be writable by it
a writable $HOME for the job user ravanan forwards the submitting process's HOME into the job (as sbatch would). The jobs now run gemma, which takes no lock and does not care; this row is kept because a step that does assume a writable $HOME would fail silently cheaply
resolvable NodeAddr Slurm's launch path needs the node address, not just a name

Practical split: RAVANAN_CACHE_DIR (ravanan's store) on a shared filesystem, RAVANAN_SCRATCH node-local (/scratch). The default, "scratch = cache dir", is only meant for single-box setups like the test container.

Caches

cache lives in scope
ravanan work reuse <store>/data/<hash>.json, keyed by (script, inputs) front end; a hit never touches Slurm

The kinship step is keyed on the animal set, not the trait. gemma -gk reads the phenotype file only to learn which individuals are analysed; the matrix comes from the genotypes. So the workflow gives the kinship step a constant-valued pheno file with the trait's missing-data pattern (kinship_pheno_file, written by write_constant_pheno_file), keyed by kinship_input_key, and gives gwa the real values. Two traits over the same animals then share all twenty K matrices: on the BXD fixture a second trait costs +1.0 MB and 7.5 s (20 GWA jobs) instead of +13 MB and 14.6 s (20 kinship + 20 GWA jobs), and its rows are byte-identical to a run that computed K from the real phenotype.

ravanan's work reuse used to never hit here, because compute_ravanan named the pheno/inputs/CWL files after a pid and a timestamp, so the (script, inputs) key was new on every call. Those three files are now named by a hash of the request's inputs (ravanan_input_key), so a repeat of the same request is answered from the store: measured on the BXD fixture, the full 20-chromosome LOCO mapping costs 14.3 s cold and 1.6 s warm single-machine, and 275.6 s cold and 1.6 s warm through Slurm in the test container (40 jobs, 7 320 rows, identical per-chromosome .assoc.txt checksums both ways). The Slurm cold run is ~6.9 s per job against ~0.36 s single-machine: on this fixture gemma itself is well under a second per chromosome, so the cluster figure is container and scheduling overhead, and Slurm only pays off with real node parallelism. The run log stays per-call, since it is what a caller streams by run_id.

There is deliberately no kinship cache. The workflow runs gemma itself (see cwl/gemma/gemma-loco.scm), one invocation per chromosome, and hands each chromosome's kinship matrix from the kinship step to the gwa step as a CWL File output -- so within a request nothing is computed twice, and a repeat request recomputes. That is the one thing gemma-wrapper's content-hash --cache-dir used to buy, and it bought it with a key that does not vary with --chromosomes (the reason the cache directory had to be one per chromosome) and with .tar.xz archives named by content hash (the reason it had to be one per input set). If repeat-request latency ever matters, the honest replacement is our own cache keyed by sha256(genotype, pheno, snps, lmm, chr) -- the chromosome included by construction -- but measure first: gemma's own kinship step on the BXD fixture is a couple of seconds.

Can the nodes rebuild the tools instead of sharing a store?

Short answer: yes, in principle, but nothing in the current setup triggers it, so today the store really does have to be visible to the nodes.

Why the current design needs the store visible:

  • ravanan builds each CWL step into a job script on the front end (gexp->script) and submits that store path (#!/bin/bash + \n + /gnu/store/…-<step>). Everything the job executes — the guile in the shebang, the profile PATH, gemma, coreutils — is a store path baked in at that moment;
  • Slurm knows nothing about Guix, so there is no substitution: a store path that is absent on the node just fails with ENOENT, the job never starts, and ravanan reports the step as failed;
  • Guix is deterministic, so a node that builds the same derivations with the same channels would produce byte-identical paths and the script would then exist — which is what makes the options below possible at all.
option what runs prerequisites caching
1. shared store nodes read the front end's /gnu/store; nothing extra runs store mounted on the nodes full (~0.9 s cached rerun)
2. realise-then-run guix build <script>.drv then exec <script> per job guix + daemon per node, identical channels only if the node store persists
3. per-workflow realise one job builds/imports every step derivation first as (2) plus job ordering as (2)
4. guix publish + (2) as (2), substituting from the front end as (2) with --substitute-urls as (2), faster cold nodes
5. ship closures guix archive --export / guix copy before the run no guix on nodes; per-run shipping within one run only

The wrapper of option 2/4 is about ten lines of change on the ravanan side: build-gexp-script in ravanan/command-line-tool.scm already builds the derivation (gexp->script returns a <derivation>) but returns only its output path, and submit-job in ravanan/slurm-api.scm assembles the script field of the job spec — so keeping the .drv path and submitting #!/bin/bash\nguix build <drv> && exec <script> is the whole mechanism. Nothing of that is implemented in this repo; the endpoint uses ravanan as-is, and compute_ravanan cannot influence how a step is launched beyond the switches it passes.

Costs to weigh before choosing it:

  • channel skew breaks it. Every node and the front end must run the same Guix commit and the same channels; a guix pull on one side changes the hashes and nothing matches any more. Pinning that across a cluster is an operational commitment, not a config file.
  • build time lands inside the job. A cold gemma/libc closure is minutes per node; the first job on each node pays it inside the Slurm time limit, and gemma's build is the same one the front end already did.
  • caching evaporates on ephemeral node stores. ravanan's free "cached rerun" (below) only holds if the node's store survives between jobs; with per-job containers or a wiped /gnu/store you re-realise the closure every time.
  • guix publish is the sane companion. Front end keeps building and serving (guix publish), nodes substitute with --substitute-urls=http://frontend:5000; that turns "rebuild on the node" into "download from the front end", which is what NFS would have given you anyway — minus the shared-filesystem dependency, plus per-node guix and a published store.

Recommendation: keep the shared (or NFS-exported) store as the default — ravanan is designed around it and the timings below assume it. If the store genuinely cannot be exported and the cluster already runs a Guix daemon with pinned channels, implement options 2 + 4 upstream in ravanan and expose it here as an opt-in config switch; until then, RAVANAN_SCRATCH should stay node-local (/scratch) while RAVANAN_CACHE_DIR stays shared.

What to expect, timing-wise

Measured on a single-box test cluster (one node, 2 CPUs); ravanan's single-machine backend runs steps synchronously, so a scatter gets no parallelism there, while Slurm runs as many as fit (2 here).

workload single-machine slurm-api
cached rerun (warm store), 2-step workflow 0.91 s 0.92 s
8 × 2 s steps 16.0 s 12.6 s
20 × 2 s steps 40.0 s 30.0 s
20 × 1 s steps 20.1 s 25.4 s
20 × 0.2 s steps 4.1 s 27.6 s

Cached runs are backend-independent (ravanan returns the stored result before it ever talks to the batch system). Fresh runs trade roughly ceil(jobs / cores) waves against a per-wave scheduling overhead of ~1–2.5 s, so Slurm only wins when there are at least two usable cores and steps are longer than a second or two; with per-chromosome gemma steps of a second or two on a 2-CPU box it can be slower than local, and on a real cluster with ≥4 effective cores it wins approximately linearly. The wrapper's Ruby startup used to be part of every one of those steps; the workflow now invokes gemma directly, which shortens each step by that much.

Slurm-side prerequisite (one-time)

The endpoint only speaks HTTP to slurmrestd, so the cluster must accept REST-submitted jobs and run them:

AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/path/to/jwt.key

slurmrestd -a rest_auth/jwt started with SLURM_JWT in its own environment (that is what makes auth_g_init pick auth/jwt for the REST daemon so rest_auth/jwt's per-request identity switch is accepted), and the JWT minted for the user the jobs should run as. Setting AuthType=auth/jwt cluster-wide instead records submissions but rejects every launch with SLURM_AUTH_NOBODY (uid 99): slurmd never loads the JWT key. gn-machines/slurm-session.md has the source-level details and gn-machines/specials/slurm-container.scm a working test cluster.