diff options
| author | Pjotr <wrk@bosa.lan> | 2026-10-24 09:33:40 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-10-24 09:33:40 +0000 |
| commit | a7ffb1cfecbdb8ac3fcc6286d7a09f7285c95d67 (patch) | |
| tree | 1cb865784d901fad979aed32694b77474019468d /docs | |
| parent | cda325b6334ef36e3f49501cfd77d68a7bd6a295 (diff) | |
| download | genenetwork4-a7ffb1cfecbdb8ac3fcc6286d7a09f7285c95d67.tar.gz | |
gemma: Make ravanan's batch system configurable (optional Slurm).
POST /compute/gemma/ravanan used to run every CWL step locally, because compute_ravanan never passed ravanan a batch system. Add a RavananBatch setting: - RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing closed on anything else, exactly like GN4_DB_MODE; - slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint, --slurm-jwt (a *file*, which is what ravanan reads) and, when set, --slurm-partition/--slurm-nice; a slurm selection without an endpoint or a JWT file refuses to start rather than 502-ing every request; - --scratch defaults to ravanan_cache_dir for slurm, so the store and the job scratch space live on the same shared filesystem (with slurm that directory has to be visible to the compute nodes); - the response's ravanan note records which backend ran. The switch building is RavananBatch::ravanan_args so it is unit testable; compute_ravanan takes the backend and the scratch directory, and the route passes them from config. docs/ravanan-slurm.md documents the keys, the shared-directory requirement, and the slurm-side auth wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in slurmrestd's environment) that makes REST-submitted jobs actually run.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/ravanan-slurm.md | 93 |
1 files changed, 93 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md new file mode 100644 index 0000000..0b120d4 --- /dev/null +++ b/docs/ravanan-slurm.md @@ -0,0 +1,93 @@ +# Running ravanan's jobs on Slurm + +`POST /compute/gemma/ravanan` hands the whole per-chromosome LOCO +workflow to [ravanan](https://git.systemreboot.net/ravanan), which runs +each step itself. By default every step is a local child process +(ravanan's `single-machine` batch system). Setting +`RAVANAN_BATCH_SYSTEM=slurm-api` makes ravanan submit each step to a +Slurm cluster through slurmrestd's REST API instead, so the CWL scatter +really fans out over the cluster's nodes. + +Nothing else about the endpoint changes: same request, same response, +same rows. The slurm switches are appended to ravanan's command line by +`RavananBatch::ravanan_args` (`src/config.rs`), and the note in the +response's `meta.notes` says which backend ran (`ravanan (…, slurm-api +backend): …`). + +## Configuration + +| key | meaning | +|---|---| +| `RAVANAN_BATCH_SYSTEM` | `single-machine` (default) or `slurm-api`; anything else refuses to start | +| `RAVANAN_SLURM_API_ENDPOINT` | slurmrestd base URL, e.g. `http://127.0.0.1:6820` (required for slurm-api) | +| `RAVANAN_SLURM_JWT` | path to a **file** holding the JWT (ravanan's `--slurm-jwt` takes a file, not the token); required for slurm-api | +| `RAVANAN_SLURM_PARTITION` | optional `--slurm-partition`, e.g. `debug` | +| `RAVANAN_SLURM_NICE` | optional integer `--slurm-nice` | +| `RAVANAN_SCRATCH` | optional `--scratch`; defaults to `RAVANAN_CACHE_DIR` when the backend is slurm | +| `RAVANAN_CACHE_DIR` | where pheno files, logs, the compiled CWL and ravanan's `--store` live | + +Selecting slurm without an endpoint or a JWT file is a startup error +(fail closed, like `GN4_DB_MODE`): ravanan would fail every step, and a +refusal at boot beats a 502 on every request. + +**With slurm, `RAVANAN_CACHE_DIR` must be on a filesystem the compute +nodes can see**, because ravanan's `--store` and `--scratch` live there +and the slurmd-side job reads and writes them. On a single box that is +just any shared directory; the same applies to the genotype/SNP/pheno +inputs, whose paths are passed to ravanan verbatim (relative paths are +resolved relative to the generated inputs YAML, so absolute paths are +the safe choice). + +Two things the cluster needs that a plain dev box may not have: + +- `gemma-wrapper` (and the `gemma` it wraps) reachable in the profile + ravanan builds from `GEMMA_RAVANAN_MANIFEST`; ravanan resolves that + manifest against the *ambient* Guix process, so `GUIX_PACKAGE_PATH` + (or a pulled `guix-bioinformatics` channel) must be set in the + server's environment, as it already must for the single-machine + backend; +- `/bin/bash` on the nodes: ravanan submits every step as + `#!/bin/bash` + the store script. + +## Example + +```sh +export RAVANAN_BATCH_SYSTEM=slurm-api +export RAVANAN_SLURM_API_ENDPOINT=http://127.0.0.1:6820 +export RAVANAN_SLURM_JWT=/etc/gn4/slurm-jwt # file with the token +export RAVANAN_SLURM_PARTITION=debug +export RAVANAN_CACHE_DIR=/srv/gn4-ravanan # shared with the nodes +./genenetwork4 +``` + +`src/config.rs`'s tests cover the switch, the argument construction and +the fail-closed cases (`ravanan_batch_*`). + +## Slurm-side requirements + +The endpoint only speaks HTTP to slurmrestd, so the cluster has to be +wired the way Slurm expects for REST-submitted jobs to actually run: + +``` +AuthType=auth/munge +AuthAltTypes=auth/jwt +AuthAltParameters=jwt_key=/path/to/jwt.key +``` + +with `slurmrestd -a rest_auth/jwt` started with `SLURM_JWT` in **its** +environment, and the JSON Web Token minted for the user the jobs should +run as. The token's identity is forwarded to slurmctld, which decodes +it; slurmd keeps using munge, which is what it can verify. Setting +`AuthType=auth/jwt` cluster-wide instead lets submissions be recorded +but every launch RPC is rejected as `SLURM_AUTH_NOBODY` (uid 99), since +slurmd never loads the JWT key. See `gn-machines/slurm-session.md` for +the source-level details and a working test container +(`gn-machines/specials/slurm-container.scm`). + +ravanan's own switches are `--batch-system`, `--slurm-api-endpoint`, +`--slurm-jwt`, `--slurm-partition`, `--slurm-nice` and `--scratch`; note +that `--slurm-api-endpoint` only accepts a URL on ravanan +`v0.2.0-92`-era checkouts after commit "ravanan: Parse +--slurm-api-endpoint into a URI" (before that the value was used as a +`uri` record while being stored as a string, so only the built-in +default `http://localhost:6820` worked). |
