about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-10-24 09:33:40 +0000
committerPjotr <wrk@bosa.lan>2026-10-24 09:33:40 +0000
commita7ffb1cfecbdb8ac3fcc6286d7a09f7285c95d67 (patch)
tree1cb865784d901fad979aed32694b77474019468d /docs
parentcda325b6334ef36e3f49501cfd77d68a7bd6a295 (diff)
downloadgenenetwork4-a7ffb1cfecbdb8ac3fcc6286d7a09f7285c95d67.tar.gz
gemma: Make ravanan's batch system configurable (optional Slurm).
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system.  Add a
RavananBatch setting:

- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
  closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
  --slurm-jwt (a *file*, which is what ravanan reads) and, when set,
  --slurm-partition/--slurm-nice; a slurm selection without an endpoint
  or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
  the job scratch space live on the same shared filesystem (with slurm
  that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.

The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config.  docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
Diffstat (limited to 'docs')
-rw-r--r--docs/ravanan-slurm.md93
1 files changed, 93 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md
new file mode 100644
index 0000000..0b120d4
--- /dev/null
+++ b/docs/ravanan-slurm.md
@@ -0,0 +1,93 @@
+# Running ravanan's jobs on Slurm
+
+`POST /compute/gemma/ravanan` hands the whole per-chromosome LOCO
+workflow to [ravanan](https://git.systemreboot.net/ravanan), which runs
+each step itself. By default every step is a local child process
+(ravanan's `single-machine` batch system). Setting
+`RAVANAN_BATCH_SYSTEM=slurm-api` makes ravanan submit each step to a
+Slurm cluster through slurmrestd's REST API instead, so the CWL scatter
+really fans out over the cluster's nodes.
+
+Nothing else about the endpoint changes: same request, same response,
+same rows. The slurm switches are appended to ravanan's command line by
+`RavananBatch::ravanan_args` (`src/config.rs`), and the note in the
+response's `meta.notes` says which backend ran (`ravanan (…, slurm-api
+backend): …`).
+
+## Configuration
+
+| key | meaning |
+|---|---|
+| `RAVANAN_BATCH_SYSTEM` | `single-machine` (default) or `slurm-api`; anything else refuses to start |
+| `RAVANAN_SLURM_API_ENDPOINT` | slurmrestd base URL, e.g. `http://127.0.0.1:6820` (required for slurm-api) |
+| `RAVANAN_SLURM_JWT` | path to a **file** holding the JWT (ravanan's `--slurm-jwt` takes a file, not the token); required for slurm-api |
+| `RAVANAN_SLURM_PARTITION` | optional `--slurm-partition`, e.g. `debug` |
+| `RAVANAN_SLURM_NICE` | optional integer `--slurm-nice` |
+| `RAVANAN_SCRATCH` | optional `--scratch`; defaults to `RAVANAN_CACHE_DIR` when the backend is slurm |
+| `RAVANAN_CACHE_DIR` | where pheno files, logs, the compiled CWL and ravanan's `--store` live |
+
+Selecting slurm without an endpoint or a JWT file is a startup error
+(fail closed, like `GN4_DB_MODE`): ravanan would fail every step, and a
+refusal at boot beats a 502 on every request.
+
+**With slurm, `RAVANAN_CACHE_DIR` must be on a filesystem the compute
+nodes can see**, because ravanan's `--store` and `--scratch` live there
+and the slurmd-side job reads and writes them. On a single box that is
+just any shared directory; the same applies to the genotype/SNP/pheno
+inputs, whose paths are passed to ravanan verbatim (relative paths are
+resolved relative to the generated inputs YAML, so absolute paths are
+the safe choice).
+
+Two things the cluster needs that a plain dev box may not have:
+
+- `gemma-wrapper` (and the `gemma` it wraps) reachable in the profile
+  ravanan builds from `GEMMA_RAVANAN_MANIFEST`; ravanan resolves that
+  manifest against the *ambient* Guix process, so `GUIX_PACKAGE_PATH`
+  (or a pulled `guix-bioinformatics` channel) must be set in the
+  server's environment, as it already must for the single-machine
+  backend;
+- `/bin/bash` on the nodes: ravanan submits every step as
+  `#!/bin/bash` + the store script.
+
+## Example
+
+```sh
+export RAVANAN_BATCH_SYSTEM=slurm-api
+export RAVANAN_SLURM_API_ENDPOINT=http://127.0.0.1:6820
+export RAVANAN_SLURM_JWT=/etc/gn4/slurm-jwt        # file with the token
+export RAVANAN_SLURM_PARTITION=debug
+export RAVANAN_CACHE_DIR=/srv/gn4-ravanan          # shared with the nodes
+./genenetwork4
+```
+
+`src/config.rs`'s tests cover the switch, the argument construction and
+the fail-closed cases (`ravanan_batch_*`).
+
+## Slurm-side requirements
+
+The endpoint only speaks HTTP to slurmrestd, so the cluster has to be
+wired the way Slurm expects for REST-submitted jobs to actually run:
+
+```
+AuthType=auth/munge
+AuthAltTypes=auth/jwt
+AuthAltParameters=jwt_key=/path/to/jwt.key
+```
+
+with `slurmrestd -a rest_auth/jwt` started with `SLURM_JWT` in **its**
+environment, and the JSON Web Token minted for the user the jobs should
+run as. The token's identity is forwarded to slurmctld, which decodes
+it; slurmd keeps using munge, which is what it can verify. Setting
+`AuthType=auth/jwt` cluster-wide instead lets submissions be recorded
+but every launch RPC is rejected as `SLURM_AUTH_NOBODY` (uid 99), since
+slurmd never loads the JWT key. See `gn-machines/slurm-session.md` for
+the source-level details and a working test container
+(`gn-machines/specials/slurm-container.scm`).
+
+ravanan's own switches are `--batch-system`, `--slurm-api-endpoint`,
+`--slurm-jwt`, `--slurm-partition`, `--slurm-nice` and `--scratch`; note
+that `--slurm-api-endpoint` only accepts a URL on ravanan
+`v0.2.0-92`-era checkouts after commit "ravanan: Parse
+--slurm-api-endpoint into a URI" (before that the value was used as a
+`uri` record while being stored as a string, so only the built-in
+default `http://localhost:6820` worked).