diff options
| author | Pjotr <wrk@bosa.lan> | 2026-10-24 15:03:28 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-10-24 15:03:28 +0000 |
| commit | b4ea4e3aaa9735c4d4342b98e3692cce52a40220 (patch) | |
| tree | 06a3b709e71a5896171240f354193fed701891ea | |
| parent | edba19046cdc6d75205d966fab8d1c4d94c19953 (diff) | |
| download | genenetwork4-b4ea4e3aaa9735c4d4342b98e3692cce52a40220.tar.gz | |
cwl: Expand the "rebuild on the nodes instead of sharing a store" section.
Spell out why the current design needs the store visible (ravanan submits a store path and Slurm does not substitute, so an absent path is an immediate ENOENT), then list the five ways out as a table -- shared store, realise-then-run wrapper, per-workflow realise job, guix publish plus wrapper, shipping closures -- with what runs, what each needs on the nodes, and what it costs in caching. Also records where the wrapper would live (build-gexp-script already builds the derivation but returns only its output path; submit-job in ravanan/slurm-api.scm assembles the script field), that nothing of it is implemented here, and the four costs that decide the question: channel skew, build time inside the job's time limit, caching on ephemeral node stores, and guix publish as the sane companion. Tables tightened so no line runs past ~170 characters.
| -rw-r--r-- | cwl/README.md | 131 |
1 files changed, 79 insertions, 52 deletions
diff --git a/cwl/README.md b/cwl/README.md index 1492d12..186a689 100644 --- a/cwl/README.md +++ b/cwl/README.md @@ -72,15 +72,15 @@ Consequently: A node does need: -| requirement | why | -|-------------------------------------------------------|-----| -| `/bin/bash` | ravanan submits `#!/bin/bash` + the store script; Slurm runs it through that shebang | -| the **same `/gnu/store`** mounted (read-only is fine) | the script, `guile`, `gemma-wrapper`, `gemma` and coreutils all come from there — nothing is "installed" on the node | -| read access to ravanan's store (`--store` dir) | inputs are interned there and staged from there into the job's working directory | -| **write access to that store's `data/` and `logs/`** | the job writes its outputs record (`WORKFLOW_OUTPUT_DATA_FILE`) and its stdout/stderr there | -| a writable **scratch** dir | the job's working directory is a temporary directory under `--scratch`; this is the only place jobs write freely ("jobs do not directly write to the shared network filesystem" is ravanan's design goal, so `/scratch` is normally node-local and fast) | -| permission as the **job user** | the identity comes from the JWT (e.g. `testuser`), so the store and scratch paths must be writable by that uid | -| resolvable `NodeAddr` | Slurm's launch path needs the node address, not just a name | +| requirement | why | +|------------------------------------------|-----| +| `/bin/bash` | ravanan submits `#!/bin/bash` + the store script; Slurm runs it via that shebang | +| same `/gnu/store` mounted (read-only ok) | the script, `guile`, `gemma-wrapper`, `gemma`, coreutils are all store paths | +| read access to ravanan's store | input files are interned there and staged from there into the job's cwd | +| write access to its `data/`, `logs/` | the job writes its outputs record and its stdout/stderr there | +| a writable **scratch** dir | the job's cwd is a temporary dir under `--scratch`: keep it node-local | +| permission as the **job user** | identity comes from the JWT (e.g. `testuser`); these paths must be writable by it | +| resolvable `NodeAddr` | Slurm's launch path needs the node address, not just a name | Practical split: `RAVANAN_CACHE_DIR` (ravanan's store + the gemma-wrapper caches) on a **shared** filesystem, `RAVANAN_SCRATCH` @@ -89,10 +89,10 @@ only meant for single-box setups like the test container. ## The two caches -| cache | lives in | scope | -|-------------------------------|-----------------------------------------------------------------------------------------------------------------|-------| -| ravanan work reuse | `<store>/data/<hash>.json`, keyed by (script, inputs) | checked on the front end *before* anything is submitted, so a hit never touches Slurm | -| gemma-wrapper's `--cache-dir` | one directory per chromosome, passed as a CWL input (today `<ravanan_cache_dir>/gemma-wrapper-cache/chr-<chr>`) | inside the job; on a cluster it must be visible to the nodes (shared, or node-local if you prefer per-node caches) | +| cache | lives in | scope | +|-----------------------------|-------------------------------------------------------------------------|-------| +| ravanan work reuse | `<store>/data/<hash>.json`, keyed by (script, inputs) | front end; a hit never touches Slurm | +| gemma-wrapper `--cache-dir` | one dir per chromosome under `<ravanan_cache_dir>/gemma-wrapper-cache/` | inside the job; nodes must see it | Two wrinkles specific to this endpoint: @@ -110,45 +110,72 @@ Two wrinkles specific to this endpoint: ## Can the nodes rebuild the tools instead of sharing a store? -Yes in principle — Guix is deterministic, so a node building the same -derivations with the same channels produces *byte-identical store paths*, -and the script ravanan submitted then simply exists. But nothing in the -current setup triggers such a build, because ravanan submits a **store -path**, and executing a store path that is absent fails immediately — -there is no automatic substitution. - -Concretely, you would need one of: - -1. **Realise-then-run wrapper.** Submit `#!/bin/bash` + `guix build - <script.drv> && exec <script>` instead of the bare script, and keep - the script's derivation path around in ravanan's slurm backend - (`build-gexp-script` already has it in hand; today it returns only - the output path). Requires `guix` **and** a `guix-daemon` on every - node, plus the identical Guix commit/channels everywhere. Costs a - build per node per step-closure — fine for a shared substitute - server (`guix publish` on the front end), painful otherwise. -2. **A realise job per workflow.** One extra Slurm job that runs `guix - build` for all the workflow's step derivations (or `guix archive - --import` from the front end) before the real jobs start. Slurm - dependency (`--dependency=afterok`) or a CWL step ordering is - needed, so this is workflow plumbing rather than a flag. -3. **`guix publish` + node daemons, no wrapper.** Front end keeps - building (ravanan is unchanged only if the paths get *to* the node - some other way); with `--substitute-urls` pointing at the front end, - a node's `guix build <drv>` fetches the closure from the substitute - server instead of compiling it. Same wrapper as (1) to trigger it. -4. **Ship the closure**: `guix archive --export` / `guix copy` the - script and profile closures to each node before the run. No guix on - the nodes, but it is per-run work and duplicates what NFS gives you. - -Why the shared store stays the default: ravanan's whole caching story -(timings in the table below) assumes step scripts and profiles are -already realised, and a shared/exported store is the standard Guix-on-HPC -answer. Rebuilding on nodes is worth it when the store genuinely cannot -be exported and the cluster already runs a Guix daemon with the same -channels — then option (1)+(3) is a small patch to ravanan's -`slurm:submit-job` (it builds the job spec in -`ravanan/slurm-api.scm`), plus `RAVANAN_SCRATCH` on node-local disk. +Short answer: **yes, in principle, but nothing in the current setup +triggers it**, so today the store really does have to be visible to the +nodes. + +Why the current design needs the store visible: + +- ravanan builds each CWL step into a *job script* on the front end + (`gexp->script`) and submits that **store path** (`#!/bin/bash` + + `\n` + `/gnu/store/…-<step>`). Everything the job executes — the + `guile` in the shebang, the profile `PATH`, `gemma-wrapper`, `gemma`, + coreutils — is a store path baked in at that moment; +- Slurm knows nothing about Guix, so there is **no substitution**: a + store path that is absent on the node just fails with `ENOENT`, the + job never starts, and ravanan reports the step as failed; +- Guix *is* deterministic, so a node that builds the same derivations + with the same channels would produce **byte-identical paths** and the + script would then exist — which is what makes the options below + possible at all. + +| option | what runs | prerequisites | caching | +|-------------------------|-------------------------------------------------------------|--------------------------------------------|---------| +| 1. shared store | nodes read the front end's `/gnu/store`; nothing extra runs | store mounted on the nodes | full (~0.9 s cached rerun) | +| 2. realise-then-run | `guix build <script>.drv` then `exec <script>` per job | guix + daemon per node, identical channels | only if the node store persists | +| 3. per-workflow realise | one job builds/imports every step derivation first | as (2) plus job ordering | as (2) | +| 4. `guix publish` + (2) | as (2), substituting from the front end | as (2) with `--substitute-urls` | as (2), faster cold nodes | +| 5. ship closures | `guix archive --export` / `guix copy` before the run | no guix on nodes; per-run shipping | within one run only | + +The wrapper of option 2/4 is about ten lines of change on the ravanan +side: `build-gexp-script` in `ravanan/command-line-tool.scm` already +builds the derivation (`gexp->script` returns a `<derivation>`) but +returns only its output path, and `submit-job` in +`ravanan/slurm-api.scm` assembles the `script` field of the job spec — +so keeping the `.drv` path and submitting +`#!/bin/bash\nguix build <drv> && exec <script>` is the whole +mechanism. Nothing of that is implemented in this repo; the endpoint +uses ravanan as-is, and `compute_ravanan` cannot influence how a step is +launched beyond the switches it passes. + +Costs to weigh before choosing it: + +- **channel skew breaks it.** Every node and the front end must run the + same Guix commit *and* the same channels; a `guix pull` on one side + changes the hashes and nothing matches any more. Pinning that across a + cluster is an operational commitment, not a config file. +- **build time lands inside the job.** A cold gemma/libc closure is + minutes per node; the first job on each node pays it inside the Slurm + time limit, and `gemma`'s build is the same one the front end already + did. +- **caching evaporates on ephemeral node stores.** ravanan's free + "cached rerun" (below) only holds if the node's store survives between + jobs; with per-job containers or a wiped `/gnu/store` you re-realise + the closure every time. +- **`guix publish` is the sane companion.** Front end keeps building + and serving (`guix publish`), nodes substitute with + `--substitute-urls=http://frontend:5000`; that turns "rebuild on the + node" into "download from the front end", which is what NFS would + have given you anyway — minus the shared-filesystem dependency, plus + per-node guix and a published store. + +Recommendation: keep the shared (or NFS-exported) store as the default — +ravanan is designed around it and the timings below assume it. If the +store genuinely cannot be exported and the cluster already runs a Guix +daemon with pinned channels, implement options 2 + 4 upstream in +ravanan and expose it here as an opt-in config switch; until then, +`RAVANAN_SCRATCH` should stay node-local (`/scratch`) while +`RAVANAN_CACHE_DIR` stays shared. ## What to expect, timing-wise |
