about summary refs log tree commit diff
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-10-24 15:03:28 +0000
committerPjotr <wrk@bosa.lan>2026-10-24 15:03:28 +0000
commitb4ea4e3aaa9735c4d4342b98e3692cce52a40220 (patch)
tree06a3b709e71a5896171240f354193fed701891ea
parentedba19046cdc6d75205d966fab8d1c4d94c19953 (diff)
downloadgenenetwork4-b4ea4e3aaa9735c4d4342b98e3692cce52a40220.tar.gz
cwl: Expand the "rebuild on the nodes instead of sharing a store" section.
Spell out why the current design needs the store visible (ravanan
submits a store path and Slurm does not substitute, so an absent path is
an immediate ENOENT), then list the five ways out as a table -- shared
store, realise-then-run wrapper, per-workflow realise job, guix publish
plus wrapper, shipping closures -- with what runs, what each needs on
the nodes, and what it costs in caching.

Also records where the wrapper would live (build-gexp-script already
builds the derivation but returns only its output path; submit-job in
ravanan/slurm-api.scm assembles the script field), that nothing of it is
implemented here, and the four costs that decide the question: channel
skew, build time inside the job's time limit, caching on ephemeral node
stores, and guix publish as the sane companion.

Tables tightened so no line runs past ~170 characters.
-rw-r--r--cwl/README.md131
1 files changed, 79 insertions, 52 deletions
diff --git a/cwl/README.md b/cwl/README.md
index 1492d12..186a689 100644
--- a/cwl/README.md
+++ b/cwl/README.md
@@ -72,15 +72,15 @@ Consequently:
 
 A node does need:
 
-| requirement                                           | why |
-|-------------------------------------------------------|-----|
-| `/bin/bash`                                           | ravanan submits `#!/bin/bash` + the store script; Slurm runs it through that shebang |
-| the **same `/gnu/store`** mounted (read-only is fine) | the script, `guile`, `gemma-wrapper`, `gemma` and coreutils all come from there — nothing is "installed" on the node |
-| read access to ravanan's store (`--store` dir)        | inputs are interned there and staged from there into the job's working directory |
-| **write access to that store's `data/` and `logs/`**  | the job writes its outputs record (`WORKFLOW_OUTPUT_DATA_FILE`) and its stdout/stderr there |
-| a writable **scratch** dir                            | the job's working directory is a temporary directory under `--scratch`; this is the only place jobs write freely ("jobs do not directly write to the shared network filesystem" is ravanan's design goal, so `/scratch` is normally node-local and fast) |
-| permission as the **job user**                        | the identity comes from the JWT (e.g. `testuser`), so the store and scratch paths must be writable by that uid |
-| resolvable `NodeAddr`                                 | Slurm's launch path needs the node address, not just a name |
+| requirement                              | why |
+|------------------------------------------|-----|
+| `/bin/bash`                              | ravanan submits `#!/bin/bash` + the store script; Slurm runs it via that shebang |
+| same `/gnu/store` mounted (read-only ok) | the script, `guile`, `gemma-wrapper`, `gemma`, coreutils are all store paths |
+| read access to ravanan's store           | input files are interned there and staged from there into the job's cwd |
+| write access to its `data/`, `logs/`     | the job writes its outputs record and its stdout/stderr there |
+| a writable **scratch** dir               | the job's cwd is a temporary dir under `--scratch`: keep it node-local |
+| permission as the **job user**           | identity comes from the JWT (e.g. `testuser`); these paths must be writable by it |
+| resolvable `NodeAddr`                    | Slurm's launch path needs the node address, not just a name |
 
 Practical split: `RAVANAN_CACHE_DIR` (ravanan's store + the
 gemma-wrapper caches) on a **shared** filesystem, `RAVANAN_SCRATCH`
@@ -89,10 +89,10 @@ only meant for single-box setups like the test container.
 
 ## The two caches
 
-| cache                         | lives in                                                                                                        | scope |
-|-------------------------------|-----------------------------------------------------------------------------------------------------------------|-------|
-| ravanan work reuse            | `<store>/data/<hash>.json`, keyed by (script, inputs)                                                           | checked on the front end *before* anything is submitted, so a hit never touches Slurm |
-| gemma-wrapper's `--cache-dir` | one directory per chromosome, passed as a CWL input (today `<ravanan_cache_dir>/gemma-wrapper-cache/chr-<chr>`) | inside the job; on a cluster it must be visible to the nodes (shared, or node-local if you prefer per-node caches) |
+| cache                       | lives in                                                                | scope |
+|-----------------------------|-------------------------------------------------------------------------|-------|
+| ravanan work reuse          | `<store>/data/<hash>.json`, keyed by (script, inputs)                   | front end; a hit never touches Slurm |
+| gemma-wrapper `--cache-dir` | one dir per chromosome under `<ravanan_cache_dir>/gemma-wrapper-cache/` | inside the job; nodes must see it |
 
 Two wrinkles specific to this endpoint:
 
@@ -110,45 +110,72 @@ Two wrinkles specific to this endpoint:
 
 ## Can the nodes rebuild the tools instead of sharing a store?
 
-Yes in principle — Guix is deterministic, so a node building the same
-derivations with the same channels produces *byte-identical store paths*,
-and the script ravanan submitted then simply exists. But nothing in the
-current setup triggers such a build, because ravanan submits a **store
-path**, and executing a store path that is absent fails immediately —
-there is no automatic substitution.
-
-Concretely, you would need one of:
-
-1. **Realise-then-run wrapper.** Submit `#!/bin/bash` + `guix build
-   <script.drv> && exec <script>` instead of the bare script, and keep
-   the script's derivation path around in ravanan's slurm backend
-   (`build-gexp-script` already has it in hand; today it returns only
-   the output path). Requires `guix` **and** a `guix-daemon` on every
-   node, plus the identical Guix commit/channels everywhere. Costs a
-   build per node per step-closure — fine for a shared substitute
-   server (`guix publish` on the front end), painful otherwise.
-2. **A realise job per workflow.** One extra Slurm job that runs `guix
-   build` for all the workflow's step derivations (or `guix archive
-   --import` from the front end) before the real jobs start. Slurm
-   dependency (`--dependency=afterok`) or a CWL step ordering is
-   needed, so this is workflow plumbing rather than a flag.
-3. **`guix publish` + node daemons, no wrapper.** Front end keeps
-   building (ravanan is unchanged only if the paths get *to* the node
-   some other way); with `--substitute-urls` pointing at the front end,
-   a node's `guix build <drv>` fetches the closure from the substitute
-   server instead of compiling it. Same wrapper as (1) to trigger it.
-4. **Ship the closure**: `guix archive --export` / `guix copy` the
-   script and profile closures to each node before the run. No guix on
-   the nodes, but it is per-run work and duplicates what NFS gives you.
-
-Why the shared store stays the default: ravanan's whole caching story
-(timings in the table below) assumes step scripts and profiles are
-already realised, and a shared/exported store is the standard Guix-on-HPC
-answer. Rebuilding on nodes is worth it when the store genuinely cannot
-be exported and the cluster already runs a Guix daemon with the same
-channels — then option (1)+(3) is a small patch to ravanan's
-`slurm:submit-job` (it builds the job spec in
-`ravanan/slurm-api.scm`), plus `RAVANAN_SCRATCH` on node-local disk.
+Short answer: **yes, in principle, but nothing in the current setup
+triggers it**, so today the store really does have to be visible to the
+nodes.
+
+Why the current design needs the store visible:
+
+- ravanan builds each CWL step into a *job script* on the front end
+  (`gexp->script`) and submits that **store path** (`#!/bin/bash` +
+  `\n` + `/gnu/store/…-<step>`). Everything the job executes — the
+  `guile` in the shebang, the profile `PATH`, `gemma-wrapper`, `gemma`,
+  coreutils — is a store path baked in at that moment;
+- Slurm knows nothing about Guix, so there is **no substitution**: a
+  store path that is absent on the node just fails with `ENOENT`, the
+  job never starts, and ravanan reports the step as failed;
+- Guix *is* deterministic, so a node that builds the same derivations
+  with the same channels would produce **byte-identical paths** and the
+  script would then exist — which is what makes the options below
+  possible at all.
+
+| option                  | what runs                                                   | prerequisites                              | caching |
+|-------------------------|-------------------------------------------------------------|--------------------------------------------|---------|
+| 1. shared store         | nodes read the front end's `/gnu/store`; nothing extra runs | store mounted on the nodes                 | full (~0.9 s cached rerun) |
+| 2. realise-then-run     | `guix build <script>.drv` then `exec <script>` per job      | guix + daemon per node, identical channels | only if the node store persists |
+| 3. per-workflow realise | one job builds/imports every step derivation first          | as (2) plus job ordering                   | as (2) |
+| 4. `guix publish` + (2) | as (2), substituting from the front end                     | as (2) with `--substitute-urls`            | as (2), faster cold nodes |
+| 5. ship closures        | `guix archive --export` / `guix copy` before the run        | no guix on nodes; per-run shipping         | within one run only |
+
+The wrapper of option 2/4 is about ten lines of change on the ravanan
+side: `build-gexp-script` in `ravanan/command-line-tool.scm` already
+builds the derivation (`gexp->script` returns a `<derivation>`) but
+returns only its output path, and `submit-job` in
+`ravanan/slurm-api.scm` assembles the `script` field of the job spec —
+so keeping the `.drv` path and submitting
+`#!/bin/bash\nguix build <drv> && exec <script>` is the whole
+mechanism. Nothing of that is implemented in this repo; the endpoint
+uses ravanan as-is, and `compute_ravanan` cannot influence how a step is
+launched beyond the switches it passes.
+
+Costs to weigh before choosing it:
+
+- **channel skew breaks it.** Every node and the front end must run the
+  same Guix commit *and* the same channels; a `guix pull` on one side
+  changes the hashes and nothing matches any more. Pinning that across a
+  cluster is an operational commitment, not a config file.
+- **build time lands inside the job.** A cold gemma/libc closure is
+  minutes per node; the first job on each node pays it inside the Slurm
+  time limit, and `gemma`'s build is the same one the front end already
+  did.
+- **caching evaporates on ephemeral node stores.** ravanan's free
+  "cached rerun" (below) only holds if the node's store survives between
+  jobs; with per-job containers or a wiped `/gnu/store` you re-realise
+  the closure every time.
+- **`guix publish` is the sane companion.** Front end keeps building
+  and serving (`guix publish`), nodes substitute with
+  `--substitute-urls=http://frontend:5000`; that turns "rebuild on the
+  node" into "download from the front end", which is what NFS would
+  have given you anyway — minus the shared-filesystem dependency, plus
+  per-node guix and a published store.
+
+Recommendation: keep the shared (or NFS-exported) store as the default —
+ravanan is designed around it and the timings below assume it. If the
+store genuinely cannot be exported and the cluster already runs a Guix
+daemon with pinned channels, implement options 2 + 4 upstream in
+ravanan and expose it here as an opt-in config switch; until then,
+`RAVANAN_SCRATCH` should stay node-local (`/scratch`) while
+`RAVANAN_CACHE_DIR` stays shared.
 
 ## What to expect, timing-wise