diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-14 16:44:07 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-14 16:44:07 +0000 |
| commit | a89781c2eaf4b503f416cc2aba2420cd15fa5289 (patch) | |
| tree | 1685790d7fde5b45da4f280210579e56fc0ae3d7 /docs | |
| parent | 05907254d44ccde0aa0c1f536d2ba838d0645402 (diff) | |
| download | genenetwork4-a89781c2eaf4b503f416cc2aba2420cd15fa5289.tar.gz | |
cache: say when ravanan's cache cannot survive a restart, and document what does
Ravanan's reuse key is (script, inputs) and both live in its --store under RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the container's. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, with the store unchanged at 14 files -- Slurm's own spool state is lost and irrelevant to reuse. The default cache dir is under /tmp, which is neither durable nor visible inside a container that bind-mounts one host path, so a job cannot read the interned inputs. The reply now carries a note when the cache dir is under /tmp, naming the fix, rather than leaving it to be discovered as a slow or failing run. docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan store, scratch, token registry, gemma-wrapper cache, Slurm state) and a path checklist; the README points at it.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/ravanan-slurm.md | 39 |
1 files changed, 39 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md index d8b40bf..bddaaf0 100644 --- a/docs/ravanan-slurm.md +++ b/docs/ravanan-slurm.md @@ -67,6 +67,45 @@ export RAVANAN_CACHE_DIR=/srv/gn4-ravanan # shared with the nodes `src/config.rs`'s tests cover the switch, the argument construction and the fail-closed cases (`ravanan_batch_*`). +## What survives a restart + +Ravanan's reuse key is `(script, inputs)`, and both halves live in +`RAVANAN_CACHE_DIR` (its `--store`). So the cache's lifetime is exactly the +lifetime of that directory — not of the container and not of Slurm: + +| state | where | survives a container restart | survives a host reboot | must the nodes see it | +|---|---|---|---|---| +| ravanan store (`--store`: interned inputs, per-step outputs, reuse records) | `RAVANAN_CACHE_DIR` | **yes, if it is under the bind-mounted share** | yes, if it is not under `/tmp` | **yes** | +| ravanan scratch | `RAVANAN_SCRATCH`, or the cache dir when unset | as above | as above | yes | +| short-token registry | `GN4_API_TOKENS_DB` (`tokens.sqlite`) | yes, if not on an ephemeral path | yes, if not under `/tmp` | no | +| gemma-wrapper cache (direct runner only) | `GEMMA_CACHE_DIR` | yes | yes, if not under `/tmp` | no | +| Slurm controller/spool state (`/var/spool/slurmctld`, `/var/spool/slurmd`) | inside the container | **no** — rebuilt at every boot | — | — | + +Measured: with the cache at `<shared mount>/gn4-ravanan-cache`, a full +20-chromosome LOCO mapping cost 13.6 s cold; after **restarting the container** +the identical request returned in 3.6 s with the store unchanged (14 files before +and after). Slurm's lost state is irrelevant to that — the reuse record is in the +store, not in `slurmctld`. + +Two ways to get this wrong, both silent until a run fails or is slow: + +- the default `RAVANAN_CACHE_DIR` is under `/tmp`, which is neither durable nor + visible inside a container that bind-mounts one host path, so jobs cannot read + the interned inputs. The reply says so in a note when the cache dir is under + `/tmp`. +- `GN4_API_TOKENS_DB` defaults to `./tokens.sqlite` in the working directory: a + service whose `WorkingDirectory` is a checkout under `/tmp`, or a directory + that is recreated, loses every short token on reboot. + +Deployment checklist for the paths: + +```sh +RAVANAN_CACHE_DIR=<shared mount>/gn4-ravanan-cache # durable, nodes can see it +RAVANAN_SCRATCH=<shared mount>/gn4-ravanan-cache # or node-local scratch +GN4_API_TOKENS_DB=<durable dir>/tokens.sqlite # mode 0600, never in the repo +GEMMA_CACHE_DIR=<durable dir>/gn4-gemma # direct runner only +``` + ## Variance components (`pve`) on this path `gemma-loco.scm`'s `gwa` step passes `-pve`, as the direct path does, and |
