about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-11-14 16:44:07 +0000
committerPjotr Prins <pjotr.public01@thebird.nl>2026-11-14 16:44:07 +0000
commita89781c2eaf4b503f416cc2aba2420cd15fa5289 (patch)
tree1685790d7fde5b45da4f280210579e56fc0ae3d7 /docs
parent05907254d44ccde0aa0c1f536d2ba838d0645402 (diff)
downloadgenenetwork4-a89781c2eaf4b503f416cc2aba2420cd15fa5289.tar.gz
cache: say when ravanan's cache cannot survive a restart, and document what does
Ravanan's reuse key is (script, inputs) and both live in its --store under
RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the
container's.  Measured: 13.6 s cold, then 3.6 s for the identical request
after restarting the container, with the store unchanged at 14 files --
Slurm's own spool state is lost and irrelevant to reuse.

The default cache dir is under /tmp, which is neither durable nor visible
inside a container that bind-mounts one host path, so a job cannot read
the interned inputs.  The reply now carries a note when the cache dir is
under /tmp, naming the fix, rather than leaving it to be discovered as a
slow or failing run.

docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan
store, scratch, token registry, gemma-wrapper cache, Slurm state) and a
path checklist; the README points at it.
Diffstat (limited to 'docs')
-rw-r--r--docs/ravanan-slurm.md39
1 files changed, 39 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md
index d8b40bf..bddaaf0 100644
--- a/docs/ravanan-slurm.md
+++ b/docs/ravanan-slurm.md
@@ -67,6 +67,45 @@ export RAVANAN_CACHE_DIR=/srv/gn4-ravanan          # shared with the nodes
 `src/config.rs`'s tests cover the switch, the argument construction and
 the fail-closed cases (`ravanan_batch_*`).
 
+## What survives a restart
+
+Ravanan's reuse key is `(script, inputs)`, and both halves live in
+`RAVANAN_CACHE_DIR` (its `--store`). So the cache's lifetime is exactly the
+lifetime of that directory — not of the container and not of Slurm:
+
+| state | where | survives a container restart | survives a host reboot | must the nodes see it |
+|---|---|---|---|---|
+| ravanan store (`--store`: interned inputs, per-step outputs, reuse records) | `RAVANAN_CACHE_DIR` | **yes, if it is under the bind-mounted share** | yes, if it is not under `/tmp` | **yes** |
+| ravanan scratch | `RAVANAN_SCRATCH`, or the cache dir when unset | as above | as above | yes |
+| short-token registry | `GN4_API_TOKENS_DB` (`tokens.sqlite`) | yes, if not on an ephemeral path | yes, if not under `/tmp` | no |
+| gemma-wrapper cache (direct runner only) | `GEMMA_CACHE_DIR` | yes | yes, if not under `/tmp` | no |
+| Slurm controller/spool state (`/var/spool/slurmctld`, `/var/spool/slurmd`) | inside the container | **no** — rebuilt at every boot | — | — |
+
+Measured: with the cache at `<shared mount>/gn4-ravanan-cache`, a full
+20-chromosome LOCO mapping cost 13.6 s cold; after **restarting the container**
+the identical request returned in 3.6 s with the store unchanged (14 files before
+and after). Slurm's lost state is irrelevant to that — the reuse record is in the
+store, not in `slurmctld`.
+
+Two ways to get this wrong, both silent until a run fails or is slow:
+
+- the default `RAVANAN_CACHE_DIR` is under `/tmp`, which is neither durable nor
+  visible inside a container that bind-mounts one host path, so jobs cannot read
+  the interned inputs. The reply says so in a note when the cache dir is under
+  `/tmp`.
+- `GN4_API_TOKENS_DB` defaults to `./tokens.sqlite` in the working directory: a
+  service whose `WorkingDirectory` is a checkout under `/tmp`, or a directory
+  that is recreated, loses every short token on reboot.
+
+Deployment checklist for the paths:
+
+```sh
+RAVANAN_CACHE_DIR=<shared mount>/gn4-ravanan-cache     # durable, nodes can see it
+RAVANAN_SCRATCH=<shared mount>/gn4-ravanan-cache       # or node-local scratch
+GN4_API_TOKENS_DB=<durable dir>/tokens.sqlite          # mode 0600, never in the repo
+GEMMA_CACHE_DIR=<durable dir>/gn4-gemma                # direct runner only
+```
+
 ## Variance components (`pve`) on this path
 
 `gemma-loco.scm`'s `gwa` step passes `-pve`, as the direct path does, and