about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-14 22:42:11 +0000
committerPjotr <wrk@bosa.lan>2026-11-14 22:42:11 +0000
commit3a62ef34c127d36fd4e04583349f64eee5de2c6f (patch)
treee60085c1ae4c9e8f085cb23c3dc0767a0f0dfb18 /docs
parenta89781c2eaf4b503f416cc2aba2420cd15fa5289 (diff)
downloadgenenetwork4-3a62ef34c127d36fd4e04583349f64eee5de2c6f.tar.gz
ravanan-slurm: what happens when Slurm goes away
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline
200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s;
container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no
partial results); the same request after Slurm returns -> 200 in 163.3 s,
i.e. the retry reuses the steps that had finished, because ravanan's store
lives in RAVANAN_CACHE_DIR and not in Slurm.
So: no automatic failover, no fallback to single-machine, nothing sticky,
and a retry that is cheaper than cold. The 502's detail is ravanan's Guile
backtrace, so the one useful line sits at its end; a pre-flight
/slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there
is no timeout on the ravanan child for a Slurm that accepts connections
and never runs the job -- both recorded as gaps.
Diffstat (limited to 'docs')
-rw-r--r--docs/ravanan-slurm.md47
1 files changed, 47 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md
index bddaaf0..391c447 100644
--- a/docs/ravanan-slurm.md
+++ b/docs/ravanan-slurm.md
@@ -250,3 +250,50 @@ the cluster actually has several nodes to fan out to (ravanan scatters one
 job per chromosome; this test container has a single node, so the steps
 serialize). On a 20-chromosome BXD run against the same container with
 default intervals, 40 jobs took 1193 s end to end.
+
+## What happens when Slurm goes away
+
+There is **no automatic failover**: nothing retries, nothing falls back to
+the `single-machine` backend, and a request in flight is not resumed. What
+there is, is a fast failure and a cheap retry. Measured on the same
+container (a 20-chromosome BXD run, 40 Slurm jobs, `slurm-api` backend,
+`POST /compute/gemma/ravanan`):
+
+| when Slurm dies | response | rows |  |
+|---|---|---|---|
+| never — baseline | `200` | 21 055 | 205.5 s |
+| before the request (`slurmrestd` unreachable) | `502 gemma_failed` | 0 | **3.7 s** |
+| 45 s into the request (container killed) | `502 gemma_failed` | 0 | **50.2 s** |
+| after the abort, Slurm back, same request | `200` | 21 051 | **163.3 s** |
+
+Four things to know:
+
+- **The failure is fast and identical either way**: ravanan raises inside
+  its Guile process — `In procedure connect: Connection refused` for
+  `POST /slurm/v0.0.41/job/submit`. The 502's `detail` is ravanan's whole
+  stderr, so the useful line is at the very end of a long backtrace. A
+  pre-flight `GET /slurm/v0.0.41/ping` would turn the first row into a
+  `503 slurm_unavailable` naming the endpoint, before any staging; that is
+  not implemented.
+- **No partial results are returned**, which is right — a half-mapped
+  genome with 0 rows and a 502 is honest. `?id=/status` tails ravanan's
+  log, not a result stream, so there is nothing half-written to read.
+- **Nothing is sticky.** The same request after Slurm returns is `200`;
+  the failed attempt leaves no lock, no half-interned store entry, nothing
+  to clean up.
+- **The retry is cheaper than cold, because ravanan's store survives**:
+  163.3 s against 205.5 s cold on the same 20 chromosomes. The steps that
+  finished before the kill are reused — `(script, inputs)` pairs live in
+  `RAVANAN_CACHE_DIR`, not in Slurm, and the kinship step is
+  trait-independent by design (`write_constant_pheno_file`), so its 20
+  jobs also serve the next trait. Practical rule for a caller: treat a 502
+  from this endpoint as retryable, and retry the *same* request.
+
+Two gaps this leaves, both worth closing before production trusts the path:
+
+- **no pre-flight**, so a dead cluster costs a `ccwl compile`, an input
+  staging and a ravanan startup before the connection is even attempted;
+- **no timeout on the ravanan child process.** Nothing here exercises a
+  Slurm that *accepts* connections and then never runs the job, and in that
+  case the request waits as long as ravanan does. A wall-clock bound with a
+  `504` would be the honest answer.