diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-14 22:42:11 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-14 22:42:11 +0000 |
| commit | 3a62ef34c127d36fd4e04583349f64eee5de2c6f (patch) | |
| tree | e60085c1ae4c9e8f085cb23c3dc0767a0f0dfb18 /docs | |
| parent | a89781c2eaf4b503f416cc2aba2420cd15fa5289 (diff) | |
| download | genenetwork4-3a62ef34c127d36fd4e04583349f64eee5de2c6f.tar.gz | |
ravanan-slurm: what happens when Slurm goes away
Measured against the slurm container, 20 chromosomes, 40 jobs: baseline 200/21 055 rows in 205.5 s; slurmrestd unreachable -> 502 in 3.7 s; container killed 45 s into a request -> 502 in 50.2 s with 0 rows (no partial results); the same request after Slurm returns -> 200 in 163.3 s, i.e. the retry reuses the steps that had finished, because ravanan's store lives in RAVANAN_CACHE_DIR and not in Slurm. So: no automatic failover, no fallback to single-machine, nothing sticky, and a retry that is cheaper than cold. The 502's detail is ravanan's Guile backtrace, so the one useful line sits at its end; a pre-flight /slurm/v0.0.41/ping would make that a 503 naming the endpoint, and there is no timeout on the ravanan child for a Slurm that accepts connections and never runs the job -- both recorded as gaps.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/ravanan-slurm.md | 47 |
1 files changed, 47 insertions, 0 deletions
diff --git a/docs/ravanan-slurm.md b/docs/ravanan-slurm.md index bddaaf0..391c447 100644 --- a/docs/ravanan-slurm.md +++ b/docs/ravanan-slurm.md @@ -250,3 +250,50 @@ the cluster actually has several nodes to fan out to (ravanan scatters one job per chromosome; this test container has a single node, so the steps serialize). On a 20-chromosome BXD run against the same container with default intervals, 40 jobs took 1193 s end to end. + +## What happens when Slurm goes away + +There is **no automatic failover**: nothing retries, nothing falls back to +the `single-machine` backend, and a request in flight is not resumed. What +there is, is a fast failure and a cheap retry. Measured on the same +container (a 20-chromosome BXD run, 40 Slurm jobs, `slurm-api` backend, +`POST /compute/gemma/ravanan`): + +| when Slurm dies | response | rows | | +|---|---|---|---| +| never — baseline | `200` | 21 055 | 205.5 s | +| before the request (`slurmrestd` unreachable) | `502 gemma_failed` | 0 | **3.7 s** | +| 45 s into the request (container killed) | `502 gemma_failed` | 0 | **50.2 s** | +| after the abort, Slurm back, same request | `200` | 21 051 | **163.3 s** | + +Four things to know: + +- **The failure is fast and identical either way**: ravanan raises inside + its Guile process — `In procedure connect: Connection refused` for + `POST /slurm/v0.0.41/job/submit`. The 502's `detail` is ravanan's whole + stderr, so the useful line is at the very end of a long backtrace. A + pre-flight `GET /slurm/v0.0.41/ping` would turn the first row into a + `503 slurm_unavailable` naming the endpoint, before any staging; that is + not implemented. +- **No partial results are returned**, which is right — a half-mapped + genome with 0 rows and a 502 is honest. `?id=/status` tails ravanan's + log, not a result stream, so there is nothing half-written to read. +- **Nothing is sticky.** The same request after Slurm returns is `200`; + the failed attempt leaves no lock, no half-interned store entry, nothing + to clean up. +- **The retry is cheaper than cold, because ravanan's store survives**: + 163.3 s against 205.5 s cold on the same 20 chromosomes. The steps that + finished before the kill are reused — `(script, inputs)` pairs live in + `RAVANAN_CACHE_DIR`, not in Slurm, and the kinship step is + trait-independent by design (`write_constant_pheno_file`), so its 20 + jobs also serve the next trait. Practical rule for a caller: treat a 502 + from this endpoint as retryable, and retry the *same* request. + +Two gaps this leaves, both worth closing before production trusts the path: + +- **no pre-flight**, so a dead cluster costs a `ccwl compile`, an input + staging and a ravanan startup before the connection is even attempted; +- **no timeout on the ravanan child process.** Nothing here exercises a + Slurm that *accepts* connections and then never runs the job, and in that + case the request waits as long as ravanan does. A wall-clock bound with a + `504` would be the honest answer. |
