diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 14:15:13 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 14:15:24 +0000 |
| commit | 51739ea82f3543ac6e28c8392d6d62957c021b3a (patch) | |
| tree | 7663b917022f709b343933ca036e6f420a3a460a /docs | |
| parent | 1f76ef2ef19ae36882236be7c016f0ab98d31f11 (diff) | |
| download | genenetwork4-51739ea82f3543ac6e28c8392d6d62957c021b3a.tar.gz | |
compute: gate R/qtl at eight concurrent scans, one per caller
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default
8, read when the gate is built) with the same one-job-per-caller rule as
GEMMA: a second submission from the same token while its scan is running
or queued is refused `409 compute_busy`; a submission that finds every
slot taken waits for one to come free. Eight because an R/qtl scan is a
single-threaded Rscript on a 96-core box; not more, because a scan on a
3 000-individual cross is minutes of CPU and the same machine serves the
database. Unlike GEMMA's single slot, concurrent scans share no cache
(R/qtl results are uncached), so the only cost is the CPU they take.
Two flaws only appeared once there was more than one slot:
- the "what is running" record was a single `Option<Active>`, so a
refusal quoted whichever job had most recently taken a slot -- robw's
409 named JoM's run id. It is now a map keyed by subject, and
`a_refusal_quotes_the_callers_own_run_id` pins it;
- `Busy::describe` took the endpoint identifier, so the message read
"your rqtl job"; the decorator takes a separate `label` ("R/qtl") from
the status path.
The test suite hit the rule itself: six compute tests in tests/rqtl.rs
share one token, and cargo runs them on several threads, so three were
refused instead of run. They now serialise on a module mutex, with a
comment saying why -- the same lesson clients need, that parallelism
means more tokens, not more requests from one.
* src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`;
`Gate::new(permits)` was already parameterised. `running` becomes a map
keyed by subject, since one record cannot name N jobs. Module doc talks
about slots rather than a single slot. Tests: eight subjects run and the
ninth waits, a caller still may not take two even with slots free, and a
refusal quotes the caller's own run id.
* src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use.
* gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are
independent.
* src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which
it needs.
* tests/rqtl.rs: compute tests serialised.
* README.md (/compute/rqtl), docs/www/ai-instruction.md (4a),
docs/www/operator-checklist.md: the numbers per endpoint, and that more
parallelism means more tokens.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/www/ai-instruction.md | 17 | ||||
| -rw-r--r-- | docs/www/operator-checklist.md | 9 |
2 files changed, 18 insertions, 8 deletions
diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index 4e8498b..bc185e2 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -447,14 +447,15 @@ against a genome-wide threshold. (gemma-wrapper itself can shuffle phenotypes with `--permutate n`; this API does not pass it — see docs/genotypes.md.) -**One job at a time, per caller.** `/compute/gemma` runs one job at a -time across all callers: a second submission from *your* token while your -job is still running — or still queued — is refused at once with -`409 compute_busy`, naming the run and linking its status endpoint; a -submission from *another* token waits for the running job to finish and -then runs. So a 409 here is not a retry-able failure: poll -`/compute/gemma/status/{id}`, and only then submit the next job. Map one -trait, read it, submit the next. +**A few jobs at a time, one per caller.** Each compute endpoint has a +fixed number of slots — **1** for `/compute/gemma`, **8** for +`/compute/rqtl` — and one job per caller within them. A second submission +from *your* token while your job is still running or queued is refused at +once with `409 compute_busy`, naming the run and linking its status +endpoint; a submission that finds every slot taken waits for one to come +free and then runs. So a 409 here is not a retry-able failure: poll the +status endpoint it names, and only then submit the next job. (If you want +more parallelism, that is more *tokens*, not more requests from one.) Results are **cached** by input, so running the same request twice costs little the second time (a cached GEMMA run on the BXD fixtures comes back diff --git a/docs/www/operator-checklist.md b/docs/www/operator-checklist.md index 14c5c82..6f49501 100644 --- a/docs/www/operator-checklist.md +++ b/docs/www/operator-checklist.md @@ -205,6 +205,15 @@ restarts — that is what happened on 2026-11-07. `GET /version` now carries `"database": "reachable" | "unreachable"` (cached ten seconds), so an up-but-useless service is visible without calling a data endpoint. +## Compute concurrency + +`POST /compute/gemma` takes one job at a time and `POST /compute/rqtl` +`RQTL_MAX_CONCURRENT` (default 8), one job per caller each; both are +in-process gates (`src/compute_gate.rs`), so they are per server process +and reset on restart -- a job's child process, however, outlives the +server. `/compute/gemma/ravanan` is not gated yet. Idle waits are bounded +only by nginx's `proxy_read_timeout`. + ## Plotting `POST /plot/rqtl` and `/plot/gemma` take a **posted mapping response** -- |
