diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 16:32:59 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 16:32:59 +0000 |
| commit | ab5fef05237309667dd2ca7f835a5fbf9c71a12a (patch) | |
| tree | 94267f7fa6c5f27fe6d5093d1280c48170ffe164 /docs | |
| parent | e934e755f14648fff6ebac07e9cf9e5b37cb3fe4 (diff) | |
| download | genenetwork4-ab5fef05237309667dd2ca7f835a5fbf9c71a12a.tar.gz | |
compute: gate GEMMA from 50 individuals, R/qtl from 1 000
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/www/ai-instruction.md | 17 | ||||
| -rw-r--r-- | docs/www/operator-checklist.md | 15 |
2 files changed, 20 insertions, 12 deletions
diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index 81f2ae8..d1d26e6 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -493,12 +493,17 @@ that rather than assembling your own. Until then, judge a GEMMA peak against the smallest `p_lrt` it reached, not against a genome-wide threshold of its own. -**Large jobs are gated, small ones are not.** The gate is there to stop -*large* computations piling up on the server, so it begins at **1 000 -individuals**: a request over a smaller panel — BXD's ~200, an older -93-strain version, any of the fixtures — takes no slot, runs alongside -whatever else is running, and is never refused. That is most exploratory -work, and it costs nothing to fire a few of those off. +**Large jobs are gated, small ones are not — and for GEMMA “large” starts +at 50 individuals.** The gate is there to stop *large* computations piling +up on the server. GEMMA runs share one cache directory (hundreds of +megabytes of kinship archives extracted by content hash) and a real panel +takes every core it can get, so `/compute/gemma` and +`/compute/gemma/ravanan` gate from **50 individuals**: in practice every +mapping request is gated, and only a handful of individuals runs free. +`/compute/rqtl` gates later, from **1 000 individuals**, because a +`scanone` over a few hundred is seconds of one core — so exploratory scans +on small crosses take no slot, run alongside whatever else is running, and +are never refused. Above the threshold, each endpoint has a fixed number of slots — **1** for `/compute/gemma`, **1** for `/compute/gemma/ravanan`, **8** for diff --git a/docs/www/operator-checklist.md b/docs/www/operator-checklist.md index 4f245d4..797f18b 100644 --- a/docs/www/operator-checklist.md +++ b/docs/www/operator-checklist.md @@ -208,9 +208,13 @@ up-but-useless service is visible without calling a data endpoint. ## Compute concurrency The gates exist to stop **large** computations piling up, so they only -apply above `compute_gate::LARGE_PANEL` (1 000 individuals); a smaller -panel takes no slot, which is why the in-process test suite no longer -serialises itself. Above it, `POST /compute/gemma` takes one job at a time, +apply above a threshold that differs by endpoint: +`compute_gate::GEMMA_LARGE_PANEL` is **50** individuals (all GEMMA runs +share `GEMMA_CACHE_DIR`, where `gemma-wrapper` extracts per-chromosome +kinship archives by content hash -- two runs at once are two writers over +the same hundreds of megabytes), `compute_gate::RQTL_LARGE_PANEL` is +**1 000** (a `scanone` over a few hundred individuals is seconds of one +core). A panel below its endpoint's threshold takes no slot. Above it, `POST /compute/gemma` takes one job at a time, `POST /compute/rqtl` `RQTL_MAX_CONCURRENT` (default 8) and `/compute/gemma/ravanan` one at a time, each one large job per caller. All three are in-process gates (`src/compute_gate.rs`), so they are per server @@ -219,9 +223,8 @@ the server. Waits are bounded only by nginx's `proxy_read_timeout`. The slot count is fixed when a gate is first built (its first large request), so raising `RQTL_MAX_CONCURRENT` takes a restart, not a reload. -Lowering the threshold is the same: `LARGE_PANEL` is a constant, and the -reason it is 1 000 is in its doc comment (measured K-matrix sizes and run -times). +Changing a threshold is the same: both are constants, and the reasons for +the numbers are in their doc comments. ## Plotting |
