about summary refs log tree commit diff
path: root/test/python
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 16:32:59 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 16:32:59 +0000
commitab5fef05237309667dd2ca7f835a5fbf9c71a12a (patch)
tree94267f7fa6c5f27fe6d5093d1280c48170ffe164 /test/python
parente934e755f14648fff6ebac07e9cf9e5b37cb3fe4 (diff)
downloadgenenetwork4-ab5fef05237309667dd2ca7f835a5fbf9c71a12a.tar.gz
compute: gate GEMMA from 50 individuals, R/qtl from 1 000
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:

- every GEMMA run goes through **one shared cache directory**
  (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
  kinship archives of hundreds of megabytes by content hash. Two runs at
  once are two writers over the same archives, and a half-extracted kinship
  another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
  individuals write 207 MB of kinship per chromosome.

So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.

Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.

* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
  (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
  the test that pinned one number now pins both and that each endpoint is
  wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
  the two thresholds and the cache-directory reason for the lower one.

277 tests pass; the same four as before do not.
Diffstat (limited to 'test/python')
0 files changed, 0 insertions, 0 deletions