diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 16:32:59 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 16:32:59 +0000 |
| commit | ab5fef05237309667dd2ca7f835a5fbf9c71a12a (patch) | |
| tree | 94267f7fa6c5f27fe6d5093d1280c48170ffe164 /src | |
| parent | e934e755f14648fff6ebac07e9cf9e5b37cb3fe4 (diff) | |
| download | genenetwork4-ab5fef05237309667dd2ca7f835a5fbf9c71a12a.tar.gz | |
compute: gate GEMMA from 50 individuals, R/qtl from 1 000
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
Diffstat (limited to 'src')
| -rw-r--r-- | src/compute_gate.rs | 130 | ||||
| -rw-r--r-- | src/routes/gemma.rs | 2 | ||||
| -rw-r--r-- | src/routes/rqtl.rs | 2 |
3 files changed, 91 insertions, 43 deletions
diff --git a/src/compute_gate.rs b/src/compute_gate.rs index c0a073e..c3ad1eb 100644 --- a/src/compute_gate.rs +++ b/src/compute_gate.rs @@ -9,6 +9,11 @@ //! than being refused. One slot for GEMMA (`gemma`, because two runs //! would fight over the same cache and the same cores), eight for R/qtl //! (`rqtl`, `RQTL_MAX_CONCURRENT`); +//! - **and only *large* jobs take a slot.** A panel below the endpoint's +//! threshold ([`GEMMA_LARGE_PANEL`], [`RQTL_LARGE_PANEL`]) runs +//! alongside everything else: the gate is for protecting the machine, +//! and a two-second request that waits behind a ten-minute one is worse +//! off than the machine would be; //! - **one job per subject**: a second submission from somebody who //! already holds a slot *or is queued for one* is refused immediately //! (`409 compute_busy`), because queueing it would only make them wait @@ -35,29 +40,42 @@ use std::time::Instant; use tokio::sync::{Semaphore, SemaphorePermit}; -/// The panel size at which a job stops being small, and starts taking the -/// endpoint's slot. +/// Where R/qtl's panel size stops being small, and starts taking one of +/// its slots. /// /// What the gate is for is **large computations**, not computation as -/// such: a reflex-sized job (the BXD panels' ~200 individuals, a 93-strain -/// version, the test fixtures) finishes in seconds, and two of them at once -/// on this box cost nothing worth serialising -- while refusing them is -/// actively harmful, because a small request then waits behind, or is -/// refused by, somebody else's minutes-long run. Above this, GEMMA's -/// kinship step is *quadratic in individuals* and writes hundreds of -/// megabytes per chromosome (measured: BXD's 235 individuals give a 660 KB -/// K matrix and a ~19 s run; UM-HET3's 6 382 give 207 MB per chromosome and -/// minutes), which is what must be one-at-a-time. +/// such: a 93-strain version or an exploratory subset finishes in seconds, +/// and two of them at once on this box cost nothing worth serialising -- +/// while refusing them is actively harmful, because a small request then +/// waits behind, or is refused by, somebody else's minutes-long run. A real +/// cross that is slow (3 281 individuals x 12 000 markers is minutes of +/// `scanone`) is well above this. +pub const RQTL_LARGE_PANEL: usize = 1000; + +/// Where GEMMA's panel size stops being small -- which is almost at once. /// -/// One threshold for every endpoint: it is the same machine, and the size -/// that makes R/qtl slow (3 281 individuals x 12 000 markers is minutes of -/// `scanone`) is in the same range. -pub const LARGE_PANEL: usize = 1000; +/// Much lower than R/qtl's, for two reasons that have nothing to do with +/// how long a small run takes: +/// +/// - **every GEMMA run goes through one shared cache directory** +/// (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts its per-chromosome +/// kinship archives by content hash. Two runs at once are two writers in +/// that directory, and the archives are hundreds of megabytes; a +/// half-extracted kinship that another run reads is a failure that looks +/// like a GEMMA bug; +/// - a real BXD WGS run is ~19 s of every core it can get (measured), and a +/// UM-HET3 one writes 207 MB of kinship per chromosome. Overlapping those +/// buys the caller little and costs the box the database is on. +/// +/// 50 rather than 0 so that a handful of individuals -- a four-way cross's +/// parents, a probe of the pipeline -- still runs without waiting. +pub const GEMMA_LARGE_PANEL: usize = 50; /// Whether a job over `individuals` individuals is large enough to be -/// gated. See [`LARGE_PANEL`]. -pub fn is_large(individuals: usize) -> bool { - individuals >= LARGE_PANEL +/// gated, for this gate's threshold. See [`RQTL_LARGE_PANEL`] and +/// [`GEMMA_LARGE_PANEL`]. +pub fn is_large(individuals: usize, threshold: usize) -> bool { + individuals >= threshold } /// Take a lock, recovering from poisoning: the guarded data is a `HashSet` @@ -129,7 +147,7 @@ impl Claim<'_> { } /// Take the endpoint's slot for a job over `individuals` individuals, - /// or stay light for a small one (see [`LARGE_PANEL`]). + /// or stay light for a small one (see [`GEMMA_LARGE_PANEL`]). /// /// This is the one gate call each compute handler makes, placed where /// the panel size is known and before the work starts. A large job may @@ -141,7 +159,7 @@ impl Claim<'_> { individuals: usize, rid: crate::response::RequestInfo, ) -> Result<(), crate::error::ApiError> { - if self.permit.is_some() || !is_large(individuals) { + if self.permit.is_some() || !self.gate.is_large(individuals) { return Ok(()); } self.claim_slot() @@ -263,6 +281,9 @@ struct Active { pub struct Gate { /// Remembered because the semaphore only reports what is left. permits: usize, + /// Panel size at which this endpoint's jobs take a slot (see + /// [`RQTL_LARGE_PANEL`], [`GEMMA_LARGE_PANEL`]). + large_panel: usize, slot: Semaphore, /// Subjects holding a slot *or waiting for one*. claimants: Mutex<HashSet<String>>, @@ -272,10 +293,12 @@ pub struct Gate { } impl Gate { - /// A gate with `permit` slots (one, for "one job at a time"). - fn new(permits: usize) -> Self { + /// A gate with `permit` slots (one, for "one job at a time") that + /// starts gating at `large_panel` individuals. + fn new(permits: usize, large_panel: usize) -> Self { Gate { permits, + large_panel, slot: Semaphore::new(permits), claimants: Mutex::new(HashSet::new()), running: Mutex::new(std::collections::HashMap::new()), @@ -287,6 +310,16 @@ impl Gate { self.permits } + /// The panel size at which this endpoint's jobs take a slot. + pub fn large_panel(&self) -> usize { + self.large_panel + } + + /// Whether a job over `individuals` individuals takes this gate's slot. + pub fn is_large(&self, individuals: usize) -> bool { + is_large(individuals, self.large_panel) + } + /// A claim that reserves nothing. /// /// This is what the `#[compute_gate(...)]` decorator makes when the @@ -395,7 +428,7 @@ pub fn rqtl() -> &'static Gate { static RQTL: OnceLock<Gate> = OnceLock::new(); RQTL.get_or_init(|| { let permits = crate::config::Config::load().rqtl_max_concurrent.max(1); - Gate::new(permits) + Gate::new(permits, RQTL_LARGE_PANEL) }) } @@ -412,13 +445,13 @@ pub fn rqtl() -> &'static Gate { /// conflict (they use different cache directories). pub fn ravanan() -> &'static Gate { static RAVANAN: OnceLock<Gate> = OnceLock::new(); - RAVANAN.get_or_init(|| Gate::new(1)) + RAVANAN.get_or_init(|| Gate::new(1, GEMMA_LARGE_PANEL)) } /// The GEMMA gate (`POST /compute/gemma`). pub fn gemma() -> &'static Gate { static GEMMA: OnceLock<Gate> = OnceLock::new(); - GEMMA.get_or_init(|| Gate::new(1)) + GEMMA.get_or_init(|| Gate::new(1, GEMMA_LARGE_PANEL)) } #[cfg(test)] @@ -427,11 +460,11 @@ mod tests { use std::time::Duration; fn leaked_gate() -> &'static Gate { - Box::leak(Box::new(Gate::new(1))) + Box::leak(Box::new(Gate::new(1, RQTL_LARGE_PANEL))) } fn leaked_gate_with(permits: usize) -> &'static Gate { - Box::leak(Box::new(Gate::new(permits))) + Box::leak(Box::new(Gate::new(permits, RQTL_LARGE_PANEL))) } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] @@ -447,32 +480,47 @@ mod tests { } #[test] - fn only_large_panels_are_gated() { - // The test fixtures' 198 individuals, a 93-strain version and a - // 235-strain WGS panel -- seconds of work, no reason to serialise - // them; UM-HET3's 6 382 and a 3 000-individual cross -- minutes and - // hundreds of megabytes per chromosome. - for small in [1, 34, 93, 198, 235, 999] { - assert!(!is_large(small), "{small} individuals is not a large job"); + fn each_endpoint_gates_at_its_own_panel_size() { + // R/qtl: a cross has to be genuinely big before it is serialised + // (3 281 individuals x 12 000 markers is minutes of scanone). + for small in [1, 93, 198, 235, 999] { + assert!( + !is_large(small, RQTL_LARGE_PANEL), + "{small} individuals is not a large R/qtl cross" + ); + } + for large in [RQTL_LARGE_PANEL, 3281, 6382, 24465] { + assert!(is_large(large, RQTL_LARGE_PANEL), "{large} is a large cross"); + } + // GEMMA: from a real panel's worth of individuals (see + // GEMMA_LARGE_PANEL -- the shared cache directory is the reason). + for small in [1, 34, 49] { + assert!(!is_large(small, GEMMA_LARGE_PANEL), "{small} still runs free"); } - for large in [LARGE_PANEL, 1000, 3281, 6382, 24465] { - assert!(is_large(large), "{large} individuals is a large job"); + for large in [GEMMA_LARGE_PANEL, 93, 198, 235, 6382] { + assert!(is_large(large, GEMMA_LARGE_PANEL), "{large} takes the slot"); } + // And the endpoints really are wired to those numbers. + assert_eq!(super::gemma().large_panel(), GEMMA_LARGE_PANEL); + assert_eq!(super::ravanan().large_panel(), GEMMA_LARGE_PANEL); + assert_eq!(super::rqtl().large_panel(), RQTL_LARGE_PANEL); } /// A light claim reserves nothing: a small job never waits and never /// refuses anybody, so two of them run at once even with one slot. #[tokio::test] async fn a_small_job_never_takes_the_slot() { - let gate = Gate::new(1); + // A gate that starts gating at 100 individuals, to keep the sizes + // here away from the endpoints' own thresholds. + let gate = Gate::new(1, 100); let mut first = gate.claim_light("alice", Some("a-1"), "compute", "job", "/status"); let mut second = gate.claim_light("alice", Some("a-2"), "compute", "job", "/status"); first - .claim_slot_or_busy(93, crate::response::RequestInfo::new(None)) + .claim_slot_or_busy(34, crate::response::RequestInfo::new(None)) .await .expect("a small job is never refused"); second - .claim_slot_or_busy(198, crate::response::RequestInfo::new(None)) + .claim_slot_or_busy(93, crate::response::RequestInfo::new(None)) .await .expect("nor is a second one from the same caller"); assert!(!first.is_held() && !second.is_held()); @@ -488,7 +536,7 @@ mod tests { /// refused with a 409 naming the first. #[tokio::test] async fn a_second_large_job_from_one_caller_is_a_409() { - let gate = Gate::new(1); + let gate = Gate::new(1, 100); let mut first = gate.claim_light("alice", Some("a-1"), "compute_gemma", "gemma", "/compute/gemma/status"); first .claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)) @@ -517,7 +565,7 @@ mod tests { /// Another caller's large job makes this one wait rather than fail. #[tokio::test] async fn a_large_job_waits_for_another_callers() { - let gate = Gate::new(1); + let gate = Gate::new(1, 100); let holder = gate.claim("alice", None).await.expect("alice's claim"); let mut bob = gate.claim_light("bob", None, "compute", "job", "/status"); // While alice holds the only slot, bob's promotion cannot finish -- diff --git a/src/routes/gemma.rs b/src/routes/gemma.rs index 04ad1bb..41b74ce 100644 --- a/src/routes/gemma.rs +++ b/src/routes/gemma.rs @@ -379,7 +379,7 @@ pub async fn compute_gemma( } // A large panel takes the one GEMMA slot here, where the size is known; // a small one runs alongside whatever else is running. See - // `compute_gate::LARGE_PANEL`. + // `compute_gate::GEMMA_LARGE_PANEL`. let panel_individuals = aligned_names.as_ref().map(Vec::len).unwrap_or(0); __gn4_claim .claim_slot_or_busy(panel_individuals, rid.clone()) diff --git a/src/routes/rqtl.rs b/src/routes/rqtl.rs index 76182ba..ee0b159 100644 --- a/src/routes/rqtl.rs +++ b/src/routes/rqtl.rs @@ -312,7 +312,7 @@ pub async fn compute_rqtl( // A large cross takes one of the R/qtl slots here, where the size is // known; a small one runs alongside whatever else is running. See - // `compute_gate::LARGE_PANEL`. + // `compute_gate::RQTL_LARGE_PANEL`. __gn4_claim .claim_slot_or_busy(samples.len(), rid.clone()) .await?; |
