about summary refs log tree commit diff
path: root/src
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 16:32:59 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 16:32:59 +0000
commitab5fef05237309667dd2ca7f835a5fbf9c71a12a (patch)
tree94267f7fa6c5f27fe6d5093d1280c48170ffe164 /src
parente934e755f14648fff6ebac07e9cf9e5b37cb3fe4 (diff)
downloadgenenetwork4-ab5fef05237309667dd2ca7f835a5fbf9c71a12a.tar.gz
compute: gate GEMMA from 50 individuals, R/qtl from 1 000
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:

- every GEMMA run goes through **one shared cache directory**
  (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
  kinship archives of hundreds of megabytes by content hash. Two runs at
  once are two writers over the same archives, and a half-extracted kinship
  another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
  individuals write 207 MB of kinship per chromosome.

So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.

Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.

* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
  (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
  the test that pinned one number now pins both and that each endpoint is
  wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
  the two thresholds and the cache-directory reason for the lower one.

277 tests pass; the same four as before do not.
Diffstat (limited to 'src')
-rw-r--r--src/compute_gate.rs130
-rw-r--r--src/routes/gemma.rs2
-rw-r--r--src/routes/rqtl.rs2
3 files changed, 91 insertions, 43 deletions
diff --git a/src/compute_gate.rs b/src/compute_gate.rs
index c0a073e..c3ad1eb 100644
--- a/src/compute_gate.rs
+++ b/src/compute_gate.rs
@@ -9,6 +9,11 @@
 //!   than being refused. One slot for GEMMA (`gemma`, because two runs
 //!   would fight over the same cache and the same cores), eight for R/qtl
 //!   (`rqtl`, `RQTL_MAX_CONCURRENT`);
+//! - **and only *large* jobs take a slot.** A panel below the endpoint's
+//!   threshold ([`GEMMA_LARGE_PANEL`], [`RQTL_LARGE_PANEL`]) runs
+//!   alongside everything else: the gate is for protecting the machine,
+//!   and a two-second request that waits behind a ten-minute one is worse
+//!   off than the machine would be;
 //! - **one job per subject**: a second submission from somebody who
 //!   already holds a slot *or is queued for one* is refused immediately
 //!   (`409 compute_busy`), because queueing it would only make them wait
@@ -35,29 +40,42 @@ use std::time::Instant;
 
 use tokio::sync::{Semaphore, SemaphorePermit};
 
-/// The panel size at which a job stops being small, and starts taking the
-/// endpoint's slot.
+/// Where R/qtl's panel size stops being small, and starts taking one of
+/// its slots.
 ///
 /// What the gate is for is **large computations**, not computation as
-/// such: a reflex-sized job (the BXD panels' ~200 individuals, a 93-strain
-/// version, the test fixtures) finishes in seconds, and two of them at once
-/// on this box cost nothing worth serialising -- while refusing them is
-/// actively harmful, because a small request then waits behind, or is
-/// refused by, somebody else's minutes-long run. Above this, GEMMA's
-/// kinship step is *quadratic in individuals* and writes hundreds of
-/// megabytes per chromosome (measured: BXD's 235 individuals give a 660 KB
-/// K matrix and a ~19 s run; UM-HET3's 6 382 give 207 MB per chromosome and
-/// minutes), which is what must be one-at-a-time.
+/// such: a 93-strain version or an exploratory subset finishes in seconds,
+/// and two of them at once on this box cost nothing worth serialising --
+/// while refusing them is actively harmful, because a small request then
+/// waits behind, or is refused by, somebody else's minutes-long run. A real
+/// cross that is slow (3 281 individuals x 12 000 markers is minutes of
+/// `scanone`) is well above this.
+pub const RQTL_LARGE_PANEL: usize = 1000;
+
+/// Where GEMMA's panel size stops being small -- which is almost at once.
 ///
-/// One threshold for every endpoint: it is the same machine, and the size
-/// that makes R/qtl slow (3 281 individuals x 12 000 markers is minutes of
-/// `scanone`) is in the same range.
-pub const LARGE_PANEL: usize = 1000;
+/// Much lower than R/qtl's, for two reasons that have nothing to do with
+/// how long a small run takes:
+///
+/// - **every GEMMA run goes through one shared cache directory**
+///   (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts its per-chromosome
+///   kinship archives by content hash. Two runs at once are two writers in
+///   that directory, and the archives are hundreds of megabytes; a
+///   half-extracted kinship that another run reads is a failure that looks
+///   like a GEMMA bug;
+/// - a real BXD WGS run is ~19 s of every core it can get (measured), and a
+///   UM-HET3 one writes 207 MB of kinship per chromosome. Overlapping those
+///   buys the caller little and costs the box the database is on.
+///
+/// 50 rather than 0 so that a handful of individuals -- a four-way cross's
+/// parents, a probe of the pipeline -- still runs without waiting.
+pub const GEMMA_LARGE_PANEL: usize = 50;
 
 /// Whether a job over `individuals` individuals is large enough to be
-/// gated. See [`LARGE_PANEL`].
-pub fn is_large(individuals: usize) -> bool {
-    individuals >= LARGE_PANEL
+/// gated, for this gate's threshold. See [`RQTL_LARGE_PANEL`] and
+/// [`GEMMA_LARGE_PANEL`].
+pub fn is_large(individuals: usize, threshold: usize) -> bool {
+    individuals >= threshold
 }
 
 /// Take a lock, recovering from poisoning: the guarded data is a `HashSet`
@@ -129,7 +147,7 @@ impl Claim<'_> {
     }
 
     /// Take the endpoint's slot for a job over `individuals` individuals,
-    /// or stay light for a small one (see [`LARGE_PANEL`]).
+    /// or stay light for a small one (see [`GEMMA_LARGE_PANEL`]).
     ///
     /// This is the one gate call each compute handler makes, placed where
     /// the panel size is known and before the work starts. A large job may
@@ -141,7 +159,7 @@ impl Claim<'_> {
         individuals: usize,
         rid: crate::response::RequestInfo,
     ) -> Result<(), crate::error::ApiError> {
-        if self.permit.is_some() || !is_large(individuals) {
+        if self.permit.is_some() || !self.gate.is_large(individuals) {
             return Ok(());
         }
         self.claim_slot()
@@ -263,6 +281,9 @@ struct Active {
 pub struct Gate {
     /// Remembered because the semaphore only reports what is left.
     permits: usize,
+    /// Panel size at which this endpoint's jobs take a slot (see
+    /// [`RQTL_LARGE_PANEL`], [`GEMMA_LARGE_PANEL`]).
+    large_panel: usize,
     slot: Semaphore,
     /// Subjects holding a slot *or waiting for one*.
     claimants: Mutex<HashSet<String>>,
@@ -272,10 +293,12 @@ pub struct Gate {
 }
 
 impl Gate {
-    /// A gate with `permit` slots (one, for "one job at a time").
-    fn new(permits: usize) -> Self {
+    /// A gate with `permit` slots (one, for "one job at a time") that
+    /// starts gating at `large_panel` individuals.
+    fn new(permits: usize, large_panel: usize) -> Self {
         Gate {
             permits,
+            large_panel,
             slot: Semaphore::new(permits),
             claimants: Mutex::new(HashSet::new()),
             running: Mutex::new(std::collections::HashMap::new()),
@@ -287,6 +310,16 @@ impl Gate {
         self.permits
     }
 
+    /// The panel size at which this endpoint's jobs take a slot.
+    pub fn large_panel(&self) -> usize {
+        self.large_panel
+    }
+
+    /// Whether a job over `individuals` individuals takes this gate's slot.
+    pub fn is_large(&self, individuals: usize) -> bool {
+        is_large(individuals, self.large_panel)
+    }
+
     /// A claim that reserves nothing.
     ///
     /// This is what the `#[compute_gate(...)]` decorator makes when the
@@ -395,7 +428,7 @@ pub fn rqtl() -> &'static Gate {
     static RQTL: OnceLock<Gate> = OnceLock::new();
     RQTL.get_or_init(|| {
         let permits = crate::config::Config::load().rqtl_max_concurrent.max(1);
-        Gate::new(permits)
+        Gate::new(permits, RQTL_LARGE_PANEL)
     })
 }
 
@@ -412,13 +445,13 @@ pub fn rqtl() -> &'static Gate {
 /// conflict (they use different cache directories).
 pub fn ravanan() -> &'static Gate {
     static RAVANAN: OnceLock<Gate> = OnceLock::new();
-    RAVANAN.get_or_init(|| Gate::new(1))
+    RAVANAN.get_or_init(|| Gate::new(1, GEMMA_LARGE_PANEL))
 }
 
 /// The GEMMA gate (`POST /compute/gemma`).
 pub fn gemma() -> &'static Gate {
     static GEMMA: OnceLock<Gate> = OnceLock::new();
-    GEMMA.get_or_init(|| Gate::new(1))
+    GEMMA.get_or_init(|| Gate::new(1, GEMMA_LARGE_PANEL))
 }
 
 #[cfg(test)]
@@ -427,11 +460,11 @@ mod tests {
     use std::time::Duration;
 
     fn leaked_gate() -> &'static Gate {
-        Box::leak(Box::new(Gate::new(1)))
+        Box::leak(Box::new(Gate::new(1, RQTL_LARGE_PANEL)))
     }
 
     fn leaked_gate_with(permits: usize) -> &'static Gate {
-        Box::leak(Box::new(Gate::new(permits)))
+        Box::leak(Box::new(Gate::new(permits, RQTL_LARGE_PANEL)))
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
@@ -447,32 +480,47 @@ mod tests {
     }
 
     #[test]
-    fn only_large_panels_are_gated() {
-        // The test fixtures' 198 individuals, a 93-strain version and a
-        // 235-strain WGS panel -- seconds of work, no reason to serialise
-        // them; UM-HET3's 6 382 and a 3 000-individual cross -- minutes and
-        // hundreds of megabytes per chromosome.
-        for small in [1, 34, 93, 198, 235, 999] {
-            assert!(!is_large(small), "{small} individuals is not a large job");
+    fn each_endpoint_gates_at_its_own_panel_size() {
+        // R/qtl: a cross has to be genuinely big before it is serialised
+        // (3 281 individuals x 12 000 markers is minutes of scanone).
+        for small in [1, 93, 198, 235, 999] {
+            assert!(
+                !is_large(small, RQTL_LARGE_PANEL),
+                "{small} individuals is not a large R/qtl cross"
+            );
+        }
+        for large in [RQTL_LARGE_PANEL, 3281, 6382, 24465] {
+            assert!(is_large(large, RQTL_LARGE_PANEL), "{large} is a large cross");
+        }
+        // GEMMA: from a real panel's worth of individuals (see
+        // GEMMA_LARGE_PANEL -- the shared cache directory is the reason).
+        for small in [1, 34, 49] {
+            assert!(!is_large(small, GEMMA_LARGE_PANEL), "{small} still runs free");
         }
-        for large in [LARGE_PANEL, 1000, 3281, 6382, 24465] {
-            assert!(is_large(large), "{large} individuals is a large job");
+        for large in [GEMMA_LARGE_PANEL, 93, 198, 235, 6382] {
+            assert!(is_large(large, GEMMA_LARGE_PANEL), "{large} takes the slot");
         }
+        // And the endpoints really are wired to those numbers.
+        assert_eq!(super::gemma().large_panel(), GEMMA_LARGE_PANEL);
+        assert_eq!(super::ravanan().large_panel(), GEMMA_LARGE_PANEL);
+        assert_eq!(super::rqtl().large_panel(), RQTL_LARGE_PANEL);
     }
 
     /// A light claim reserves nothing: a small job never waits and never
     /// refuses anybody, so two of them run at once even with one slot.
     #[tokio::test]
     async fn a_small_job_never_takes_the_slot() {
-        let gate = Gate::new(1);
+        // A gate that starts gating at 100 individuals, to keep the sizes
+        // here away from the endpoints' own thresholds.
+        let gate = Gate::new(1, 100);
         let mut first = gate.claim_light("alice", Some("a-1"), "compute", "job", "/status");
         let mut second = gate.claim_light("alice", Some("a-2"), "compute", "job", "/status");
         first
-            .claim_slot_or_busy(93, crate::response::RequestInfo::new(None))
+            .claim_slot_or_busy(34, crate::response::RequestInfo::new(None))
             .await
             .expect("a small job is never refused");
         second
-            .claim_slot_or_busy(198, crate::response::RequestInfo::new(None))
+            .claim_slot_or_busy(93, crate::response::RequestInfo::new(None))
             .await
             .expect("nor is a second one from the same caller");
         assert!(!first.is_held() && !second.is_held());
@@ -488,7 +536,7 @@ mod tests {
     /// refused with a 409 naming the first.
     #[tokio::test]
     async fn a_second_large_job_from_one_caller_is_a_409() {
-        let gate = Gate::new(1);
+        let gate = Gate::new(1, 100);
         let mut first = gate.claim_light("alice", Some("a-1"), "compute_gemma", "gemma", "/compute/gemma/status");
         first
             .claim_slot_or_busy(6382, crate::response::RequestInfo::new(None))
@@ -517,7 +565,7 @@ mod tests {
     /// Another caller's large job makes this one wait rather than fail.
     #[tokio::test]
     async fn a_large_job_waits_for_another_callers() {
-        let gate = Gate::new(1);
+        let gate = Gate::new(1, 100);
         let holder = gate.claim("alice", None).await.expect("alice's claim");
         let mut bob = gate.claim_light("bob", None, "compute", "job", "/status");
         // While alice holds the only slot, bob's promotion cannot finish --
diff --git a/src/routes/gemma.rs b/src/routes/gemma.rs
index 04ad1bb..41b74ce 100644
--- a/src/routes/gemma.rs
+++ b/src/routes/gemma.rs
@@ -379,7 +379,7 @@ pub async fn compute_gemma(
     }
     // A large panel takes the one GEMMA slot here, where the size is known;
     // a small one runs alongside whatever else is running. See
-    // `compute_gate::LARGE_PANEL`.
+    // `compute_gate::GEMMA_LARGE_PANEL`.
     let panel_individuals = aligned_names.as_ref().map(Vec::len).unwrap_or(0);
     __gn4_claim
         .claim_slot_or_busy(panel_individuals, rid.clone())
diff --git a/src/routes/rqtl.rs b/src/routes/rqtl.rs
index 76182ba..ee0b159 100644
--- a/src/routes/rqtl.rs
+++ b/src/routes/rqtl.rs
@@ -312,7 +312,7 @@ pub async fn compute_rqtl(
 
     // A large cross takes one of the R/qtl slots here, where the size is
     // known; a small one runs alongside whatever else is running. See
-    // `compute_gate::LARGE_PANEL`.
+    // `compute_gate::RQTL_LARGE_PANEL`.
     __gn4_claim
         .claim_slot_or_busy(samples.len(), rid.clone())
         .await?;