diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 16:23:03 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 16:23:14 +0000 |
| commit | e934e755f14648fff6ebac07e9cf9e5b37cb3fe4 (patch) | |
| tree | b7b921fe7e83bed348d65f3ecda3cb284479cd19 /src | |
| parent | 0769a0ea0a8654e5e50e03365545c8ccf58d6b91 (diff) | |
| download | genenetwork4-e934e755f14648fff6ebac07e9cf9e5b37cb3fe4.tar.gz | |
api: describe case attributes, and gate large computations only
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
Diffstat (limited to 'src')
| -rw-r--r-- | src/compute_gate.rs | 365 | ||||
| -rw-r--r-- | src/covariates.rs | 37 | ||||
| -rw-r--r-- | src/db.rs | 72 | ||||
| -rw-r--r-- | src/routes/gemma.rs | 134 | ||||
| -rw-r--r-- | src/routes/groups.rs | 11 | ||||
| -rw-r--r-- | src/routes/rqtl.rs | 75 | ||||
| -rw-r--r-- | src/rqtl.rs | 1 |
7 files changed, 501 insertions, 194 deletions
diff --git a/src/compute_gate.rs b/src/compute_gate.rs index 376a7a6..c0a073e 100644 --- a/src/compute_gate.rs +++ b/src/compute_gate.rs @@ -35,6 +35,31 @@ use std::time::Instant; use tokio::sync::{Semaphore, SemaphorePermit}; +/// The panel size at which a job stops being small, and starts taking the +/// endpoint's slot. +/// +/// What the gate is for is **large computations**, not computation as +/// such: a reflex-sized job (the BXD panels' ~200 individuals, a 93-strain +/// version, the test fixtures) finishes in seconds, and two of them at once +/// on this box cost nothing worth serialising -- while refusing them is +/// actively harmful, because a small request then waits behind, or is +/// refused by, somebody else's minutes-long run. Above this, GEMMA's +/// kinship step is *quadratic in individuals* and writes hundreds of +/// megabytes per chromosome (measured: BXD's 235 individuals give a 660 KB +/// K matrix and a ~19 s run; UM-HET3's 6 382 give 207 MB per chromosome and +/// minutes), which is what must be one-at-a-time. +/// +/// One threshold for every endpoint: it is the same machine, and the size +/// that makes R/qtl slow (3 281 individuals x 12 000 markers is minutes of +/// `scanone`) is in the same range. +pub const LARGE_PANEL: usize = 1000; + +/// Whether a job over `individuals` individuals is large enough to be +/// gated. See [`LARGE_PANEL`]. +pub fn is_large(individuals: usize) -> bool { + individuals >= LARGE_PANEL +} + /// Take a lock, recovering from poisoning: the guarded data is a `HashSet` /// or an `Option`, both still valid after a panic elsewhere, and wedging /// the gate (or leaking an entry) is worse than trusting them. @@ -73,12 +98,102 @@ impl Drop for Reservation<'_> { } } -/// A claim on the slot. +/// A claim on the slot -- in two stages. +/// +/// The decorator makes a *light* claim as soon as the request arrives +/// ([`Gate::claim_light`]): cheap, and it reserves nothing, because at +/// that point nobody knows whether the job is large. Once the handler has +/// resolved the panel it calls [`Claim::claim_slot_or_busy`], which takes +/// the slot for a large job and does nothing at all for a small one. pub struct Claim<'a> { gate: &'a Gate, subject: String, - // Held for the claim's lifetime; dropping it lets the next waiter in. - _permit: SemaphorePermit<'a>, + /// The run id the request named, for the refusal's `next` link and for + /// a later refusal that names *this* job. + run_id: Option<String>, + /// Where the 409's `detail`, `operation_id` and `next` come from -- + /// carried here so a handler promotes its claim without repeating + /// them. + endpoint: &'static str, + label: &'static str, + status_prefix: &'static str, + /// Held for the claim's lifetime, once taken; dropping it lets the + /// next waiter in. `None` for a light claim and for a small job. + permit: Option<SemaphorePermit<'a>>, +} + +impl Claim<'_> { + /// Whether this claim is holding a slot. + pub fn is_held(&self) -> bool { + self.permit.is_some() + } + + /// Take the endpoint's slot for a job over `individuals` individuals, + /// or stay light for a small one (see [`LARGE_PANEL`]). + /// + /// This is the one gate call each compute handler makes, placed where + /// the panel size is known and before the work starts. A large job may + /// have to wait for another user's job to finish, and a *second* large + /// job from a caller who already holds one is refused here -- queuing + /// it would only make that caller wait twice. + pub async fn claim_slot_or_busy( + &mut self, + individuals: usize, + rid: crate::response::RequestInfo, + ) -> Result<(), crate::error::ApiError> { + if self.permit.is_some() || !is_large(individuals) { + return Ok(()); + } + self.claim_slot() + .await + .map_err(|busy| busy_error(busy, self.endpoint, self.label, self.status_prefix, rid)) + } + + /// The slot itself: register the subject, then wait for a permit. + async fn claim_slot(&mut self) -> Result<(), Busy> { + { + let mut claimants = lock_recover(&self.gate.claimants); + if !claimants.insert(self.subject.clone()) { + let running = lock_recover(&self.gate.running); + let active = running.get(&self.subject); + return Err(Busy { + run_id: active.and_then(|active| active.run_id.clone()), + elapsed: active + .map(|active| active.started.elapsed()) + .unwrap_or_default(), + }); + } + } + // From here to the permit, cancellation must undo the insertion. + let mut reservation = Reservation { + gate: self.gate, + subject: self.subject.clone(), + held: true, + }; + // Another subject's job may be running: this is where a different + // user waits (indefinitely -- the request timeout is the bound). + let permit = match self.gate.slot.acquire().await { + Ok(permit) => permit, + Err(_) => { + // The semaphore is never closed in this code; the + // reservation's drop undoes the insertion either way. + return Err(Busy { + run_id: None, + elapsed: std::time::Duration::ZERO, + }); + } + }; + reservation.hand_over(); + lock_recover(&self.gate.running).insert( + self.subject.clone(), + Active { + run_id: self.run_id.clone(), + started: Instant::now(), + }, + ); + self.permit = Some(permit); + Ok(()) + } } impl std::fmt::Debug for Claim<'_> { @@ -91,7 +206,12 @@ impl std::fmt::Debug for Claim<'_> { impl Drop for Claim<'_> { fn drop(&mut self) { - // Field drop order matters here: this runs before `_permit` is + // Only a *held* claim registered itself; a light one must not + // delete a registration that belongs to somebody's live job. + if self.permit.is_none() { + return; + } + // Field drop order matters here: this runs before `permit` is // released, so the next waiter cannot see a stale entry or a // stale `running` record. lock_recover(&self.gate.claimants).remove(&self.subject); @@ -167,102 +287,55 @@ impl Gate { self.permits } - /// Claim the slot for `subject`, waiting for another user's job to - /// finish if need be. + /// A claim that reserves nothing. /// - /// `None` when `subject` already holds a claim (running or queued) -- - /// the caller turns that into a `409`. - pub async fn claim(&self, subject: &str, run_id: Option<&str>) -> Result<Claim<'_>, Busy> { - { - let mut claimants = lock_recover(&self.claimants); - if !claimants.insert(subject.to_string()) { - let running = lock_recover(&self.running); - let active = running.get(subject); - return Err(Busy { - run_id: active.and_then(|active| active.run_id.clone()), - elapsed: active - .map(|active| active.started.elapsed()) - .unwrap_or_default(), - }); - } - } - // From here to the permit, cancellation must undo the insertion. - let mut reservation = Reservation { - gate: self, - subject: subject.to_string(), - held: true, - }; - - // Another subject's job may be running: this is where a different - // user waits (indefinitely -- the request timeout is the bound). - let permit = match self.slot.acquire().await { - Ok(permit) => permit, - Err(_) => { - // The semaphore is never closed in this code; the - // reservation's drop undoes the insertion either way. - return Err(Busy { - run_id: None, - elapsed: std::time::Duration::ZERO, - }); - } - }; - // The permit is ours: the Claim (not the reservation) now owns the - // entry's removal. - reservation.hand_over(); - - lock_recover(&self.running).insert( - subject.to_string(), - Active { - run_id: run_id.map(str::to_string), - started: Instant::now(), - }, - ); - Ok(Claim { + /// This is what the `#[compute_gate(...)]` decorator makes when the + /// request arrives: at that point the panel is not resolved, so the + /// gate cannot know whether the job is large. The handler promotes it + /// with [`Claim::claim_slot_or_busy`] where the size *is* known, and a + /// small job never touches the semaphore at all. + pub fn claim_light( + &self, + subject: &str, + run_id: Option<&str>, + endpoint: &'static str, + label: &'static str, + status_prefix: &'static str, + ) -> Claim<'_> { + Claim { gate: self, subject: subject.to_string(), - _permit: permit, - }) + run_id: run_id.map(str::to_string), + endpoint, + label, + status_prefix, + permit: None, + } } -} -impl Gate { - /// Claim, or the `409` a handler should return: the policy (status, - /// code, wording, and the `next` link to the job in the way) lives - /// here rather than in each endpoint, so it is covered by this - /// module's tests. + /// Take the slot for `subject` unconditionally, waiting for another + /// user's job to finish if need be. /// - /// `#[compute_gate(...)]` (see `gnauth-macros`) expands to a call to - /// this and [`Gate::run`], which is why an endpoint carries no gate - /// code of its own. - pub async fn claim_or_busy( - &self, - subject: &str, - run_id: Option<&str>, - rid: crate::response::RequestInfo, - endpoint: &str, - label: &str, - status_prefix: &str, - ) -> Result<Claim<'_>, crate::error::ApiError> { - self.claim(subject, run_id).await.map_err(|busy| { - let mut error = crate::error::ApiError::new( - axum::http::StatusCode::CONFLICT, - "compute_busy", - "A compute job is already running", - busy.describe(label), - endpoint, - ) - .with_request_id(rid); - if let Some(run_id) = busy.run_id.as_deref() { - error = error.with_next(&format!("{status_prefix}/{run_id}")); - } - error - }) + /// This is where [`Claim::claim_slot_or_busy`] ends up for a large job, + /// and what this module's tests use to exercise the policy directly. + /// `None` when `subject` already holds a claim (running or queued) -- + /// the caller turns that into a `409`. + pub async fn claim(&self, subject: &str, run_id: Option<&str>) -> Result<Claim<'_>, Busy> { + let mut claim = self.claim_light(subject, run_id, "compute", "job", "/compute/status"); + claim.claim_slot().await?; + Ok(claim) } +} +impl Gate { /// Run `job` in a task that owns `claim`, so the slot is held for as /// long as the work is -- not as long as the *request* is (a client /// disconnect does not kill the child process, see /// [`crate::streaming`]). + /// + /// This is the last thing a gated handler does with its claim: promote + /// it if the job is large ([`Claim::claim_slot_or_busy`]), then wrap the + /// call that actually computes. pub async fn run<T, F>(&self, claim: Claim<'static>, job: F) -> Result<T, crate::error::ApiError> where T: Send + 'static, @@ -284,6 +357,31 @@ impl Gate { } } +/// The `409` a handler should return for a [`Busy`] gate: the policy +/// (status, code, wording, and the `next` link to the job in the way) +/// lives here rather than in each endpoint, so it is covered by this +/// module's tests. +pub fn busy_error( + busy: Busy, + endpoint: &str, + label: &str, + status_prefix: &str, + rid: crate::response::RequestInfo, +) -> crate::error::ApiError { + let mut error = crate::error::ApiError::new( + axum::http::StatusCode::CONFLICT, + "compute_busy", + "A compute job is already running", + busy.describe(label), + endpoint, + ) + .with_request_id(rid); + if let Some(run_id) = busy.run_id.as_deref() { + error = error.with_next(&format!("{status_prefix}/{run_id}")); + } + error +} + /// The R/qtl gate (`POST /compute/rqtl`): `RQTL_MAX_CONCURRENT` slots /// (default 8), one job per caller. /// @@ -348,6 +446,97 @@ mod tests { assert!(gate.claim("alice", None).await.is_ok()); } + #[test] + fn only_large_panels_are_gated() { + // The test fixtures' 198 individuals, a 93-strain version and a + // 235-strain WGS panel -- seconds of work, no reason to serialise + // them; UM-HET3's 6 382 and a 3 000-individual cross -- minutes and + // hundreds of megabytes per chromosome. + for small in [1, 34, 93, 198, 235, 999] { + assert!(!is_large(small), "{small} individuals is not a large job"); + } + for large in [LARGE_PANEL, 1000, 3281, 6382, 24465] { + assert!(is_large(large), "{large} individuals is a large job"); + } + } + + /// A light claim reserves nothing: a small job never waits and never + /// refuses anybody, so two of them run at once even with one slot. + #[tokio::test] + async fn a_small_job_never_takes_the_slot() { + let gate = Gate::new(1); + let mut first = gate.claim_light("alice", Some("a-1"), "compute", "job", "/status"); + let mut second = gate.claim_light("alice", Some("a-2"), "compute", "job", "/status"); + first + .claim_slot_or_busy(93, crate::response::RequestInfo::new(None)) + .await + .expect("a small job is never refused"); + second + .claim_slot_or_busy(198, crate::response::RequestInfo::new(None)) + .await + .expect("nor is a second one from the same caller"); + assert!(!first.is_held() && !second.is_held()); + // Nothing was registered, so the slot is still free for a large job. + let large = gate + .claim("alice", Some("a-3")) + .await + .expect("the slot was never taken"); + assert!(large.is_held()); + } + + /// The same caller cannot hold two large jobs, and the second one is + /// refused with a 409 naming the first. + #[tokio::test] + async fn a_second_large_job_from_one_caller_is_a_409() { + let gate = Gate::new(1); + let mut first = gate.claim_light("alice", Some("a-1"), "compute_gemma", "gemma", "/compute/gemma/status"); + first + .claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)) + .await + .expect("first large job"); + assert!(first.is_held()); + let mut second = gate.claim_light("alice", Some("a-2"), "compute_gemma", "gemma", "/compute/gemma/status"); + let error = second + .claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)) + .await + .expect_err("the same caller cannot run two large jobs"); + assert_eq!(error.status, axum::http::StatusCode::CONFLICT); + assert_eq!(error.code, "compute_busy"); + assert_eq!( + error.next.as_deref(), + Some("/compute/gemma/status/a-1"), + "the refusal names the job in the way" + ); + drop(first); + second + .claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)) + .await + .expect("and the slot is free once the first job ends"); + } + + /// Another caller's large job makes this one wait rather than fail. + #[tokio::test] + async fn a_large_job_waits_for_another_callers() { + let gate = Gate::new(1); + let holder = gate.claim("alice", None).await.expect("alice's claim"); + let mut bob = gate.claim_light("bob", None, "compute", "job", "/status"); + // While alice holds the only slot, bob's promotion cannot finish -- + // and it is not refused either (no error, just no result). + let blocked = tokio::time::timeout( + Duration::from_millis(100), + bob.claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)), + ) + .await; + assert!(blocked.is_err(), "bob waits for alice's job, and is not refused"); + // Cancelling that wait leaves nothing behind (see the cancelled- + // waiter test), so the slot comes free to bob when alice ends. + drop(holder); + bob.claim_slot_or_busy(6382, crate::response::RequestInfo::new(None)) + .await + .expect("the slot is free once alice's job ends"); + assert!(bob.is_held(), "bob holds the slot while his job runs"); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn another_subject_waits_instead_of_being_refused() { let gate = leaked_gate(); diff --git a/src/covariates.rs b/src/covariates.rs index b269dc7..ed30af0 100644 --- a/src/covariates.rs +++ b/src/covariates.rs @@ -283,6 +283,15 @@ pub struct Spec { pub values: Option<Vec<NamedValue>>, } +/// A group's case-attribute columns for this request: the values aligned +/// onto the samples being mapped, and the descriptions that explain the +/// names. +#[derive(Default)] +struct AttributeColumns { + columns: HashMap<String, Vec<Option<String>>>, + descriptions: HashMap<String, String>, +} + /// One inline covariate value, by sample name. #[derive(Debug, Clone, serde::Deserialize, utoipa::ToSchema)] #[schema(as = CovariateValue)] @@ -333,8 +342,8 @@ pub async fn resolve( .filter_map(|spec| spec.case_attribute.as_deref()) .map(str::trim) .collect(); - let attribute_values = if wanted.is_empty() { - HashMap::new() + let attributes = if wanted.is_empty() { + AttributeColumns::default() } else { let Some(group) = group else { return Err(ApiError::new( @@ -402,7 +411,7 @@ pub async fn resolve( .iter() .map(|row| (row.strain.as_str(), &row.case_attributes)) .collect(); - let mut columns: HashMap<String, Vec<Option<String>>> = HashMap::new(); + let mut found = AttributeColumns::default(); for attribute in wanted { let column = names .iter() @@ -414,16 +423,21 @@ pub async fn resolve( .filter(|value| !value.is_empty()) }) .collect(); - columns.insert(attribute.to_string(), column); + if let Some(description) = attributes.descriptions.get(attribute) { + found + .descriptions + .insert(attribute.to_string(), description.clone()); + } + found.columns.insert(attribute.to_string(), column); } - columns + found }; for spec in specs { let (name, kind, values, coding, levels) = match (&spec.case_attribute, &spec.values) { (Some(attribute), None) => { let attribute = attribute.trim(); - let Some(column) = attribute_values.get(attribute) else { + let Some(column) = attributes.columns.get(attribute) else { return Err(ApiError::new( StatusCode::UNPROCESSABLE_ENTITY, "covariate_unavailable", @@ -565,8 +579,17 @@ pub async fn resolve( // Just the levels: how the runner turns them into columns is the // runner's business (GEMMA's own note names the dummies it was // given; R/qtl's wrapper expands the one column itself). + // The attribute's own description, if the database has one: the + // names are not self-explanatory (`SeqCvge` is a sequencing depth, + // not a sequence coverage *phenotype*), and a covariate whose + // meaning is guessed is a covariate fitted for the wrong reason. + let meaning = attributes + .descriptions + .get(&name) + .map(|description| format!(" ({description})")) + .unwrap_or_default(); notes.push(format!( - "Covariate {name}: {}{}, {present} of {} samples have a value{complete}{}", + "Covariate {name}{meaning}: {}{}, {present} of {} samples have a value{complete}{}", kind.as_str(), coding .as_deref() diff --git a/src/db.rs b/src/db.rs index 4919264..283f4bf 100644 --- a/src/db.rs +++ b/src/db.rs @@ -576,6 +576,15 @@ pub struct TraitCaseAttribute { pub value: Option<String>, } +/// One attribute of one trait: its values on that trait's samples, and +/// what the attribute means (`CaseAttribute.Description`, advertised +/// because the names are not self-explanatory -- `SeqCvge` is "Depth of +/// DNA sequencing for 1 male of each strain in 2016"). +pub struct TraitCaseAttributeLookup { + pub description: Option<String>, + pub values: Vec<TraitCaseAttribute>, +} + /// A trait's sample names, in the order its values come back. pub fn get_group_trait_samples(group_id: u16, trait_id: u32) -> Result<Vec<String>, DbError> { Ok(get_group_trait_values(group_id, trait_id)? @@ -593,17 +602,18 @@ pub fn get_trait_case_attribute( group_id: u16, trait_id: u32, attribute: &str, -) -> Result<Option<Vec<TraitCaseAttribute>>, DbError> { +) -> Result<Option<TraitCaseAttributeLookup>, DbError> { let pool = pool()?; let mut conn = pool.get_conn()?; - let defined: Option<String> = conn.exec_first( - "SELECT Name FROM CaseAttribute \ + let defined: Option<(String, String)> = conn.exec_first( + "SELECT Name, Description FROM CaseAttribute \ WHERE InbredSetId = :group_id AND Name = :attribute LIMIT 1", mysql::params! { "group_id" => group_id, "attribute" => attribute }, )?; - if defined.is_none() { + let Some((_, description)) = defined else { return Ok(None); - } + }; + let description = description.trim().to_string(); let samples = get_group_trait_samples(group_id, trait_id)?; // The attribute's values for the whole group, then aligned onto the // trait's samples: the case-attribute table is keyed by @@ -626,15 +636,16 @@ pub fn get_trait_case_attribute( (strain, value) }) .collect(); - Ok(Some( - samples + Ok(Some(TraitCaseAttributeLookup { + description: (!description.is_empty()).then_some(description), + values: samples .into_iter() .map(|name| { let value = by_name.get(&name).cloned().filter(|v| !v.is_empty()); TraitCaseAttribute { name, value } }) .collect(), - )) + })) } /// One published (phenotype) trait's metadata: its PublishXRef mapping @@ -985,26 +996,36 @@ pub struct StrainCaseAttributes { } /// A group's case attributes: every attribute name defined for it -/// (`CaseAttribute.Name`) and, for each strain that has at least one -/// value set, that strain's values -- combined into one response -/// rather than GN3's separate .../names and .../values endpoints, -/// since neither list is large enough on its own to need splitting. +/// (`CaseAttribute.Name`), what each one means (`CaseAttribute.Description`), +/// and, for each strain that has at least one value set, that strain's +/// values -- combined into one response rather than GN3's separate +/// .../names and .../values endpoints, since neither list is large enough +/// on its own to need splitting. The descriptions are what make the names +/// usable: `SeqCvge` is unguessable, and its description -- "Depth of DNA +/// sequencing for 1 male of each strain in 2016" -- says both what it is +/// and why it is not a phenotype. #[derive(Serialize, utoipa::ToSchema)] pub struct CaseAttributes { pub names: Vec<String>, + /// Attribute name -> description, for the names that have one. A + /// description can be empty in the database; those are left out + /// rather than served as "". + pub descriptions: std::collections::BTreeMap<String, String>, pub values: Vec<StrainCaseAttributes>, } -/// Fetch every case-attribute name defined for a group, matching -/// GeneNetwork3's gn3.api.case_attributes.__case_attribute_labels_by_inbred_set__ -/// (trimmed to just the name -- Description isn't used by genenetwork4 -/// yet, and CaseAttributeId/InbredSetId are already implied by this -/// being scoped to one group). -fn get_group_case_attribute_names(group_id: u16) -> Result<Vec<String>, DbError> { +/// Fetch every case-attribute name and description defined for a group, +/// matching GeneNetwork3's +/// gn3.api.case_attributes.__case_attribute_labels_by_inbred_set__ (which +/// selects the name only; `CaseAttribute.Description` is what genenetwork4 +/// adds -- CaseAttributeId/InbredSetId are already implied by this being +/// scoped to one group). +fn get_group_case_attribute_definitions(group_id: u16) -> Result<Vec<(String, String)>, DbError> { let pool = pool()?; let mut conn = pool.get_conn()?; conn.exec( - "SELECT Name FROM CaseAttribute WHERE InbredSetId = :group_id ORDER BY Name", + "SELECT Name, Description FROM CaseAttribute \ + WHERE InbredSetId = :group_id ORDER BY Name", mysql::params! { "group_id" => group_id }, ) .map_err(DbError::from) @@ -1063,8 +1084,19 @@ fn get_group_case_attribute_values(group_id: u16) -> Result<Vec<StrainCaseAttrib /// Fetch a group's case attributes: every attribute name defined for /// it, and every strain's values for them. See `CaseAttributes`. pub fn get_group_case_attributes(group_id: u16) -> Result<CaseAttributes, DbError> { + let definitions = get_group_case_attribute_definitions(group_id)?; + let mut names = Vec::with_capacity(definitions.len()); + let mut descriptions = std::collections::BTreeMap::new(); + for (name, description) in definitions { + let description = description.trim().to_string(); + if !description.is_empty() { + descriptions.insert(name.clone(), description); + } + names.push(name); + } Ok(CaseAttributes { - names: get_group_case_attribute_names(group_id)?, + names, + descriptions, values: get_group_case_attribute_values(group_id)?, }) } diff --git a/src/routes/gemma.rs b/src/routes/gemma.rs index 66caf57..04ad1bb 100644 --- a/src/routes/gemma.rs +++ b/src/routes/gemma.rs @@ -18,6 +18,7 @@ use gnauth_macros::{compute_gate, gnauth}; use crate::auth::AuthToken; use crate::config::Config; +use crate::compute_gate; use crate::covariates; use crate::error::{ApiError, Unauthorized}; use crate::genotypes; @@ -376,6 +377,14 @@ pub async fn compute_gemma( ) .with_request_id(rid)); } + // A large panel takes the one GEMMA slot here, where the size is known; + // a small one runs alongside whatever else is running. See + // `compute_gate::LARGE_PANEL`. + let panel_individuals = aligned_names.as_ref().map(Vec::len).unwrap_or(0); + __gn4_claim + .claim_slot_or_busy(panel_individuals, rid.clone()) + .await?; + // Covariates are per individual, so they need the panel's names: the // same individuals, in the same order, as the phenotype file above. // Resolved here rather than beside the panel because the note it @@ -392,17 +401,38 @@ pub async fn compute_gemma( ) .await?; } - gemma::compute( - &config.gemma_wrapper_cmd, - &config.gemma_cache_dir, - &genotype_file, - &snps_file, - &trait_values, - &covariate_list, - lmm, - params.id.as_deref(), - ) - .await + let wrapper = config.gemma_wrapper_cmd.clone(); + let cache_dir = config.gemma_cache_dir.clone(); + let run_id = params.id.clone(); + let rid_for_job = rid.clone(); + let trait_values_for_job = trait_values.clone(); + let covariates_for_job = covariate_list.clone(); + compute_gate::gemma() + .run(__gn4_claim, async move { + gemma::compute( + &wrapper, + &cache_dir, + &genotype_file, + &snps_file, + &trait_values_for_job, + &covariates_for_job, + lmm, + run_id.as_deref(), + ) + .await + .map_err(|e| match e { + GemmaError::RunIdInUse(id) => ApiError::new( + StatusCode::CONFLICT, + "run_id_in_use", + "Run id already in use", + format!("id '{id}' is already in use; pick a different id."), + "compute_gemma", + ) + .with_request_id(rid_for_job.clone()), + other => gemma_error_to_api_error(other, rid_for_job.clone()), + }) + }) + .await .map(|output| { let num_rows = output.rows.as_array().map(Vec::len).unwrap_or(0); let mut envelope = ctx @@ -487,17 +517,6 @@ pub async fn compute_gemma( } Json(envelope) }) - .map_err(|e| match e { - GemmaError::RunIdInUse(id) => ApiError::new( - StatusCode::CONFLICT, - "run_id_in_use", - "Run id already in use", - format!("id '{id}' is already in use; pick a different id."), - "compute_gemma", - ) - .with_request_id(rid.clone()), - other => gemma_error_to_api_error(other, rid.clone()), - }) } #[utoipa::path(post, path = "/compute/gemma/ravanan", @@ -560,25 +579,49 @@ pub async fn compute_gemma_ravanan( } } + // ravanan is gated like the direct GEMMA run, and by the same size + // rule: an explicit genotype_file request carries no panel, so it is + // sized by the trait values it was given. + let trait_values = request.trait_values.clone().unwrap_or_default(); + __gn4_claim + .claim_slot_or_busy(trait_values.len(), rid.clone()) + .await?; + let config = Config::load(); - gemma::compute_ravanan( - &config.ravanan_cmd, - &config.ravanan_batch, - &config.ccwl_cmd, - &config.gemma_ravanan_workflow, - &config.gemma_ravanan_manifest, - &config.ravanan_cache_dir, - &config.ravanan_scratch, - request - .genotype_file - .as_deref() - .unwrap_or_default(), - request.snps_file.as_deref().unwrap_or_default(), - &request.trait_values.clone().unwrap_or_default(), - lmm, - params.id.as_deref(), - ) - .await + let genotype_file = request.genotype_file.clone().unwrap_or_default(); + let snps_file = request.snps_file.clone().unwrap_or_default(); + let run_id = params.id.clone(); + let job_rid = rid.clone(); + compute_gate::ravanan() + .run(__gn4_claim, async move { + gemma::compute_ravanan( + &config.ravanan_cmd, + &config.ravanan_batch, + &config.ccwl_cmd, + &config.gemma_ravanan_workflow, + &config.gemma_ravanan_manifest, + &config.ravanan_cache_dir, + &config.ravanan_scratch, + &genotype_file, + &snps_file, + &trait_values, + lmm, + run_id.as_deref(), + ) + .await + .map_err(|e| match e { + GemmaError::RunIdInUse(id) => ApiError::new( + StatusCode::CONFLICT, + "run_id_in_use", + "Run id already in use", + format!("id '{id}' is already in use; pick a different id."), + "compute_gemma_ravanan", + ) + .with_request_id(job_rid.clone()), + other => gemma_error_to_api_error(other, job_rid.clone()), + }) + }) + .await .map(|output| { let num_rows = output.rows.as_array().map(Vec::len).unwrap_or(0); let mut envelope = ctx @@ -592,17 +635,6 @@ pub async fn compute_gemma_ravanan( } Json(envelope) }) - .map_err(|e| match e { - GemmaError::RunIdInUse(id) => ApiError::new( - StatusCode::CONFLICT, - "run_id_in_use", - "Run id already in use", - format!("id '{id}' is already in use; pick a different id."), - "compute_gemma_ravanan", - ) - .with_request_id(rid.clone()), - other => gemma_error_to_api_error(other, rid.clone()), - }) } #[utoipa::path(get, path = "/compute/gemma/status/{run_id}", diff --git a/src/routes/groups.rs b/src/routes/groups.rs index b8fdc5a..45c4b13 100644 --- a/src/routes/groups.rs +++ b/src/routes/groups.rs @@ -705,6 +705,11 @@ pub struct TraitCaseAttributeValues { pub group: String, pub trait_id: u32, pub attribute: String, + /// What the attribute means, as the database records it (`null` when + /// it has no description). Read this before interpreting a name like + /// `SeqCvge` or `Epoch`. + #[serde(skip_serializing_if = "Option::is_none")] + pub description: Option<String>, /// One entry per sample of the trait, in the trait's own order; /// `value` is `null` where the sample has no value for the attribute. pub values: Vec<TraitCaseAttributeRow>, @@ -761,7 +766,7 @@ pub async fn group_trait_case_attribute( return Ok(None); }; db::get_trait_case_attribute(group_row.id, trait_id, &attribute_for_query) - .map(|values| values.map(|values| (group_row.name, values))) + .map(|found| found.map(|found| (group_row.name, found))) }) .await .map_err(|e| { @@ -776,7 +781,8 @@ pub async fn group_trait_case_attribute( })? .map_err(|e| ApiError::from_db(e, "group_trait_case_attribute").with_request_id(rid.clone()))? .map(|found| { - let (group_name, values) = found; + let (group_name, found) = found; + let values = found.values; let mut counts: std::collections::HashMap<String, usize> = std::collections::HashMap::new(); for row in &values { if let Some(value) = &row.value { @@ -793,6 +799,7 @@ pub async fn group_trait_case_attribute( group: group_name, trait_id, attribute, + description: found.description, values: values .into_iter() .map(|row| TraitCaseAttributeRow { diff --git a/src/routes/rqtl.rs b/src/routes/rqtl.rs index 3b82977..76182ba 100644 --- a/src/routes/rqtl.rs +++ b/src/routes/rqtl.rs @@ -15,6 +15,7 @@ use gnauth_macros::{compute_gate, gnauth}; use crate::auth::AuthToken; use crate::config::Config; +use crate::compute_gate; use crate::covariates; use crate::error::{ApiError, Unauthorized}; use crate::genotypes; @@ -309,21 +310,54 @@ pub async fn compute_rqtl( ) .await?; - rqtl::compute( - &config.rqtl_wrapper_path, - &config.rqtl_cache_dir, - &genotype_file, - &request.trait_name, - &samples, - &covariate_list, - request.model.as_deref().unwrap_or("normal"), - request.method.as_deref().unwrap_or("hk"), - request.nperm.unwrap_or(0), - &scale, - request.control.as_deref(), - params.id.as_deref(), - ) - .await + // A large cross takes one of the R/qtl slots here, where the size is + // known; a small one runs alongside whatever else is running. See + // `compute_gate::LARGE_PANEL`. + __gn4_claim + .claim_slot_or_busy(samples.len(), rid.clone()) + .await?; + + let wrapper_path = config.rqtl_wrapper_path.clone(); + let cache_dir = config.rqtl_cache_dir.clone(); + let trait_name = request.trait_name.clone(); + let samples_for_job = samples.clone(); + let covariates_for_job = covariate_list.clone(); + let model = request.model.clone().unwrap_or_else(|| "normal".to_string()); + let method = request.method.clone().unwrap_or_else(|| "hk".to_string()); + let nperm = request.nperm.unwrap_or(0); + let control = request.control.clone(); + let run_id = params.id.clone(); + let rid_for_job = rid.clone(); + compute_gate::rqtl() + .run(__gn4_claim, async move { + rqtl::compute( + &wrapper_path, + &cache_dir, + &genotype_file, + &trait_name, + &samples_for_job, + &covariates_for_job, + &model, + &method, + nperm, + &scale, + control.as_deref(), + run_id.as_deref(), + ) + .await + .map_err(|e| match e { + RqtlError::RunIdInUse(id) => ApiError::new( + StatusCode::CONFLICT, + "run_id_in_use", + "Run id already in use", + format!("id '{id}' is already in use; pick a different id."), + "compute_rqtl", + ) + .with_request_id(rid_for_job.clone()), + other => rqtl_error_to_api_error(other, rid_for_job.clone()), + }) + }) + .await .map(|output| { let notes = output.notes.clone(); let num_rows = output.markers.len(); @@ -414,17 +448,6 @@ pub async fn compute_rqtl( } Json(envelope) }) - .map_err(|e| match e { - RqtlError::RunIdInUse(id) => ApiError::new( - StatusCode::CONFLICT, - "run_id_in_use", - "Run id already in use", - format!("id '{id}' is already in use; pick a different id."), - "compute_rqtl", - ) - .with_request_id(rid.clone()), - other => rqtl_error_to_api_error(other, rid.clone()), - }) } #[utoipa::path(get, path = "/compute/rqtl/status/{run_id}", diff --git a/src/rqtl.rs b/src/rqtl.rs index 7267870..8724a61 100644 --- a/src/rqtl.rs +++ b/src/rqtl.rs @@ -90,6 +90,7 @@ impl From<RunLogError> for RqtlError { /// One sample's name and trait value (`None` for missing -- written as /// the literal string `x`, the R script's own `na.strings`, not `NA` /// like `crate::gemma`'s pheno file convention). +#[derive(Debug, Clone)] pub struct Sample { pub name: String, pub value: Option<f64>, |
