diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 15:39:12 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 15:39:12 +0000 |
| commit | 336f18e1bc22cb330052dc1ff45d1d06073bfcc3 (patch) | |
| tree | 4c029b46500edab067b35424b9e9ebbc7156221a /src | |
| parent | fb41143226f7e0231f863a8226ba077c7415b71f (diff) | |
| download | genenetwork4-336f18e1bc22cb330052dc1ff45d1d06073bfcc3.tar.gz | |
compute: fit covariates in GEMMA, and share them with R/qtl
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
Diffstat (limited to 'src')
| -rw-r--r-- | src/covariates.rs | 938 | ||||
| -rw-r--r-- | src/gemma.rs | 65 | ||||
| -rw-r--r-- | src/lib.rs | 1 | ||||
| -rw-r--r-- | src/routes/gemma.rs | 61 | ||||
| -rw-r--r-- | src/routes/rqtl.rs | 305 | ||||
| -rw-r--r-- | src/rqtl.rs | 242 |
6 files changed, 1089 insertions, 523 deletions
diff --git a/src/covariates.rs b/src/covariates.rs new file mode 100644 index 0000000..de025a6 --- /dev/null +++ b/src/covariates.rs @@ -0,0 +1,938 @@ +//! Covariates, shared by the two engines that can fit them. +//! +//! A covariate is a per-individual number fitted alongside the trait. The +//! two runners take them differently, which is why this module exists: +//! +//! - R/qtl's wrapper wants extra **phenotype-file columns** plus a +//! `--covarstruct` file saying which are categorical, and expands more +//! than two levels itself (`write_covarstruct_file`); +//! - GEMMA wants a separate **covariate file**, positional over the +//! genotype panel, numeric only, and knows nothing about factors -- so a +//! categorical covariate has to be dummy-expanded here +//! (`write_gemma_covariates_file`). +//! +//! Both get the same inputs: the caller names a case attribute of the +//! group (`{"case_attribute": "Sex"}`), sends named values, or maps text +//! explicitly, and [`resolve`] aligns whatever arrives onto the samples +//! being mapped and codes the text (see [`code_case_attribute`]). A +//! missing value is `None`, which each runner turns into its own missing +//! marker -- and both then drop that individual, per the measured +//! behaviour of the engines themselves (R/qtl: `na.strings="x"` in the +//! wrapper; GEMMA: a literal `NA` in the covariate file, after which its +//! log says "number of analyzed individuals" one lower). +//! +//! Covariates are never inferred from database text except through +//! [`code_case_attribute`]'s explicit rules: 2-20 distinct values are +//! coded in sorted order and the coding is reported in the response's +//! `notes`, so two runs of "the same" covariate can be compared. + +use std::collections::{BTreeMap, HashMap}; +use std::io; +use std::path::Path; + +use axum::http::StatusCode; + +use crate::db; +use crate::error::ApiError; +use crate::response::RequestInfo; + +/// Whether a covariate is a factor or a number. R/qtl reads this from the +/// generated `--covarstruct` file; GEMMA has no factors, so it decides how +/// many columns [`write_gemma_covariates_file`] writes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CovariateKind { + /// Two or more levels: R/qtl's wrapper dummy-expands more than two + /// itself; GEMMA always gets k-1 dummies. + Categorical, + /// A number, used as-is. + Numerical, +} + +impl CovariateKind { + pub fn as_str(self) -> &'static str { + match self { + CovariateKind::Categorical => "categorical", + CovariateKind::Numerical => "numerical", + } + } +} + +/// One covariate, already on the samples' scale: `values` is positional, +/// one entry per sample, `None` where that sample has no covariate value +/// (which both engines turn into a dropped individual, not a zero). +#[derive(Debug, Clone)] +pub struct Covariate { + pub name: String, + pub kind: CovariateKind, + pub values: Vec<Option<f64>>, + /// The distinct levels, in the order they were coded, for a + /// categorical covariate: GEMMA gets k-1 dummy columns named after + /// them (`sex.M`), where R/qtl expands the single column itself. + /// Empty for a numerical covariate. + pub levels: Vec<Level>, +} + +/// The most distinct values an un-mapped text attribute may have and +/// still be coded automatically. Beyond this a covariate is refused +/// rather than silently turned into an arbitrary ordering. +pub const MAX_AUTO_LEVELS: usize = 20; + +/// Turn one case attribute's text values into a covariate: numeric +/// columns pass through, anything else is coded 0..k-1 and reported as +/// categorical. +/// +/// An explicit `map` is honoured exactly (and must cover every value +/// present); without one, text is only coded when it has between 2 and +/// [`MAX_AUTO_LEVELS`] distinct values. +pub fn code_case_attribute( + values: &[Option<String>], + map: Option<&BTreeMap<String, f64>>, +) -> Result<CodedAttribute, String> { + if let Some(map) = map { + let mut coded = Vec::with_capacity(values.len()); + let mut missing_mapping: Vec<&str> = Vec::new(); + for value in values { + let coded_value = match value.as_deref() { + None => None, + Some(text) => match map.get(text) { + Some(number) => Some(*number), + None => { + missing_mapping.push(text); + None + } + }, + }; + coded.push(coded_value); + } + if !missing_mapping.is_empty() { + missing_mapping.sort_unstable(); + missing_mapping.dedup(); + return Err(format!( + "the map does not cover {}; add mappings for those or leave the \ + attribute out", + missing_mapping + .iter() + .map(|value| format!("'{value}'")) + .collect::<Vec<_>>() + .join(", ") + )); + } + let levels = map.len(); + if levels < 2 { + return Err("the map has fewer than two levels, so the covariate cannot vary".to_string()); + } + return Ok(CodedAttribute { + values: coded, + kind: CovariateKind::Categorical, + coding: Some( + map.iter() + .map(|(value, number)| format!("{value}={number}")) + .collect::<Vec<_>>() + .join(", "), + ), + levels: map + .iter() + .map(|(name, code)| Level::new(name.clone(), *code)) + .collect(), + }); + } + + let mut levels: Vec<&str> = values + .iter() + .filter_map(|value| value.as_deref()) + .filter(|value| !value.is_empty()) + .collect(); + levels.sort_unstable(); + levels.dedup(); + if levels.len() < 2 { + return Err(format!( + "the attribute has {} usable level(s) among these samples ({}), so it cannot \ + be a covariate", + levels.len(), + if levels.is_empty() { + "no values at all".to_string() + } else { + format!("all '{}'", levels[0]) + } + )); + } + // All numeric? Use them as they are. + let numbers: Option<Vec<f64>> = levels + .iter() + .map(|level| level.trim().parse::<f64>().ok()) + .collect(); + if numbers.is_some() { + let coded = values + .iter() + .map(|value| { + value + .as_deref() + .map(str::trim) + .filter(|text| !text.is_empty()) + .and_then(|text| text.parse::<f64>().ok()) + }) + .collect(); + return Ok(CodedAttribute { + values: coded, + kind: CovariateKind::Numerical, + coding: None, + levels: Vec::new(), + }); + } + if levels.len() > MAX_AUTO_LEVELS { + return Err(format!( + "the attribute has {} distinct values, too many to code automatically (limit \ + {}); pass an explicit `map`", + levels.len(), + MAX_AUTO_LEVELS + )); + } + let mut mapping = std::collections::BTreeMap::new(); + for (index, level) in levels.iter().enumerate() { + mapping.insert((*level).to_string(), index as f64); + } + let coded = values + .iter() + .map(|value| { + value + .as_deref() + .filter(|text| !text.is_empty()) + .and_then(|text| mapping.get(text).copied()) + }) + .collect(); + Ok(CodedAttribute { + values: coded, + kind: CovariateKind::Categorical, + coding: Some( + mapping + .iter() + .map(|(value, number)| format!("{value}={number}")) + .collect::<Vec<_>>() + .join(", "), + ), + levels: mapping + .iter() + .map(|(name, code)| Level::new(name.clone(), *code)) + .collect(), + }) +} + +/// One level of a categorical covariate: what it is called, and the code +/// the values carry for it. Both are needed -- an explicit `map` may code +/// `HFD=1` where `LFD=0`, so the level's position in the sorted list is +/// not its code. +#[derive(Debug, Clone, PartialEq)] +pub struct Level { + pub name: String, + pub code: f64, +} + +impl Level { + fn new(name: impl Into<String>, code: f64) -> Self { + Level { + name: name.into(), + code, + } + } +} + +/// What [`code_case_attribute`] produced. +#[derive(Debug)] +pub struct CodedAttribute { + pub values: Vec<Option<f64>>, + pub kind: CovariateKind, + pub coding: Option<String>, + /// The levels, in coded order (empty for a numerical result). + pub levels: Vec<Level>, +} + +/// Write the wrapper's `--covarstruct` file: one `name<TAB>kind` row per +/// covariate, the format `read.table(opt$covarstruct, sep="\t", +/// header=FALSE)` expects. +pub fn write_covarstruct_file(path: &Path, covariates: &[Covariate]) -> io::Result<String> { + let mut contents = String::new(); + for covariate in covariates { + contents.push_str(&format!( + "{}\t{}\n", + covariate.name, + covariate.kind.as_str() + )); + } + std::fs::write(path, &contents)?; + Ok(contents) +} + +/// How many covariates one request may fit. +pub const MAX_COVARIATES: usize = 8; + +/// One covariate in a request: a case attribute of the group, by name, or +/// explicit per-sample numbers. Both runners take this same shape. +#[derive(Debug, Clone, serde::Deserialize, utoipa::ToSchema)] +#[schema(as = CovariateSpec)] +pub struct Spec { + /// Case attribute to use, e.g. "Sex". The server reads it for the + /// group, aligns it onto the panel and codes its text. + pub case_attribute: Option<String>, + /// Optional explicit coding for a case attribute, e.g. {"F": 0, "M": 1}. + pub map: Option<BTreeMap<String, f64>>, + /// For an inline covariate: its name. + pub name: Option<String>, + /// For an inline covariate: "categorical" (default) or "numerical". + pub kind: Option<String>, + /// For an inline covariate: one value per sample, matched by name. + pub values: Option<Vec<NamedValue>>, +} + +/// One inline covariate value, by sample name. +#[derive(Debug, Clone, serde::Deserialize, utoipa::ToSchema)] +#[schema(as = CovariateValue)] +pub struct NamedValue { + pub name: String, + pub value: Option<f64>, +} + +/// Resolve request specs onto the samples being mapped. +/// +/// Returns the covariates in request order and one `notes` line each +/// (what was coded, how many samples have a value, and -- when +/// `has_trait` says which samples carry a trait value -- how many have +/// both, i.e. the complete cases the fit will use). `group` is where a +/// `case_attribute` is read from; without one, only inline covariates are +/// possible. +pub async fn resolve( + specs: &[Spec], + sample_names: &[String], + group: Option<&str>, + has_trait: Option<&[bool]>, + rid: RequestInfo, +) -> Result<(Vec<Covariate>, Vec<String>), ApiError> { + // `RequestInfo` is not `Copy`; every error path here needs its own. + let rid_at = |_: ()| rid.clone(); + let mut covariates: Vec<Covariate> = Vec::new(); + let mut notes: Vec<String> = Vec::new(); + if specs.is_empty() { + return Ok((covariates, notes)); + } + if specs.len() > MAX_COVARIATES { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "too_many_covariates", + "Too many covariates", + format!( + "{} covariates given; at most {MAX_COVARIATES} are accepted.", + specs.len() + ), + "covariates", + ) + .with_request_id(rid_at(()))); + } + + // Case attributes come from the group, once for the whole request. + let wanted: Vec<&str> = specs + .iter() + .filter_map(|spec| spec.case_attribute.as_deref()) + .map(str::trim) + .collect(); + let attribute_values = if wanted.is_empty() { + HashMap::new() + } else { + let Some(group) = group else { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "covariate_unavailable", + "Covariate unavailable", + "a case attribute needs a `group` on the request; with a genotype_file, pass the covariate's values inline" + .to_string(), + "covariates", + ) + .with_request_id(rid_at(()))); + }; + let group_owned = group.to_string(); + let names = sample_names.to_vec(); + let attributes = tokio::task::spawn_blocking(move || { + let Some(row) = db::get_group_by_name(&group_owned)? else { + return Ok::<_, db::DbError>(None); + }; + db::get_group_case_attributes(row.id).map(Some) + }) + .await + .map_err(|e| { + ApiError::new( + StatusCode::INTERNAL_SERVER_ERROR, + "internal_error", + "Internal error", + e.to_string(), + "covariates", + ) + .with_request_id(rid_at(())) + })? + .map_err(|e| ApiError::from_db(e, "covariates").with_request_id(rid_at(())))?; + let Some(attributes) = attributes else { + return Err(ApiError::new( + StatusCode::NOT_FOUND, + "group_not_found", + "Group not found", + "the request's group does not exist, so its case attributes cannot be read" + .to_string(), + "covariates", + ) + .with_request_id(rid_at(()))); + }; + for attribute in &wanted { + if !attributes.names.iter().any(|name| name == attribute) { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "covariate_unavailable", + "Covariate unavailable", + format!( + "case attribute '{attribute}' is not defined for this group; it has: {}", + if attributes.names.is_empty() { + "none".to_string() + } else { + attributes.names.join(", ") + } + ), + "covariates", + ) + .with_request_id(rid_at(()))); + } + } + let by_strain: HashMap<&str, &HashMap<String, String>> = attributes + .values + .iter() + .map(|row| (row.strain.as_str(), &row.case_attributes)) + .collect(); + let mut columns: HashMap<String, Vec<Option<String>>> = HashMap::new(); + for attribute in wanted { + let column = names + .iter() + .map(|name| { + by_strain + .get(name.as_str()) + .and_then(|attributes| attributes.get(attribute)) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + }) + .collect(); + columns.insert(attribute.to_string(), column); + } + columns + }; + + for spec in specs { + let (name, kind, values, coding, levels) = match (&spec.case_attribute, &spec.values) { + (Some(attribute), None) => { + let attribute = attribute.trim(); + let Some(column) = attribute_values.get(attribute) else { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "covariate_unavailable", + "Covariate unavailable", + format!("case attribute '{attribute}' could not be read"), + "covariates", + ) + .with_request_id(rid_at(()))); + }; + let coded = code_case_attribute(column, spec.map.as_ref()).map_err(|detail| { + ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "invalid_covariate", + "Invalid covariate", + format!("case attribute '{attribute}': {detail}"), + "covariates", + ) + .with_request_id(rid_at(())) + })?; + ( + attribute.to_string(), + coded.kind, + coded.values, + coded.coding, + coded.levels, + ) + } + (None, Some(values)) => { + let Some(name) = spec.name.clone() else { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "invalid_covariate", + "Invalid covariate", + "an inline covariate needs a `name`".to_string(), + "covariates", + ) + .with_request_id(rid_at(()))); + }; + let by_name: HashMap<&str, Option<f64>> = values + .iter() + .map(|value| (value.name.as_str(), value.value)) + .collect(); + let coded: Vec<Option<f64>> = sample_names + .iter() + .map(|name| by_name.get(name.as_str()).copied().flatten()) + .collect(); + if coded.iter().all(Option::is_none) { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "covariate_unavailable", + "Covariate unavailable", + format!( + "none of the {} values of covariate '{name}' match a sample of the trait", + values.len() + ), + "covariates", + ) + .with_request_id(rid_at(()))); + } + let (kind, levels) = match spec + .kind + .as_deref() + .unwrap_or("categorical") + .to_ascii_lowercase() + .as_str() + { + "categorical" => { + // Codes come from the values themselves, sorted, so + // the dummy columns (GEMMA) have stable names. + let mut distinct: Vec<f64> = coded.iter().filter_map(|v| *v).collect(); + distinct.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal)); + distinct.dedup(); + let levels = distinct + .iter() + .map(|code| Level::new(format!("{code}"), *code)) + .collect(); + (CovariateKind::Categorical, levels) + } + "numerical" => (CovariateKind::Numerical, Vec::new()), + other => { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "invalid_covariate", + "Invalid covariate", + format!( + "covariate '{name}': kind '{other}' is not 'categorical' or 'numerical'" + ), + "covariates", + ) + .with_request_id(rid_at(()))); + } + }; + (name, kind, coded, None, levels) + } + (Some(_), Some(_)) => { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "invalid_covariate", + "Invalid covariate", + "give either case_attribute or values for a covariate, not both".to_string(), + "covariates", + ) + .with_request_id(rid_at(()))); + } + (None, None) => { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "invalid_covariate", + "Invalid covariate", + "a covariate needs a case_attribute, or a name and values".to_string(), + "covariates", + ) + .with_request_id(rid_at(()))); + } + }; + if covariates.iter().any(|covariate| covariate.name == name) { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "duplicate_covariate", + "Duplicate covariate", + format!("covariate '{name}' is given more than once."), + "covariates", + ) + .with_request_id(rid_at(()))); + } + let present = values.iter().filter(|value| value.is_some()).count(); + let complete = has_trait + .map(|has_trait| { + format!( + ", {} with both the trait and the covariate", + values + .iter() + .zip(has_trait) + .filter(|(value, has_trait)| value.is_some() && **has_trait) + .count() + ) + }) + .unwrap_or_default(); + notes.push(format!( + "Covariate {name}: {}{}, {present} of {} samples have a value{complete}{}", + kind.as_str(), + coding + .as_deref() + .map(|coding| format!(" coded {coding}")) + .unwrap_or_default(), + values.len(), + if kind == CovariateKind::Categorical && levels.len() > 2 { + format!( + "; {} dummy column(s) named {name}.<level>", + levels.len() - 1 + ) + } else { + String::new() + } + )); + covariates.push(Covariate { + name, + kind, + values, + levels, + }); + } + Ok((covariates, notes)) +} + +/// What GEMMA's covariate file turned out to be. +pub struct GemmaCovariates { + /// The file's contents: no header, no sample names, tab separated. + pub contents: String, + /// The columns written, in order (`sex.M`, `Age`, ...). + pub columns: Vec<String>, + /// Columns left out because they do not vary among the individuals + /// that will enter the fit, with why. + pub dropped: Vec<String>, + /// For each covariate, how many individuals have no value for it -- + /// GEMMA drops those, exactly as it drops a missing phenotype. + pub missing: Vec<(String, usize)>, +} + +/// Write GEMMA's covariate file: no header, no sample names, one row per +/// individual in the panel's order, one column per covariate -- and, for a +/// categorical covariate, k-1 dummy columns (`sex.M`) instead of its +/// level codes, because GEMMA reads numbers and knows nothing about +/// factors (R/qtl's wrapper does that expansion for itself). +/// +/// A missing value is written as `NA`, which GEMMA treats as a missing +/// phenotype does: that individual leaves the fit (measured). +/// +/// `analysed` says which rows have a phenotype, so a dummy for a level +/// nobody analysed carries has to go: an all-zero column is singular and +/// GEMMA stops with `LUDecomp ... input domain error` rather than +/// dropping it the way R would. Dropping it changes nothing about the +/// model -- a level with no observations contributes no term. +pub fn write_gemma_covariates_file( + path: &Path, + covariates: &[Covariate], + analysed: &[bool], +) -> std::io::Result<GemmaCovariates> { + // A row enters the fit only if it has a phenotype and a value for + // every covariate (GEMMA drops it otherwise). + let mut in_fit: Vec<bool> = analysed.to_vec(); + in_fit.resize( + covariates + .first() + .map(|covariate| covariate.values.len()) + .unwrap_or(0), + false, + ); + for covariate in covariates { + for (keep, value) in in_fit.iter_mut().zip(&covariate.values) { + if value.is_none() { + *keep = false; + } + } + } + + let mut columns: Vec<(String, Vec<Option<f64>>)> = Vec::new(); + let mut dropped: Vec<String> = Vec::new(); + let mut missing: Vec<(String, usize)> = Vec::new(); + for covariate in covariates { + let absent = covariate.values.iter().filter(|value| value.is_none()).count(); + if absent > 0 { + missing.push((covariate.name.clone(), absent)); + } + let mut candidates: Vec<(String, Vec<Option<f64>>)> = Vec::new(); + match covariate.kind { + CovariateKind::Numerical => { + candidates.push((covariate.name.clone(), covariate.values.clone())); + } + CovariateKind::Categorical => { + // The first level is the reference: k-1 dummies. + for level in covariate.levels.iter().skip(1) { + let indicator = covariate + .values + .iter() + .map(|value| value.map(|code| if code == level.code { 1.0 } else { 0.0 })) + .collect(); + candidates.push((format!("{}.{}", covariate.name, level.name), indicator)); + } + } + } + for (name, values) in candidates { + // Constant (or empty) across the rows that matter: no term. + let kept: Vec<f64> = values + .iter() + .zip(&in_fit) + .filter(|(_, keep)| **keep) + .filter_map(|(value, _)| *value) + .collect(); + let varies = kept + .first() + .map(|first| kept.iter().any(|value| value != first)) + .unwrap_or(false); + if varies { + columns.push((name, values)); + } else { + dropped.push(name); + } + } + } + + let rows = covariates + .first() + .map(|covariate| covariate.values.len()) + .unwrap_or(0); + let mut contents = String::new(); + for row in 0..rows { + for (index, (_, values)) in columns.iter().enumerate() { + if index > 0 { + contents.push('\t'); + } + match values.get(row).copied().flatten() { + Some(value) => contents.push_str(&value.to_string()), + None => contents.push_str("NA"), + } + } + if !columns.is_empty() { + contents.push('\n'); + } + } + std::fs::write(path, &contents)?; + Ok(GemmaCovariates { + contents, + columns: columns.into_iter().map(|(name, _)| name).collect(), + dropped, + missing, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn levels(levels: &[Level]) -> Vec<String> { + levels.iter().map(|level| level.name.clone()).collect() + } + + fn texts(values: &[&str]) -> Vec<Option<String>> { + values + .iter() + .map(|value| Some(value.to_string())) + .collect() + } + + #[test] + fn text_covariates_are_coded_and_numeric_ones_pass_through() { + let sex = texts(&["F", "M", "", "F"]); + let coded = code_case_attribute(&sex, None).unwrap(); + assert_eq!(coded.kind, CovariateKind::Categorical); + assert_eq!(coded.values, vec![Some(0.0), Some(1.0), None, Some(0.0)]); + assert_eq!(coded.coding.as_deref(), Some("F=0, M=1")); + // The levels, in coded order: what a dummy expansion names itself + // after. + assert_eq!( + levels(&coded.levels), + vec!["F".to_string(), "M".to_string()] + ); + + let ages = texts(&["183", "200"]); + let coded = code_case_attribute(&ages, None).unwrap(); + assert_eq!(coded.kind, CovariateKind::Numerical); + assert_eq!(coded.values, vec![Some(183.0), Some(200.0)]); + assert!(coded.levels.is_empty()); + + // One level, or none, cannot be a covariate. + let error = code_case_attribute(&texts(&["Live", "Live"]), None).unwrap_err(); + assert!(error.contains("cannot be a covariate"), "{error}"); + } + + #[test] + fn an_explicit_map_is_honoured_and_must_be_complete() { + let values = texts(&["HFD", "LFD"]); + let mut map = BTreeMap::new(); + map.insert("LFD".to_string(), 0.0); + map.insert("HFD".to_string(), 1.0); + let coded = code_case_attribute(&values, Some(&map)).unwrap(); + assert_eq!(coded.kind, CovariateKind::Categorical); + assert_eq!(coded.values, vec![Some(1.0), Some(0.0)]); + assert_eq!(coded.coding.as_deref(), Some("HFD=1, LFD=0")); + // Ordered by name, coded by the map: HFD=1 comes before LFD=0, so + // a dummy has to key on the code, not the position. + assert_eq!( + levels(&coded.levels), + vec!["HFD".to_string(), "LFD".to_string()] + ); + assert_eq!(coded.levels[0].code, 1.0); + + map.remove("LFD"); + let error = code_case_attribute(&values, Some(&map)).unwrap_err(); + assert!(error.contains("'LFD'"), "{error}"); + } + + #[test] + fn too_many_levels_asks_for_an_explicit_map() { + let values: Vec<Option<String>> = + (0..MAX_AUTO_LEVELS + 1).map(|i| Some(format!("level-{i}"))).collect(); + let error = code_case_attribute(&values, None).unwrap_err(); + assert!(error.contains("pass an explicit `map`"), "{error}"); + } + + fn write_analysed(covariates: &[Covariate], analysed: &[bool]) -> GemmaCovariates { + let path = std::env::temp_dir().join(format!( + "gn4-covariates-{}-{}", + std::process::id(), + covariates.len() + )); + let written = write_gemma_covariates_file(&path, covariates, analysed).unwrap(); + let _ = std::fs::remove_file(&path); + written + } + + #[test] + fn gemma_gets_one_column_per_numerical_covariate_and_na_for_a_missing_one() { + // No header, no names: GEMMA reads the file positionally. + let written = write_analysed( + &[Covariate { + name: "Age".to_string(), + kind: CovariateKind::Numerical, + values: vec![Some(183.0), None, Some(200.5)], + levels: Vec::new(), + }], + &[true, true, true], + ); + assert_eq!(written.contents, "183\nNA\n200.5\n"); + assert_eq!(written.columns, vec!["Age".to_string()]); + assert_eq!(written.missing, vec![("Age".to_string(), 1)]); + } + + #[test] + fn a_two_level_covariate_becomes_one_dummy_named_after_the_second_level() { + let written = write_analysed( + &[Covariate { + name: "Sex".to_string(), + kind: CovariateKind::Categorical, + values: vec![Some(0.0), Some(1.0), None, Some(0.0)], + levels: vec![Level::new("F", 0.0), Level::new("M", 1.0)], + }], + &[true, true, true, true], + ); + // F is the reference, so only M's dummy is written -- and the + // individual with no value is NA in it, so GEMMA drops them. + assert_eq!(written.columns, vec!["Sex.M".to_string()]); + assert_eq!(written.contents, "0\n1\nNA\n0\n"); + } + + #[test] + fn a_three_level_covariate_becomes_two_dummies_and_a_missing_value_stays_missing() { + let written = write_analysed( + &[ + Covariate { + name: "Diet".to_string(), + kind: CovariateKind::Categorical, + values: vec![Some(0.0), Some(1.0), Some(2.0), None], + levels: vec![ + Level::new("HFD", 0.0), + Level::new("LFD", 1.0), + Level::new("CR", 2.0), + ], + }, + Covariate { + name: "Age".to_string(), + kind: CovariateKind::Numerical, + values: vec![Some(1.0), Some(2.0), Some(3.0), Some(4.0)], + levels: Vec::new(), + }, + ], + &[true, true, true, true], + ); + assert_eq!( + written.columns, + vec!["Diet.LFD".to_string(), "Diet.CR".to_string(), "Age".to_string()] + ); + assert_eq!(written.contents, "0\t0\t1\n1\t0\t2\n0\t1\t3\nNA\tNA\t4\n"); + } + + #[test] + fn a_dummy_no_analysed_individual_carries_is_dropped() { + // The level exists in the panel but none of the individuals with + // a phenotype is in it: an all-zero column is singular, and GEMMA + // stops on it rather than dropping the term as R would. + let written = write_analysed( + &[Covariate { + name: "Status".to_string(), + kind: CovariateKind::Categorical, + values: vec![Some(0.0), Some(0.0), Some(1.0), Some(2.0)], + levels: vec![ + Level::new("Cryo", 0.0), + Level::new("Live", 1.0), + Level::new("Ext", 2.0), + ], + }], + // Only the Cryo and Live individuals have a phenotype. + &[true, true, true, false], + ); + assert_eq!(written.columns, vec!["Status.Live".to_string()]); + assert_eq!(written.dropped, vec!["Status.Ext".to_string()]); + assert_eq!(written.contents, "0\n0\n1\n0\n"); + + // Nothing varying at all leaves no covariates to fit. + let written = write_analysed( + &[Covariate { + name: "Status".to_string(), + kind: CovariateKind::Categorical, + values: vec![Some(0.0), Some(1.0)], + levels: vec![Level::new("Cryo", 0.0), Level::new("Live", 1.0)], + }], + &[true, false], + ); + assert!(written.columns.is_empty(), "{:?}", written.columns); + assert_eq!(written.contents, ""); + } + + #[test] + fn an_inline_categorical_is_expanded_from_its_own_codes() { + // No names are known for inline values, so the dummies are named + // after the codes, which is all a caller can then report. + let spec = Spec { + case_attribute: None, + map: None, + name: Some("Batch".to_string()), + kind: None, + values: Some(vec![ + NamedValue { name: "A".to_string(), value: Some(0.0) }, + NamedValue { name: "B".to_string(), value: Some(1.0) }, + NamedValue { name: "C".to_string(), value: Some(2.0) }, + ]), + }; + let runtime = tokio::runtime::Builder::new_current_thread().build().unwrap(); + let names = vec!["A".to_string(), "B".to_string(), "C".to_string()]; + let (covariates, notes) = runtime + .block_on(resolve( + &[spec], + &names, + None, + Some(&[true, true, true]), + RequestInfo::new(None), + )) + .unwrap(); + assert_eq!( + levels(&covariates[0].levels), + vec!["0".to_string(), "1".to_string(), "2".to_string()] + ); + assert!(notes[0].contains("dummy column(s) named Batch.<level>"), "{notes:?}"); + assert!( + notes[0].contains("3 with both the trait and the covariate"), + "the complete cases are reported: {notes:?}" + ); + let written = write_analysed(&covariates, &[true, true, true]); + assert_eq!(written.columns, vec!["Batch.1".to_string(), "Batch.2".to_string()]); + assert_eq!(written.contents, "0\t0\n1\t0\n0\t1\n"); + } +} diff --git a/src/gemma.rs b/src/gemma.rs index 2647d9c..2c6c7d2 100644 --- a/src/gemma.rs +++ b/src/gemma.rs @@ -213,6 +213,7 @@ async fn compute_gwa( snps_file: &str, lmm: u8, k_json_path: &str, + covariate_file: Option<&str>, log_path: Option<&Path>, ) -> Result<Value, GemmaError> { let lmm_str = lmm.to_string(); @@ -221,6 +222,7 @@ async fn compute_gwa( genotype_file, pheno_file, snps_file, + covariate_file, &lmm_str, k_json_path, ); @@ -374,10 +376,11 @@ fn gwa_args<'a>( genotype_file: &'a str, pheno_file: &'a str, snps_file: &'a str, + covariate_file: Option<&'a str>, lmm_str: &'a str, k_json_path: &'a str, ) -> Vec<&'a str> { - vec![ + let mut args = vec![ "--json", "--cache-dir", cache_dir, @@ -391,9 +394,17 @@ fn gwa_args<'a>( pheno_file, "-a", snps_file, - "-lmm", - lmm_str, - ] + ]; + // Covariates reach only the GWA step: -gk (kinship) takes none, and + // GEMMA's own key for the K cache drops the phenotype pair, so the + // kinship computed once is still shared across covariate sets. + if let Some(covariate_file) = covariate_file { + args.push("-c"); + args.push(covariate_file); + } + args.push("-lmm"); + args.push(lmm_str); + args } /// Run the full gemma-wrapper flow -- compute kinship, then run the LMM @@ -411,6 +422,7 @@ pub async fn compute( genotype_file: &str, snps_file: &str, trait_values: &TraitValues, + covariates: &[crate::covariates::Covariate], lmm: u8, run_id: Option<&str>, ) -> Result<GemmaOutput, GemmaError> { @@ -420,6 +432,44 @@ pub async fn compute( write_pheno_file(&pheno_path, trait_values).map_err(GemmaError::Io)?; let pheno_file = pheno_path.to_string_lossy().into_owned(); + // GEMMA's own covariate file: positional over the panel, numeric, with + // `NA` for a missing value -- which it treats like a missing phenotype + // and drops the individual for (see crate::covariates). + let covariate_path = Path::new(cache_dir).join(format!("{unique}-covariates.txt")); + let mut notes: Vec<String> = Vec::new(); + let covariate_file = if covariates.is_empty() { + None + } else { + let written = crate::covariates::write_gemma_covariates_file( + &covariate_path, + covariates, + &trait_values.iter().map(Option::is_some).collect::<Vec<bool>>(), + ) + .map_err(GemmaError::Io)?; + for (name, absent) in &written.missing { + notes.push(format!( + "Missing covariate values: {name} has none for {absent} individual(s), which \ + GEMMA drops, as it drops a missing phenotype." + )); + } + if !written.dropped.is_empty() { + notes.push(format!( + "Covariate column(s) left out because they do not vary among the individuals \ + fitted: {}.", + written.dropped.join(", ") + )); + } + if written.columns.is_empty() { + notes.push( + "No covariate varies among the individuals fitted, so GEMMA ran without one." + .to_string(), + ); + None + } else { + Some(covariate_path.to_string_lossy().into_owned()) + } + }; + // One log file for the whole request (both steps); see // crate::streaming::create_run_log for why this can't silently // collide with another run's. @@ -455,6 +505,7 @@ pub async fn compute( genotype_file, &pheno_file, snps_file, + covariate_file.as_deref(), &lmm_str, &k_json_path.to_string_lossy(), ) @@ -468,6 +519,7 @@ pub async fn compute( snps_file, lmm, &k_json_path.to_string_lossy(), + covariate_file.as_deref(), log_path.as_deref(), ) .await?; @@ -490,10 +542,12 @@ pub async fn compute( let (columns, rows) = parse_assoc_files(&extract_dir, basenames)?; let _ = std::fs::remove_dir_all(&extract_dir); // best effort; a leftover scratch dir is harmless + let mut output_notes = vec![kinship_note, gwa_note]; + output_notes.extend(notes); Ok(GemmaOutput { columns, rows: Value::Array(rows), - notes: vec![kinship_note, gwa_note], + notes: output_notes, }) } .await; @@ -967,6 +1021,7 @@ mod tests { "geno.txt.gz", "pheno.txt", "snps.txt", + None, "9", "/cache/k.json", ); diff --git a/src/lib.rs b/src/lib.rs index c8e13b4..6edeed6 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -8,6 +8,7 @@ use utoipa_axum::routes; pub mod auth; pub mod compute_gate; +pub mod covariates; pub mod config; pub mod db; pub mod embedded; diff --git a/src/routes/gemma.rs b/src/routes/gemma.rs index 895bc36..38f0a8f 100644 --- a/src/routes/gemma.rs +++ b/src/routes/gemma.rs @@ -18,6 +18,7 @@ use gnauth_macros::{compute_gate, gnauth}; use crate::auth::AuthToken; use crate::config::Config; +use crate::covariates; use crate::error::{ApiError, Unauthorized}; use crate::genotypes; use crate::gemma::{self, GemmaError}; @@ -53,6 +54,12 @@ pub struct GemmaRequest { pub genotype_version: Option<String>, /// Include parents and F1s. Off by default, as in GeneNetwork2/3. pub include_parents_f1: Option<bool>, + /// Covariates to fit alongside the trait: a case attribute of the + /// group by name, or explicit per-sample numbers. Same shape as + /// `/compute/rqtl`'s; GEMMA needs `group`, since the individuals have + /// to be named to align them. + #[serde(default)] + pub covariates: Vec<crate::covariates::Spec>, /// One phenotype value per individual, in the prepared genotype /// file's order; `null` for a missing value. Alternatively give /// `samples`, which is matched onto the panel by name. @@ -138,6 +145,8 @@ pub async fn compute_gemma( let mut extra_notes: Vec<String> = Vec::new(); let request_group = request.group.clone(); let request_genotype_version = request.genotype_version.clone(); + let request_covariates = request.covariates.clone(); + let request_covariates_present = !request_covariates.is_empty(); // GEMMA's request carries no trait name; R/qtl's does, so the // suggestion below labels the trait by its group. let request_trait_label = format!("{}::gemma", request.group.clone().unwrap_or_default()); @@ -147,6 +156,10 @@ pub async fn compute_gemma( .map(|sample| serde_json::json!({"name": sample.name, "value": sample.value})) .collect() }); + let mut covariate_list: Vec<covariates::Covariate> = Vec::new(); + let mut covariate_notes: Vec<String> = Vec::new(); + let mut aligned_names: Option<Vec<String>> = None; + let mut aligned_group = String::new(); let (genotype_file, snps_file, trait_values): (String, String, Vec<Option<f64>>) = match ( request.genotype_file.clone(), @@ -192,7 +205,7 @@ pub async fn compute_gemma( let wanted = group.clone(); let had_samples = request.samples.is_some(); let supplied_values = request.trait_values.clone(); - let (prepared, aligned, matched, resolved) = + let (prepared, aligned, matched, resolved, kept_names) = tokio::task::spawn_blocking(move || { let resolved = genotypes::resolve(&directory, &wanted, version.as_deref())?; let excluded = if include_parents_f1 { @@ -208,6 +221,10 @@ pub async fn compute_gemma( .map(|(index, _)| index) .collect(); let (aligned, matched) = genotypes::align(&resolved.panel, &keep, &values); + let kept_names: Vec<String> = keep + .iter() + .filter_map(|index| resolved.panel.get(*index).cloned()) + .collect(); let prepared = genotypes::prepare( &directory, &resolved, @@ -215,7 +232,9 @@ pub async fn compute_gemma( &keep, &cache_dir, )?; - Ok::<_, genotypes::GenotypeError>((prepared, aligned, matched, resolved)) + Ok::<_, genotypes::GenotypeError>(( + prepared, aligned, matched, resolved, kept_names, + )) }) .await .map_err(|e| { @@ -232,6 +251,10 @@ pub async fn compute_gemma( genotype_error_to_api_error(e, "compute_gemma", rid.clone()) })?; let individuals = prepared.individuals; + // Kept for the covariate resolve below: it has to run after + // the trait values exist (see there). + aligned_names = Some(kept_names); + aligned_group = group.clone(); let trait_values = if had_samples { if matched == 0 { return Err(ApiError::new( @@ -301,12 +324,43 @@ pub async fn compute_gemma( .with_request_id(rid)); } }; + // Covariates are aligned by sample name, so only a group-scoped request + // can have them: with explicit files there is nothing to align against. + if request_covariates_present && request_group.is_none() { + return Err(ApiError::new( + StatusCode::UNPROCESSABLE_ENTITY, + "covariate_unavailable", + "Covariate unavailable", + "covariates need a `group` on the request: individual names are what they are \ + aligned against, and explicit genotype_file/snps_file requests carry none" + .to_string(), + "compute_gemma", + ) + .with_request_id(rid)); + } + // Covariates are per individual, so they need the panel's names: the + // same individuals, in the same order, as the phenotype file above. + // Resolved here rather than beside the panel because the note it + // reports counts the individuals that have *both* the trait and the + // covariate -- the complete cases the fit will actually use. + if let Some(names) = &aligned_names { + let has_trait: Vec<bool> = trait_values.iter().map(Option::is_some).collect(); + (covariate_list, covariate_notes) = covariates::resolve( + &request_covariates, + names, + Some(&aligned_group), + Some(&has_trait), + rid.clone(), + ) + .await?; + } gemma::compute( &config.gemma_wrapper_cmd, &config.gemma_cache_dir, &genotype_file, &snps_file, &trait_values, + &covariate_list, lmm, params.id.as_deref(), ) @@ -322,6 +376,9 @@ pub async fn compute_gemma( for note in &output.notes { envelope = envelope.with_note(note); } + for note in &covariate_notes { + envelope = envelope.with_note(note); + } for note in &extra_notes { envelope = envelope.with_note(note); } diff --git a/src/routes/rqtl.rs b/src/routes/rqtl.rs index bfb979f..3b82977 100644 --- a/src/routes/rqtl.rs +++ b/src/routes/rqtl.rs @@ -15,7 +15,7 @@ use gnauth_macros::{compute_gate, gnauth}; use crate::auth::AuthToken; use crate::config::Config; -use crate::db; +use crate::covariates; use crate::error::{ApiError, Unauthorized}; use crate::genotypes; use crate::response::{Ctx, Envelope, RequestInfo, path_segment}; @@ -62,7 +62,7 @@ pub struct RqtlRequest { /// (`/groups/{group}/traits/{id}/case-attributes/{attribute}` shows /// the values it will code). #[serde(default)] - pub covariates: Vec<RqtlCovariate>, + pub covariates: Vec<covariates::Spec>, /// One entry per individual being mapped. With `group` given, they are /// matched onto the panel **by name** (unknown names are dropped, and /// missing individuals get `null`), so a trait's own values are @@ -89,27 +89,6 @@ pub struct RqtlRequest { } /// How many covariates one request may fit. -const MAX_COVARIATES: usize = 8; - -/// One covariate: either a case attribute of the group, by name, or -/// explicit per-sample numbers. -#[derive(Deserialize, utoipa::ToSchema)] -pub struct RqtlCovariate { - /// Case attribute to use, e.g. "Sex". The server reads it for the - /// group, aligns it onto the panel and codes its text. - pub case_attribute: Option<String>, - /// Optional explicit coding for a case attribute, e.g. {"F": 0, "M": 1}. - /// Without one, text with 2-20 levels is coded 0..k-1 (in sorted - /// order) and numeric text is used as numbers. - pub map: Option<std::collections::BTreeMap<String, f64>>, - /// For an inline covariate: its name. - pub name: Option<String>, - /// For an inline covariate: "categorical" (default) or "numerical". - pub kind: Option<String>, - /// For an inline covariate: one value per sample, matched by name - /// (with `group`) or in `samples`' order (with `genotype_file`). - pub values: Option<Vec<RqtlSample>>, -} #[utoipa::path(post, path = "/compute/rqtl", summary = "Map a QTL from a genotype file and trait values using R/qtl", @@ -316,271 +295,19 @@ pub async fn compute_rqtl( .with_request_id(rid)); } }; - // Covariates, on the same sample scale as `samples`. - let mut covariates: Vec<rqtl::Covariate> = Vec::new(); - let mut covariate_notes: Vec<String> = Vec::new(); - if !request.covariates.is_empty() { - if request.covariates.len() > MAX_COVARIATES { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "too_many_covariates", - "Too many covariates", - format!( - "{} covariates given; at most {MAX_COVARIATES} are accepted.", - request.covariates.len() - ), - "compute_rqtl", - ) - .with_request_id(rid)); - } - let sample_names: Vec<String> = samples.iter().map(|s| s.name.clone()).collect(); - // Case attributes come from the group; a request that named files - // instead of a group has no attribute source. - let attribute_values: std::collections::HashMap<String, Vec<Option<String>>> = - match request_group.clone() { - Some(group) => { - let directory = Config::load().genotype_files_dir; - let _ = directory; // resolved below, in one blocking call - let wanted: Vec<(String, Option<std::collections::BTreeMap<String, f64>>)> = - request - .covariates - .iter() - .filter_map(|covariate| { - covariate.case_attribute.clone().map(|attribute| { - (attribute, covariate.map.clone()) - }) - }) - .collect(); - if wanted.is_empty() { - std::collections::HashMap::new() - } else { - let names = sample_names.clone(); - let fetched = tokio::task::spawn_blocking(move || { - let Some(group_row) = db::get_group_by_name(&group)? else { - return Ok::<_, db::DbError>(None); - }; - db::get_group_case_attributes(group_row.id).map(Some) - }) - .await - .map_err(|e| { - ApiError::new( - StatusCode::INTERNAL_SERVER_ERROR, - "internal_error", - "Internal error", - e.to_string(), - "compute_rqtl", - ) - .with_request_id(rid.clone()) - })? - .map_err(|e| { - ApiError::from_db(e, "compute_rqtl").with_request_id(rid.clone()) - })?; - let Some(attributes) = fetched else { - return Err(ApiError::new( - StatusCode::NOT_FOUND, - "group_not_found", - "Group not found", - "the request's group does not exist, so its case attributes \n cannot be read" - .to_string(), - "compute_rqtl", - ) - .with_request_id(rid)); - }; - // An attribute the group does not define is a different - // error from one it defines but leaves empty for these - // samples: say which, and list what it has. - for (attribute, _) in &wanted { - if !attributes.names.iter().any(|name| name == attribute) { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "covariate_unavailable", - "Covariate unavailable", - format!( - "case attribute '{attribute}' is not defined for this \ - group; it has: {}", - if attributes.names.is_empty() { - "none".to_string() - } else { - attributes.names.join(", ") - } - ), - "compute_rqtl", - ) - .with_request_id(rid)); - } - } - let by_strain: std::collections::HashMap<String, &std::collections::HashMap<String, String>> = - attributes - .values - .iter() - .map(|row| (row.strain.clone(), &row.case_attributes)) - .collect(); - let mut values = std::collections::HashMap::new(); - for (attribute, _) in &wanted { - let column = names - .iter() - .map(|name| { - by_strain - .get(name) - .and_then(|attributes| attributes.get(attribute)) - .map(|value| value.trim().to_string()) - .filter(|value| !value.is_empty()) - }) - .collect(); - values.insert(attribute.clone(), column); - } - values - } - } - None => std::collections::HashMap::new(), - }; - - for spec in &request.covariates { - let (name, kind_text, values, coding_text) = - match (&spec.case_attribute, &spec.values) { - (Some(attribute), None) => { - let attribute_name = attribute.trim().to_string(); - let Some(column) = attribute_values.get(&attribute_name) else { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "covariate_unavailable", - "Covariate unavailable", - format!( - "case attribute '{attribute_name}' needs a `group` on the \ - request; with a genotype_file, pass the values inline" - ), - "compute_rqtl", - ) - .with_request_id(rid)); - }; - let (coded, kind, coding) = - rqtl::code_case_attribute(column, spec.map.as_ref()).map_err(|detail| { - ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "invalid_covariate", - "Invalid covariate", - format!("case attribute '{attribute_name}': {detail}"), - "compute_rqtl", - ) - .with_request_id(rid.clone()) - })?; - (attribute_name, kind, coded, coding) - } - (None, Some(values)) => { - let inline_name = spec.name.clone().ok_or_else(|| { - ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "invalid_covariate", - "Invalid covariate", - "an inline covariate needs a `name`".to_string(), - "compute_rqtl", - ) - .with_request_id(rid.clone()) - })?; - let by_name: std::collections::HashMap<&str, Option<f64>> = values - .iter() - .map(|value| (value.name.as_str(), value.value)) - .collect(); - let coded: Vec<Option<f64>> = sample_names - .iter() - .map(|name| by_name.get(name.as_str()).copied().flatten()) - .collect(); - if coded.iter().all(Option::is_none) { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "covariate_unavailable", - "Covariate unavailable", - format!( - "none of the {} values of covariate '{inline_name}' match a \ - sample of the trait", - values.len() - ), - "compute_rqtl", - ) - .with_request_id(rid)); - } - let kind = match spec.kind.as_deref().unwrap_or("categorical") { - kind if kind.eq_ignore_ascii_case("categorical") => { - rqtl::CovariateKind::Categorical - } - kind if kind.eq_ignore_ascii_case("numerical") => { - rqtl::CovariateKind::Numerical - } - other => { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "invalid_covariate", - "Invalid covariate", - format!( - "covariate '{inline_name}': kind '{other}' is not \ - 'categorical' or 'numerical'" - ), - "compute_rqtl", - ) - .with_request_id(rid)); - } - }; - (inline_name, kind, coded, None) - } - (Some(_), Some(_)) => { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "invalid_covariate", - "Invalid covariate", - "give either case_attribute or values for a covariate, not both" - .to_string(), - "compute_rqtl", - ) - .with_request_id(rid)); - } - (None, None) => { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "invalid_covariate", - "Invalid covariate", - "a covariate needs a case_attribute, or a name and values".to_string(), - "compute_rqtl", - ) - .with_request_id(rid)); - } - }; - if covariates - .iter() - .any(|covariate| covariate.name == name) - { - return Err(ApiError::new( - StatusCode::UNPROCESSABLE_ENTITY, - "duplicate_covariate", - "Duplicate covariate", - format!("covariate '{name}' is given more than once."), - "compute_rqtl", - ) - .with_request_id(rid)); - } - let present = values.iter().filter(|value| value.is_some()).count(); - let complete = values - .iter() - .zip(samples.iter()) - .filter(|(covariate, sample)| covariate.is_some() && sample.value.is_some()) - .count(); - let coding_note = format!( - "Covariate {name}: {}{}, {present} of {} samples have a value, {complete} with \ -both the trait and the covariate", - kind_text.as_str(), - coding_text - .as_deref() - .map(|coding| format!(" coded {coding}")) - .unwrap_or_default(), - values.len() - ); - covariate_notes.push(coding_note); - covariates.push(rqtl::Covariate { - name, - kind: kind_text, - values, - }); - } - } + // Covariates, on the same sample scale as `samples` (see + // src/covariates.rs: the spec shape, alignment and coding are shared + // with /compute/gemma). + let sample_names: Vec<String> = samples.iter().map(|s| s.name.clone()).collect(); + let has_trait: Vec<bool> = samples.iter().map(|sample| sample.value.is_some()).collect(); + let (covariate_list, covariate_notes) = covariates::resolve( + &request.covariates, + &sample_names, + request_group.as_deref(), + Some(&has_trait), + rid.clone(), + ) + .await?; rqtl::compute( &config.rqtl_wrapper_path, @@ -588,7 +315,7 @@ both the trait and the covariate", &genotype_file, &request.trait_name, &samples, - &covariates, + &covariate_list, request.model.as_deref().unwrap_or("normal"), request.method.as_deref().unwrap_or("hk"), request.nperm.unwrap_or(0), diff --git a/src/rqtl.rs b/src/rqtl.rs index d31ce32..7267870 100644 --- a/src/rqtl.rs +++ b/src/rqtl.rs @@ -43,6 +43,10 @@ use std::io; use std::path::Path; +use crate::covariates::{Covariate, write_covarstruct_file}; +#[cfg(test)] +use crate::covariates::CovariateKind; + use serde::Serialize; use serde_json::Value; @@ -91,180 +95,6 @@ pub struct Sample { pub value: Option<f64>, } -/// How a covariate is declared to the wrapper's `--covarstruct` file. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum CovariateKind { - /// Two or more levels; the wrapper dummy-expands more than two. - Categorical, - /// A number, used as-is. - Numerical, -} - -impl CovariateKind { - pub fn as_str(self) -> &'static str { - match self { - CovariateKind::Categorical => "categorical", - CovariateKind::Numerical => "numerical", - } - } -} - -/// One covariate, already on the samples' scale: `values` is positional, -/// one entry per sample, `None` where that sample has no covariate value -/// (which R/qtl turns into a dropped individual, not a zero). -#[derive(Debug, Clone)] -pub struct Covariate { - pub name: String, - pub kind: CovariateKind, - pub values: Vec<Option<f64>>, -} - -/// Turn one case attribute's text values into a covariate: numeric -/// columns pass through, anything else is coded 0..k-1 and reported as -/// categorical. -/// -/// An explicit `map` is honoured exactly (and must cover every value -/// present); without one, text is only coded when it has between 2 and -/// [`MAX_AUTO_LEVELS`] distinct values, so a free-text attribute is -/// refused rather than silently turned into an arbitrary ordering. -pub const MAX_AUTO_LEVELS: usize = 20; - -pub fn code_case_attribute( - values: &[Option<String>], - map: Option<&std::collections::BTreeMap<String, f64>>, -) -> Result<(Vec<Option<f64>>, CovariateKind, Option<String>), String> { - if let Some(map) = map { - let mut coded = Vec::with_capacity(values.len()); - let mut missing_mapping: Vec<&str> = Vec::new(); - for value in values { - let coded_value = match value.as_deref() { - None => None, - Some(text) => match map.get(text) { - Some(number) => Some(*number), - None => { - missing_mapping.push(text); - None - } - }, - }; - coded.push(coded_value); - } - if !missing_mapping.is_empty() { - missing_mapping.sort_unstable(); - missing_mapping.dedup(); - return Err(format!( - "the map does not cover {}; add mappings for those or leave the \ - attribute out", - missing_mapping - .iter() - .map(|value| format!("'{value}'")) - .collect::<Vec<_>>() - .join(", ") - )); - } - let levels = map.len(); - if levels < 2 { - return Err("the map has fewer than two levels, so the covariate cannot vary".to_string()); - } - return Ok(( - coded, - CovariateKind::Categorical, - Some( - map.iter() - .map(|(value, number)| format!("{value}={number}")) - .collect::<Vec<_>>() - .join(", "), - ), - )); - } - - let mut levels: Vec<&str> = values - .iter() - .filter_map(|value| value.as_deref()) - .filter(|value| !value.is_empty()) - .collect(); - levels.sort_unstable(); - levels.dedup(); - if levels.len() < 2 { - return Err(format!( - "the attribute has {} usable level(s) among these samples ({}), so it cannot \ - be a covariate", - levels.len(), - if levels.is_empty() { - "no values at all".to_string() - } else { - format!("all '{}'", levels[0]) - } - )); - } - // All numeric? Use them as they are. - let numbers: Option<Vec<f64>> = levels - .iter() - .map(|level| level.trim().parse::<f64>().ok()) - .collect(); - if numbers.is_some() { - let coded = values - .iter() - .map(|value| { - value - .as_deref() - .map(str::trim) - .filter(|text| !text.is_empty()) - .and_then(|text| text.parse::<f64>().ok()) - }) - .collect(); - return Ok((coded, CovariateKind::Numerical, None)); - } - if levels.len() > MAX_AUTO_LEVELS { - return Err(format!( - "the attribute has {} distinct values, too many to code automatically (limit \ - {}); pass an explicit `map`", - levels.len(), - MAX_AUTO_LEVELS - )); - } - let mut mapping = std::collections::BTreeMap::new(); - for (index, level) in levels.iter().enumerate() { - mapping.insert((*level).to_string(), index as f64); - } - let coded = values - .iter() - .map(|value| { - value - .as_deref() - .filter(|text| !text.is_empty()) - .and_then(|text| mapping.get(text).copied()) - }) - .collect(); - Ok(( - coded, - CovariateKind::Categorical, - Some( - mapping - .iter() - .map(|(value, number)| format!("{value}={number}")) - .collect::<Vec<_>>() - .join(", "), - ), - )) -} - -/// Write the wrapper's `--covarstruct` file: one `name<TAB>kind` row per -/// covariate, the format `read.table(opt$covarstruct, sep="\t", -/// header=FALSE)` expects. -pub fn write_covarstruct_file(path: &Path, covariates: &[Covariate]) -> io::Result<String> { - let mut contents = String::new(); - for covariate in covariates { - contents.push_str(&format!( - "{}\t{}\n", - covariate.name, - covariate.kind.as_str() - )); - } - std::fs::write(path, &contents)?; - Ok(contents) -} - /// Write the phenotype file R/qtl's wrapper script expects: /// `read.table(..., na.strings = "x", header = TRUE)`, a `Sample` /// column plus one named trait column. @@ -702,11 +532,22 @@ mod tests { name: "Sex".to_string(), kind: CovariateKind::Categorical, values: vec![Some(0.0), Some(1.0)], + levels: vec![ + crate::covariates::Level { + name: "F".to_string(), + code: 0.0, + }, + crate::covariates::Level { + name: "M".to_string(), + code: 1.0, + }, + ], }, Covariate { name: "Age".to_string(), kind: CovariateKind::Numerical, values: vec![Some(183.0), None], + levels: Vec::new(), }, ]; let contents = write_pheno_file(&path, "trait", &samples, &covariates).unwrap(); @@ -722,59 +563,6 @@ mod tests { } #[test] - fn text_covariates_are_coded_and_numeric_ones_pass_through() { - let sex: Vec<Option<String>> = ["F", "M", "", "F"] - .iter() - .map(|value| Some(value.to_string())) - .collect(); - let (coded, kind, coding) = code_case_attribute(&sex, None).unwrap(); - assert_eq!(kind, CovariateKind::Categorical); - assert_eq!(coded, vec![Some(0.0), Some(1.0), None, Some(0.0)]); - assert_eq!(coding.as_deref(), Some("F=0, M=1")); - - let ages: Vec<Option<String>> = ["183", "200"] - .iter() - .map(|value| Some(value.to_string())) - .collect(); - let (coded, kind, _) = code_case_attribute(&ages, None).unwrap(); - assert_eq!(kind, CovariateKind::Numerical); - assert_eq!(coded, vec![Some(183.0), Some(200.0)]); - - // One level, or none, cannot be a covariate. - let constant: Vec<Option<String>> = vec![Some("Live".to_string()), Some("Live".to_string())]; - let error = code_case_attribute(&constant, None).unwrap_err(); - assert!(error.contains("cannot be a covariate"), "{error}"); - } - - #[test] - fn an_explicit_map_is_honoured_and_must_be_complete() { - let values: Vec<Option<String>> = ["HFD", "LFD"] - .iter() - .map(|value| Some(value.to_string())) - .collect(); - let mut map = std::collections::BTreeMap::new(); - map.insert("LFD".to_string(), 0.0); - map.insert("HFD".to_string(), 1.0); - let (coded, kind, coding) = code_case_attribute(&values, Some(&map)).unwrap(); - assert_eq!(kind, CovariateKind::Categorical); - assert_eq!(coded, vec![Some(1.0), Some(0.0)]); - assert_eq!(coding.as_deref(), Some("HFD=1, LFD=0")); - - map.remove("LFD"); - let error = code_case_attribute(&values, Some(&map)).unwrap_err(); - assert!(error.contains("'LFD'"), "{error}"); - } - - #[test] - fn too_many_levels_asks_for_an_explicit_map() { - let values: Vec<Option<String>> = (0..MAX_AUTO_LEVELS + 1) - .map(|index| Some(format!("level-{index}"))) - .collect(); - let error = code_case_attribute(&values, None).unwrap_err(); - assert!(error.contains("pass an explicit `map`"), "{error}"); - } - - #[test] fn parse_mapping_output_strips_quotes_from_name_and_chr_only() { let contents = "\"\",\"chr\",\"pos\",\"lod\"\n\"rs1\",\"1\",3.01,0.092\n\"rs2\",\"2\",5.01,0.013\n"; |
