diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-10 12:19:05 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-10 12:19:05 +0000 |
| commit | e0bbc56324cc7a6bd563f0dadd7163a93b8b4ad8 (patch) | |
| tree | 6e1fb728c0e149c0baf3f0caddd0b829b4454f0b /src | |
| parent | 8f4a9512d7fb4ac518341f23a4b062ccf80e2a7d (diff) | |
| download | genenetwork4-e0bbc56324cc7a6bd563f0dadd7163a93b8b4ad8.tar.gz | |
genotypes: call frequencies, marker names, and a location per marker
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
Diffstat (limited to 'src')
| -rw-r--r-- | src/frequencies.rs | 439 | ||||
| -rw-r--r-- | src/genotypes.rs | 41 | ||||
| -rw-r--r-- | src/lib.rs | 8 | ||||
| -rw-r--r-- | src/routes/frequencies.rs | 353 | ||||
| -rw-r--r-- | src/routes/genotypes.rs | 293 | ||||
| -rw-r--r-- | src/routes/mod.rs | 1 |
6 files changed, 1092 insertions, 43 deletions
diff --git a/src/frequencies.rs b/src/frequencies.rs new file mode 100644 index 0000000..971451f --- /dev/null +++ b/src/frequencies.rs @@ -0,0 +1,439 @@ +//! Genotype frequencies, counted from a `.geno` and reported as a map. +//! +//! A `.geno` cell is a genotype *call* (`B`, `D`, `H`, `A`–`H` for the DO's +//! founder alleles), or -- in a file that has been through a dosage or +//! probability pipeline -- a number. Calls are counted as they are; a column +//! of probabilities is binned, because 0.4187 and 0.4191 are the same +//! statement about a marker and listing them separately hides the shape of +//! the distribution. The default bin is 0.05 wide (twenty bins over [0, 1]), +//! and a file with only a handful of distinct numbers (`0`, `0.5`, `1` for a +//! biallelic dosage) is reported exactly: binning three values would lose the +//! very thing the caller is looking at. +//! +//! The genotypes themselves are counted and discarded, never returned -- the +//! same rule `/groups/{group}/genotypes` follows. What a caller gets is the +//! distribution, which is also what makes a panel's quality legible: a marker +//! that is 99 % missing, or calls that are not the codes its header declares, +//! shows up here and nowhere else in the API. +//! +//! Two costs are bounded on purpose. Cells are counted through an interner, so +//! a 100-million-cell panel costs one hash lookup per cell and one allocation +//! per *distinct* call, not per cell. And a file larger than [`MAX_SCAN_BYTES`] +//! is refused rather than read: `HSNIH-Palmer_true.geno` is 3.7 GB (134 927 +//! markers × 6 147 individuals), which is not a diagnosis anyone waits for. + +use std::collections::{BTreeMap, HashMap}; +use std::io::{BufRead, BufReader}; +use std::path::Path; + +use crate::genotypes::MAX_SCAN_BYTES; + +/// Cells that are not calls. `U` is GeneNetwork's "unknown", the rest are +/// what the files in the wild use for the same thing. +const MISSING: [&str; 9] = ["", "U", "N", "NA", "x", "X", "?", "-", "."]; + +/// How many distinct numeric values a column may have before it is treated +/// as probabilities and binned rather than listed exactly. +const EXACT_LIMIT: usize = 20; + +/// Why a tally could not be produced. +#[derive(Debug)] +pub enum FreqError { + /// No such file, no such marker in it, or no calls at all. + NotFound(String), + /// The file is bigger than [`MAX_SCAN_BYTES`]. + TooLarge(String), + /// The file could not be read. + Io(String), +} + +impl std::fmt::Display for FreqError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + FreqError::NotFound(what) | FreqError::TooLarge(what) | FreqError::Io(what) => { + write!(f, "{what}") + } + } + } +} + +/// The distribution of calls over the cells that carry one. +#[derive(Debug, Clone, PartialEq)] +pub struct Tally { + /// Call (or bin label) -> frequency, summing to 1 over `called`. + pub frequencies: BTreeMap<String, f64>, + /// Cells that carry a call. + pub called: u64, + /// Cells whose call is missing. + pub missing: u64, + /// Marker rows read (one, when a single marker was asked for). + pub markers: usize, + /// Bin width, when the calls were numeric probabilities and got binned. + pub bin_width: Option<f64>, +} + +/// Where the counted marker sits, when one was named. +#[derive(Debug, Clone, PartialEq)] +pub struct Counted { + pub tally: Tally, + pub chr: Option<String>, + pub cm: Option<f64>, + pub mb: Option<f64>, +} + +fn is_missing(cell: &str) -> bool { + MISSING.contains(&cell) +} + +/// Counts cells through an interner: one hash lookup per cell, one allocation +/// per distinct call. A probability-valued file switches to binning the moment +/// its distinct values pass [`EXACT_LIMIT`], so a dosages file cannot fill +/// memory with distinct floats. +struct Counter { + width: f64, + calls: Vec<String>, + counts: Vec<u64>, + index: HashMap<String, usize>, + bins: BTreeMap<usize, u64>, + binned: bool, +} + +impl Counter { + fn new(width: f64) -> Self { + Self { + width, + calls: Vec::new(), + counts: Vec::new(), + index: HashMap::new(), + bins: BTreeMap::new(), + binned: false, + } + } + + fn add(&mut self, cell: &str) { + if self.binned { + match self.bin_of(cell) { + Some(index) => *self.bins.entry(index).or_insert(0) += 1, + // A mixed file: a code among probabilities keeps its own key. + None => self.add_exact(cell), + } + return; + } + self.add_exact(cell); + if self.calls.len() > EXACT_LIMIT + && self + .calls + .iter() + .all(|call| call.parse::<f64>().map(|v| (0.0..=1.0).contains(&v)).unwrap_or(false)) + { + self.binned = true; + let calls = std::mem::take(&mut self.calls); + let counts = std::mem::take(&mut self.counts); + self.index.clear(); + for (call, count) in calls.into_iter().zip(counts) { + if let Some(index) = self.bin_of(&call) { + *self.bins.entry(index).or_insert(0) += count; + } + } + } + } + + fn add_exact(&mut self, cell: &str) { + match self.index.get(cell) { + Some(slot) => self.counts[*slot] += 1, + None => { + self.index.insert(cell.to_string(), self.calls.len()); + self.calls.push(cell.to_string()); + self.counts.push(1); + } + } + } + + /// The bin a value falls in. + fn bin_of(&self, call: &str) -> Option<usize> { + let value = call + .parse::<f64>() + .ok() + .filter(|value| (0.0..=1.0).contains(value))?; + let bins = ((1.0 / self.width).round() as usize).max(1); + Some(((value / self.width).floor() as usize).min(bins - 1)) + } + + fn total(&self) -> u64 { + self.counts.iter().sum::<u64>() + self.bins.values().sum::<u64>() + } + + fn finish(self, called: u64) -> (BTreeMap<String, f64>, Option<f64>) { + let mut frequencies: BTreeMap<String, f64> = BTreeMap::new(); + if self.binned { + for (index, count) in &self.bins { + let low = *index as f64 * self.width; + let high = (low + self.width).min(1.0); + frequencies.insert(format!("{low:.2}-{high:.2}"), proportion(*count, called)); + } + for (call, count) in self.calls.iter().zip(&self.counts) { + frequencies.insert(call.clone(), proportion(*count, called)); + } + return (frequencies, Some(self.width)); + } + let mut ordered: Vec<(&String, &u64)> = self.calls.iter().zip(&self.counts).collect(); + // Numeric calls sort by value; anything else alphabetically. + ordered.sort_by(|left, right| match (left.0.parse::<f64>(), right.0.parse::<f64>()) { + (Ok(a), Ok(b)) => a.partial_cmp(&b).unwrap_or(std::cmp::Ordering::Equal), + _ => left.0.cmp(right.0), + }); + for (call, count) in ordered { + frequencies.insert(call.clone(), proportion(*count, called)); + } + (frequencies, None) + } +} + +/// A frequency, rounded so that a client's sum does not wobble in the last +/// digit. +fn proportion(count: u64, total: u64) -> f64 { + ((count as f64 / total as f64) * 1e6).round() / 1e6 +} + +/// Count the calls in `geno`, over the whole panel or one marker. +pub fn tally(geno: &Path, only: Option<&str>, bin_width: Option<f64>) -> Result<Counted, FreqError> { + let metadata = std::fs::metadata(geno) + .map_err(|error| FreqError::NotFound(format!("{}: {error}", geno.display())))?; + if only.is_none() && metadata.len() > MAX_SCAN_BYTES { + return Err(FreqError::TooLarge(format!( + "{} is {} MB, past the {} MB a whole-panel tally reads; ask for one marker instead, \ + or pick a smaller version of the panel", + geno.display(), + metadata.len() / (1024 * 1024), + MAX_SCAN_BYTES / (1024 * 1024) + ))); + } + let file = std::fs::File::open(geno) + .map_err(|error| FreqError::Io(format!("{}: {error}", geno.display())))?; + let width = bin_width + .filter(|width| *width > 0.0 && *width <= 1.0) + .unwrap_or(0.05); + let mut counter = Counter::new(width); + let mut missing = 0u64; + let mut markers = 0usize; + let mut seen_header = false; + let mut start = 3usize; + let mut position: Option<(String, Option<f64>, Option<f64>)> = None; + let mut found = only.is_none(); + for line in BufReader::with_capacity(1 << 20, file).lines() { + let line = line.map_err(|error| FreqError::Io(format!("{}: {error}", geno.display())))?; + let line = line.trim_end_matches('\r'); + if line.trim().is_empty() || line.starts_with('#') || line.starts_with('@') { + continue; + } + let columns: Vec<&str> = line.split('\t').collect(); + if !seen_header { + seen_header = true; + start = if columns.len() > 3 && columns[3] == "Mb" { + 4 + } else { + 3 + }; + continue; + } + let name = columns.get(1).copied().unwrap_or("").trim(); + if let Some(wanted) = only { + if name != wanted { + continue; + } + found = true; + let number = |index: usize| -> Option<f64> { + columns + .get(index) + .and_then(|value| value.trim().parse().ok()) + }; + position = Some(( + columns.first().copied().unwrap_or("").trim().to_string(), + number(2), + number(3).or_else(|| number(2)), + )); + } + markers += 1; + for cell in columns.iter().skip(start) { + let cell = cell.trim(); + if is_missing(cell) { + missing += 1; + } else { + counter.add(cell); + } + } + if only.is_some() { + // One row was the whole request. + break; + } + } + if !seen_header { + return Err(FreqError::NotFound(format!( + "{}: no header line", + geno.display() + ))); + } + if !found { + return Err(FreqError::NotFound(format!( + "{}: no marker '{}'", + geno.display(), + only.unwrap_or_default() + ))); + } + let called = counter.total(); + if called == 0 { + return Err(FreqError::NotFound(format!( + "{}: no genotype calls{}", + geno.display(), + only.map(|name| format!(" for '{name}'")).unwrap_or_default() + ))); + } + let (frequencies, bin_width) = counter.finish(called); + let (chr, cm, mb) = match position { + Some((chr, cm, mb)) => (Some(chr), cm, mb), + None => (None, None, None), + }; + Ok(Counted { + tally: Tally { + frequencies, + called, + missing, + markers, + bin_width, + }, + chr, + cm, + mb, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn counter_of(calls: &[&str]) -> Counter { + let mut counter = Counter::new(0.05); + for call in calls { + counter.add(call); + } + counter + } + + #[test] + fn calls_are_reported_exactly_and_sum_to_one() { + let mut counter = Counter::new(0.05); + for (call, times) in [("B", 60), ("D", 38), ("H", 2)] { + for _ in 0..times { + counter.add(call); + } + } + let called = counter.total(); + assert_eq!(called, 100); + let (frequencies, width) = counter.finish(called); + assert_eq!(width, None, "three distinct calls are not binned"); + assert_eq!(frequencies["B"], 0.6); + assert_eq!(frequencies["D"], 0.38); + assert_eq!(frequencies["H"], 0.02); + let sum: f64 = frequencies.values().sum(); + assert!((sum - 1.0).abs() < 1e-6, "{sum}"); + } + + #[test] + fn numbers_are_ordered_by_value_not_by_string() { + let counter = counter_of(&[ + "0.5", "0.5", "0.5", "0.5", "0.5", "0.5", "0.5", "0.5", "0.5", "0.5", "0", "0", "0", + "0", "0", "0", "0", "0", "0", "0", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", + ]); + let called = counter.total(); + let (frequencies, width) = counter.finish(called); + assert_eq!(width, None, "a dosage with three values stays exact"); + let order: Vec<&String> = frequencies.keys().collect(); + assert_eq!(order, vec!["0", "0.5", "1"]); + } + + #[test] + fn probabilities_are_binned_at_a_twentieth() { + let values: Vec<String> = (0..25) + .map(|step| format!("{:.4}", step as f64 / 25.0)) + .collect(); + let mut counter = Counter::new(0.05); + for value in &values { + for _ in 0..4 { + counter.add(value); + } + } + counter.add("1"); + let called = counter.total(); + let (frequencies, width) = counter.finish(called); + assert_eq!(width, Some(0.05)); + assert_eq!(frequencies.len(), 20, "twenty bins over [0, 1]"); + assert!(frequencies.contains_key("0.00-0.05")); + assert!(frequencies.contains_key("0.95-1.00")); + let sum: f64 = frequencies.values().sum(); + // Each frequency is rounded to 6 decimal places, so twenty of them can + // drift by a few parts per million. + assert!((sum - 1.0).abs() < 1e-4, "{sum}"); + } + + #[test] + fn a_requested_bin_width_is_honoured() { + let values: Vec<String> = (0..40).map(|step| format!("{}", step as f64 / 40.0)).collect(); + let mut counter = Counter::new(0.1); + for value in &values { + counter.add(value); + } + let called = counter.total(); + let (frequencies, width) = counter.finish(called); + assert_eq!(width, Some(0.1)); + assert_eq!(frequencies.len(), 10, "ten bins, as asked"); + } + + #[test] + fn a_code_among_probabilities_keeps_its_own_key() { + let values: Vec<String> = (0..5000) + .map(|step| format!("{:.4}", step as f64 / 5000.0)) + .collect(); + let mut counter = Counter::new(0.05); + for value in &values { + counter.add(value); + } + assert!(counter.binned, "5000 distinct probabilities are binned"); + assert!(counter.calls.is_empty(), "and dropped from the exact side"); + counter.add("A"); + let called = counter.total(); + let (frequencies, width) = counter.finish(called); + assert_eq!(width, Some(0.05)); + assert!(frequencies.contains_key("0.00-0.05")); + assert!((frequencies["A"] - 1.0 / 5001.0).abs() < 1e-6, "{}", frequencies["A"]); + let sum: f64 = frequencies.values().sum(); + assert!((sum - 1.0).abs() < 1e-4, "{sum}"); + } + + #[test] + fn a_geno_panel_reports_its_calls() { + let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/test_data/rqtl_sample1.geno"); + let counted = tally(&path, None, None).unwrap(); + assert!(counted.tally.called > 0); + assert_eq!(counted.tally.markers, 6, "the fixture's six markers"); + let sum: f64 = counted.tally.frequencies.values().sum(); + assert!((sum - 1.0).abs() < 1e-6, "{sum}"); + assert!( + counted.tally.frequencies.keys().all(|call| call.len() <= 2), + "calls, not positions: {:?}", + counted.tally.frequencies.keys() + ); + } + + #[test] + fn one_marker_reads_only_its_own_row() { + let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/test_data/rqtl_sample1.geno"); + let counted = tally(&path, Some("rs6269442"), None).unwrap(); + assert_eq!(counted.tally.markers, 1); + assert_eq!(counted.chr.as_deref(), Some("1")); + assert!(counted.tally.called > 0); + assert!( + tally(&path, Some("nonesuch"), None).is_err(), + "an absent marker is an error, not an empty tally" + ); + } +} diff --git a/src/genotypes.rs b/src/genotypes.rs index 28d1266..46503ff 100644 --- a/src/genotypes.rs +++ b/src/genotypes.rs @@ -115,6 +115,47 @@ impl Resolved { } } +/// The largest `.geno` a whole-file read will touch: the markers listing, a +/// panel-wide tally. Above this the answer is another call (one marker, a +/// smaller version), not a wait -- `HSNIH-Palmer_true.geno` is 3.7 GB. +pub const MAX_SCAN_BYTES: u64 = 256 * 1024 * 1024; + +/// The `.geno` a request named when no group lists it: a bare version +/// (`BXD.4`), a `.geno` name, or a path relative to the genotype files +/// directory. +pub fn resolve_named(root: &Path, requested: &str) -> Result<PathBuf, GenotypeError> { + let name = requested.trim(); + if name.is_empty() || name.contains("..") { + return Err(GenotypeError::NotFound(format!( + "'{requested}' is not a genotype file name" + ))); + } + let genotype_dir = root.join("genotype"); + let mut candidates = Vec::new(); + if name.ends_with(".geno") { + candidates.push(genotype_dir.join(name)); + candidates.push(root.join(name)); + } else { + candidates.push(genotype_dir.join(format!("{name}.geno"))); + candidates.push(genotype_dir.join(name)); + candidates.push(root.join(name)); + } + for path in &candidates { + if path.is_file() { + return Ok(path.clone()); + } + } + Err(GenotypeError::NotFound(format!( + "no genotype file '{name}' under {}; looked for {}", + root.display(), + candidates + .iter() + .map(|path| path.display().to_string()) + .collect::<Vec<_>>() + .join(", ") + ))) +} + fn io_err(path: &Path, error: std::io::Error) -> GenotypeError { GenotypeError::Io(format!("{}: {error}", path.display())) } diff --git a/src/lib.rs b/src/lib.rs index bd7f964..19bdf8b 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -13,6 +13,7 @@ pub mod config; pub mod db; pub mod embedded; pub mod fitqtl; +pub mod frequencies; pub mod genotypes; pub mod normalize; mod plot; @@ -122,9 +123,16 @@ pub fn app() -> Router { .routes(routes!(routes::plot::plot_rqtl)) .routes(routes!(routes::plot::plot_gemma)) .routes(routes!(routes::phewas::group_phewas)) + .routes(routes!(routes::frequencies::group_version_frequencies)) + .routes(routes!(routes::frequencies::group_marker_frequencies)) + .routes(routes!(routes::frequencies::unlisted_frequencies)) + .routes(routes!(routes::frequencies::unlisted_marker_frequencies)) .routes(routes!(routes::genotypes::group_genotypes)) .routes(routes!(routes::genotypes::group_genotype_version)) .routes(routes!(routes::genotypes::group_genotype_markers)) + .routes(routes!(routes::genotypes::group_marker_location)) + .routes(routes!(routes::genotypes::unlisted_genotype_markers)) + .routes(routes!(routes::genotypes::unlisted_marker_location)) .routes(routes!(routes::gemma::compute_gemma)) .routes(routes!(routes::gemma::compute_gemma_ravanan)) .routes(routes!(routes::gemma::gemma_status)) diff --git a/src/routes/frequencies.rs b/src/routes/frequencies.rs new file mode 100644 index 0000000..40c3877 --- /dev/null +++ b/src/routes/frequencies.rs @@ -0,0 +1,353 @@ +//! Genotype call frequencies. +//! +//! `/groups/{group}/genotypes/{version}/frequencies` is the canonical shape: +//! a tally is a fact about one version of one group's panel, so it belongs +//! beside `/markers` in the version's own subtree, where the group's +//! permissions, the group-to-version check and the "available: …" message on +//! a wrong version all already apply. +//! +//! A file no group lists -- a panel in preparation, a fixture, the DO's `do/` +//! cross -- is served from `/groups/unlisted/genotypes/{name}/…`, a namespace +//! rather than a group: one prefix for the whole genotype surface, so an +//! agent that has learned where genotype facts live does not have to learn a +//! second spelling for the same tally. `unlisted` is not a group and +//! `/groups/unlisted` has no record; only its genotype subtree answers. +//! +//! "What is this panel made of?" is a question the listing cannot answer and +//! the compute endpoints do not: they report how many markers and +//! individuals a file has, never how they are called. A panel whose calls are +//! not the codes its header declares, or a marker that is 99 % missing, is +//! invisible until a scan comes back flat -- and then the search for the +//! cause starts at the wrong end. +//! +//! None of these serve genotypes. A tally is counted and discarded, so what +//! leaves the server is a distribution, not a copy of the matrix. + +use std::path::PathBuf; + +use axum::Json; +use axum::extract::{Path, Query}; +use axum::http::StatusCode; +use serde::Deserialize; + +use gnauth_macros::gnauth; + +use crate::auth::AuthToken; +use crate::config::Config; +use crate::error::{ApiError, Unauthorized}; +use crate::frequencies::{self, Counted, FreqError}; +use crate::genotypes; +use crate::response::{Ctx, Envelope, RequestInfo}; + +#[derive(Deserialize, utoipa::IntoParams)] +pub struct FrequencyParams { + /// Tally one marker instead of the whole file. The marker route is the + /// same call with the marker in the path. + pub marker: Option<String>, + /// Bin width for probability-valued calls (default 0.05: twenty bins + /// over [0, 1]). Ignored for call-valued and low-cardinality files. + pub bin: Option<f64>, +} + +#[derive(Deserialize, utoipa::IntoParams)] +pub struct MarkerFrequencyParams { + /// Bin width for probability-valued calls (default 0.05). + pub bin: Option<f64>, +} + +#[derive(Deserialize, utoipa::IntoParams)] +pub struct NamedFrequencyParams { + /// Tally one marker instead of the whole file. + pub marker: Option<String>, + /// Bin width for probability-valued calls (default 0.05). + pub bin: Option<f64>, +} + +#[utoipa::path(get, path = "/groups/{group}/genotypes/{version}/frequencies", + summary = "Genotype call frequencies for one genotype version", + description = "Counts the calls in the version's `.geno` and returns their frequencies, e.g. \ +{\"B\": 0.62, \"D\": 0.36, \"H\": 0.02}: what the panel is actually called as, which the \ +listing and the compute endpoints never say. The genotypes themselves are counted and \ +discarded. A file whose cells are probabilities is binned at 0.05 (twenty bins over [0, 1]); \ +pass ?bin= for another width. Use ?marker= for one marker.", + params( + ("group" = String, Path, description = "InbredSet name, e.g. BXD"), + ("version" = String, Path, description = "Version from /groups/{group}/genotypes"), + FrequencyParams + ), + responses( + (status = 200, description = "A call -> frequency map (counts in notes)", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such group, version, or marker", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn group_version_frequencies( + ctx: Ctx, + Path((group, version)): Path<(String, String)>, + Query(params): Query<FrequencyParams>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir.clone(); + let label = format!("{group} {version}"); + let path = resolve_version(root, group, version, rid.clone()).await?; + let envelope = count(ctx, path, params.marker, params.bin, label, rid).await?; + Ok(Json(envelope)) +} + +#[utoipa::path(get, path = "/groups/{group}/genotypes/{version}/markers/{marker}/frequencies", + summary = "Genotype call frequencies for one marker", + description = "The same tally as the version's `/frequencies`, for a single marker: what the \ +calls at one locus look like across the panel. This is how a caller sees whether a candidate \ +peak's marker is nearly all missing -- a flat LOD from a silent marker and a flat LOD from real \ +data look the same in a scan.", + params( + ("group" = String, Path, description = "InbredSet name, e.g. BXD"), + ("version" = String, Path, description = "Version from /groups/{group}/genotypes"), + ("marker" = String, Path, description = "Marker name as the genotype file spells it"), + MarkerFrequencyParams + ), + responses( + (status = 200, description = "A call -> frequency map for one marker", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such group, version, or marker", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn group_marker_frequencies( + ctx: Ctx, + Path((group, version, marker)): Path<(String, String, String)>, + Query(params): Query<MarkerFrequencyParams>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir.clone(); + let label = format!("{group} {version}"); + let path = resolve_version(root, group, version, rid.clone()).await?; + let envelope = count(ctx, path, Some(marker), params.bin, label, rid).await?; + Ok(Json(envelope)) +} + +#[utoipa::path(get, path = "/groups/unlisted/genotypes/{name}/frequencies", + summary = "Genotype call frequencies for a genotype file no group lists", + description = "The same tally as `/groups/{group}/genotypes/{version}/frequencies`, for a \ +file the listing does not know about: a panel in preparation, a fixture, the DO's `do/` cross. \ +A bare name is looked up as `genotype/{name}.geno`; a `.geno` name or a path relative to the \ +genotype files directory also resolves. Prefer the group-scoped route where a group exists -- it \ +validates the pair and says what versions do.", + params(("name" = String, Path, description = "Version (e.g. BXD.4), .geno name, or path relative to the genotype files directory"), NamedFrequencyParams), + responses( + (status = 200, description = "A call -> frequency map (counts in notes)", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such genotype file, or no such marker in it", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn unlisted_frequencies( + ctx: Ctx, + Path(name): Path<String>, + Query(params): Query<NamedFrequencyParams>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir.clone(); + let requested = name.clone(); + let path: PathBuf = tokio::task::spawn_blocking(move || { + genotypes::resolve_named(std::path::Path::new(&root), &requested) + }) + .await + .map_err(|error| internal(error.to_string(), rid.clone()))? + .map_err(|error| map_genotype_error(error, rid.clone()))?; + let envelope = count(ctx, path, params.marker, params.bin, name, rid).await?; + Ok(Json(envelope)) +} + +/// `/genotypes/marker/{marker}?genotype=…`: the marker route for a named file, +/// kept because a file outside the listing has no group-scoped path. +#[utoipa::path(get, path = "/groups/unlisted/genotypes/{name}/markers/{marker}/frequencies", + summary = "One marker's call frequencies in a genotype file no group lists", + description = "The same tally as `/groups/{group}/genotypes/{version}/markers/{marker}/frequencies`, \ +for a file no group lists; `{name}` is the version, `.geno` name, or path relative to the \ +genotype files directory.", + params( + ("name" = String, Path, description = "Version, .geno name, or path relative to the genotype files directory"), + ("marker" = String, Path, description = "Marker name as the genotype file spells it"), + MarkerFrequencyParams + ), + responses( + (status = 200, description = "A call -> frequency map for one marker", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such genotype file, or no such marker in it", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn unlisted_marker_frequencies( + ctx: Ctx, + Path((name, marker)): Path<(String, String)>, + Query(params): Query<MarkerFrequencyParams>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir.clone(); + let requested = name.clone(); + let path: PathBuf = tokio::task::spawn_blocking(move || { + genotypes::resolve_named(std::path::Path::new(&root), &requested) + }) + .await + .map_err(|error| internal(error.to_string(), rid.clone()))? + .map_err(|error| map_genotype_error(error, rid.clone()))?; + let envelope = count(ctx, path, Some(marker), params.bin, name, rid).await?; + Ok(Json(envelope)) +} + +/// Resolve `group`+`version` the way every other group endpoint does, so a +/// wrong version is reported with the ones that exist. +async fn resolve_version( + root: String, + group: String, + version: String, + rid: RequestInfo, +) -> Result<PathBuf, ApiError> { + tokio::task::spawn_blocking(move || { + genotypes::resolve(std::path::Path::new(&root), &group, Some(&version)) + .map(|resolved| resolved.geno) + }) + .await + .map_err(|error| internal(error.to_string(), rid.clone()))? + .map_err(|error| map_genotype_error(error, rid)) +} + +/// Count, off the async runtime: a large panel is seconds of CSV. +async fn count( + ctx: Ctx, + path: PathBuf, + marker: Option<String>, + bin: Option<f64>, + label: String, + rid: RequestInfo, +) -> Result<Envelope<serde_json::Value>, ApiError> { + let counted = tokio::task::spawn_blocking({ + let marker = marker.clone(); + move || frequencies::tally(&path, marker.as_deref(), bin) + }) + .await + .map_err(|error| internal(error.to_string(), rid.clone()))? + .map_err(|error| map_error(error, rid.clone()))?; + Ok(envelope(ctx, counted, &label, marker.as_deref())) +} + +/// The response: the frequency map as `data`, the counts that produced it in +/// `notes`. +fn envelope( + ctx: Ctx, + counted: Counted, + file: &str, + marker: Option<&str>, +) -> Envelope<serde_json::Value> { + let tally = counted.tally; + let cells = tally.called + tally.missing; + let missing_share = if cells == 0 { + 0.0 + } else { + ((tally.missing as f64 / cells as f64) * 1e4).round() / 100.0 + }; + let mut envelope = ctx.envelope(serde_json::json!(tally.frequencies)); + envelope.meta.num_rows = Some(tally.frequencies.len()); + envelope = match marker { + Some(marker) => envelope.with_note(&format!( + "Marker {marker}{} in {file}: the calls across the panel, as frequencies", + match (counted.chr.as_deref(), counted.mb) { + (Some(chr), Some(mb)) => format!(" (chr{chr}, {mb:.2} Mb)"), + (Some(chr), None) => format!(" (chr{chr})"), + _ => String::new(), + } + )), + None => envelope.with_note(&format!( + "{file}: {} marker(s), {cells} genotype cell(s)", + tally.markers + )), + }; + envelope = envelope.with_note(&format!( + "{} call(s) counted, {} missing ({missing_share} %); frequencies are over the calls that \ + are present, so they sum to 1", + tally.called, tally.missing, + )); + if let Some(width) = tally.bin_width { + envelope = envelope.with_note(&format!( + "The calls are probabilities, so they are binned {width:.2} wide (keys are the bin's \ + edges, e.g. \"0.05-0.10\"); pass ?bin= for another width" + )); + } else if marker.is_none() { + envelope = envelope.with_note( + "Calls are reported exactly, as the file spells them; missing codes (U, N, NA, an \ + empty cell) are counted as missing, not as a call", + ); + } + envelope.with_note( + "The genotypes themselves are counted and discarded: this endpoint reports a \ + distribution, not the matrix.", + ) +} + +fn internal(message: String, rid: RequestInfo) -> ApiError { + ApiError::new( + StatusCode::INTERNAL_SERVER_ERROR, + "internal_error", + "Internal error", + message, + "genotype_frequencies", + ) + .with_request_id(rid) +} + +fn map_error(error: FreqError, rid: RequestInfo) -> ApiError { + match error { + FreqError::NotFound(message) => ApiError::new( + StatusCode::NOT_FOUND, + "genotype_not_found", + "Genotype file or marker not found", + message, + "genotype_frequencies", + ) + .with_request_id(rid), + FreqError::TooLarge(message) => ApiError::new( + StatusCode::PAYLOAD_TOO_LARGE, + "genotype_too_large", + "Panel too large to tally whole", + message, + "genotype_frequencies", + ) + .with_request_id(rid), + FreqError::Io(message) => ApiError::new( + StatusCode::INTERNAL_SERVER_ERROR, + "genotype_unreadable", + "Genotype file could not be read", + message, + "genotype_frequencies", + ) + .with_request_id(rid), + } +} + +/// `genotypes::resolve` failures (group scoping) keep their own codes. +fn map_genotype_error(error: genotypes::GenotypeError, rid: RequestInfo) -> ApiError { + match error { + genotypes::GenotypeError::NotFound(message) => ApiError::new( + StatusCode::NOT_FOUND, + "genotype_not_found", + "Genotype file or marker not found", + message, + "genotype_frequencies", + ) + .with_request_id(rid), + genotypes::GenotypeError::Io(message) => ApiError::new( + StatusCode::INTERNAL_SERVER_ERROR, + "genotype_unreadable", + "Genotype file could not be read", + message, + "genotype_frequencies", + ) + .with_request_id(rid), + } +} diff --git a/src/routes/genotypes.rs b/src/routes/genotypes.rs index 7f632bd..c81abc2 100644 --- a/src/routes/genotypes.rs +++ b/src/routes/genotypes.rs @@ -281,18 +281,21 @@ pub async fn group_genotype_version( } #[utoipa::path(get, path = "/groups/{group}/genotypes/{version}/markers", - summary = "The markers one genotype version carries, paged", - description = "Marker names and positions (chr, cM, Mb) only: enough to frame a mapping \ -result, not the genotype matrix. Filter with ?chr=1 and page with ?page/?limit.", + summary = "The marker names one genotype version carries, paged", + description = "The names only -- the list a caller iterates. Positions are a separate call, \ +`/groups/{group}/genotypes/{version}/markers/{marker}/location`, so a page of names stays small \ +and a position lookup is one marker rather than a table. Filter with ?chr=1 and page with \ +?page/?limit.", params( ("group" = String, Path, description = "InbredSet name, e.g. BXD"), ("version" = String, Path, description = "Version from /groups/{group}/genotypes"), MarkerParams ), responses( - (status = 200, description = "A page of markers, positional rows (meta.columns names them)", body = Envelope<serde_json::Value>), + (status = 200, description = "A page of marker names, positional rows (meta.columns names them)", body = Envelope<serde_json::Value>), Unauthorized, (status = 404, description = "No such group or version", body = ApiError), + (status = 413, description = "The genotype file is too large to read whole", body = ApiError), (status = 422, description = "limit out of range", body = ApiError), (status = 500, description = "The genotype file could not be read", body = ApiError) ) @@ -305,34 +308,163 @@ pub async fn group_genotype_markers( _token: AuthToken, ) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { let rid = ctx.request_info(); - let limit = params.limit.unwrap_or(100).clamp(1, 10_000) as usize; - let page = params.page.unwrap_or(1).max(1) as usize; - let chr = params.chr.clone(); - let directory = Config::load().genotype_files_dir; let wanted_group = group.clone(); let wanted_version = version.clone(); - let markers = tokio::task::spawn_blocking(move || { - let resolved = genotypes::resolve( + let geno = tokio::task::spawn_blocking(move || { + genotypes::resolve( std::path::Path::new(&directory), &wanted_group, Some(&wanted_version), - )?; - genotypes::file_meta(&resolved.geno).map(|meta| meta.markers) + ) + .map(|resolved| resolved.geno) }) .await - .map_err(|e| { - ApiError::new( - axum::http::StatusCode::INTERNAL_SERVER_ERROR, - "internal_error", - "Internal error", - e.to_string(), - "group_genotype_markers", + .map_err(|error| internal_error(error.to_string(), "group_genotype_markers", rid.clone()))? + .map_err(|error| genotype_error_to_api_error(error, "group_genotype_markers", rid.clone()))?; + let prefix = format!( + "/groups/{}/genotypes/{}/markers", + path_segment(&group), + path_segment(&version) + ); + Ok(Json( + marker_names(ctx, geno, ¶ms, &prefix, rid).await?, + )) +} + +#[utoipa::path(get, path = "/groups/{group}/genotypes/{version}/markers/{marker}/location", + summary = "One marker's position, as named fields", + description = "`name`, `chr`, `cM` and `Mb` for one marker of this version, as an object \ +rather than a row: the companion to the marker-name list, and the same shape as the \ +frequencies endpoints' answers.", + params( + ("group" = String, Path, description = "InbredSet name, e.g. BXD"), + ("version" = String, Path, description = "Version from /groups/{group}/genotypes"), + ("marker" = String, Path, description = "Marker name as the genotype file spells it") + ), + responses( + (status = 200, description = "The marker's position", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such group, version or marker", body = ApiError), + (status = 413, description = "The genotype file is too large to read whole", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn group_marker_location( + ctx: Ctx, + Path((group, version, marker)): Path<(String, String, String)>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let directory = Config::load().genotype_files_dir; + let wanted_group = group.clone(); + let wanted_version = version.clone(); + let geno = tokio::task::spawn_blocking(move || { + genotypes::resolve( + std::path::Path::new(&directory), + &wanted_group, + Some(&wanted_version), ) - .with_request_id(rid.clone()) - })? - .map_err(|e| genotype_error_to_api_error(e, "group_genotype_markers", rid.clone()))?; + .map(|resolved| resolved.geno) + }) + .await + .map_err(|error| internal_error(error.to_string(), "group_marker_location", rid.clone()))? + .map_err(|error| genotype_error_to_api_error(error, "group_marker_location", rid.clone()))?; + let prefix = format!( + "/groups/{}/genotypes/{}/markers", + path_segment(&group), + path_segment(&version) + ); + Ok(Json(marker_location(ctx, geno, &marker, &prefix, rid).await?)) +} +#[utoipa::path(get, path = "/groups/unlisted/genotypes/{name}/markers", + summary = "The marker names in a genotype file no group lists, paged", + description = "The names only, for a file outside the group listing (a panel in \ +preparation, a fixture, the DO's `do/` cross). `{name}` is the version, `.geno` name, or a path \ +relative to the genotype files directory.", + params( + ("name" = String, Path, description = "Version, .geno name, or path relative to the genotype files directory"), + MarkerParams + ), + responses( + (status = 200, description = "A page of marker names, positional rows (meta.columns names them)", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such genotype file", body = ApiError), + (status = 413, description = "The genotype file is too large to read whole", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn unlisted_genotype_markers( + ctx: Ctx, + Path(name): Path<String>, + Query(params): Query<MarkerParams>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir; + let geno = tokio::task::spawn_blocking({ + let name = name.clone(); + move || genotypes::resolve_named(std::path::Path::new(&root), &name) + }) + .await + .map_err(|error| internal_error(error.to_string(), "unlisted_genotype_markers", rid.clone()))? + .map_err(|error| { + genotype_error_to_api_error(error, "unlisted_genotype_markers", rid.clone()) + })?; + let prefix = format!("/groups/unlisted/genotypes/{}/markers", path_segment(&name)); + Ok(Json( + marker_names(ctx, geno, ¶ms, &prefix, rid).await?, + )) +} + +#[utoipa::path(get, path = "/groups/unlisted/genotypes/{name}/markers/{marker}/location", + summary = "One marker's position in a genotype file no group lists", + description = "`name`, `chr`, `cM` and `Mb` for one marker of a file outside the group \ +listing.", + params( + ("name" = String, Path, description = "Version, .geno name, or path relative to the genotype files directory"), + ("marker" = String, Path, description = "Marker name as the genotype file spells it") + ), + responses( + (status = 200, description = "The marker's position", body = Envelope<serde_json::Value>), + Unauthorized, + (status = 404, description = "No such genotype file or marker", body = ApiError), + (status = 413, description = "The genotype file is too large to read whole", body = ApiError) + ) +)] +#[gnauth(token_only)] +pub async fn unlisted_marker_location( + ctx: Ctx, + Path((name, marker)): Path<(String, String)>, + _token: AuthToken, +) -> Result<Json<Envelope<serde_json::Value>>, ApiError> { + let rid = ctx.request_info(); + let root = Config::load().genotype_files_dir; + let geno = tokio::task::spawn_blocking({ + let name = name.clone(); + move || genotypes::resolve_named(std::path::Path::new(&root), &name) + }) + .await + .map_err(|error| internal_error(error.to_string(), "unlisted_marker_location", rid.clone()))? + .map_err(|error| genotype_error_to_api_error(error, "unlisted_marker_location", rid.clone()))?; + let prefix = format!("/groups/unlisted/genotypes/{}/markers", path_segment(&name)); + Ok(Json(marker_location(ctx, geno, &marker, &prefix, rid).await?)) +} + +/// Read a `.geno`'s markers and answer with one page of names. +async fn marker_names( + ctx: Ctx, + geno: std::path::PathBuf, + params: &MarkerParams, + prefix: &str, + rid: crate::response::RequestInfo, +) -> Result<Envelope<serde_json::Value>, ApiError> { + let limit = params.limit.unwrap_or(100).clamp(1, 10_000) as usize; + let page = params.page.unwrap_or(1).max(1) as usize; + let chr = params.chr.clone(); + guard_size(&geno, "markers", rid.clone())?; + let markers = read_markers(geno, "markers", rid.clone()).await?; let filtered: Vec<genotypes::Marker> = markers .into_iter() .filter(|marker| chr.as_deref().is_none_or(|wanted| marker.chr == wanted)) @@ -342,19 +474,14 @@ pub async fn group_genotype_markers( .into_iter() .skip((page - 1) * limit) .take(limit) - .map(|marker| serde_json::json!([marker.name, marker.chr, marker.cm, marker.mb])) + .map(|marker| serde_json::json!([marker.name])) .collect(); let num_rows = rows.len(); - let mut envelope = ctx .envelope(serde_json::Value::Array(rows)) - .with_columns( - ["name", "chr", "cM", "Mb"].iter().map(|s| s.to_string()).collect(), - num_rows, - ) + .with_columns(vec!["name".to_string()], num_rows) .with_note( - "Markers only: name, chromosome, cM and Mb from the genotype file. The genotypes \ - themselves are not served.", + "Marker names as the genotype file spells them. A position is one call per marker: append /{marker}/location. The genotypes themselves are never served.", ); if let Some(chr) = &chr { envelope = envelope.with_note(&format!( @@ -365,27 +492,107 @@ pub async fn group_genotype_markers( if page < pages { envelope = envelope.with_next( &format!( - "/groups/{}/genotypes/{}/markers?page={}&limit={}{}", - path_segment(&group), - path_segment(&version), + "{prefix}?page={}&limit={}{}", page + 1, limit, chr.as_deref() .map(|c| format!("&chr={}", path_segment(c))) .unwrap_or_default() ), - "The next page of markers", + "The next page of marker names", ); } - envelope = envelope.with_next( - &format!( - "/groups/{}/genotypes/{}", - path_segment(&group), - path_segment(&version) - ), - "This version's metadata (labels, individuals, chromosomes)", - ); - Ok(Json(envelope)) + Ok(envelope) +} + +/// One marker's position, as named fields. +async fn marker_location( + ctx: Ctx, + geno: std::path::PathBuf, + marker: &str, + prefix: &str, + rid: crate::response::RequestInfo, +) -> Result<Envelope<serde_json::Value>, ApiError> { + guard_size(&geno, "location", rid.clone())?; + let markers = read_markers(geno, "location", rid.clone()).await?; + let found = markers + .into_iter() + .find(|candidate| candidate.name == marker) + .ok_or_else(|| { + ApiError::new( + axum::http::StatusCode::NOT_FOUND, + "marker_not_found", + "Marker not found", + format!("no marker '{marker}' in this genotype file"), + "marker_location", + ) + .with_request_id(rid.clone()) + })?; + Ok(ctx + .envelope(serde_json::json!({ + "name": found.name, + "chr": found.chr, + "cM": found.cm, + "Mb": found.mb, + })) + .with_note( + "Positions come from the genotype file itself: cM from its genetic map, Mb from its physical one, either of which may be absent.", + ) + .with_next(prefix, "The marker names this file carries")) +} + +/// A whole-file read needs a size guard, or a 3.7 GB panel turns a listing into +/// a ten-minute request. +fn guard_size( + geno: &std::path::Path, + operation: &'static str, + rid: crate::response::RequestInfo, +) -> Result<(), ApiError> { + let size = std::fs::metadata(geno) + .map_err(|error| internal_error(format!("{}: {error}", geno.display()), operation, rid.clone()))? + .len(); + if size > genotypes::MAX_SCAN_BYTES { + return Err(ApiError::new( + axum::http::StatusCode::PAYLOAD_TOO_LARGE, + "genotype_too_large", + "Panel too large to read whole", + format!( + "{} is {} MB, past the {} MB a listing reads; ask for one marker's location, or pick a smaller version", + geno.display(), + size / (1024 * 1024), + genotypes::MAX_SCAN_BYTES / (1024 * 1024) + ), + operation, + ) + .with_request_id(rid)); + } + Ok(()) +} + +async fn read_markers( + geno: std::path::PathBuf, + operation: &'static str, + rid: crate::response::RequestInfo, +) -> Result<Vec<genotypes::Marker>, ApiError> { + tokio::task::spawn_blocking(move || genotypes::file_meta(&geno).map(|meta| meta.markers)) + .await + .map_err(|error| internal_error(error.to_string(), operation, rid.clone()))? + .map_err(|error| genotype_error_to_api_error(error, operation, rid)) +} + +fn internal_error( + message: String, + operation: &'static str, + rid: crate::response::RequestInfo, +) -> ApiError { + ApiError::new( + axum::http::StatusCode::INTERNAL_SERVER_ERROR, + "internal_error", + "Internal error", + message, + operation, + ) + .with_request_id(rid) } /// Resolve a version and read its metadata; shared by the detail handler. diff --git a/src/routes/mod.rs b/src/routes/mod.rs index 87486e0..c601f2d 100644 --- a/src/routes/mod.rs +++ b/src/routes/mod.rs @@ -9,6 +9,7 @@ pub mod datasets; pub mod dev; pub mod fitqtl; pub mod gemma; +pub mod frequencies; pub mod genotypes; pub mod groups; pub mod mapping_methods; |
