about summary refs log tree commit diff
path: root/tests
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 15:39:12 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 15:39:12 +0000
commit336f18e1bc22cb330052dc1ff45d1d06073bfcc3 (patch)
tree4c029b46500edab067b35424b9e9ebbc7156221a /tests
parentfb41143226f7e0231f863a8226ba077c7415b71f (diff)
downloadgenenetwork4-336f18e1bc22cb330052dc1ff45d1d06073bfcc3.tar.gz
compute: fit covariates in GEMMA, and share them with R/qtl
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.

The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.

A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.

Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.

Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.

* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
  `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
  `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
  (moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
  file, `-c` after `-a` and before `-lmm`, and the notes about dropped
  individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
  values so the note can count complete cases; `RqtlCovariate` replaced by
  the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
  levels; inline codes; a dropped degenerate column) and an end-to-end
  `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
  `Status`, which also pins that `-c` is in the GWA command and not in the
  kinship one, and that the covariate changes the fit rather than reusing
  the plain run's cache. The gemma tests now take a lock: the gate answers
  a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
  points at the shared shape and explains `x` vs `NA`), docs/www/
  ai-instruction.md (both endpoints; what differs in `notes`).

Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.

272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
Diffstat (limited to 'tests')
-rw-r--r--tests/gemma.rs181
1 files changed, 181 insertions, 0 deletions
diff --git a/tests/gemma.rs b/tests/gemma.rs
index 4ded3ab..b1a145d 100644
--- a/tests/gemma.rs
+++ b/tests/gemma.rs
@@ -12,6 +12,19 @@ mod common;
 
 /// True only when gemma-wrapper is actually runnable. `-v` is cheap and
 /// does not touch the cache dir or invoke the real GEMMA binary.
+/// The tests that run a real GEMMA job take this lock first: the compute
+/// gate refuses a second job from the same caller with 409 instead of
+/// queueing it, and every test shares the one token.
+static COMPUTE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+/// Take [`COMPUTE_LOCK`], ignoring poisoning (one failing test must not
+/// fail the rest).
+fn serialised() -> std::sync::MutexGuard<'static, ()> {
+    COMPUTE_LOCK
+        .lock()
+        .unwrap_or_else(|poisoned| poisoned.into_inner())
+}
+
 fn gemma_wrapper_available() -> bool {
     std::process::Command::new("gemma-wrapper")
         .arg("-v")
@@ -52,6 +65,7 @@ fn bxd_trait_values() -> Vec<Option<f64>> {
 
 #[tokio::test]
 async fn compute_gemma_maps_a_qtl_from_the_bxd_fixtures() {
+    let _serial = serialised();
     if !gemma_wrapper_available() {
         eprintln!(
             "skipping compute_gemma_maps_a_qtl_from_the_bxd_fixtures: gemma-wrapper not on PATH"
@@ -451,3 +465,170 @@ async fn compute_gemma_rejects_a_run_id_already_in_use() {
 
     std::fs::remove_file(std::path::Path::new(&cache_dir).join(format!("{run_id}.log"))).unwrap();
 }
+
+/// GEMMA with a covariate: a case attribute of the group, dummy-expanded
+/// into a covariate file of its own. Needs gemma-wrapper, a database and
+/// the genotype files.
+///
+/// BXD, its classic `BXD.4` panel (3 811 markers, so this is a mechanism
+/// test rather than a mapping) and the `Status` case attribute, which has
+/// five levels among this trait's 34 strains -- so four dummy columns, and
+/// a check that a categorical covariate really is expanded rather than
+/// fitted as one ordinal number.
+#[tokio::test]
+async fn compute_gemma_fits_a_case_attribute_as_a_covariate() {
+    let _serial = serialised();
+    if !gemma_wrapper_available() {
+        eprintln!(
+            "skipping compute_gemma_fits_a_case_attribute_as_a_covariate: gemma-wrapper not on PATH"
+        );
+        return;
+    }
+    let Some(token) = common::auth_token().await else {
+        eprintln!(
+            "skipping compute_gemma_fits_a_case_attribute_as_a_covariate: no token (see docs/gn-auth.md)"
+        );
+        return;
+    };
+    if !common::db_available() {
+        eprintln!(
+            "skipping compute_gemma_fits_a_case_attribute_as_a_covariate: no usable database"
+        );
+        return;
+    }
+    let directory = genenetwork4::config::Config::load().genotype_files_dir;
+    if !std::path::Path::new(&directory).is_dir() {
+        eprintln!(
+            "skipping compute_gemma_fits_a_case_attribute_as_a_covariate: no genotype files"
+        );
+        return;
+    }
+
+    let response = common::get_with_token(
+        "GET /groups/BXD/traits/10001/values",
+        "/groups/BXD/traits/10001/values",
+        &token,
+    )
+    .await;
+    assert_eq!(response.status(), StatusCode::OK);
+    let bytes = response.into_body().collect().await.unwrap().to_bytes();
+    let samples: Vec<Value> = common::data(&bytes);
+
+    // Runs one request and hands back its status and body, so each step
+    // below reads as "post this, look at that".
+    let post = |samples: Vec<Value>, covariates: Value| {
+        let body = serde_json::json!({
+            "group": "BXD",
+            "genotype_version": "BXD.4",
+            "samples": samples,
+            "covariates": covariates,
+        });
+        let token = token.clone();
+        async move {
+            let response =
+                common::post_json_with_token("POST /compute/gemma", "/compute/gemma", &token, &body)
+                    .await;
+            let status = response.status();
+            let bytes = response.into_body().collect().await.unwrap().to_bytes();
+            (status, bytes)
+        }
+    };
+
+    // The same trait without covariates, for comparison.
+    let (status, bytes) = post(samples.clone(), serde_json::json!([])).await;
+    assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&bytes));
+    let plain: Value = serde_json::from_slice(&bytes).unwrap();
+
+    let (status, bytes) = post(
+        samples.clone(),
+        serde_json::json!([{"case_attribute": "Status"}]),
+    )
+    .await;
+    assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&bytes));
+    let with_status: Value = serde_json::from_slice(&bytes).unwrap();
+    assert!(!with_status["data"].as_array().unwrap().is_empty());
+
+    let notes: Vec<&str> = with_status["notes"]
+        .as_array()
+        .unwrap()
+        .iter()
+        .map(|note| note.as_str().unwrap())
+        .collect();
+    let covariate_note = notes
+        .iter()
+        .find(|note| note.starts_with("Covariate Status:"))
+        .expect("a note saying what the covariate was");
+    assert!(covariate_note.contains("categorical"), "{covariate_note}");
+    assert!(
+        covariate_note.contains("coded"),
+        "the coding is reported: {covariate_note}"
+    );
+    // A categorical covariate is expanded, never fitted as one ordinal
+    // number: several levels, one reference, a dummy for each of the rest.
+    assert!(
+        covariate_note.contains("dummy column(s) named Status.<level>"),
+        "a categorical covariate is dummy-expanded, not fitted as one number: {covariate_note}"
+    );
+    // A covariate only ever reaches the GWA step.
+    let kinship = notes
+        .iter()
+        .find(|note| note.starts_with("Kinship:"))
+        .expect("the kinship command line");
+    assert!(!kinship.contains(" -c "), "{kinship}");
+    let gwa = notes
+        .iter()
+        .find(|note| note.starts_with("GWA:"))
+        .expect("the GWA command line");
+    assert!(gwa.contains(" -c "), "{gwa}");
+    assert!(gwa.contains("covariates.txt"), "{gwa}");
+
+    // The covariate changed the fit: the wrapper hashes its file
+    // arguments, so this also proves the covariate file was part of the
+    // key rather than a cached no-covariate run being reused.
+    let peak_p = |envelope: &Value| {
+        let columns: Vec<&str> = envelope["meta"]["columns"]
+            .as_array()
+            .unwrap()
+            .iter()
+            .map(|c| c.as_str().unwrap())
+            .collect();
+        let index = columns.iter().position(|c| *c == "p_lrt").expect("p_lrt");
+        envelope["data"]
+            .as_array()
+            .unwrap()
+            .iter()
+            .filter_map(|row| row[index].as_f64())
+            .fold(f64::INFINITY, f64::min)
+    };
+    assert!(
+        (peak_p(&plain) - peak_p(&with_status)).abs() > 0.0,
+        "the covariate must change the fit, not reuse the plain run: {} vs {}",
+        peak_p(&plain),
+        peak_p(&with_status)
+    );
+
+    // Covariates are aligned by name, so explicit files cannot have them.
+    let body = serde_json::json!({
+        "genotype_file": test_data_path("BXD_geno.txt.gz"),
+        "snps_file": test_data_path("BXD_snps.txt"),
+        "trait_values": bxd_trait_values(),
+        "covariates": [{"case_attribute": "Epoch"}],
+    });
+    let response =
+        common::post_json_with_token("POST /compute/gemma (paths)", "/compute/gemma", &token, &body)
+            .await;
+    assert_eq!(response.status(), StatusCode::UNPROCESSABLE_ENTITY);
+    let bytes = response.into_body().collect().await.unwrap().to_bytes();
+    let error: Value = serde_json::from_slice(&bytes).unwrap();
+    assert_eq!(error["code"], "covariate_unavailable");
+
+    // An attribute the group does not define is refused, listing its own.
+    let (status, bytes) = post(samples, serde_json::json!([{"case_attribute": "Sex"}])).await;
+    assert_eq!(status, StatusCode::UNPROCESSABLE_ENTITY);
+    let error: Value = serde_json::from_slice(&bytes).unwrap();
+    assert_eq!(error["code"], "covariate_unavailable");
+    assert!(
+        error["detail"].as_str().unwrap().contains("Status"),
+        "the error lists what the group has: {error}"
+    );
+}