From 336f18e1bc22cb330052dc1ff45d1d06073bfcc3 Mon Sep 17 00:00:00 2001 From: Pjotr Date: Sun, 8 Nov 2026 15:39:12 +0000 Subject: compute: fit covariates in GEMMA, and share them with R/qtl `POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit. --- docs/www/ai-instruction.md | 48 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-) (limited to 'docs') diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index fcc5c4c..fe033d4 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -55,6 +55,25 @@ built from a working tree with uncommitted changes: compare the part before `-dirty`, and quote `source_digest` if you need to identify the exact build. +**A version change means refetch this document.** `/version` reports the +build the *server* is running, and this deployment is updated in place — +so a `git` value you have not seen before (or one that moved during your +session) is the signal that the page you are reading may be out of date. +When that happens, do not keep working from your copy: refresh it and +re-read it before the next call. + +```sh +git -C genenetwork4 pull --depth 1 # or re-run the clone above +less genenetwork4/docs/www/ai-instruction.md +``` + +Endpoints, parameters and defaults get added between revisions (covariate +support in `/compute/rqtl` and `/compute/gemma` arrived that way), and a +stale copy does not announce itself: the API simply accepts a request +without the new option, or answers with fields the old text does not +mention. Re-reading a file is cheap; a wrong conclusion from a stale +description is not. + ``` 1. You are given a token → by the GeneNetwork team, out of band 2. Send it on every request → Authorization: Bearer @@ -131,7 +150,10 @@ returns `git` (the commit the running binary was built from; a `-dirty` suffix means that build had uncommitted changes), `dirty`, `source_digest` (which identifies the build when a commit hash cannot), `database` (`reachable` or `unreachable`) and `date` (the *commit's* date, -not the build time). If `database` is `unreachable`, every data endpoint +not the build time). If `git` is not the revision you started your session +with, the server has been updated under you: refetch this document (see +"Get this document" above) and re-read it before going on. If `database` +is `unreachable`, every data endpoint will fail with `503 db_unavailable` or `500 database_error` and there is nothing you can do but report it — do not retry in a loop and do not conclude the resource does not exist. @@ -418,12 +440,14 @@ A mapping's response ends with what to do next: naming them, so a Manhattan plot needs no reshaping. **Covariates.** Add `covariates` to fit the study's own metadata — sex, -diet, batch — alongside the trait. The easiest form names a case -attribute (§3), and the server codes it. It is also the smallest: nothing -is posted, where an inline covariate sends a value per individual — for a -panel of thousands that is the difference between a few hundred bytes and -a multi-megabyte body, and bodies are capped at 64 MiB (`413`, in plain -text, if you pass it): +diet, batch — alongside the trait, in **both** `/compute/rqtl` and +`/compute/gemma`. The easiest form names a case attribute (§3), and the +server codes it. It is also the smallest: nothing is posted, where an +inline covariate sends a value per individual — for a panel of thousands +that is the difference between a few hundred bytes and a multi-megabyte +body, and bodies are capped at 64 MiB (`413`, in plain text, if you pass +it). Covariates need a `group` on the request: individuals are aligned by +name, which an explicit `genotype_file` does not carry (`422`): ```json "covariates": [{"case_attribute": "Sex"}] @@ -440,6 +464,16 @@ matters, or `{"name": …, "kind": "numerical", "values": […]}` when you already have numbers. An attribute the group does not define is a `422` listing the ones it has. +The two engines fit the same covariates differently, which changes what +you should expect in `notes`: R/qtl takes each categorical covariate as +one column and dummy-expands it itself, while GEMMA gets a covariate file +of its own with k‑1 dummy columns named after their levels +(`Status.Live`) — GEMMA has no factors — and a column that does not vary +among the individuals fitted is left out, with a note saying so (an +all-zero column is singular, and GEMMA stops rather than dropping the +term). Either way, covariates reach only the association step and not the +kinship, so adding one does not recompute the relatedness matrix. + **GEMMA has no permutation test.** Its `actions` therefore suggests the *R/qtl* permutation run on the same group and the same values, and says so; judge a GEMMA peak against the smallest `p_lrt` it reached, not -- cgit 1.4.1