# genenetwork4 A Rust/Axum-based API server for GeneNetwork. ## Endpoints | Method | Path | Description | |--------|---------------------------------------------------|-----------------------------------| | GET | `/` | Greeting. | | GET | `/version` | Build info (hash, date). | | GET | `/species` | All species. | | GET | `/species/{species}` | One species by name or id. | | GET | `/species/{species}/datasets` | ProbeSet datasets in a species. | | GET | `/groups` | All groups (filterable). | | GET | `/mapping-methods` | Mapping tools offered. | | GET | `/accessions/{accession}` | Resolve a GN accession number. | | GET | `/groups/{group}` | One group, by name or Id. | | GET | `/groups/{group}/parents-and-f1s` | The family's parental design. | | GET | `/groups/{group}/strains` | Strains/samples in a group. | | GET | `/groups/{group}/case-attributes` | Case-attribute names and values. | | GET | `/groups/{group}/traits` | Published (phenotype) traits. | | GET | `/groups/{group}/traits/{trait_id}` | A published trait's metadata. | | GET | `/groups/{group}/traits/{trait_id}/values` | A published trait's values. | | GET | `/groups/{group}/datasets` | ProbeSet datasets in a group. | | GET | `/publications/{pubmed_id}/traits` | Traits a publication reports. | | GET | `/datasets/{dataset}` | One expression dataset. | | GET | `/datasets/{dataset}/probesets` | Expression traits in a dataset. | | GET | `/datasets/{dataset}/probesets/{probeset}` | An expression trait's metadata. | | GET | `/datasets/{dataset}/probesets/{probeset}/values` | An expression trait's values. | | GET | `/traits/{dataset}` | Every trait in a dataset. | | GET | `/traits/{dataset}/{trait}` | One trait's metadata. | | GET | `/traits/{dataset}/{trait}/values` | One trait's values. | | POST | `/compute/correlation` | Correlate a trait (stub; see [docs/correlations.md](docs/correlations.md)). | | POST | `/compute/gemma` | Map a QTL with GEMMA (see below). | | GET | `/compute/gemma/status/{run_id}` | Poll a gemma run's progress. | | POST | `/compute/gemma/ravanan` | Map a QTL via ravanan (see below).| | POST | `/compute/rqtl` | Map a QTL with R/qtl (see below). | | POST | `/compute/bnw` | Learn a Bayesian network (BNW). | | POST | `/compute/bnw/features` | Score each module as a trait. | | GET | `/compute/bnw/examples[/{name}]` | BNW's worked examples, ready to post. | | POST | `/plot/rqtl` | The R/qtl Manhattan plot, as SVG. | | POST | `/plot/gemma` | The GEMMA Manhattan plot, as SVG. | | POST | `/plot/bnw` | The BNW network, as SVG. | | GET | `/compute/rqtl/status/{run_id}` | Poll an rqtl run's progress. | | POST | `/get-test-token` | Get a test bearer token (local). | | GET | `/tokens` | List short API tokens (local). | | POST | `/tokens` | Mint a short API token (local). | | DELETE | `/tokens/{id}` | Revoke a short API token (local). | | GET | `/tokens/who` | Describe the presented credential (local). | | GET | `/search` | Search traits (free-text). | | GET | `/openapi.json` | OpenAPI 3.1 document. | Every endpoint above requires a valid gn-auth bearer token (`Authorization: Bearer `) except `/`, `/version`, `/openapi.json` and `/get-test-token` itself -- see docs/gn-auth.md and the "Development" section below (`POST /get-test-token`) for how to get one for local testing. **Short API tokens.** Beside gn-auth's JWT there is a second credential format for agents and scripts: `gn4ai_` + 64-char opaque tokens, stored as SHA-256 hashes in one SQLite database (`tokens.sqlite`, `GN4_API_TOKENS_DB`, chmod 0600, gitignored). Nothing is stored *in* the token, so revocation is deleting a row and takes effect on the next request; a token in a log line reveals nothing; and if the registry is lost every short token stops working. Mint, list, revoke and introspect with `scripts/gn4-token.py create|list|revoke|who`; the token is printed once and never again. Administration lives at `/tokens`, `/tokens/{id}` and `/tokens/who`, which **answer only a direct caller on this machine** (TCP peer address, plus a refusal for anything claiming to be proxied) -- minting credentials is not reachable from the network, and there is no switch to change that. A short token reads **public data only**: it works everywhere a token suffices, and on `#[gnauth(secured)]` endpoints for public rows; a confidential dataset answers 403 telling the caller to use a JWT, since gn-auth cannot validate an opaque token. Send it as `Authorization: Bearer gn4ai_...` and never in a query string, which would land in nginx's log and in `next` links. **What a request looks like in the log.** One JSON line per request on stderr (`GN4_ACCESS_LOG`), carrying the credentials, never the credential: ```json {"event":"request","ts":"2026-11-14T14:02:11Z","method":"POST", "path":"/compute/gemma/ravanan","status":200,"took_ms":912, "request_id":"…","user":"zachs","scheme":"opaque","token":"agent-ravanan-3"} ``` `user` is the gn-auth user id, so the line already answers *who*; `scheme` and `token` say *with which credential* (absent for a JWT, which has no registry handle). `scripts/gn4-log.py` reads such a log and shows it as a table with the handle resolved against `tokens.sqlite` — the subject as the registry has it now, plus the token's status *today* (`[revoked]`, `[expired]`, `[unknown]`) and its note, which is how "an agent was working at 14:02" is told apart from "an agent was working at 14:02 with a credential revoked since". The path is logged without its query string, and no secret or hash is ever written. A missing or invalid token is a 401; which further, resource-level authorization (if any) an endpoint checks once a token is valid is a separate, per-endpoint matter covered in each entry below. Every success response is wrapped in one envelope (see [Success responses](#success-responses) below), except `/` (a plain-text greeting) and `/openapi.json`. The example responses below are real ones, with long lists and strings abbreviated by `...`, and `took_ms` and `request_id` shortened to plausible values. Details, including example responses, below. ### Following the API without memorising it Every success envelope carries `next`: the calls that lead on from what you just read. It is deliberately exhaustive rather than minimal -- a genotype version leads to its frequencies, its marker names and its scans; a tally leads to the markers and to one marker's location and tally; a marker leads to the PheWAS at it; a trait leads to its values, its genotypes and its group; `/version` leads to the listings. A caller that knows only `/version` can walk to anything. A scan is the case that matters most: its reply points at the *class counts* behind its own peak (`.../markers/{peak}/classes?trait={trait_id}`), because "LOD 2.93" reads the same whether it rests on two animals or a hundred, and `/compute/qtl2` says in `notes` that a `riself` panel drops heterozygous calls before the scan while GEMMA keeps them at dosage 0.5. `next` is a promise that following a link is a `GET` that returns 200, which is what makes blind following safe. So a link that would need a parameter the server cannot fill is *not* offered (`/groups/{g}/phewas` without a locus is a 422, and is left out); a template such as `.../markers/{marker}/location` carries a placeholder for the caller to complete; and calls that are not `GET`s (the `POST`s that run a scan) live in `actions`, never in `next`. A response never suggests itself. `tests/links.rs` walks the graph from a dozen seeds and fails if any link does not answer 200. The convention that follows from this: **a reply always offers a next step, and an assistant relaying it should name one** — a single sentence after any job, taken from that reply's `next`/`actions`, unless the user asks for the result alone. The server's part of that bargain is that the suggestion is always *reachable*; the assistant's part is that it is one sentence and it is the link the reply actually carries. See [docs/www/SKILL.md](docs/www/SKILL.md#always-offer-the-next-step). - `GET /` - returns `Hello, world!`. - `GET /version` - returns a JSON record describing the running build: ```json { "data": { "name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0, "release": 1, "date": "20261107", "git": "9fa0303", "dirty": false, "source_digest": "2f7f77ed4ace92c1", "branch": "main", "database": "reachable" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [] } ``` `date` and `git` are the HEAD commit's date and short hash, embedded at build time (see `build.rs`); they describe the commit the binary was built from, not the time it happened to be compiled. `git` carries a `-dirty` suffix when that build had uncommitted changes -- so a binary can never claim to be a commit its code does not match -- and `source_digest` is a hash over the compiled source files, which tells two builds of the same commit apart where a commit hash cannot. `branch` is the branch HEAD was on, or `HEAD` when detached. `database` is `reachable` or `unreachable`, checked at most every ten seconds (`db::health`): a service that answers `/version` while every data endpoint fails is exactly the state that went unnoticed for twenty minutes on 2026-11-07, and this field makes it visible. `build.rs` declares `.git/HEAD`, the ref it points at and `.git/index` as rerun triggers, so a rebuild after a commit re-stamps rather than reusing the old hash; committing without rebuilding does not move the stamp, which is the point. - `GET /species` - returns a JSON array of all species in the `Species` table, mirroring GeneNetwork3's `gn3.db.species.get_all_species`: ```json { "data": [ {"name": "human", "menu_name": "Human (Homo sapiens, hg19)", "family": "Vertebrates"}, {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species= to list one species' groups"} ] } ``` Needs a live MariaDB connection, configured via the `SQL_URI` environment variable (same config key GeneNetwork3 uses), e.g. `mysql://user:password@host/db_webqtl`. Defaults to `mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl` if unset. - `GET /species/{species}` - returns every column of the matching row of the `Species` table, matched against `Name`, `FullName` or `SpeciesName` (matching GeneNetwork2's `wqflask.api.router.get_species_info`, which returns the same match logic but only a 4-column subset). 404 if no species matches. ```json { "data": { "id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus", "family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090, "order_id": 15 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /species/{species}/datasets` - returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of `/groups/{group}/datasets`. GeneNetwork2's own species-scoped variant of `wqflask.api.router.get_datasets_for_group` still requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as `/groups/{group}/datasets` -- this returns dataset metadata (names, `public`/`confidentiality` flags), not trait data -- but a token is still required, like every endpoint. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse", "description": "The species these datasets belong to"} ] } ``` - `GET /groups` - returns a JSON array of all groups in the `InbredSet` table, matching the shape of GeneNetwork2's `GET /api/v_pre1/groups` response: ```json { "data": [ {"id": 25, "species_id": 4, "display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "name": "AD-cases-controls", "full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "public": 2, "mapping_method_id": null, "genetic_type": null, "family": null, "species": "human", "mapping_method": null}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species", "description": "The species the groups belong to (filter groups with ?species=)"} ] } ``` `family` is the population category and `species` the species' short name as `/species` lists it. `mapping_method` is the name of the mapping tool offered for the group (the `MappingMethod` row its `mapping_method_id` points to, see `GET /mapping-methods`), null for a group with none. Three optional filters, combined with AND and all case-insensitive: `family` (contains), `species` (exact short name) and `q` (contained in the name, display name or full name). For example `/groups?family=Reference Populations` lists the reference populations, and `/groups?species=rat&q=hxb` the rat HXB/BXH sets. "Group" is GeneNetwork's own alias for `InbredSet` -- see `docs/genenetwork_datatypes.md` for why. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}` - returns every column of the matching row of the `InbredSet` table, matched against `InbredSetName`, `Name`, `FullName` or numeric `Id` (matching GeneNetwork2's `wqflask.api.router.get_group_info`, which matches the same way but not by Id, plus gn-guile's `gn.data.population:population-info`, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (see `docs/genenetwork_datatypes.md`); only the MariaDB column names themselves stay `InbredSet*`. 404 if no group matches. ```json { "data": { "id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD", "species_id": 1, "full_name": "BXD Family", "public": 2, "mapping_method_id": "1", "genetic_type": "riset", "family": "Reference Populations (replicate average, SE, N)", "family_order": 1, "menu_order_id": 0.0, "group_code": "BXD", "description": "

...

", "mapping_method": "qtlreaper" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}, {"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"}, {"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/strains` - returns the strains/samples belonging to a group, matching gn-guile's `gn.data.strains:strain-id-names`. 404 if no group matches. Takes an optional `used_for_mapping` query parameter to filter to only strains used for mapping (gn-guile's `strain-id-names` takes the same flag as a keyword argument): ```json { "data": [ {"id": 1, "name": "B6D2F1", "used_for_mapping": false}, {"id": 2, "name": "C57BL/6J", "used_for_mapping": false}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these strains belong to"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"} ] } ``` `GET /groups/BXD/strains?used_for_mapping=true` excludes BXD's parent strains (`B6D2F1`, `C57BL/6J`, `DBA/2J`), which aren't used for mapping. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/case-attributes` - returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3's `gn3.api.case_attributes` splits this into separate `.../names` and `.../values` endpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns `200` with both lists empty, not a 404. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "names": ["Epoch", "RRID", ...], "values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...] }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"} ] } ``` - `GET /groups/{group}/phewas` - a PheWAS around one position: every published trait of the group whose QTL peak (`PublishXRef.Locus`) is a marker within `window` base pairs of it, strongest first. Centre it on a marker (`?locus=rs30524230`, e.g. a peak from a mapping) or on a position (`?chr=9&mb=29.940961`); the window is in **base pairs** and centred, so `window=10000` is 10 kb up- and downstream (default 10000). `?genotype_version=` picks whose marker table defines the window (default: the group's current one). The answer carries the markers the window covered as well as the traits, so an empty `traits` list is readable as "nothing peaks that close" rather than a failure. Each hit has `trait_id`, `dataset`, `abbreviation`, `description`, its peak `locus`/`chr`/`mb`, `lrs`, `lod` (= `lrs` / 4.604), `mean`, `additive`, `nsamples`, `pubmed_id` and the GeneNetwork trait `url`. Example: `?locus=rs30524230&window=10000` returns 4 markers and 10 BXD traits, from LOD 99.9 down to 2.7. - `GET /groups/{group}/genotypes` - the genotype files this group can be mapped with, newest first: each `version` (e.g. `BXD.8`), the `file` name (`BXD.8.geno`), the panel's `title`, `individuals`, `markers`, whether `gemma` and `rqtl` can run on it, and which one is `current` (what a mapping request gets by default). The order comes from the deployment's `genotype/.json`, whose `genofile` list is newest-first, the same rule GeneNetwork2 uses; groups with no JSON (HET3-ITP) fall back to the highest version number and the listing's `notes` says so. `POST /compute/rqtl` / `/compute/gemma` take `group` plus an optional `genotype_version` instead of paths. `GET /groups/{group}/genotypes/{version}` is the metadata of one of them: the file name, the header's own `@`-labels (`type`, `mat`, `pat`, ...), the panel's `individuals` in file order and the `strains` a mapping actually uses (parents/F1s dropped), `num_individuals`, `num_strains`, `markers`, and per-chromosome marker counts. `GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr=` is the list of **marker names** -- one column, paged, filterable by chromosome -- and a position is one call per marker: `.../markers/{marker}/location` returns `{"name", "chr", "cM", "Mb"}` as named fields, so a client iterating names never carries a position table and a position lookup is a lookup. So a caller can see the panel, the markers and the header it is mapping against; the **genotype calls themselves are never served**, only summaries of them. See [docs/genotypes.md](docs/genotypes.md). `GET /groups/{group}/genotypes/{version}/frequencies` is how the panel is actually *called*, which nothing else in the API says: ```json {"data": {"B": 0.4889, "D": 0.492109, "H": 0.018991}, "notes": ["BXD BXD.4: 3811 marker(s), 354423 genotype cell(s)", "352741 call(s) counted, 1682 missing (0.47 %); …"]} ``` The map is call -> frequency over the calls that are present, so it sums to 1; missing codes (`U`, `N`, `NA`, an empty cell) are counted separately and reported in `notes`, not as a call. A file whose cells are probabilities is **binned**, 0.05 wide by default (twenty bins over [0, 1], keys are the bin edges: `"0.05-0.10"`), because 0.4187 and 0.4191 are the same statement about a marker; `?bin=0.1` changes the width. A file with only a handful of distinct numbers (`0`, `0.5`, `1` for a biallelic dosage) is reported exactly, since binning three values would hide the thing being looked at. `GET /groups/{group}/genotypes/{version}/markers/{marker}/frequencies` is the same tally for one locus (`?marker=` on the panel route does the same thing), which is how a caller checks whether a flat LOD came from silence or from a nearly-all-missing marker: both look identical in a scan. Counting is bounded: cells go through an interner, so a 100-million-cell panel costs one hash lookup per cell and one allocation per *distinct* call, and a `.geno` over 256 MB is refused (`413 genotype_too_large`) rather than read -- `HSNIH-Palmer_true.geno` is 3.7 GB (134 927 markers x 6 147 individuals), which is not a diagnosis anyone waits for. `GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait={id}` answers the next question -- *what does the trait look like in each genotype class at this marker* -- without serving genotypes: the calls are joined to the trait's values inside the server and reduced to `n`, `mean`, `sd`, `min` and `max` per class, with that trait's authorization applied. It is how you tell a peak that rests on two animals from one that rests on a hundred, and it is what explains why GEMMA and R/qtl2 peak at different markers on DOL trait 10003: GET .../markers/JAX00418132/classes?trait=10003 {"A": {"n": 2, "mean": 1.8, "sd": 0.1414, "min": 1.7, "max": 1.9}, "H": {"n": 20, "mean": 2.525, "sd": 1.1652, "min": 1.7, "max": 5.3}, "B": {"n": 171, "mean": 3.9491, "sd": 1.1038, "min": 1.7, "max": 6.7}} The A class is two animals and they are the trait's two lowest values; R/qtl2, which drops `H` on a `riself` panel (R/qtl's `convert2riself`), keeps 173 animals there but only 2 of one class, so the marker is nearly monomorphic for it. GEMMA keeps the 20 hets at dosage 0.5, which is where its sharper p-value comes from. Nothing in the payload names an animal. `GET /groups/unlisted/genotypes/{name}/frequencies` is the same tally for a file no group lists (a panel in preparation, a fixture, the DO's `do/` cross), and `.../markers/{marker}/frequencies` for one locus. `unlisted` is a namespace, not a group: `/groups/unlisted` has no record, only its genotype subtree answers, and `{name}` is the version, `.geno` name, or a path relative to the genotype files directory. One prefix for the whole genotype surface beats a second spelling for the same tally. The markers come the same way: `/groups/unlisted/genotypes/{name}/markers` (names) and `.../markers/{marker}/location` (position). - `GET /probesets/{name}` - **every dataset that carries a probeset with this name** (indexed on `ProbeSet.Name`): one row per dataset with the panel (`group`, `species`, `platform`) and the probeset's mapping statistics there (`chr`, `mb`, `mean`, `lrs`, `lod`, `locus`, `additive`). Suffixed variants match too -- `1427571_at` also finds `1427571_at_A`, the older chips -- and each row's `name` says which one it was, so the reply never leaves you guessing. `?group=BXD`, `?species=mouse` and `?platform=M430` (substring: there is no platform id in this schema) narrow it; `?page=` and `?limit=` page it, and `notes` carries the count that matters. GET /probesets/1427571_at → 155 rows over 155 datasets (Shh on chr5 28.46 Mb), e.g. ["SA_M2_1104_R","BXD","mouse","Striatum_M430_V2_RMA_November04", "1427571_at","Shh","5",28.4579,6.039,16.318,3.544, "rs52243480",0.129] This is the lookup a probeset id needed: it is not a Xapian field, so `/search` cannot answer it and the only alternative was guessing a dataset and calling `/datasets/{dataset}/probesets/{name}`. - `GET /genes/{symbol}/traits` - the same rows, filtered on `ProbeSet.Symbol` instead: **every expression trait of a gene, across datasets**, with a real total (unlike `/search`'s capped list), grouped by dataset, and `?sort=lod` / `?min_lod=` / `?group=` / `?species=` / `?platform=` so "BXD datasets where Shh is a strong eQTL" is one call. `GET /genes/Shh/traits?sort=lod` → 1 225 rows over 17 datasets and 13 probesets, strongest first (`HXB_Liver_1208 / 1387584_at`, LOD 18.46). - `GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=…` - **the genotypes of the animals at one marker of a confirmed hit**, which is the one place this API returns genotypes. It is fenced, because a genotype matrix is the thing GeneNetwork does not hand out: * the server **re-runs the R/qtl scan** for the trait (the same machinery and gate as `/compute/rqtl`) and takes the peak from *that* run; * the marker must lie within `?window=` Mb of it (default 10) -- a marker elsewhere is `422 not_the_peak`, quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach `?min_lod=` (default 3.0), or the answer is `422 not_a_hit`; * only animals **that have a value** for the trait are listed, and only this marker's calls -- nothing else of the matrix leaves. ```json {"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856, "peak": {"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856}, "counts": {"A": 110, "B": 8, "H": 74}, "animals": [{"individual": "TB001", "genotype": "A", "value": 4.3}, …]} ``` Which is the data behind R/qtl's Fig 4.1, and `POST /plot/genotypes` draws it: post that reply's `animals` (or its `genotypes` map) with the trait's values from `/groups/{g}/traits/{id}/values`, and get an SVG with one point per animal, genotype class across, value up, class means marked: ```json {"marker": "UNCHS026345", "genotypes": {"TB001": "A", "TB002": "A", …}, "values": [{"name": "TB001", "value": 4.3}, …]} ``` A class with a couple of animals in it is visible at a glance -- the same warning `.../markers/{marker}/classes` puts in numbers. - `GET /genes/{symbol}/aliases` - **Wikidata's aliases for a gene symbol**, the list a trait page shows next to the symbol, plus the `label` (the entity's canonical symbol in that species) and a `links` array of **external gene pages** built for that symbol and species -- RGD, MGI, Alliance, NCBI Gene, Wikidata, UniProt, STRING, GeneMANIA and, per species, GTEx/Open Targets/ Protein Atlas/GWAS Catalog (human) or BioGPS/Allen Brain Atlas (mouse). Two SPARQL queries against `WIKIDATA_ENDPOINT` (the entity, then its `skos:altLabel`s), kept in an LMDB file, `WIKIDATA_CACHE_DIR/gene-aliases.lmdb`, for `WIKIDATA_TTL_SECONDS` (seven days by default). The entity query matches the label case-insensitively, because human genes are labelled `SHH` and mouse ones `Shh`, and picks mouse, rat or human deterministically -- `?species=human` gets the ortholog whose labels GeneNetwork2's page shows: GET /genes/Shh/aliases {"data": {"symbol": "Shh", "wikidata_id": "Q14860079", "species": "mouse", "aliases": ["9530036O11Rik","Dsh","Hhg1","Hx","Hxl3","M100081", "ShhNC","sonic hedgehog"], "fetched_at": 1794315892, "stale": false}} When Wikidata cannot be reached the cached answer is served with `"stale": true` and the envelope marked `partial` -- the last value, not an empty list: an empty alias list and an unreachable upstream mean different things, and only one of them is about the gene. `?refresh=true` forces a refetch (an operator's switch). The cache is one file plus LMDB's lock companion, written `0600`. The `label` is what makes the human-only services reachable from a rodent symbol: `?species=human` on a mouse `Shh` gives `label: "SHH"`, an RGD link for human, and a GTEx link that resolves -- a mouse symbol in GTEx is an empty report. A cached entity written before the label was kept is refreshed rather than served (`stale` tells you which happened). **Both serve metadata, not values.** A token buys the index; the values are per dataset and authorized per dataset, and the `next` link goes to `/datasets/{dataset}/probesets/{name}/values`, which checks. There is no endpoint that returns values across datasets, deliberately: that is bulk work and `/groups/{group}/traits` is the bulk call. - `GET|POST /baskets`, `GET|PATCH|DELETE /baskets/{name}` - **your own column of values**, stored in your own SQLite file, and mapped by name. This is how data gets *changed* without changing the database: take a trait's values, edit one or more of them (or build a covariate column that does not exist), and map that instead. MariaDB is never written to. ```sh curl -s "$API/groups/BXD/traits/10001/values" > wbc.json # post the column back as a basket (jq: .data is the values array) curl -s -X POST "$API/baskets" -d '{"name":"wbc-edit","kind":"trait", "base_dataset":"BXD","base_trait":10001,"values":[...]}' curl -s -X PATCH "$API/baskets/wbc-edit" -d '{"values":[...]}' # revision 2 curl -s -X POST "$API/compute/qtl2" -d '{"group":"BXD", "genotype_version":"BXD.4","trait_name":"basket:wbc-edit","basket":"wbc-edit"}' ``` Choices worth knowing: * **The full column is stored**, not a diff: the caller has the original beside it and the comparison is theirs, while a mapping needs one complete column. * **Text with a parsed copy**: `raw` is what was typed (`5.2`, `NA`), `value` the number or null, `status` which it is -- so `NA` and `0` stay different claims. A column containing neither a number nor a missing marker (`NA`, `x`, `-`, `.`, `?`) is refused with the offending samples named, never silently dropped. * **An automatic version history**: revision 1 is the column as first stored and every `PATCH` adds one, with its own digest and Unix timestamp — earlier revisions are kept, not overwritten, so `GET /baskets/{name}/revisions` is the history, `?revision=N` reads an old column, and a mapping can pin one: `{"basket": "wbc-edit@2"}` maps revision 2 even after the basket has moved on. Every reply records which revision and digest ran (`Basket 'wbc-edit' revision 2 (digest 9559a70e…)`), so two runs on one name can never look alike. * **One SQLite file per user, named for them** — `BASKET_DIR/.sqlite` (0600, WAL), where `` is the token's subject: tokens rotate, subjects do not, so a new token finds the same baskets and an operator can see whose file it is. A subject that is not filename-safe is sanitised and disambiguated with a short digest. Baskets are private: `GET /baskets` only ever lists yours, and nothing here needs a database the API may write to. * **Every compute endpoint** takes `basket` in place of its value field -- `/compute/{rqtl,gemma,qtl2,fitqtl}` -- and giving both is a 422 rather than a silent choice. `PATCH` replaces the column, it does not merge: leaving a value out drops it, deliberately (the dropped revision is still in the history). - `POST /matrices`, `GET|DELETE /matrices/{name}` - **upload new traits and covariates as one named table**, checked against an existing genotype file. The shape is the one people have: a header row, then one row per individual, the first column naming them. ``` id,body_weight,glucose,batch BXD1,23.100,9.400,2 BXD2,18.700,8.100,1 ... ``` POST /matrices {"name":"my-traits","genotype_version":"BXD.4", "contents":"id,bw,glucose\nBXD1,23.1,9.4\n...", "kinds":{"batch":"covariate"}} The upload is checked against `genotype_version`'s panel **before** it is stored: the same number of individuals, and every name one of them. 92 of 93, or a name the panel does not have, is a 422 that says which -- a mapping that silently drops individuals would be worse than an error. `kinds` marks columns that are covariates; the rest are traits. Each column becomes a **basket** named `.`, so a mapping names it the way it names any basket, on any engine, and inherits the revision history and digest: POST /compute/qtl2 {"group":"BXD","genotype_version":"BXD.4", "basket":"my-traits.bw"} Everything lands in the uploader's own SQLite file -- MariaDB is never written to -- and `GET /matrices/{name}/values` reads the table back (`null` where a value was missing). A script writes an example table from a genotype file's own header, so the fixture passes the check by construction: `scripts/example-matrix.py --genotype --out example-matrix.csv`. - `POST /uploads/genotypes` - **upload a genotype matrix that updates an existing population**, named `$USER-GENONAME`. Tab-delimited: the header names the individuals from column 4 on, column 1 is the marker, 2 the chromosome, 3 the position in Mb. ``` marker chr pos BXD1 BXD2 ... rs6269442 1 3.4921 B D ``` This is an *update*, not an ingest, and the checks say so: the base version must exist, at most **50** individuals may be new, at least **one** must overlap (`none of the 93 individuals are in the base population: an upload updates a population`), and at most **200 000** markers -- beyond that *the sysadmin has to organize a file that size*. Values must be the same **kind** as the base's (single letters, whole numbers, or probabilities), because a mapping engine reads the result the way it reads the population it updates; a letters panel full of `0/1` is refused, naming both kinds. Stored as text in the uploader's own SQLite file, and written out on demand as a GN `.geno` (Chr, Locus, cM, Mb, individuals) carrying the base's `@`-metadata: `GET /uploads/genotypes/{name}/geno` returns the path for the `genotype_file` parameter of the compute endpoints, which now accepts values **by name** (`samples` or a `basket`) as well as positionally: ``` POST /compute/qtl2 {"genotype_file":"","samples":[{"name":"BXD1","value":61.4}, …]} ``` An upload of one chromosome is a poor thing to scan: a leave-one-chromosome- out kinship has nothing to leave out, so the scan comes back flat or empty. `scripts/example-genotype-upload.py` therefore spreads its markers across chromosomes; use `"kinship": "overall"` if a file really has only one. `GET /uploads/genotypes` lists yours, `.../markers` pages the marker table, `DELETE` removes one. `scripts/example-genotype-upload.py` turns any `.geno` into a valid update (subset of markers, a few individuals dropped or added). - `GET /groups/{group}/traits` - returns every published (phenotype) trait belonging to a group, the listing equivalent of `/groups/{group}/traits/{trait_id}` (same fields, same join, no `trait_id` filter). 404 if no group matches. If the group has any published traits, its `PublishFreeze` row is checked the same way as `/traits/{dataset}` (see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's own `wqflask.api.router.fetch_traits`, which defaults to unbounded too. Needs `SQL_URI`, same as `/species` above. The optional `abbreviation` parameter keeps only the traits with that short name, e.g. `/groups/BXD/traits?abbreviation=5HT_MB_M` finds trait 17985 (an unknown abbreviation is an empty list). ```json { "data": [ {"id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD", "description": "The group these traits belong to"} ] } ``` - `GET /groups/{group}/traits/{trait_id}` - returns one published (phenotype) trait's `PublishXRef` mapping stats plus its `Phenotype` description/units and `Publication` reference, based on the data explored in `precompute-publishdata.gmi` (e.g. its "More metadata" section, which identifies exactly this join as worth surfacing). `trait_id` is `PublishXRef.Id`, scoped by group for the same reason as `/groups/{group}/traits/{trait_id}/values`. 404 if no group matches, or if the group has no such trait; 403 if the trait's `PublishFreeze` isn't public and the caller isn't authorized for it (see docs/gn-auth.md). ```json { "data": { "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"}, {"href": "/groups/BXD", "description": "The group this trait belongs to"}, {"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"} ] } ``` `dataset` is the trait's `PublishFreeze`, `abbreviation` its short name (`Phenotype.Post_publication_abbreviation`, else the pre-publication one) and `url` its page on GeneNetwork's website. `lod` is `lrs` / 4.604. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute}` - one case attribute's values (`Sex`, `Age`, `Tissue`, `Batch`, ...) for exactly the samples a trait's values cover, **in that trait's order**, so it lines up with `/values` without re-sorting -- the covariate source, kept off the values endpoint. `values` carries the text as recorded (`F`, `Live`), `null` where a sample has none; `num_values` counts those that have one, and `distinct` lists each value with its count, commonest first, which is what says whether it is usable as a covariate (a constant one is not). Attributes are per group -- `BXD` defines `Epoch, RRID, SeqCvge, Status` and no `Sex`, while `BXD-Longevity` and several other BXD-family studies do -- and an unknown one is a 404 whose `next` is the group's attribute list. Example: `/groups/BXD-Longevity/traits/10001/case-attributes/Sex` returns 3281 values in `AgeAtDeath`'s sample order, `F` × 3101, `M` × 180. - `GET /groups/{group}/traits/{trait_id}/values` - returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2's `wqflask.api.router.trait_sample_data` (the Publish branch) and gn-guile's `gn.data.dataset:get-bxd-publish-name-value-dict`, generalized to any group rather than hardcoding BXD. `trait_id` is the trait's `PublishXRef.Id`; it alone doesn't identify a trait uniquely across groups (only the pair of group and `PublishXRef.Id` does), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait's `PublishFreeze` isn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a `#[gnauth(secured)]` endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits. **`?normalize=` / `?winsor_quantile=` / `?winsor_fences=`** apply the same pipeline `/compute/*` takes (see "Normalize the trait first" below), over exactly the samples returned, and say what ran in `notes`. The mapping suggestions in `actions` carry the same fields, so fetching z-scores and then following the suggestion maps the numbers that were shown rather than the raw ones: ```sh curl "$API/groups/BXD/traits/10010/values?normalize=zscore" # notes: "Normalized: zscore ... over 28 values; mean 0.000000, sd 1.000000" # actions[*].body.normalize == "zscore" ``` A pipeline that cannot apply (both winsorising rules at once, a log of a non-positive value, nothing to divide by) is a `422 invalid_normalization`. ```json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/datasets` - returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2's `wqflask.api.router.get_datasets_for_group`. 404 if no group matches. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these datasets belong to"} ] } ``` `tissue` is what the dataset was measured in (the `Tissue` its `ProbeFreeze` points to), also on `GET /datasets/{dataset}` and `GET /species/{species}/datasets`. - `GET /mapping-methods` - the mapping tools GeneNetwork offers, the `MappingMethod` table in id order: `qtlreaper`, `Rqtl`, `HappyR`, `PLINK` and `FastMap`. It includes tools no group currently uses (PLINK, FastMap); a group's `mapping_method` (see `GET /groups`) names the one offered for it. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 1, "name": "qtlreaper"}, {"id": 2, "name": "Rqtl"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"} ] } ``` - `GET /accessions/{accession}` - resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without the `GN` prefix. It names the dataset (`InfoFiles.InfoPageName`), its page title, its `category` (`publish` for a phenotype dataset, `probeset` for an expression dataset, null if the name matches neither) and the `group` it belongs to. 400 if the text is not a number, 404 if no dataset has the accession. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "accession": 602, "name": "BXDPublish", "title": "BXD Published Phenotypes", "category": "publish", "group": "BXD" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group the dataset belongs to"}, {"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"} ] } ``` - `GET /publications/{pubmed_id}/traits` - returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as `/groups/{group}/traits`). 404 if no publication has that id; a publication with no traits is an empty list. Unlike `/groups/{group}/traits`, the datasets are authorization-checked as in `GET /traits/{dataset}`: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 17967, "phenotype_id": 16502, "publication_id": 13584, "mean": 6038.655919692095, "locus": "rsm10000012868", "lrs": 10.535408722907198, "lod": 2.2883164037591652, "additive": -647.5889092087912, "nsamples": 34, "pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824, "title": "Evaluation of heritable determinants of blood and brain serotonin homeos...", "authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...", "journal": "Genes, brain, and behavior", "volume": null, "pages": "247-260", "month": "Mar", "year": "2014", "dataset": "BXDPublish", "abbreviation": "5HT_Blood_M", "url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"} ] } ``` - `GET /datasets/{dataset}` - returns every column of the matching row of the `ProbeSetFreeze` table, matched against `Name`, `Name2`, `FullName`, `ShortName` or numeric `Id` (matching GeneNetwork2's `wqflask.api.router.get_dataset_info`'s matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under `/groups/{group}`. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M", "name2": "BXDMicroArray_ProbeSet_August03", "full_name": "Brain U74Av2 08/03 MAS5", "short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01", "public": 0, "confidentiality": 0, "data_scale": "log2", "tissue": "Brain mRNA" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"} ] } ``` - `GET /datasets/{dataset}/probesets` - returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of `/datasets/{dataset}/probesets/{probeset}` (same fields, same join, no `probeset` filter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (`auth::authorise_dataset`) rather than once per probeset -- see docs/gn-auth.md. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"} ] } ``` - `GET /datasets/{dataset}/probesets/{probeset}` - returns one expression (ProbeSet) trait's `ProbeSetXRef` mapping stats plus its `ProbeSet` gene annotation, matching GeneNetwork2's `wqflask.api.router.get_trait_info`'s ProbeSet branch. `ProbeSetXRef` has no single Id column of its own (its primary key is `DataId`; the natural key is the pair `(ProbeSetFreezeId, ProbeSetId)`), so a trait here is identified by dataset name plus `ProbeSet.Name` rather than a single numeric id, unlike `/groups/{group}/traits/{trait_id}`. 404 if no dataset matches, or if the dataset has no such probeset. ```json { "data": { "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"}, {"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /datasets/{dataset}/probesets/{probeset}/values` - returns one expression trait's value in each strain in the dataset, matching GeneNetwork2's `wqflask.api.router.trait_sample_data`'s ProbeSet branch, addressed the same way as `/datasets/{dataset}/probesets/{probeset}`. 404 if no dataset matches, or if the dataset has no such probeset. ```json { "data": [ {"name": "B6D2F1", "value": 5.742000102996826}, {"name": "BXD1", "value": 6.414000034332275}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"} ] } ``` Needs `SQL_URI`, same as `/species` above. ### `/traits` -- ProbeSet and Publish data through one URL shape Everything above addresses ProbeSet (expression) and Publish (phenotype) traits differently: `/datasets/{dataset}/probesets/...` vs. `/groups/{group}/traits/...`. A trait is a trait either way -- a dataset, a trait within it, and that trait's per-strain values -- so `/traits/...` addresses both the same way, `(dataset, trait)`, matching GeneNetwork2's own generic `wqflask.api.router.trait_sample_data`/`get_trait_info`. This is additional, not a replacement: the endpoints above are unchanged and keep working exactly as documented. `dataset` is resolved against `ProbeSetFreeze` first, then `PublishFreeze` (`db::resolve_trait_dataset`) -- whichever matches decides the category, so a bare trait name/id is never guessed across categories, only the (much smaller, auditable) set of dataset names is. A Publish dataset can be named either way GeneNetwork2 itself accepts: the `PublishFreeze`'s own `Name`/`ShortName`/`Id` (e.g. `"BXDPublish"`), or its owning group's `Name`/`InbredSetName`/ `FullName`/`Id` (e.g. `"BXD"`) -- a group normally has only one `PublishFreeze`, so the two are near-interchangeable in practice. - `GET /traits/{dataset}` - every trait in `dataset`, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (`auth::authorise_dataset`), same as `/datasets/{dataset}/probesets`. This is still more than `/groups/{group}/traits` checks for the same phenotype data: both endpoints require a token now, but only this one also checks `PublishFreeze.public`/`confidentiality` and, for a non-public dataset, gn-auth -- `/groups/{group}/traits` (unchanged) stops at the token. ```json { "data": [ {"category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] } ``` or, for a Publish dataset: ```json { "data": [ {"category": "publish", "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] } ``` - `GET /traits/{dataset}/{trait}` - one trait's metadata, tagged by category (`ProbeSetInfo`'s gene annotation or `TraitInfo`'s publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other). `trait` is `ProbeSet.Name` for a ProbeSet dataset, or `PublishXRef.Id` for a Publish dataset (a non-numeric `trait` against a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"} ] } ``` - `GET /traits/{dataset}/{trait}/values` - that trait's value in each strain, as `{name, value}` pairs regardless of category (`TraitValue` and `ProbeSetValue` are identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as `/traits/{dataset}/{trait}`. ```json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"} ] } ``` - `GET /search` - free-text search across GN traits (see `docs/search-engines.md` for switching between the gn3 proxy and the native engine); hits keep gn3's field names and are wrapped in the envelope like every other response. Parameters: `q` (xapian syntax, e.g. `serotonin AND midbrain AND group:BXD`), `type` (`phenotype` or `gene`), `limit` (default 20, at most 100), and optionally `sort` (`lrs` or `lod`, which order identically) with `order` (`desc`, the default, or `asc`), and `page` (default 1) for the next `limit` hits. Three fields are added to every hit: `lod` (`lrs` / 4.604) and, for phenotype hits, `locus` (the peak marker's name) and `abbreviation` (the trait's short name), both from the database (null for other hits, or when the database is unreachable). `pubmed_id` is gn3's. With `sort`, gn3 is asked for its first 1000 matches by relevance, those are ranked, and `page`/`limit` are applied to the ranked list, so `page=2` continues where page 1 stopped; hits with no `lrs` come last. Without `sort`, `page` is gn3's own. For example, the highest-LOD trait about serotonin in BXD midbrain: `/search?q=serotonin AND midbrain AND group:BXD&sort=lod`. Tissue and sex are not fields: they appear in each hit's description, so match them there. Needs a reachable gn3, configured via the `GN3_URL` config key (default `http://127.0.0.1:5003/api/`). ```json { "data": [ {"name": "17994", "dataset": "BXDPublish", "dataset_fullname": "BXD Published Phenotypes", "group": "BXD", "species": "mouse", "authors": ["Ye R", "Carneiro AM", ...], "year": 2014, "description": "Central nervous system, protein expression: Midbrain serotonin transport...", "mean": 1627.7004466337316, "additive": 233.98946836805567, "lrs": 16.49128337950074, "lod": 3.5819468678324804, "locus": "rsm10000013922", "abbreviation": "SERT_MB_M", "geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD", "pubmed_id": 24405699, "pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."} ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"}, {"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"} ] } ``` - `POST /plot/rqtl`, `POST /plot/gemma` - the drawing step after a mapping: **post the mapping's own response back** (the JSON `/compute/rqtl` or `/compute/gemma` returned, as-is) and get the SVG the deployment's plotter draws from it (`scripts/plot-rqtl.py`, `scripts/plot-gemma.py`; `src/plot.rs` runs them). Nothing is recomputed -- a plot is one script run, and it can be redrawn from a result you kept. `/plot/rqtl` also accepts just the `data` member; `/plot/gemma` needs the whole response, because its plotter reads `meta.columns` to find the `p_lrt` column. The reply is `image/svg+xml`, not the JSON envelope -- an image is not JSON -- and the plotter's one-line peak summary comes back in `x-plot-summary`. R/qtl plots LOD with the permutation thresholds as dashed lines when the result carried them. GEMMA plots `-log10(p_lrt)` with **no line unless one is asked for**: `?bonferroni=true` adds GEMMA's own Bonferroni (computed by the plotter as 0.05 over the analysed markers), and `?thresholds=` adds one computed elsewhere, each labelled. The title names the trait: taken from the posted response's own `Trait: ...` note (which `/compute/rqtl` and `/compute/gemma` both write first), or from `?trait=` when you post only `data`; `?trait=` wins over the note, and `?title=` replaces the whole title. Two traits over the same panel draw two different traces -- the plot is drawn from the rows you posted, and nothing is cached between calls. Configuration: `PLOT_PYTHON` (default `python3`), `RQTL_PLOTTER`, `GEMMA_PLOTTER`. Both mapping responses suggest these calls in `actions`. - `GET /openapi.json` - returns an OpenAPI 3.1 document describing the endpoints above; see [OpenAPI](#openapi) below. ### `/compute/gemma` Maps a QTL by running the real `gemma-wrapper`/GEMMA binaries: a K-matrix step, then an LMM association step, both `--loco`, over a genotype file, a SNP annotation file and a list of trait values. See `src/gemma.rs` for the invocation itself and why it's shaped this way (found by running GeneNetwork2's and GeneNetwork3's own gemma code for real against the BXD fixtures -- GeneNetwork3's always errors out; GeneNetwork2's non-LOCO branch does too; only `--loco` works). **Individuals with no genotypes are left out.** A panel can name individuals whose genotype row is empty -- `BXD-AE.4` has 136 of 344, `BXD-Heart-Metals.4` 200 of 481 -- and an empty row makes the kinship matrix singular, which GEMMA reports as `eigenvalues close to zero` and then a LAPACK `LUDecomp ... input domain error`, a message that says nothing about the cause. Those individuals are dropped before the fit and counted in `notes`; they contribute nothing either way, and the panel maps in seconds once they are gone. **Normalize the trait first.** `normalize` changes the values' scale before mapping -- `none` (default), `log2`, `log10`, `sqrt`, `negate`, `zscore`, `robust_zscore`, or `rank` -- and `winsor_quantile` / `winsor_fences` handle the tails. They run in that fixed order (**transform, then winsorise, then standardise**), over exactly the individuals that enter the fit, and the response says what ran: ```json "normalize": "rank", "winsor_fences": 1.5 ``` - `rank` is GeneNetwork2's "Quantile" (`qnorm`): `Φ⁻¹((i−0.5)/n)` by rank, ties sharing their average rank -- *not* array quantile normalisation. Accepting `qnorm` and `invert` (for `negate`) is deliberate: those are GN2's own names. `zscore` uses the **population** standard deviation, as GN2 does (`scipy.stats.mstats.zscore`), so the numbers match its trait page; `robust_zscore` is `(x−median)/(1.4826·MAD)` and is ours. - `winsor_quantile: 0.05` clips each tail at the 5th/95th percentile; `winsor_fences: 1.5` clips at `Q1 − 1.5·IQR` / `Q3 + 1.5·IQR`, the rule GN2 uses to *flag* outliers -- and a count of those is reported whether or not anything is clipped. The two are mutually exclusive (`422`). - `log2`/`log10` on a non-positive value, `sqrt` on a negative one, or a standardisation with nothing to divide by is a `422`; missing values stay missing and are never imputed. GN2's silent `sqrt` "add 1 in [0,1]" quirk is deliberately not reproduced. - `beta`, `se` and `additive` come out on the transformed scale -- a `zscore` additive is in standard deviations -- so quote the note with the number. **Name what you are mapping.** `trait_name` is optional -- GEMMA's values are positional, so it needs no name to compute -- but it is what the response reports (`Trait: BXD::20010`, first in `notes`), what the PheWAS and R/qtl suggestions carry, and what `/plot/gemma` titles the picture with. Without it they fall back to `::gemma`, so two mappings over one panel are distinguishable in the data but identically titled. **No permutations.** GEMMA's permutation test is not run here and `nperm`/`permutations`/`permutate` are **refused** (`422 permutations_not_supported`), not ignored -- an ignored flag is worse than a refusal, because the caller believes it got thresholds. The reason is cost: `gemma-wrapper --permutate` shuffles the phenotype file and repeats the entire LOCO association scan per permutation, so a thousand is thousands of full scans -- hours to days on a real panel, against the one slot this endpoint has. R/qtl does have the test, on the same panel and the same values: `POST /compute/rqtl` with `"nperm": 100` returns the suggestive and significant LOD thresholds, and every successful GEMMA response already carries that call, pre-filled, in `actions`. Judge a GEMMA peak against the smallest `p_lrt` it reached, never against a genome-wide threshold of its own. **Covariates.** `covariates` fits one or more covariates alongside the trait -- the same request shape `/compute/rqtl` takes (below), each either a **case attribute of the group** by name, which the server reads, aligns onto the panel and codes, or explicit per-sample numbers: ```json "covariates": [ {"case_attribute": "Sex"}, {"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}}, {"name": "age_days", "kind": "numerical", "values": [{"name": "F3092", "value": 183.0}, …]} ] ``` GEMMA wants numbers, positional over the genotype panel, in a file of its own (`-c`), so `src/covariates.rs` writes one: one row per individual, no header, no sample names, and a categorical covariate **dummy-expanded** into k‑1 columns named after their levels (`Sex.M`) -- GEMMA knows nothing about factors, unlike R/qtl's wrapper, which does that expansion itself. The first level is the reference. A level none of the fitted individuals carries is left out (an all-zero column is singular and GEMMA stops with `LUDecomp` rather than dropping the term the way R would), and the response says so in `notes`. Text with 2–20 distinct values is coded 0..k‑1 in sorted order; numeric text is passed through as `numerical`; more than 20 levels, one level, or an explicit `map` that misses a value is a `422` asking for a `map`. An unknown attribute is a `422` listing the group's own. A missing value is written `NA`, which GEMMA treats as it does a missing phenotype -- it drops that individual (measured; GeneNetwork2 instead writes `-9`, which GEMMA fits as a value). Both the coding, how many individuals have a value, and any column left out are reported in `notes`. Covariates reach **only the association step**, never the K-matrix step, and `gemma-wrapper` hashes its file arguments, so a different covariate set is a different cache entry while the kinship it was computed against still is not recomputed. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`). **Concurrency: large jobs, not all jobs.** Gating exists to keep large computations from piling up on this box, and for GEMMA that begins at **50 individuals** (`compute_gate::GEMMA_LARGE_PANEL`) -- i.e. at the smallest real panel, so in practice every mapping request is gated, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. Two reasons, neither about how long a small run takes: every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash, so two runs at once are two writers over the same archives; and a real BXD WGS run is ~19 s of every core it can get, while UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So GEMMA runs **one job at a time, one per caller**: a second submission from the same token while its run is running or queued is refused at once with `409 compute_busy` naming that run, and a submission from another caller waits for the slot. R/qtl is gated much later (1 000 individuals; see below) because a `scanone` over a few hundred individuals is seconds of one core and its results are uncached scratch of its own. **Concurrency.** A large scan -- **1 000+ individuals** (`compute_gate::RQTL_LARGE_PANEL`), where a `scanone` starts to be minutes rather than seconds -- takes one of `RQTL_MAX_CONCURRENT` slots (default 8), one per caller: a second large submission from the same token while its scan is running or queued is refused at once with `409 compute_busy` naming that run, and a submission that finds every slot taken waits for one to come free. Eight because a scan is a single-threaded `Rscript` and this box has cores to spare; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. The count is fixed when the gate is built (at the first large request), so a change needs a restart. Smaller crosses are not gated at all. **Permutations.** `nperm` runs R/qtl's permutation test, and the response carries `permutation: {suggestive, significant, values}`. A scan without it suggests one with `nperm: 100` in `actions` -- the thresholds a peak needs to be judged against -- and `next` carries a PheWAS at the peak the run found. **No permutation test, and requesting one is an error.** GEMMA reports a p-value per marker; `gemma-wrapper --permutate n` does exist, but it repeats the whole LOCO scan per permutation, so it is hours of the one slot this endpoint has. `nperm`, `permutations` and `permutate` are therefore refused (`422 permutations_not_supported`) rather than ignored, and a peak is judged against the smallest `p_lrt` it reached. The response's `actions` carries the R/qtl permutation run over the same panel and values, pre-filled, which is where genome-wide thresholds come from here. **Give it a group, not a path.** `group` (an InbredSet name) plus an optional `genotype_version` makes the server resolve the genotype files itself -- `GET /groups/{group}/genotypes` lists what is available and which version is current, and `src/genotypes.rs` applies GeneNetwork2's rule for that. `trait_values` may then be replaced by named `samples`, which are matched onto the panel by name; parents and F1s are dropped unless `include_parents_f1` is set, as in GeneNetwork2/3. `genotype_file` + `snps_file` remain as an explicit override, for files outside that layout. **If you name the files yourself**, they are BIMBAM, in GeneNetwork2's own layout: `genotype/bimbam/_geno.txt` (the genotype matrix) together with `_snps.txt` (marker/chromosome). The `genotype/.geno` that R/qtl takes is *not* a GEMMA input. For BXD's current panel that is `bimbam/BXD.8_geno.txt` + `bimbam/BXD.8_snps.txt` -- 235 individuals × 21 056 markers. `trait_values` must then have **one entry per individual, in the genotype file's column order**, `null` where the trait has no value; GEMMA matches by position, so a short or reordered list is a wrong answer, not an error. Expanding a trait's values onto the panel is the caller's job today (the API does not re-index yet); see [docs/genotypes.md](docs/genotypes.md) for the conventions, which version counts as current, and the measured runs -- 17.1 s for the BXD WGS panel (21 055 marker rows) against the 19 s the 7 320-marker fixtures take. This runs synchronously inside the request -- there is no job queue -- and it runs sequentially, not per-chromosome in parallel. **Not feasible past roughly 100,000 markers at this point.** The BXD fixtures (7,320 markers, LOCO across 20 chromosomes) already take about 19 seconds; some real GeneNetwork datasets have upwards of 1 million markers, which this endpoint cannot handle yet -- the request would simply run for a very long time inside one HTTP call. Making that feasible needs either parallelising the per-chromosome LOCO jobs (`gemma-wrapper --parallel`, which gn-guile's own precompute script already uses) or moving the computation out of the request/response cycle entirely (a job queue, or an offline precompute step, the way gn-guile's own `scripts/precompute/run-gemma.scm` already does it). ```json { "data": [ [1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, -0.07665315, 0.2178407], ... [1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, -0.21408755, 0.0003520103] ], "meta": { "state": "complete", "took_ms": 18456, "request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc", "columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "additive", "p_lrt"], "num_rows": 7320, "num_cols": 12 }, "notes": [ "Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.", "Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk", "GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9" ], "next": [] } ``` (`data` has one row per marker -- 7,320 rows in this real run against the BXD fixtures, trimmed to two here. Each row is positional, in `meta.columns`' order, rather than a `{"chr": ..., "rs": ..., ...}` object per row -- cuts the repeated key names out of a response this large. **The parental design is an endpoint, not a guess, and `/groups/{group}` carries its headline.** That reply's note reads *"Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default"* — the two strain names answer "which cross is this?" without a second request — and `GET /groups/{group}/parents-and-f1s` reports a family's parents and F1s from the deployment's `parents_and_f1s.json` (ported from GeneNetwork2's `ParInfo`, all 24 groups), plus the `@type` of its genotype file, the individuals each role names when the metadata names them, and — the part a caller actually needs — which members of the *current panel* the default rule drops, and which the `F1` name heuristic catches. For an unlisted group it says `source: "name heuristic"` and warns that its parents are unknown, so nothing is assumed. For an RI set it also explains a subtlety: the metadata names the *design's* F1s (`B6D2F1`) while the panel carries *specific* intercross F1 individuals (`C57BL/6JxBXD037F1`), so the exclusions come from the name rule and `excluded_by_name` lists precisely which. **Choosing individuals.** Parents and F1s are excluded by default, as in GeneNetwork2/3 (`include_parents_f1: true` turns that off). On top of that rule any request that maps over a group's panel takes a selection: ```json {"group": "BXD", "include_parents_f1": true, "individuals": {"include": ["C57BL/6JxBXD037F1"], "exclude": ["BXD48a"]}} ``` `exclude` wins over `include`, and an unknown name is a `422 invalid_individuals` whose message lists the panel — a silently dropped animal changes a kinship matrix without anyone noticing. The reason both exist: the parent/F1 list is *data* (`parents_and_f1s.json`, seven groups) that a request cannot reach, while F1 names differ per group (`B6D2F1` for BXD, `BHF1`/`HBF1` for BXH, `C57BL/6JxBXD037F1` for intercross F1s). Worth knowing before turning it on: **including parents/F1s changes the kinship, not the fit.** GEMMA computes kinship without reading phenotypes, so they shape the relatedness structure; the association still uses only the individuals that have a trait value. The reply says both numbers — *"197 individuals … ; the kinship uses all 197, the association uses the 33 with a trait value"* — and a request can therefore legitimately have 234 individuals in the genotype file and 34 in the fit. `additive` is the effect size, sitting next to the score: **`−beta/2`**, which is what GeneNetwork's own tables and CSV call the additive effect (`gn2/wqflask/marker_regression/gemma_mapping.py`). Halved because half the difference between the two homozygotes is the usual additive effect; negated because GN2's BIMBAM conversion codes `@mat` as 1, so GEMMA's `beta` is per *maternal* copy while GeneNetwork's published sign is anchored on the other allele. `beta` and `se` are left where GEMMA put them -- `se` is what makes the effect interpretable (`beta ± 1.96·se` is a 95 % interval) -- and `p_lrt` stays the **last** column. These describe *this run* on these samples and this genotype version; they are not the `additive` that trait info reports, which came from the published mapping (trait 20010: this run `−0.54`, published `+2.118`). The last two `notes` are the exact gemma-wrapper invocations used (paths shortened here for readability) -- `-gk` only on the kinship step, `-lmm` only on the GWA step, never both together; see `src/gemma.rs`'s `compute_gwa` doc comment for why that distinction matters. `lmm` defaults to 9, matching GeneNetwork2's and gn-guile's own convention; see the field's own doc comment in `src/routes/gemma.rs` for what's actually been verified about it.) One further note is GEMMA's own variance component for the **null** model, which it writes only when asked (`-pve`, which this build passes on the GWA step; accepted alongside `-lmm 9`, verified against GEMMA 0.98.6). Under `--loco` there is one per chromosome -- twenty different kinship matrices -- so the note carries **every** value, not a summary: GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451, chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to 0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg 23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933, ve 0.8661). The unrounded numbers are also appended to the run log as `variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612` lines, so a `?id=` run's `/compute/gemma/status` stream shows them while the run is still in flight; and gemma-wrapper keeps each chromosome's own `.log.txt` beside its archive (`5eb53fa`), named by the same hash. `pve` is the proportion of variance the kinship explains -- the same quantity as `/compute/bulklmm`'s `h2`, but per chromosome instead of genome-wide -- and it is scale-free, so compare it with `h2` rather than comparing `vg` against `σ²_e`: a trait measured in thousandths has variances in thousandths. When the estimate sits on a boundary GEMMA prints `-nan` for `se(pve)`; that is reported as "not available", because a boundary estimate has no standard error, and a `pve` of exactly 0 or 1 is a direction rather than a proportion. #### `/compute/gemma/status/{run_id}` -- polling a run's progress Given the endpoint above can genuinely take minutes, pass `?id=` on the `POST` and poll this on a second connection while it's in flight to show progress, the same log-tailing shape GeneNetwork2/3 use for R/qtl (`gn3/computations/streaming.py`, `gn3/api/streaming.py`): the run's `id` becomes a log file (`{cache_dir}/{id}.log`); gemma-wrapper's own stderr (its `--json`-mode progress banner, chromosome list, "Executing: ...", per-step info -- its stdout carries only the one final JSON record, so there's nothing to stream there) is appended to it line by line as both the kinship and GWA steps run; once the whole request finishes, `"GN4: computation complete"` is appended as a sentinel so a poller can tell "still running" from "done". `run_id` must be 1-64 ASCII letters/digits/`-`/`_` -- it's a path component, so this is enforced on both this endpoint and the `?id=` that names it. `id` must also be unique: if a log already exists for it (another run still in progress, or a finished one never polled/cleaned up), the `POST` is refused with 409 rather than silently overwriting it. ``` POST /compute/gemma?id=my-run-1 (on one connection; blocks until done) GET /compute/gemma/status/my-run-1?peek=0 (on another, while the first is in flight) ``` ```json { "data": { "run_id": "my-run-1", "lines": [ "gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024", "Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022", "Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..." ], "pointer": 612 }, "meta": {"state": "complete", "took_ms": 1, "request_id": "..."}, "notes": [], "next": [] } ``` Pass `pointer` back as `?peek=` on the next poll; `lines` is empty (not an error) when nothing new has been written since. An unknown `run_id` (never started, or the main request hasn't written its log yet) is a 404, not an empty response -- genuinely different from "nothing new yet". #### `/compute/gemma/ravanan` -- the same mapping, run per-chromosome via ravanan Same request/response shape as `/compute/gemma` -- `group` (plus `genotype_version`), an uploaded genotype (`genotype_upload`), or `genotype_file` + `snps_file`; values as `samples` (named, aligned by name) or `trait_values` (positional), or a `basket` -- resolved by the same code as the direct runner (`resolve_inputs`), so the two cannot drift. Two fields are **refused rather than ignored**, because this workflow has no step for them: `covariates` (`422 covariates_not_supported`, use `/compute/gemma`) and `normalize` (`422 normalization_not_supported`). The same two steps (LOCO kinship, then LOCO GWA), but run by [ravanan](https://github.com/vepiteski/ravanan) driving a CWL workflow compiled on every call (via `ccwl compile`, see below) from the ccwl source at `cwl/gemma/gemma-loco.scm`, instead of this server's own direct subprocess calls -- an alternative runner for the same computation, kept side by side with `/compute/gemma` so the two can be compared (correctness, timing). **Ravanan's cache is only as durable as `RAVANAN_CACHE_DIR`.** Ravanan's reuse key is `(script, inputs)`, and both live in its `--store` under that directory, so the cache survives a container restart if — and only if — the directory is on a host path the container bind-mounts. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, store unchanged. The default is under `/tmp`, which is neither durable nor visible inside the container; the reply carries a note when it sees that. Same reasoning for `GN4_API_TOKENS_DB` (`./tokens.sqlite` by default) and `GEMMA_CACHE_DIR`. See [docs/ravanan-slurm.md](docs/ravanan-slurm.md#what-survives-a-restart). **K is shared across traits.** The `kinship` step is given a constant-valued phenotype file carrying only the trait's missing-data pattern, because `gemma -gk` uses `-p` solely to learn which individuals are analysed; `gwa` gets the real values. Two traits over the same animals therefore reuse all twenty K matrices -- on the BXD fixture a second trait adds ~1 MB of store and 20 GWA jobs rather than ~13 MB and a second kinship scatter. **This runner calls `gemma` itself, not `gemma-wrapper`** (the direct runner still calls the wrapper, which is what makes the comparison worth making). GEMMA 0.98.6 has `-loco ` natively for both `-gk` and `-lmm`, taking the chromosome to leave out by name and requiring the BIMBAM annotation file (`-a`) that every genotype shape here supplies -- so gemma-wrapper's LOCO orchestration only duplicates the tool. Its parallel fan-out was never used on this path anyway: ravanan scatters one `gemma -gk|-lmm ... -loco ` job per chromosome, and the kinship matrix flows from the kinship job to the GWA job as a CWL `File` output rather than through a wrapper cache. The wrapper's content-hash K cache is the one thing given up; a repeat request recomputes, which is what the per-chromosome scatter costs anyway. This workflow is written in [ccwl](https://ccwl.systemreboot.net/) at `cwl/gemma/gemma-loco.scm`, not raw CWL; `compute_ravanan` compiles it to CWL with `ccwl compile` fresh on every call (cheap -- pure macro expansion, no Guix build involved) into a generated, throwaway file under `ravanan_cache_dir`, never checked into the repo. `compute_ravanan` computes the chromosome list from `snps_file` itself (`chromosomes_from`) and passes it as the workflow's `chromosomes` input, since CWL's `scatter` needs a concrete array at invocation time -- and GEMMA's `-loco` takes exactly that chromosome name. `ravanan_cmd`, `ccwl_cmd`, `gemma_ravanan_workflow` (the ccwl source, not a CWL file), `gemma_ravanan_manifest` (ravanan's own required `--guix-manifest`, a Guix manifest naming `gemma`, at `cwl/gemma/manifest.scm`) and `ravanan_cache_dir` (its own cache directory, separate from `gemma_cache_dir`, for pheno files, run logs, the compiled workflow, and ravanan's own `--store`) are configurable the same way as the rest of `src/config.rs`'s layered config. Its genotype and SNP inputs follow the same convention as `/compute/gemma` -- the `bimbam/_geno.txt` + `_snps.txt` pair, not the `.geno` file (see [docs/genotypes.md](docs/genotypes.md)) -- and the values must likewise line up one for one with that file's individuals. ravanan's batch system is configurable too: by default each CWL step is a local child process, and `RAVANAN_BATCH_SYSTEM=slurm-api` submits every step to a Slurm cluster through slurmrestd instead (endpoint, JWT file, partition, nice and scratch directory via `RAVANAN_SLURM_*` / `RAVANAN_SCRATCH`). With slurm, `ravanan_cache_dir` must be on a filesystem the compute nodes can see, since ravanan's `--store` and scratch live there. See [docs/ravanan-slurm.md](docs/ravanan-slurm.md) for the keys, the fail-closed rules and the slurm-side auth requirements, and [cwl/README.md](cwl/README.md) for how a request gets from the API to a job on a node (and what the nodes themselves need). `GET /compute/gemma/status/{run_id}` is shared between both endpoints and checks whichever cache directory the `run_id` actually logged to. Verified end to end on 2026-11-12 against the Slurm container from `gn-machines` (`specials/slurm-container.scm`): a `group: BXD` request became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the SNP map's chromosomes), all completed, 21 055 rows. See [docs/ravanan-slurm.md](docs/ravanan-slurm.md) for what the host and the nodes must provide -- including the two requirements that bit during that run: `ravanan` must be a build with the `--slurm-api-endpoint` fix, and the *server's* `HOME` must be writable by the job user inside the cluster. The reply carries the same variance-component note `/compute/gemma` does (`-pve`, every chromosome). Since this workflow runs `gemma` itself, each job's `.log.txt` is a first-class CWL output and the note is read straight out of those files -- no archive to unpack and no wrapper-named copy to depend on. See [docs/ravanan-slurm.md](docs/ravanan-slurm.md). **Concurrency.** Like `/compute/gemma` -- same 50-individual threshold, since ravanan writes into the same cache directory -- only a large request claims a slot. One ravanan workflow at a time, one per caller: a second large submission while one runs (or is queued) is refused at once with `409 compute_busy`, naming the run and linking `/compute/gemma/status/{id}` (which ravanan shares with `/compute/gemma`); a submission from another caller waits for the slot. One slot because ravanan already fans its own per-chromosome steps out -- over the cluster, when the slurm backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from `/compute/gemma`'s: they are different runners of the same computation with their own cache directories. Exploratory, and genuinely rough around the edges -- three real bugs turned up building this, none of them in `/compute/gemma`'s own direct-subprocess path: - ravanan resolves `gemma-wrapper` by building a Guix profile from `--guix-manifest` against the *ambient* Guix process (`GUIX_PACKAGE_PATH`, no `--guix-channels`) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package in `guix-bioinformatics` too new for the pinned base Guix. This needs `GUIX_PACKAGE_PATH` set in whatever environment starts the genenetwork4 server itself (inherited by the `ravanan` child process genenetwork4 spawns); it is not something `compute_ravanan` sets. - ravanan's `--store` needs `files/`, `logs/` and `data/` subdirectories to already exist; it does not create them itself (`compute_ravanan` does now). - A step scattered over exactly one input (`kinship`'s `chr`) needs an explicit `scatterMethod` even though CWL itself only requires one for *multiple* scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it. - Every chromosome's `gwa.cwl` run shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in `--chromosomes`), so the 20 scattered `archive` outputs all collide under ravanan's optional `--output-directory` symlink-everything-into-one-flat-dir step; `compute_ravanan` simply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own `--store` via its stdout JSON record, never through `--output-directory`. A real end-to-end run against the BXD fixtures (same rows as the `/compute/gemma` example above, confirmed byte-for-byte including `rs31443144`'s exact values) took **16.7s** wall-clock scattered across 20 chromosomes this way, vs. `/compute/gemma`'s own ~19s and the earlier sequential (non-scattered) ravanan workflow's ~21s -- a real, if modest, win from letting ravanan's own scheduler fan the chromosomes out instead of going through gemma-wrapper's internal GNU `parallel` call, on top of no longer needing that path's own `PARALLEL_SHELL` workaround at all (each scattered job's `--no-parallel` gemma-wrapper call never invokes GNU `parallel` in the first place, so the bash-`time`-keyword bug it worked around does not arise here). Those 16.7s are with ravanan's default **`single-machine`** batch system, where a step is just a local child process. With `RAVANAN_BATCH_SYSTEM=slurm-api` every step additionally pays Slurm dispatch -- submit, scheduler tick, launch, poll -- which on a small input dwarfs the mapping itself: three chromosomes, measured on the test container (`gn-machines/specials/slurm-container.scm`, one node, two CPUs), are **2.3s** cold / **0.7s** warm on `/compute/gemma`, **4.7s** / **3.5s** on ravanan single-machine, and **33s** on ravanan+Slurm with a 5s scheduler interval (**158s** with Slurm's defaults), while each job's own `gemma-wrapper` run took under a second. Same rows in every case. See [docs/ravanan-slurm.md](docs/ravanan-slurm.md) for the breakdown and what to tune. ### `/compute/qtl2` A genome scan with **R/qtl2**: genotype probabilities from a hidden Markov model, and a linear mixed model with leave-one-chromosome-out kinship. It is the middle ground between `/compute/rqtl` (Haley-Knott, no kinship) and `/compute/gemma` (an LMM, but biallelic BIMBAM with mean-imputed genotypes), and it is how multiparent populations are analysed. ```json { "group": "BXD", "genotype_version": "BXD.4", "trait_name": "BXD::10001", "samples": [{"name": "BXD1", "value": 61.4}, …], "kinship": "loco", // loco (default) | overall | none "model": "normal", // normal (default) | binary "cores": 4 } ``` Rows are `marker, chr, pos (Mb), lod` — the same shape `/compute/rqtl` returns, so **`/plot/rqtl` draws it unchanged**. On DOL trait 10003 the three engines tell the same story with different emphasis, which is what modeling relatedness buys: R/qtl's HK scan peaks on chr9 at LOD 5.86 and does not see chr16 at all; GEMMA's LMM prefers chr16 at −log10 p 7.15; R/qtl2's LMM finds chr9 (LOD 4.85), chr19 (4.76) and chr16 (4.24) on one footing. **Autosomes only.** The X needs a sex covariate to be fitted properly, and the Y and mitochondrion degenerate the per-chromosome kinship — GEMMA's LOCO run is given the same restriction, for the same reason. The `notes` say how many markers were left out. **Multiparent crosses.** A multiparent population cannot come from a `.geno`: that file carries two codes per marker (`B`/`D`, `A`/`H`/`B`), which is a collapse, not a founder allele. It has to ship its own qtl2 control file with founder genotypes, so the endpoint also takes a cross by name: ```json {"rqtl2_cross": "do", "trait_name": "DO::WBC", "samples": [{"name": "JAXW202.2129", "value": 4.34}, …]} ``` The name is looked up as `genotype/rqtl2/{name}.json`, then `{name}/{name}.json`, then `{name}.json` under `GENOTYPE_FILES`; `do/` is the one such cross this deployment has (742 individuals, 8 founders A–H, 6 413 markers, WBC and NEUT). The samples are matched to the cross by name and anything outside its population is left out (and counted in `notes`). A multiparent scan is a **different claim** from a biallelic one: every marker is fitted with one coefficient per founder, so the LOD asks whether *any* founder differs, not which. Reading a multiparent LOD as a strain effect is the usual mistake; per-founder effect sizes need `scan1coef`, which this endpoint does not return yet. Measured on the DO (`WBC`, 8 cores): genotype probabilities 174 s and 1.39 GB, cached per cross and keyed by the control file *and its directory*; LOCO kinship 15 s, cached; the scan itself 21 s. `scan1` is run on one core for a multiparent cross deliberately — its forked path stalls here on allele probabilities (25+ minutes with 8 cores and no CPU consumed, against 21 s serial). The autosomes-only rule applies here too: the DO's X is left out (369 markers) rather than fitted without a sex covariate. **Panel conversion.** The cross is generated from this deployment's own `.geno` file and cached (`QTL2_CACHE_DIR`), because the shipped `genotype/rqtl2/` control files are not usable: they name paths relative to a directory the data is not in, declare no genotype-code map (every genotype reads as missing) and no orientation (individuals read as markers) — four traps that each produce a plausible flat scan rather than an error. The conversion also drops heterozygous calls on a `riself` panel, as R/qtl's `convert2riself` does. Configuration: `QTL2_SCRIPT_PATH`, `QTL2_CACHE_DIR`. ### Gene link-outs A gene symbol is useful beyond our own pages, and SKILL.md carries the table of external services that still resolve (checked 2026-11-11): RGD for **both mouse and rat**, MGI, the Alliance of Genome Resources, NCBI Gene, Wikidata (the QID that `/genes/{symbol}/aliases` already returns), UniProt, Ensembl, GTEx, STRING, GeneMANIA, Open Targets, the Human Protein Atlas, the GWAS Catalog, BioGPS, the Allen Brain Atlas and GeneCup — with the retired ones named so they are not offered by mistake (HomoloGene, UniGene, Genotation, GeneBridge, PhenoGen). Two conventions that keep the links honest: **quote the species** the symbol came from, and prefer RGD/GeneMANIA(rat) for a rat symbol and GTEx/Open Targets/ Protein Atlas for a human one — a mouse symbol in GTEx resolves to an empty report, because GTEx is human-only. ### Why R/qtl and R/qtl2 disagree on a `riset` panel They read the same file and can end up fitting different predictors, so a peak in one is not a bug in the other. The mechanism, with BXD trait 16627 as the worked case (BXD.8, 35 phenotyped animals, all measured on production): | marker | Mb | R/qtl | R/qtl2 `riself` | R/qtl2 `f2` | |---|---|---|---|---| | `rsm10000000889` | 5.652 | 9.59 | 3.79 | 9.98 | | `rs33299937` | 5.653 | 9.59 | 3.73 | 9.68 | | `rs13476332` | 6.311 | 9.59 | 3.70 | — | **What R/qtl does.** `scripts/rqtl_wrapper.R` reads the cross with `genotypes = c(@mat, @het, @pat)` — codes 1, 2, 3 — and then, for a `riset` panel, calls `convert2riself`, which is *adaptive*: ```r gtab <- table(pull.geno(cross)) usethree <- !(gtab["3"] < gtab["2"]) # is code 3 at least as common as code 2? if (usethree) { dat[!is.na(dat) & dat != 1 & dat != 3] <- NA; dat[dat == 3] <- 2 } else { dat[!is.na(dat) & dat != 1 & dat != 2] <- NA } ``` It keeps the **two most numerous codes** and drops the third (warning `Omitting N genotypes with code==2`), then recodes to 1/2. No HMM, no probabilities: `scanone` fits each marker's own calls. **What R/qtl2 does.** `riself`'s state space is `AA, BB` (two states, no het), so a heterozygous call is not representable and becomes missing; the HMM then re-imputes it from flanking markers, and `scan1` fits *probabilities*, not calls. On BXD.8 the conversion drops **610 684 calls, 12.7 % of the matrix** (the reply says so). At a marker with 14 % H, that is a different predictor from R/qtl's, which is why the peak survives in one engine and thins out in the other — not a failure of either. `genotype_model: "f2"` keeps the calls as a third genotype (`AA, AB, BB`) and recovers R/qtl's number (9.68 against 9.59). It is a **diagnostic**: it fits het as a real genotype *and* assumes an outbred intercross design, which an inbred line is not. Its value is that it separates "the conversion cost the information" from "the model cannot see the locus". **What the proper model is, and where it stands.** qtl2 has general RIL types — `genril2` / `genail2`, from qtl2 0.18: *“handle any number of founders; include the number of founders in the cross type… `genril` assumes the progeny are inbred lines (RIL), while `genail` assumes the progeny have two random chromosomes (AIL)”*, with `cross_info` of length `1 + n_founders`: the first column the number of **generations**, the rest founder frequencies. Both are supported by the qtl2 0.42 this deployment runs (checked through `qtl2:::check_crosstype()`), and `genril2` is the model that fits a `riset` panel's residual heterozygosity honestly. It is **not wired yet**, and the reason is concrete rather than a matter of effort: a founder-based cross type expects individual genotypes coded as *alleles/pairs* of founders (as the DO's `A`–`H` are), not the `1/2/3` of a two-parent cross; a prototype with numeric codes had every genotype read as missing. The remaining work is therefore exact: build `founder_geno` from the two parental alleles (for a two-parent cross every marker is homozygous founder 1 vs founder 2 by definition of the coding), write `cross_info` with the generation count, and code `H` as the founder *pair* rather than code 2. Then compare trait 16627 against 9.59/9.68 — and check DOL through the same path, because its `H` is not residual heterozygosity at all: it is a collapsed DO, where `H` means "two different founders" and the honest model is `crosstype: "do"` with founder genotypes the collapse destroyed. **What to do when the engines disagree**, in order: `.../markers/{marker}/classes?trait=` to see the class sizes and means; `.../frequencies` for the H share at the marker; `genotype_model: "f2"` to see what keeping the calls would give; and quote which model ran, because a reply says `Cross: riself` or `Cross: f2`. ### `/compute/bulklmm` A genome scan that can be asked for **thresholds**. GEMMA's LOCO scan is the reference for a biallelic panel, but a GEMMA permutation repeats the whole scan, which is why `/compute/gemma` refuses `nperm`; the usual stand-in is R/qtl's `nperm`, whose null shuffles phenotypes as if the individuals were unrelated. [BulkLMM.jl](https://github.com/senresearch/BulkLMM.jl) fits one LMM per marker with a global kinship matrix and permutes by *rotating* the data through the kinship's eigen decomposition — relatedness included — so a threshold costs seconds: ```json {"group": "BXD", "genotype_version": "BXD.8", "trait_name": "BXD::16627", "samples": [{"name": "BXD1", "value": 1.5}, …], "nperm": 200, "seed": 42} ``` BXD trait 16627, 35 phenotyped animals, 21 056 markers, 22 s: ``` peak rsm10000000888 chr2 4.92 Mb LOD 9.59 beta 0.0027 h2 3.0e-16 permutation thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47 ``` which is the trait's published locus (`PublishXRef.Locus` `rsm10000000888`, LOD 9.59) and a genome-wide-significant peak against a threshold this API could not compute before. Rows are `marker, chr, pos (Mb), lod, beta, h2`. **Thresholds on another engine's plot.** `/plot/gemma` draws **no reference line by default** — a line on a plot is an annotation, not part of the result, and a reader should be told where each one came from. GEMMA's Bonferroni line is one more opt-in labelling (`?bonferroni=true`); a threshold computed here travels with the plot, labelled, because the two are different nulls: ``` POST /plot/gemma?thresholds=3.50,3.65&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05 { the /compute/gemma reply } ``` Each line is drawn dashed in the series colour with its label and value (`bulklmm alpha 0.05: 3.65`), and the Bonferroni line only appears when it is asked for — so a plot shows exactly the lines someone chose, each named. `/compute/bulklmm`'s reply already offers that href with its α = 0.05 threshold filled in, so the picture is one link away. `/plot/bulklmm` draws two reference lines for it: the genome-wide α = 0.05 and the stricter α = 0.01. The reply also reports α = 0.10, but that is the loosest bar in the set — drawing it invites reading it as the line to clear — so it appears only on request (`?all_thresholds=true`, or an explicit `?thresholds=3.50&threshold_labels=alpha 0.10`). GEMMA's Bonferroni line is opt-in in the same spirit: a reference line is an annotation, not a result. **About cores.** BulkLMM's single-trait `scan()` — what this endpoint calls — is **serial in Julia by design**: the package threads `bulkscan()` (many traits) and its helpers, not `scan()` (checked in its source: `Threads.@threads` appears in `bulkscan.jl` and `bulkscan_helpers.jl`, not in `scan.jl`). What was parallel before was BLAS, not the scan. The driver therefore splits the markers into independent blocks and scans them on Julia threads (`--blocks`, default = `BULKLMM_THREADS`), which is exact — the LODs and permutation thresholds are identical to a serial run, verified bit-for-bit at 1 and 4 blocks — and caches the parsed matrix and the kinship per panel: cold, 1 block : read 1.3 s, scan 9.0 s, total 20.9 s cold, 4 blocks: read 0.4 s, scan 7.5 s, total 18.3 s (same LOD 9.589938) The gain is modest at this size because the LMM loop is only about 8 s of the run and each block re-estimates the variance components; the rest is Julia startup. **Where the cores pay is `bulkscan()`** — a hundred traits in one process, threaded over trait blocks. That is the shape an eQTL endpoint wants, and it is the next thing worth adding here. Two differences from GEMMA are stated in every reply rather than implied: * **a global kinship matrix, not leave-one-chromosome-out**, so its p-values are not GEMMA's; * **`h2_null` says how much relatedness it found**. Here it is ~0, so the fit is effectively a fixed-effects model — which is exactly why it agrees with R/qtl on this trait where GEMMA's LOCO scan differs. See [docs/bulklmm.md](docs/bulklmm.md) for the model comparison, the measured core behaviour and the packaging options. The engine is a Julia subprocess (`BULKLMM_CMD`, `BULKLMM_ENV`, `BULKLMM_DEPOT`, `BULKLMM_SCRIPT`, `BULKLMM_THREADS`, `BULKLMM_CACHE_DIR`) with the driver in `scripts/bulklmm_scan.jl`. The Julia depot is prepared once, deliberately, and is small: 79 packages, 139 MB, no GPU/CUDA stack — BulkLMM's closure is CSV, DataFrames, Distributions, Optim and what those need. Every request runs with `JULIA_PKG_OFFLINE=true`, so a scan can never download anything. ### `/compute/bnw` A Bayesian network over a set of variables, learned by BNW (the Bayesian Network Webserver): which traits belong together under a causal ordering. The computation runs in BNW's own headless driver — the `bnw` program in the `gn-bnw` repository — so this endpoint only prepares a matrix, calls out, and parses the reply (`BNW_CMD`, `BNW_BIN_DIR`, `BNW_CACHE_DIR`). ```json { "group": "BXD", "traits": [10001, 10003, 10038], "maxparents": 4, "k": 1, "tiers": 3, "threshold": 0.5 } ``` or, for a matrix the database does not hold (a basket's columns, a handful of probe sets): ```json { "values": [ {"name": "Trait1", "samples": [{"name": "BXD1", "value": 1.2}, …]}, {"name": "Trait2", "samples": [{"name": "BXD1", "value": 0.4}, …]} ] } ``` Rows are positional with `meta.columns`: ``` from, to, posterior, in_structure ``` One row per candidate edge (variables²−variables of them: self-edges are dropped), strongest posterior first. `posterior` is the share of the *k*-best networks that contain the edge; `in_structure` says whether it survived `threshold`. The four parameters default to BNW's own (`maxparents` 4, `k` 1, `tiers` 3, `threshold` 0.5 — from `create_tiers_gom_part1.php`), and the reply names the values used, because a network at threshold 0.5 and one at 0.9 are different answers. **A posterior is not a p-value.** It says how often the search put that arrow there in *this sample* — a statement about correlation under a causal ordering, not about surprise, and not an effect size. At most 40 variables (the search is exponential in in-degree) and at least 3 samples complete across every variable: BNW models a complete multivariate normal, so a sample missing any value is dropped and the reply says how many survived. `POST /plot/bnw` draws it from the posted reply, the same way `/plot/gemma` does — an SVG back, nothing recomputed, the plotter's one-line summary in `x-plot-summary`. The default figure is the **network**: nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge posterior, and the below-threshold edges as faint dotted lines so a reader sees what was *considered* as well as what survived. `?kind=posterior` draws every candidate edge as a cell in a matrix instead, which is the view that says how close a rejected edge came to the threshold. Pure-Python SVG (`BNW_PLOTTER`, run by `PLOT_PYTHON`), so no graphviz is needed in the deployment. **From a network to a mappable trait: `POST /compute/bnw/features`.** Every node with at least one parent at the threshold is a *module* — the node and its parents — scored per sample by the **first principal component** of the module's standardised members, sign-corrected so the score rises with the node. Without `?module=` the reply is one row per module, `module, members, size, variance_explained` (the share of the module's variance the component explains, which says whether the score is a fair summary). With `?module=NAME` it is that module's per-sample scores in the `{name, value}` shape every mapping endpoint already takes — post them as `samples` to `/compute/rqtl`, `/compute/gemma` or `/compute/qtl2` and map the module like any trait. That is the chain BNW exists for in GeneNetwork: structure → module → QTL. A module is a correlation structure, not a QTL: what maps is where its members' shared variation comes from, not which member causes which. **BNW's own worked examples: `GET /compute/bnw/examples`** and `GET /compute/bnw/examples/{name}`. The five datasets its tutorial and examples pages present (synthetic 2-genotype/6-trait, the scientific 5-node network at 100 and 500 samples, BXD spleen immune genes, the time series) are listed with their sizes, and each one is served as a matrix in the `values` shape `/compute/bnw` takes — with a ready-to-post action carrying the body. `?columns=` restricts the matrix, which the spleen example needs (14 variables, and a caller may want fewer). `genotype_column` warns that a column is a genotype code: this driver is continuous-only, so such a column enters the network as a number, as the web application's GeneNetwork path treats it. The files are read from `BNW_EXAMPLES_DIR` (a gn-bnw checkout's `sourcecodes/example_datasets`); a deployment without it answers `404 no_examples` rather than pretending. The driver's contract is documented in [gn-bnw/info_files/gn4-integration-plan.md](../gn-bnw/info_files/gn4-integration-plan.md); `tests/bnw.rs` checks the endpoint against the answer the BNW *web application* gives for its own six-trait example, so "it works" means the same edges, not merely a 200. ### `/compute/fitqtl` `/compute/rqtl` says *where* a QTL is (a LOD per marker); this says *how much*, by fitting R/qtl's `fitqtl` at the markers you name -- normally the peaks a scan already reported, via `GET /groups/{group}/genotypes/{version}/markers`. It is separate because it is a separate question, asked about a few positions rather than a genome: `fitqtl` fits one model per marker, so a genome-wide version would be thousands of fits (the cap is 100 markers per request). ```json { "group": "BXD", "genotype_version": "BXD.4", "trait_name": "BXD::10001", "samples": [{"name": "BXD1", "value": 61.4}, …], "markers": ["mCV24409492", "rs8253516"] } ``` Rows are positional with `meta.columns`: ``` marker, chr, pos, lod, pct_var, additive, se, t, p ``` `additive` is fitqtl's estimate for the QTL term. R/qtl codes the QTL genotype ±1, so this is half the difference between the two homozygote classes -- the same convention GeneNetwork's own `additive` uses, and on BXD trait 10001 it reproduces the published value to six decimals (`2.394472` here against GN2's `2.394444`, from the same peak region). `pct_var` is the model's percentage of variance explained and `p` its p-value; `se` and `t` belong to the effect. Markers not in the genotype file, or that fitqtl cannot fit, are absent rather than reported as zero. **Covariates** work exactly as on `/compute/rqtl` -- `covariates: [{"case_attribute": "Status"}]`, inline values, or an explicit `map` -- and are fitted alongside the QTL, so the effect reported is the one *after* conditioning. Two details worth knowing: a covariate that is spread across the panel but constant among the individuals a trait covers is left out, with the columns named in `notes` (R/qtl stops with "system is exactly singular" otherwise), and the LOD then no longer equals the scan's, because it is a different model. The cross-check that matters: without covariates, `fitqtl`'s `lod` at a marker **equals** `scanone`'s LOD at that marker, to every digit the CSV carries -- the test suite pins it. Same alignment rules as `/compute/rqtl` (values matched onto the panel by name, parents/F1s excluded unless `include_parents_f1`), the same `normalize`/`winsor_*` pipeline (and the effects then come out on that scale), and the same R/qtl gate. Configuration: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`), `RQTL_CACHE_DIR` for its scratch files. ### `/compute/rqtl` Maps a QTL by running GeneNetwork3's own `scripts/rqtl_wrapper.R` (via `Rscript`), copied into this repo verbatim: a `scanone` genome scan, plus permutation thresholds if `nperm > 0`. Same request/response shape as `/compute/gemma` -- synchronous, `?id=`/`GET .../status/{run_id}` progress-streaming reusing the same infrastructure (`src/streaming.rs`) -- see `src/rqtl.rs` for the invocation itself, its documented scope (no interval mapping or pair-scan `scantwo` -- just the common single-scan case), and a fix made relative to GeneNetwork3: its own cache-key hashing of the genotype/phenotype files never actually looks at their contents (a key-name mismatch between `generate_rqtl_cmd` and its real caller), so two different phenotypes mapped with the same other options could silently reuse each other's cached result there; fixed here by always hashing both files' contents. **Covariates.** The same `covariates` list as `/compute/gemma` (above) -- case attributes by name, or per-sample values -- fitted here with `--addcovar` and a generated `--covarstruct` file. R/qtl's wrapper takes categorical covariates as a single column and dummy-expands more than two levels itself, so a level with no observations is its problem, not a singular column: no columns are dropped on this path. A missing value is `x` (the marker the wrapper calls missing), so R/qtl drops those individuals; the coding and the count of complete cases come back in `notes`. **Give it a group, not a path.** `group` (an InbredSet name) plus an optional `genotype_version` resolves the `.geno` server-side; see `GET /groups/{group}/genotypes`. `samples` are then matched onto the panel **by name** -- unknown names dropped, missing individuals `null` -- and parents/F1s are excluded unless `include_parents_f1` is set. The resolved file, version, panel size and how many values aligned come back in `notes`, so a run is reproducible without the caller ever naming a path. `genotype_file` is still accepted as an explicit override. **If you name the file yourself**, R/qtl takes the GN genotype file itself -- `genotype/.geno`, e.g. `genotype/BXD.8.geno` (235 individuals × 21 057 markers) or `genotype/HET3-ITP.12.geno` (6 447 individuals × 894 markers) -- and reads its format directly. Which engine a group offers, and which of its genotype files is current, are GeneNetwork2/3 conventions summarised in [docs/genotypes.md](docs/genotypes.md); `InbredSet.MappingMethodId` is what decides the engine (BXD = both, HET3-ITP = R/qtl only -- UM-HET3 is a 4-way cross). Measured on this deployment: BXD with the WGS panel, 235 samples of which 34 carry values, 21 056 markers in 7.5 s; UM-HET3 lifespan over 6 447 individuals, 893 markers in 83 s, peak LOD 7.95 at chr 1, 24.04 Mb. **Important:** R/qtl's cross file format has no per-sample name matching of its own -- `samples` must list values in exactly the genotype file's sample-column order, and it must have as many entries as the genotype file has individuals. The second half of that used to go unchecked (as it does in GeneNetwork3): nine values scanned against a six-individual cross came back as a complete, plausible-looking set of LOD scores belonging to nobody. `scripts/rqtl_wrapper.R` now stops with "the phenotype file has 9 samples but the genotype file has 6 individuals" instead, which surfaces as a `502 rqtl_failed`. It still cannot tell whether the *order* is right -- that is the caller's responsibility, and the reason a trait's values and its genotype file have to come from the same panel. ```json { "data": { "markers": [ ["rs31443144", "1", 3.010274, 0.0923496], ["rs6269442", "1", 3.492195, 0.1131095] ], "permutation": null }, "meta": { "state": "complete", "took_ms": 420, "request_id": "...", "columns": ["name", "chr", "pos", "lod"], "num_rows": 6, "num_cols": 4 }, "notes": [ "markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.", "Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename -output.csv" ], "next": [] } ``` (Real output, against the tiny 6-marker/6-sample BXD riset fixture in `tests/test_data/`, trimmed to two markers. Each marker row is positional, in `meta.columns`' order, rather than a `{"name": ..., "chr": ..., ...}` object per row -- same space-saving shape `/compute/gemma` uses for its own (much larger) rows; a 4-way cross's extra mean/SE effect columns, if present, trail each row and aren't named in `meta.columns` (see `MarkerResult`'s doc comment in `src/rqtl.rs`). `permutation` is `null` unless `nperm` was set above 0; when it is, it's `{"values": [...], "suggestive": ..., "significant": ...}`, the 67th- and 95th-percentile thresholds GeneNetwork3 itself uses -- computed here with `numpy.percentile`'s default linear interpolation, reproduced in `src/rqtl.rs`'s `percentile`.) ## Genotype files `/compute/rqtl` and `/compute/gemma` take **paths on the server**, so which panel file is current, in which format, and in which sample order is a deployment question rather than an API one. That convention (GN2's per-group `genotype/.json`, whose `genofile` list is newest-first; `genotype/bimbam/_geno.txt` for GEMMA; the `.geno` header for the sample order; `InbredSet.MappingMethodId` for which engine a group offers) is written up in [`docs/genotypes.md`](docs/genotypes.md), together with the three mappings measured against this deployment. ## OpenAPI `GET /openapi.json` serves an OpenAPI 3.1 document covering every route. It's built with [utoipa](https://docs.rs/utoipa) and [utoipa-axum](https://docs.rs/utoipa-axum): - Each handler carries a `#[utoipa::path(...)]` attribute describing its method, path, parameters and responses; each response/request struct derives `utoipa::ToSchema` (see `VersionInfo`, `db::Species`, `db::SpeciesDetail`, `error::ApiError` in `src/lib.rs`/`src/db.rs`/ `src/error.rs`). - Routes are registered with `utoipa_axum::routes!(handler)` instead of plain `axum::routing::get(handler)`, on an `OpenApiRouter` instead of a plain `Router`, so each route's metadata is collected as it's added. - `ApiDoc` (a `#[derive(utoipa::OpenApi)]` struct) supplies the document's `info` (title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's. - `OpenApiRouter::split_for_parts()` splits the builder back into a plain `axum::Router` (returned from `app()`) and the assembled `utoipa::openapi::OpenApi` value, which the `/openapi.json` handler then just serializes with `Json`. - `ApiDoc`'s `#[openapi(security(("bearer_token" = [])), ...)]` makes a token the default for every operation, so an endpoint that needs one doesn't repeat `security(...)` itself -- only the handful that must NOT require one (`/`, `/version`, `/get-test-token`) override it with an empty `security()` of their own. `error::Unauthorized` (a `#[derive(utoipa::IntoResponses)]` type) is the shared 401 response every gated endpoint references by name in its own `responses(...)` list (`responses(Unauthorized, (status = 200, ...), ...)`) instead of writing out `(status = 401, description = "...", body = ApiError)` by hand each time. Because the attributes are declared alongside the handlers they describe, the spec can't silently drift out of sync with the code the way a hand-maintained spec file could -- adding or changing a route without updating its `#[utoipa::path(...)]` is a compile error (the macro checks the response/schema types exist) or an obviously incomplete diff, not something to notice weeks later. The document is assembled once, when `app()` runs at process startup (not at `cargo build` time, and not freshly on every request): the `#[utoipa::path]`/`#[derive(ToSchema)]` macros expand to Rust code at compile time, but the actual merging of all routes' metadata into one `OpenApi` value happens the first time `app()` executes. Every `GET /openapi.json` request after that just serializes the same already-built value. ## Success responses Every success response is one JSON envelope (`src/response.rs`): the data, next to facts about the response, remarks on the data, and suggested follow-up calls. It is the success counterpart of the error shape below. ```json { "data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."}, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] } ``` - `data` is the response proper: exactly what each endpoint below documents. - `meta.state` is `complete`, or `partial` when the response is usable but something was left out or unavailable -- `notes` says what. For example `/search` is `partial` when the database could not be reached (hits come back without `locus` and `abbreviation`) or when a sorted search ranked a full 1000-match window and more matches exist; `/publications/{pubmed_id}/traits` is `partial` when traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure. - `meta.took_ms` is the server's time for the request, from when it started handling it; it excludes the network. `meta.request_id` is the request's id, also in the `x-request-id` response header and in any error body, so a report can be correlated with logs. - `x-gn4-user` names the caller on every **authenticated** response: the token's `sub` (e.g. `pjotrp`). It is there for the proxy's access log -- `user=$upstream_http_x_gn4_user` in the `gn4` log format -- so the deployment can answer "who is using this, successfully" rather than guessing from addresses and user agents. Unauthenticated responses (`/version`, `/openapi.json`, a `401`) carry no such header, and the caller only ever sees an identity it already presented. - The same identity goes into this server's own access log: one JSON line per request on stderr — `{"event":"request","method","path","status","took_ms", "request_id","user"}` — with `user` absent when no token was verified, the path stripped of its query string, and the token itself never logged. `GN4_ACCESS_LOG=false` turns it off. The `event: error` lines for 5xx go to the same stream and carry a `request_id`, so a failure and the request that caused it are one `grep` apart. - `notes` are server-authored remarks on `data`, e.g. `lod = lrs / 4.604` on responses that carry `lod`. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's. - `next` lists follow-up calls, each a relative `href` that resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large). `GET /search` suggests the next page and the top hit's metadata. - `actions` lists suggested calls that are **not** plain `GET`s, so they cannot be `next` links: each carries `method`, `href`, a `description`, and a `body` where the server can build one. It is present only when there is something to suggest, so most responses keep the four members above. Today the per-strain values endpoints (`/groups/{group}/traits/{id}/values`, `/datasets/{dataset}/probesets/ {probeset}/values`, `/traits/{dataset}/{trait}/values`) use it to hand back the values they just returned, ready to map: ```json "actions": [ {"method": "POST", "href": "/compute/rqtl", "description": "Map 'BXDPublish::10001' with R/qtl (scanone): this body carries the 34 values above as `samples`, in the same order. Replace the `genotype_file` placeholder with a path that exists on the server. …", "body": {"genotype_file": "", "trait_name": "BXDPublish::10001", "samples": [{"name": "BXD1", "value": 61.400001525878906}, "…"]}}, {"method": "POST", "href": "/compute/gemma", "description": "…", "body": {"genotype_file": "<…>", "snps_file": "<…>", "trait_values": [61.400001525878906, "…"]}} ] ``` The file paths stay bracketed placeholders on purpose: they have to exist on the server's own filesystem, which the server cannot know on the caller's behalf (see the `/compute` sections below). `/` returns the plain text `Hello, world!` and `/openapi.json` the OpenAPI document itself; neither is wrapped. In `/openapi.json` every other success response is an `Envelope_*` schema (a test enforces this). ## Error handling Every endpoint uses the same `ApiError` JSON shape on error, modeled on RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see `src/error.rs`). `/species` and `/species/{species}` can actually produce one today (database errors, or a 404 for an unknown species); `/` and `/version` document the same 500 response in `/openapi.json` for consistency, even though neither has a failure path currently (no database access, no I/O): ```json { "type": "/errors/species_not_found", "title": "Species not found", "status": 404, "code": "species_not_found", "detail": "No species matching 'nonexistent'.", "operation_id": "species_by_name", "spec_url": "/openapi.json", "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f", "took_ms": 2, "next": "/species" } ``` `next` is omitted (`null`) when there's no obviously related endpoint to suggest, e.g. a hypothetical error from `/`: ```json { "type": "/errors/internal_error", "title": "Internal error", "status": 500, "code": "internal_error", "detail": "example failure for /", "operation_id": "hello_world", "spec_url": "/openapi.json", "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f", "took_ms": 1, "next": null } ``` - `type` and `spec_url` are relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allows `type` to be a relative URI reference. `spec_url` is always `/openapi.json`, on every error from every endpoint. - `operation_id` is the OpenAPI operation that failed (same name as in `/openapi.json`). - `request_id` is the request's `x-request-id`, generated per request (or echoed back if the client already sent one) by `SetRequestIdLayer`/`PropagateRequestIdLayer` in `src/lib.rs`, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token. - `took_ms` is the server's time for the request up to the error, the same measure as a success response's `meta.took_ms` (see [Success responses](#success-responses)). RFC 7807 allows extension members like this one, `request_id`, `code` and `next` next to its standard ones. - `next` is an optional hint at a related endpoint to try instead; e.g. a 404 from `/species/{species}` sets `next` to `/species`, so a caller with a wrong species name can discover the valid ones. ## Running with cargo Requires a Rust toolchain with cargo (edition 2024). ``` cargo run ``` The server listens on `0.0.0.0:3000` by default. Set `GENENETWORK4_PORT` to use a different port, or `GENENETWORK4_BIND` to bind a specific address instead of every interface: ``` GENENETWORK4_PORT=8080 cargo run GENENETWORK4_BIND=127.0.0.1 cargo run ``` ### This deployment runs two instances: 3000 (production) and 3001 (development) The reference deployment keeps a development instance next to the live one, on the same box, so work can be tried out without disturbing what the site is serving: | instance | port | binary | run caches | proxied by nginx | |---|---|---|---|---| | **production** | `3000` | `target/release/genenetwork4` | defaults (`/tmp/gn4-gemma`, `/tmp/gn4-rqtl`, …) | yes, at `/gn4/` (see [docs/nginx.md](docs/nginx.md)) | | **development** | `3001` | `target/debug/genenetwork4` | `/tmp/gn4-dev/{gemma,rqtl,ravanan}` | no | They differ only in the port, the build profile and those cache directories; both use the same read-only `SQL_URI`, the same Xapian index and the same auth stub, so a request behaves the same on either. The separate caches matter: a GEMMA or R/qtl run on the development instance must not fill, or be sped up by, production's warm cache. ```sh P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT') # needs R + gemma-wrapper on PATH # production PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \ GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3000 GENENETWORK4_TEST_TOKEN=false \ AUTH_SERVER_URL=http://127.0.0.1:8081/ \ SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \ XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \ nohup ./target/release/genenetwork4 > /tmp/gn4-3000.log 2>&1 & # pid in /tmp/gn4.pid # development: same, plus the port and its own caches PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \ GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3001 GENENETWORK4_TEST_TOKEN=false \ AUTH_SERVER_URL=http://127.0.0.1:8081/ \ SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \ XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \ GEMMA_CACHE_DIR=/tmp/gn4-dev/gemma RQTL_CACHE_DIR=/tmp/gn4-dev/rqtl \ RAVANAN_CACHE_DIR=/tmp/gn4-dev/ravanan \ nohup ./target/debug/genenetwork4 > /tmp/gn4-dev.log 2>&1 & # pid in /tmp/gn4-dev.pid ``` Both need the `PATH`/`R_LIBS_SITE` pair (see "Building with cargo, outside a bare toolchain" below) or R/qtl fails with `there is no package called 'optparse'`, and both need the auth stub on `127.0.0.1:8081` to serve `/auth/public-jwks` (see "Tests"). Anything the server keeps in memory is per instance: the pool, the JWKS and authorisation caches, and any in-process compute limit. So the two instances do not serialise against each other — a mapping on `:3001` runs alongside one on `:3000` — which is the point of the split, and a good reason not to treat the development instance as a capacity test of what production will do. ### Building with cargo, outside a bare toolchain `cargo build` in the package's own dependency shell is the supported way to build; a bare `cargo build` against a distro toolchain fails, and one of the reasons cannot be fixed by installing packages, since it is a Xapian version conflict (see [Running with Guix](#running-with-guix) for the pin in `guix.scm`). ``` GUIX_PACKAGE_PATH= \ guix shell -D -f guix.scm -- bash -c ' export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib # autocxx: libclang 13, not 22 export DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib # libduckdb-sys: no duckdb.pc export DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include cargo build --release' ``` - `GUIX_PACKAGE_PATH` (or `-L`) is for `(gn packages gemma)`; without it `guix.scm` fails to load with "no code for module (gn packages gemma)". - The three exported variables are exactly what `guix.scm`'s `set-duckdb-paths` phase exports for the packaged build. Without `LIBCLANG_PATH` the build stops in `xapian-rs`'s build script with "Unable to find libclang"; without the `DUCKDB_*` pair `libduckdb-sys` links a `-lduckdb` with no search path. - The resulting binary resolves every shared library from the shell's store paths on its own (Guix's linker wrapper records them as a RUNPATH), so running it needs no `LD_LIBRARY_PATH` -- but it does stop working once those store paths are garbage-collected. ### Project memory: why this code looks like this The repository carries an `ai-memory` store (`ai-memory.sqlite`, gitignored) that links each commit to the reasoning behind it, plus an optional Xapian topic index over both. It is how a session that starts with an empty context recovers what the last one worked out — measured at 16 ms for a text query and about 100 ms through the topic index: ```sh ai-memory -C . search 'pve' # reasoning, full-text ai-memory -C . why gemma.compute_compute # commits that touched a symbol, with its reasoning ai-memory -C . --index ~/.cache/gn4-topics search 'topic:ravanan-slurm' ``` Operational runbooks live there too — for example the Slurm container from `gn-machines`, its JWT, the environment `/compute/gemma/ravanan` needs, and the fastest endpoint test — because the commands are the part a reader cannot reconstruct from a diff. The store is local and per repository: it survives a cleared session, not a lost machine, so every conclusion is also in a commit message and in these docs. ## Development Every endpoint needs a bearer token (see the note under [Endpoints](#endpoints)), which normally means getting one from gn-auth yourself. For local development, genenetwork4 can hand you one instead: `POST /get-test-token` calls gn-auth's own password grant (`POST /auth/token`) for gn-auth's well-known development user and returns the result. It needs no setting and is not tied to a mode. ``` cargo run ``` Get a token, then use it against a real endpoint: ``` curl -X POST http://127.0.0.1:3000/get-test-token ``` ```json { "data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"}, "meta": {"state": "complete", "took_ms": 41, "request_id": "..."}, "notes": [], "next": [] } ``` ``` TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])') curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species ``` ```json { "data": [ {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": {"state": "complete", "took_ms": 2, "request_id": "..."}, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species= to list one species' groups"} ] } ``` The endpoint is registered by default and listed in `/openapi.json` (`GENENETWORK4_TEST_TOKEN=false` switches it off, see below), and decides per request whether to answer: only a caller on this machine gets a token. That means the TCP connection's peer address is in `127.0.x.x` (or is `::1`), and the request carries no `X-Forwarded-For`, `Forwarded` or `X-Real-IP` header. Anything else, including a request whose peer address is unknown, is a `403`. The peer address comes from the connection itself, never from a header a client can set; the proxy headers are used only to refuse a request that claims to have been forwarded. This is why the check also refuses forwarded requests: a reverse proxy running on the same host connects from `127.0.0.1` for every client, so without it the whole internet would look local. A proxy that forwards without setting any of those headers would defeat the check, so don't put one in front of a server whose gn-auth has the dev user (on a production gn-auth the dev user normally doesn't exist, and the request just fails with a `502`). ### Behind nginx or in a container [`docs/nginx.md`](docs/nginx.md) is the full deployment recipe: serving genenetwork4 under a path prefix such as `https://host/gn4/` from a loopback port, the long `proxy_read_timeout` `/compute/*` needs, and how the root-relative links in responses are kept under the prefix. What follows here is only about `POST /get-test-token`. The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here): - **Container on a bridge network, port published.** The server sees the bridge gateway (something like `172.17.0.1`) as the peer, not `127.0.0.1`. Every request through the published port is a `403`, whether it comes via nginx or from the host itself. It still works from inside the container, e.g. `docker exec curl -X POST http://127.0.0.1:3000/get-test-token`. - **Container sharing the host's network, nginx on the same host.** nginx connects from `127.0.0.1`, so the peer check passes for every client. The server then relies on refusing requests that carry `X-Forwarded-For`, `Forwarded` or `X-Real-IP`, and a minimal `proxy_pass` sets none of them. Without one of the measures below, anyone who can reach the site could get a token. For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third: 1. **Block the path in nginx**, independent of genenetwork4: ```nginx location = /get-test-token { return 404; } ``` 2. **Make nginx say it is forwarding**, so genenetwork4 refuses proxied requests (any one of the three headers is enough): ```nginx proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ``` 3. **Switch the endpoint off in genenetwork4 itself** with `GENENETWORK4_TEST_TOKEN=false` (an environment variable, or a `GENENETWORK4_TEST_TOKEN=false` line in `GN4_CONF`). The route is then not registered at all: `POST /get-test-token` is a `404` and the endpoint is absent from `/openapi.json`. The setting fails closed: only `true`, `1`, `yes` or `on` (any case) keep the endpoint on, and anything else -- `false`, an empty value, or a typo such as `flase` -- turns it off. It is on by default, so a local setup needs nothing. Also relevant: the token is for gn-auth's dev user (`GN_AUTH_DEV_USERNAME`). On a production gn-auth that user normally doesn't exist, so a request that does get through fails with a `502` instead of returning a token. That depends on gn-auth's data, not on anything in genenetwork4. The credentials used (gn-auth's dev client id/secret and dev user email/password, from `flask init-dev-clients`/`gn_auth/wsgi.py`) are publicly known and the same for every gn-auth checkout by default -- override `GN_AUTH_DEV_CLIENT_ID`, `GN_AUTH_DEV_CLIENT_SECRET`, `GN_AUTH_DEV_USERNAME` and `GN_AUTH_DEV_PASSWORD` if your gn-auth instance's dev fixtures differ. Needs a gn-auth reachable at `AUTH_SERVER_URL` with `AUTHLIB_INSECURE_TRANSPORT` set (gn-auth itself refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev setting, not something genenetwork4 needs to know about). The token has no `exp` claim, and neither genenetwork4 nor gn-auth expires it by time or for lack of use. It stops verifying only when the key that signed it leaves gn-auth's published key set: gn-auth signs with a key for 7 days (`JWKS_ROTATION_AGE_DAYS`) and deletes it 14 days after its creation (`JWKS_DELETION_AGE_DAYS`), so a token lasts roughly 7 to 14 days, depending on where in the key's life it was issued. genenetwork4 caches the key set for an hour. When requests start failing with `401 invalid_token`, fetch a new token. (`exp` is optional in genenetwork4's own validation, which is why a token without one like this lives as long as its signing key does -- see "Tokens are stateless" below.) The test suite uses this same endpoint automatically -- see [Tests](#tests) below. Configuration is layered: built-in defaults, then `GN4_CONF` and `GN4_SECRETS` files of `KEY=VALUE` lines, then environment variables (which always win). Keys: `SQL_URI` (MariaDB), `SQL_CHECK_HEALTH` (see below), `GN3_URL` (GeneNetwork3 base URL for proxy endpoints), `GN3_TIMEOUT_SECS` (upstream call timeout, default 30), `GENENETWORK4_PORT`, `GENENETWORK4_BIND`, and `GENENETWORK4_TEST_TOKEN` (`false` turns `POST /get-test-token` off; see "Behind nginx or in a container"). ### Tokens are stateless Nothing about a bearer token is stored server side: there is no session table, no list of issued tokens, no `jti` denylist. Verification is signature + `iss` + `exp`/`nbf`, and that is all, so **restarting genenetwork4 loses no tokens** -- a token minted before a redeploy keeps working after it, as does one minted by a stub that has since been restarted (its key is fixed). What the process does keep is two caches, and restarting only empties them: - the signing keys (`JWKS_TTL`, an hour) -- how long a fetched copy of the auth server's key set is reused before being re-fetched; - authorisation answers (`AUTHORISATION_TTL`, a minute), per `(subject, resource)`. Neither is a lifetime. The token's own `exp` is the only thing that ends its life: it is validated on every request (`validation.validate_exp` in `src/auth.rs`, with jsonwebtoken's default 60-second leeway), it is never extended, and there is no refresh token. The hour matters in one case only: if the auth server is unreachable, a cached key set keeps verification working for up to `JWKS_TTL`, and past that every request is `401 invalid_token` with a `fetching .../auth/public-jwks` detail -- the tokens are still valid, the key lookup is not. Run the auth server under the same supervision as genenetwork4 rather than from a shell. Consequences worth knowing: - **Restarting is not revocation.** Since no state exists, restarting genenetwork4 or the auth server leaves every outstanding token valid. To withdraw one, wait for its `exp` (hence short lifetimes for tokens you may need to pull, e.g. `scripts/gn-auth-stub.py --expires-in`). To withdraw all of them at once, change the signing key or `AUTH_SERVER_URL` -- `iss` is compared against that string verbatim, so a changed host, scheme or trailing slash invalidates everything with `InvalidIssuer`. - **`exp` is optional.** `required_spec_claims.clear()` means a token without `exp` is accepted and never expires. gn-auth and `scripts/gn-auth-stub.py` always set one; a hand-minted token (the test key is in this repository) that omits it is a permanent credential. Stub-minted tokens behave the same way (see [Tests](#tests)): `exp` is whatever `--expires-in` said at mint time, and restarting the stub does not invalidate them -- it always signs with the committed test key. ### Request body size The API accepts bodies up to **64 MiB** (`BODY_LIMIT_BYTES` in `src/lib.rs`). axum's own default is 2 MiB, which a legitimate request exceeds: `/plot/{rqtl,gemma}` is *posted a whole mapping response* (a real GEMMA run on the BXD WGS panel is ~1.9 MB; a genome-sized one is tens of MB), and `/compute/*` carries one value per individual plus up to eight covariates. Over the limit, axum answers `413 Failed to buffer the request body: length limit exceeded` before any handler runs. Two consequences worth knowing: - a proxy in front must allow at least as much (`client_max_body_size 64m` in `etc/nginx-vesuvio.pu.ac.ke.conf`), or it refuses first with a 413 the API never sees; - a request does not need to be that large. For covariates, prefer `{"case_attribute": "Sex"}` -- the server reads the attribute and aligns it, so nothing is posted -- over sending a value for every individual in the panel. Only the samples the trait has are used, so posting values for the rest is pure payload. ### Database connections `SQL_URI` is used through one shared `mysql::Pool`. Two of the driver's per-checkout round trips are configurable: - `SQL_CHECK_HEALTH` (default `true`) pings a connection taken from the pool and replaces it when the ping fails. Leave it on unless the database is far away: turning it off saves a round trip, but then a connection the server closed while idle surfaces as `500 database_error` on whichever request happens to draw it -- measured here as ten consecutive failures, one per dead pooled socket, until the process was restarted. - `reset_connection` is off unconditionally: a `COM_RESET_CONNECTION` per checkout, with nothing to reset (the queries are read-only). Even with the ping on, a socket can die between the ping and the query; when a database error names a connection loss (`Broken pipe`, `Connection reset`, `server has gone away`, ...), the pool is dropped so the next request builds a fresh one (`db::is_connection_loss` / `db::forget_pool`, called from `ApiError::from_db`, the one place database errors pass through). `tests/db_recovery.rs` kills every pooled session and asserts the following requests still succeed. ## Running with Guix ``` guix build -f guix.scm ``` builds the package (running both test suites, see below) and prints the resulting store path; the binary is at `/bin/genenetwork4`. The `xapian` input is pinned to 1.4.29 (`xapian-1.4` in `guix.scm`): the `xapian-rs` git revision `Cargo.toml` pins was written against Xapian 1.4, generating bindings for `Xapian::InMemory::open`, and Xapian 2.x removed it (`xapian/dbfactory.h` no longer declares the namespace; 2.x spells it as the `Xapian::DB_BACKEND_INMEMORY` flag). 2.x also needs C++17, where autocxx hands the parser a fixed `-std=c++14` and 1.4's headers are happy with it. Guix's own `xapian` is 2.1.0 now, so without the pin the `xapian-rs` build script stops with "the 'generate' or 'generate_pod' directive for 'Xapian::InMemory::open' did not result in any code being generated" before compiling anything of this project's. The pin is a plain `url-fetch`/`sha256` for `xapian-core-1.4.29.tar.xz` with 1.4's own `make check-inmemory check-remoteprog check-multi check-glass check-chert` test targets; drop it once `xapian-rs` moves to Xapian 2. The same pin is what `guix shell -D` and a manual `cargo build` see (see "Building with cargo, outside a bare toolchain" above). ``` guix shell -D -f guix.scm ``` drops you into a shell with all build dependencies available (cargo, rust, python, git), for running `cargo build`/`cargo test`/`cargo run` directly. This shell is not isolated from the network or from your normal environment, which `cargo build`/`cargo run` need to reach crates.io. A fully isolated container (`-C`) normally needs `-N` to still reach the network from inside it, but on at least one guix build (1.5.0-6.64d4de2) combining `-D` with `-N` causes `-D` to silently stop providing the package's build dependencies (the shell then has only `genenetwork4` itself, not cargo/rust/etc.). If `guix shell -C -D -N -f guix.scm` fails with "cargo: command not found", drop `-C`/`-N` and use the plain `guix shell -D -f guix.scm` above instead. ## Tests Two test suites exercise the same endpoints: - `tests/*.rs` - Rust integration tests that call the router in-process (no socket), via `tower::ServiceExt::oneshot`. Run with: ``` cargo test ``` Both suites check the envelope on every success response: the Rust helper `tests/common::data(&body)` and the Python helper `server_test_case.envelope_data(body)` verify its shape and return `data`. `tests/envelope.rs` (and `test/python/test_envelope.py`) follow every `next` link the endpoints suggest; `tests/envelope_partial.rs` checks the `partial` state without a database; `tests/openapi.rs` checks that every JSON success response is documented as an envelope. - `test/python/test_*.py` - Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with: ``` cargo build --release GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python ``` Either suite can fetch its own token from a live gn-auth automatically (see "Development" above) -- set `AUTH_SERVER_URL` before running either command above, and nothing further needs setting up by hand. A run without a reachable gn-auth still works, skipping (not failing) every test that needed a token it couldn't get -- set `GN4_TEST_AUTH_TOKEN` instead if you'd rather provide one manually. Both suites also run automatically as part of `guix build -f guix.scm` (the Rust tests in the `check` phase, the Python tests in a `python-rest-test` phase after install). Every test under `tests/*.rs`/`test/python/test_*.py` for an endpoint that touches the database (which is most of them -- `hello_world`, `version`, `openapi` and `get_test_token`'s own disabled/wrong-config cases are the exceptions) checks for a live MariaDB on `127.0.0.1:3306` first (see `SQL_URI` above) and skips with a message rather than failing when none is reachable, since the Guix build sandbox has no network/database access. Likewise, any test for an endpoint that requires a token (every endpoint except those four) checks for one first and skips if none is available -- see "Development mode" above for how the test suite gets one automatically. The Rust tests in `tests/*.rs` that hit the database issue their requests through `tests/common::get`, which times the request and prints a warning to stderr if it takes over a second -- e.g. `WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s)`. Warnings only show with `cargo test -- --nocapture` (or on a failing test), since cargo captures stdout/stderr for passing tests otherwise. ### Against a real database and index Those skips are the sandbox's, not the code's. Supply the three things the tests look for -- a live MariaDB, a Xapian index, a token -- and the commands above run the whole suite rather than a small subset. In a development environment with all three, `cargo test` reports 225 tests passed / 0 failed (the handful of skips left are listed below) and the Python suite 94 tests, OK, no skips. 1. **Database**: whatever `SQL_URI` points at, by default `mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl`. `tests/common::db_available()` really connects, so wrong credentials skip exactly like an absent server does. 2. **Index**: `XAPIAN_DB_PATH` for a dev index, plus `GN4_FULL_INDEX` for a full/production one. Only a full index is treated as a count oracle: `tests/search_native_db.rs` then requires the engine's `dataset:bxdpublish` match set to equal the database's `PublishXRef` ids for BXD exactly, id for id -- the invariant a dev snapshot famously fails (it is missing ~1.7k traits and holds ~350 ids the database no longer has). A dev index only has to overlap, and its counts are then printed as reference data, never asserted. When both are set the full one wins. 3. **Token**: either a real gn-auth at `AUTH_SERVER_URL` (see "Development"), or `scripts/gn-auth-stub.py`, which stands in for one: it serves the JWKS matching the committed `test/data/test_rsa_private.pem` (the same fixture pair `search_native_db` spins up in-process), answers `/auth/token` so `POST /get-test-token` works, and answers both authorisation endpoints with "no privileges" -- the stranger case the non-public endpoints' 403 tests assert. It is test scaffolding, not a gn-auth replacement: a committed test key, and no knowledge of users or groups. `--grant-resource-id ID` gives its token `group:resource:view-resource` on one resource id, for the owner-side tests, which want `GN4_TEST_RESOURCE_ID=ID` to match. Left ungranted, a stub token is a **public-data-only token**, which is useful beyond the tests (handing a caller or an agent something to browse the API with). Only-public is not a property of the token: genenetwork4 decides from the data's own `public`/`confidentiality`/`create_time` columns, and asks this authorisation lookup about anything that is not public (`src/auth.rs`'s `check`). No privileges therefore means public datasets read normally and every non-public one is a 403 -- the token's `scope` is never consulted for data access. Dataset *metadata* (`/species/{species}/datasets`, `/groups/{group}/datasets`) stays visible either way: those endpoints are documented as returning names and flags, not trait data, and are not gated on fine-grained authorisation. `--mint-only` mints for one subject and exits without listening: how a token is issued for one more user without disturbing the running stub (see docs/www/operator-checklist.md, "Token issuance"). `--expires-in` sets the token's lifetime (`30d`, `24h`, `60m`, or plain seconds; default 24h). A month-long one is fine because the stub's key is fixed and never rotates: nothing invalidates the token but its own `exp`, and it stays verifiable for as long as anything serving `test/data/test_jwks.json` is up (genenetwork4 caches the key set for an hour, so a brief gap is survivable): ``` scripts/gn-auth-stub.py --port 8099 --expires-in 30d --token-file /tmp/public-only.token & AUTH_SERVER_URL=http://127.0.0.1:8099/ GENENETWORK4_PORT=3999 cargo run ``` Note the port: `iss` has to match the server's `AUTH_SERVER_URL`, so a token minted for `:8099` is rejected by a server left on the default `:8081`, and vice versa. ``` scripts/gn-auth-stub.py --grant-resource-id 424242 --token-file /tmp/gn4-token.txt & export GN4_TEST_AUTH_TOKEN=$(cat /tmp/gn4-token.txt) export GN4_TEST_RESOURCE_ID=424242 export GN4_FULL_INDEX=/path/to/full-index XAPIAN_DB_PATH=/path/to/full-index GUIX_PACKAGE_PATH= \ guix shell -D -f guix.scm -- bash -c ' export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib \ DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib \ DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include cargo test --no-fail-fast -- --nocapture' cargo build --release GENENETWORK4_BINARY=target/release/genenetwork4 \ python3 -m unittest discover -v test/python ``` The stub listens on `127.0.0.1:8081`, which is the default `AUTH_SERVER_URL` -- including its trailing slash, since that string is compared verbatim against the token's `iss`. `--port`/`--issuer` cover the case where `AUTH_SERVER_URL` says something else. What still skips with all three in place, and why: - one `config` unit test asserts what an *unset* `XAPIAN_DB_PATH` means, so setting the variable at all skips it; - `src/indexer/cache.rs`'s RDF-cache test needs an authoritative RDF cache directory, which this does not provide; - `compute_gemma_ravanan_*` needs ravanan, gemma-wrapper and ccwl on `PATH`; - `tests/search_native_dev_corpus.rs` records a dev corpus, so it needs `GN4_DEV_CORPUS_DIR` as well as an index.