about summary refs log tree commit diff

genenetwork4

A Rust/Axum-based API server for GeneNetwork.

Endpoints

Method Path Description
GET / Greeting.
GET /version Build info (hash, date).
GET /species All species.
GET /species/{species} One species by name or id.
GET /species/{species}/datasets ProbeSet datasets in a species.
GET /groups All groups (filterable).
GET /mapping-methods Mapping tools offered.
GET /accessions/{accession} Resolve a GN accession number.
GET /groups/{group} One group, by name or Id.
GET /groups/{group}/parents-and-f1s The family's parental design.
GET /groups/{group}/strains Strains/samples in a group.
GET /groups/{group}/case-attributes Case-attribute names and values.
GET /groups/{group}/traits Published (phenotype) traits.
GET /groups/{group}/traits/{trait_id} A published trait's metadata.
GET /groups/{group}/traits/{trait_id}/values A published trait's values.
GET /groups/{group}/datasets ProbeSet datasets in a group.
GET /publications/{pubmed_id}/traits Traits a publication reports.
GET /datasets/{dataset} One expression dataset.
GET /datasets/{dataset}/probesets Expression traits in a dataset.
GET /datasets/{dataset}/probesets/{probeset} An expression trait's metadata.
GET /datasets/{dataset}/probesets/{probeset}/values An expression trait's values.
GET /traits/{dataset} Every trait in a dataset.
GET /traits/{dataset}/{trait} One trait's metadata.
GET /traits/{dataset}/{trait}/values One trait's values.
POST /compute/correlation Correlate a trait (stub; see docs/correlations.md).
POST /compute/gemma Map a QTL with GEMMA (see below).
GET /compute/gemma/status/{run_id} Poll a gemma run's progress.
POST /compute/gemma/ravanan Map a QTL via ravanan (see below).
POST /compute/rqtl Map a QTL with R/qtl (see below).
POST /compute/bnw Learn a Bayesian network (BNW).
POST /compute/bnw/features Score each module as a trait.
GET /compute/bnw/examples[/{name}] BNW's worked examples, ready to post.
POST /plot/rqtl The R/qtl Manhattan plot, as SVG.
POST /plot/gemma The GEMMA Manhattan plot, as SVG.
POST /plot/bnw The BNW network, as SVG.
GET /compute/rqtl/status/{run_id} Poll an rqtl run's progress.
POST /get-test-token Get a test bearer token (local).
GET /tokens List short API tokens (local).
POST /tokens Mint a short API token (local).
DELETE /tokens/{id} Revoke a short API token (local).
GET /tokens/who Describe the presented credential (local).
GET /search Search traits (free-text).
GET /openapi.json OpenAPI 3.1 document.

Every endpoint above requires a valid gn-auth bearer token (Authorization: Bearer <token>) except /, /version, /openapi.json and /get-test-token itself -- see docs/gn-auth.md and the "Development" section below (POST /get-test-token) for how to get one for local testing.

Short API tokens. Beside gn-auth's JWT there is a second credential format for agents and scripts: gn4ai_ + 64-char opaque tokens, stored as SHA-256 hashes in one SQLite database (tokens.sqlite, GN4_API_TOKENS_DB, chmod 0600, gitignored). Nothing is stored in the token, so revocation is deleting a row and takes effect on the next request; a token in a log line reveals nothing; and if the registry is lost every short token stops working. Mint, list, revoke and introspect with scripts/gn4-token.py create|list|revoke|who; the token is printed once and never again. Administration lives at /tokens, /tokens/{id} and /tokens/who, which answer only a direct caller on this machine (TCP peer address, plus a refusal for anything claiming to be proxied) -- minting credentials is not reachable from the network, and there is no switch to change that. A short token reads public data only: it works everywhere a token suffices, and on #[gnauth(secured)] endpoints for public rows; a confidential dataset answers 403 telling the caller to use a JWT, since gn-auth cannot validate an opaque token. Send it as Authorization: Bearer gn4ai_... and never in a query string, which would land in nginx's log and in next links.

What a request looks like in the log. One JSON line per request on stderr (GN4_ACCESS_LOG), carrying the credentials, never the credential:

{"event":"request","ts":"2026-11-14T14:02:11Z","method":"POST",
 "path":"/compute/gemma/ravanan","status":200,"took_ms":912,
 "request_id":"…","user":"zachs","scheme":"opaque","token":"agent-ravanan-3"}

user is the gn-auth user id, so the line already answers who; scheme and token say with which credential (absent for a JWT, which has no registry handle). scripts/gn4-log.py reads such a log and shows it as a table with the handle resolved against tokens.sqlite — the subject as the registry has it now, plus the token's status today ([revoked], [expired], [unknown]) and its note, which is how "an agent was working at 14:02" is told apart from "an agent was working at 14:02 with a credential revoked since". The path is logged without its query string, and no secret or hash is ever written.

A missing or invalid token is a 401; which further, resource-level authorization (if any) an endpoint checks once a token is valid is a separate, per-endpoint matter covered in each entry below.

Every success response is wrapped in one envelope (see Success responses below), except / (a plain-text greeting) and /openapi.json. The example responses below are real ones, with long lists and strings abbreviated by ..., and took_ms and request_id shortened to plausible values.

Details, including example responses, below.

Following the API without memorising it

Every success envelope carries next: the calls that lead on from what you just read. It is deliberately exhaustive rather than minimal -- a genotype version leads to its frequencies, its marker names and its scans; a tally leads to the markers and to one marker's location and tally; a marker leads to the PheWAS at it; a trait leads to its values, its genotypes and its group; /version leads to the listings. A caller that knows only /version can walk to anything.

A scan is the case that matters most: its reply points at the class counts behind its own peak (.../markers/{peak}/classes?trait={trait_id}), because "LOD 2.93" reads the same whether it rests on two animals or a hundred, and /compute/qtl2 says in notes that a riself panel drops heterozygous calls before the scan while GEMMA keeps them at dosage 0.5.

next is a promise that following a link is a GET that returns 200, which is what makes blind following safe. So a link that would need a parameter the server cannot fill is not offered (/groups/{g}/phewas without a locus is a 422, and is left out); a template such as .../markers/{marker}/location carries a placeholder for the caller to complete; and calls that are not GETs (the POSTs that run a scan) live in actions, never in next. A response never suggests itself. tests/links.rs walks the graph from a dozen seeds and fails if any link does not answer 200.

The convention that follows from this: a reply always offers a next step, and an assistant relaying it should name one — a single sentence after any job, taken from that reply's next/actions, unless the user asks for the result alone. The server's part of that bargain is that the suggestion is always reachable; the assistant's part is that it is one sentence and it is the link the reply actually carries. See docs/www/SKILL.md.

  • GET / - returns Hello, world!.
  • GET /version - returns a JSON record describing the running build:

json { "data": { "name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0, "release": 1, "date": "20261107", "git": "9fa0303", "dirty": false, "source_digest": "2f7f77ed4ace92c1", "branch": "main", "database": "reachable" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [] }

date and git are the HEAD commit's date and short hash, embedded at build time (see build.rs); they describe the commit the binary was built from, not the time it happened to be compiled. git carries a -dirty suffix when that build had uncommitted changes -- so a binary can never claim to be a commit its code does not match -- and source_digest is a hash over the compiled source files, which tells two builds of the same commit apart where a commit hash cannot. branch is the branch HEAD was on, or HEAD when detached. database is reachable or unreachable, checked at most every ten seconds (db::health): a service that answers /version while every data endpoint fails is exactly the state that went unnoticed for twenty minutes on 2026-11-07, and this field makes it visible. build.rs declares .git/HEAD, the ref it points at and .git/index as rerun triggers, so a rebuild after a commit re-stamps rather than reusing the old hash; committing without rebuilding does not move the stamp, which is the point.

  • GET /species - returns a JSON array of all species in the Species table, mirroring GeneNetwork3's gn3.db.species.get_all_species:

json { "data": [ {"name": "human", "menu_name": "Human (Homo sapiens, hg19)", "family": "Vertebrates"}, {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"} ] }

Needs a live MariaDB connection, configured via the SQL_URI environment variable (same config key GeneNetwork3 uses), e.g. mysql://user:password@host/db_webqtl. Defaults to mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl if unset.

  • GET /species/{species} - returns every column of the matching row of the Species table, matched against Name, FullName or SpeciesName (matching GeneNetwork2's wqflask.api.router.get_species_info, which returns the same match logic but only a 4-column subset). 404 if no species matches.

json { "data": { "id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus", "family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090, "order_id": 15 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] }

Needs SQL_URI, same as /species above.

  • GET /species/{species}/datasets - returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of /groups/{group}/datasets. GeneNetwork2's own species-scoped variant of wqflask.api.router.get_datasets_for_group still requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as /groups/{group}/datasets -- this returns dataset metadata (names, public/confidentiality flags), not trait data -- but a token is still required, like every endpoint. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse", "description": "The species these datasets belong to"} ] }

  • GET /groups - returns a JSON array of all groups in the InbredSet table, matching the shape of GeneNetwork2's GET /api/v_pre1/groups response:

json { "data": [ {"id": 25, "species_id": 4, "display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "name": "AD-cases-controls", "full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "public": 2, "mapping_method_id": null, "genetic_type": null, "family": null, "species": "human", "mapping_method": null}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species", "description": "The species the groups belong to (filter groups with ?species=<name>)"} ] }

family is the population category and species the species' short name as /species lists it. mapping_method is the name of the mapping tool offered for the group (the MappingMethod row its mapping_method_id points to, see GET /mapping-methods), null for a group with none. Three optional filters, combined with AND and all case-insensitive: family (contains), species (exact short name) and q (contained in the name, display name or full name). For example /groups?family=Reference Populations lists the reference populations, and /groups?species=rat&q=hxb the rat HXB/BXH sets.

"Group" is GeneNetwork's own alias for InbredSet -- see docs/genenetwork_datatypes.md for why. Needs SQL_URI, same as /species above.

  • GET /groups/{group} - returns every column of the matching row of the InbredSet table, matched against InbredSetName, Name, FullName or numeric Id (matching GeneNetwork2's wqflask.api.router.get_group_info, which matches the same way but not by Id, plus gn-guile's gn.data.population:population-info, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (see docs/genenetwork_datatypes.md); only the MariaDB column names themselves stay InbredSet*. 404 if no group matches.

json { "data": { "id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD", "species_id": 1, "full_name": "BXD Family", "public": 2, "mapping_method_id": "1", "genetic_type": "riset", "family": "Reference Populations (replicate average, SE, N)", "family_order": 1, "menu_order_id": 0.0, "group_code": "BXD", "description": "<p>...</p>", "mapping_method": "qtlreaper" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}, {"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"}, {"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"} ] }

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/strains - returns the strains/samples belonging to a group, matching gn-guile's gn.data.strains:strain-id-names. 404 if no group matches. Takes an optional used_for_mapping query parameter to filter to only strains used for mapping (gn-guile's strain-id-names takes the same flag as a keyword argument):

json { "data": [ {"id": 1, "name": "B6D2F1", "used_for_mapping": false}, {"id": 2, "name": "C57BL/6J", "used_for_mapping": false}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these strains belong to"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"} ] }

GET /groups/BXD/strains?used_for_mapping=true excludes BXD's parent strains (B6D2F1, C57BL/6J, DBA/2J), which aren't used for mapping. Needs SQL_URI, same as /species above.

  • GET /groups/{group}/case-attributes - returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3's gn3.api.case_attributes splits this into separate .../names and .../values endpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns 200 with both lists empty, not a 404. Needs SQL_URI, same as /species above.

json { "data": { "names": ["Epoch", "RRID", ...], "values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...] }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"} ] }

  • GET /groups/{group}/phewas - a PheWAS around one position: every published trait of the group whose QTL peak (PublishXRef.Locus) is a marker within window base pairs of it, strongest first. Centre it on a marker (?locus=rs30524230, e.g. a peak from a mapping) or on a position (?chr=9&mb=29.940961); the window is in base pairs and centred, so window=10000 is 10 kb up- and downstream (default 10000). ?genotype_version= picks whose marker table defines the window (default: the group's current one). The answer carries the markers the window covered as well as the traits, so an empty traits list is readable as "nothing peaks that close" rather than a failure. Each hit has trait_id, dataset, abbreviation, description, its peak locus/chr/mb, lrs, lod (= lrs / 4.604), mean, additive, nsamples, pubmed_id and the GeneNetwork trait url. Example: ?locus=rs30524230&window=10000 returns 4 markers and 10 BXD traits, from LOD 99.9 down to 2.7.

  • GET /groups/{group}/genotypes - the genotype files this group can be mapped with, newest first: each version (e.g. BXD.8), the file name (BXD.8.geno), the panel's title, individuals, markers, whether gemma and rqtl can run on it, and which one is current (what a mapping request gets by default). The order comes from the deployment's genotype/<group>.json, whose genofile list is newest-first, the same rule GeneNetwork2 uses; groups with no JSON (HET3-ITP) fall back to the highest version number and the listing's notes says so. POST /compute/rqtl / /compute/gemma take group plus an optional genotype_version instead of paths.

GET /groups/{group}/genotypes/{version} is the metadata of one of them: the file name, the header's own @-labels (type, mat, pat, ...), the panel's individuals in file order and the strains a mapping actually uses (parents/F1s dropped), num_individuals, num_strains, markers, and per-chromosome marker counts. GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr= is the list of marker names -- one column, paged, filterable by chromosome -- and a position is one call per marker: .../markers/{marker}/location returns {"name", "chr", "cM", "Mb"} as named fields, so a client iterating names never carries a position table and a position lookup is a lookup. So a caller can see the panel, the markers and the header it is mapping against; the genotype calls themselves are never served, only summaries of them. See docs/genotypes.md.

GET /groups/{group}/genotypes/{version}/frequencies is how the panel is actually called, which nothing else in the API says:

json {"data": {"B": 0.4889, "D": 0.492109, "H": 0.018991}, "notes": ["BXD BXD.4: 3811 marker(s), 354423 genotype cell(s)", "352741 call(s) counted, 1682 missing (0.47 %); …"]}

The map is call -> frequency over the calls that are present, so it sums to 1; missing codes (U, N, NA, an empty cell) are counted separately and reported in notes, not as a call. A file whose cells are probabilities is binned, 0.05 wide by default (twenty bins over [0, 1], keys are the bin edges: "0.05-0.10"), because 0.4187 and 0.4191 are the same statement about a marker; ?bin=0.1 changes the width. A file with only a handful of distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported exactly, since binning three values would hide the thing being looked at. GET /groups/{group}/genotypes/{version}/markers/{marker}/frequencies is the same tally for one locus (?marker= on the panel route does the same thing), which is how a caller checks whether a flat LOD came from silence or from a nearly-all-missing marker: both look identical in a scan.

Counting is bounded: cells go through an interner, so a 100-million-cell panel costs one hash lookup per cell and one allocation per distinct call, and a .geno over 256 MB is refused (413 genotype_too_large) rather than read -- HSNIH-Palmer_true.geno is 3.7 GB (134 927 markers x 6 147 individuals), which is not a diagnosis anyone waits for. GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait={id} answers the next question -- what does the trait look like in each genotype class at this marker -- without serving genotypes: the calls are joined to the trait's values inside the server and reduced to n, mean, sd, min and max per class, with that trait's authorization applied. It is how you tell a peak that rests on two animals from one that rests on a hundred, and it is what explains why GEMMA and R/qtl2 peak at different markers on DOL trait 10003:

  GET .../markers/JAX00418132/classes?trait=10003
  {"A": {"n": 2,   "mean": 1.8,    "sd": 0.1414, "min": 1.7, "max": 1.9},
   "H": {"n": 20,  "mean": 2.525,  "sd": 1.1652, "min": 1.7, "max": 5.3},
   "B": {"n": 171, "mean": 3.9491, "sd": 1.1038, "min": 1.7, "max": 6.7}}

The A class is two animals and they are the trait's two lowest values; R/qtl2, which drops H on a riself panel (R/qtl's convert2riself), keeps 173 animals there but only 2 of one class, so the marker is nearly monomorphic for it. GEMMA keeps the 20 hets at dosage 0.5, which is where its sharper p-value comes from. Nothing in the payload names an animal.

GET /groups/unlisted/genotypes/{name}/frequencies is the same tally for a file no group lists (a panel in preparation, a fixture, the DO's do/ cross), and .../markers/{marker}/frequencies for one locus. unlisted is a namespace, not a group: /groups/unlisted has no record, only its genotype subtree answers, and {name} is the version, .geno name, or a path relative to the genotype files directory. One prefix for the whole genotype surface beats a second spelling for the same tally. The markers come the same way: /groups/unlisted/genotypes/{name}/markers (names) and .../markers/{marker}/location (position).

  • GET /probesets/{name} - every dataset that carries a probeset with this name (indexed on ProbeSet.Name): one row per dataset with the panel (group, species, platform) and the probeset's mapping statistics there (chr, mb, mean, lrs, lod, locus, additive). Suffixed variants match too -- 1427571_at also finds 1427571_at_A, the older chips -- and each row's name says which one it was, so the reply never leaves you guessing. ?group=BXD, ?species=mouse and ?platform=M430 (substring: there is no platform id in this schema) narrow it; ?page= and ?limit= page it, and notes carries the count that matters.

    GET /probesets/1427571_at → 155 rows over 155 datasets (Shh on chr5 28.46 Mb), e.g. ["SA_M2_1104_R","BXD","mouse","Striatum_M430_V2_RMA_November04", "1427571_at","Shh","5",28.4579,6.039,16.318,3.544, "rs52243480",0.129]

This is the lookup a probeset id needed: it is not a Xapian field, so /search cannot answer it and the only alternative was guessing a dataset and calling /datasets/{dataset}/probesets/{name}.

  • GET /genes/{symbol}/traits - the same rows, filtered on ProbeSet.Symbol instead: every expression trait of a gene, across datasets, with a real total (unlike /search's capped list), grouped by dataset, and ?sort=lod / ?min_lod= / ?group= / ?species= / ?platform= so "BXD datasets where Shh is a strong eQTL" is one call. GET /genes/Shh/traits?sort=lod → 1 225 rows over 17 datasets and 13 probesets, strongest first (HXB_Liver_1208 / 1387584_at, LOD 18.46).

  • GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=… - the genotypes of the animals at one marker of a confirmed hit, which is the one place this API returns genotypes. It is fenced, because a genotype matrix is the thing GeneNetwork does not hand out:

  • the server re-runs the R/qtl scan for the trait (the same machinery and gate as /compute/rqtl) and takes the peak from that run;

  • the marker must lie within ?window= Mb of it (default 10) -- a marker elsewhere is 422 not_the_peak, quoting the peak's marker, chromosome, Mb and LOD;
  • the peak must reach ?min_lod= (default 3.0), or the answer is 422 not_a_hit;
  • only animals that have a value for the trait are listed, and only this marker's calls -- nothing else of the matrix leaves.

json {"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856, "peak": {"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856}, "counts": {"A": 110, "B": 8, "H": 74}, "animals": [{"individual": "TB001", "genotype": "A", "value": 4.3}, …]}

Which is the data behind R/qtl's Fig 4.1, and POST /plot/genotypes draws it: post that reply's animals (or its genotypes map) with the trait's values from /groups/{g}/traits/{id}/values, and get an SVG with one point per animal, genotype class across, value up, class means marked:

json {"marker": "UNCHS026345", "genotypes": {"TB001": "A", "TB002": "A", …}, "values": [{"name": "TB001", "value": 4.3}, …]}

A class with a couple of animals in it is visible at a glance -- the same warning .../markers/{marker}/classes puts in numbers.

  • GET /genes/{symbol}/aliases - Wikidata's aliases for a gene symbol, the list a trait page shows next to the symbol, plus the label (the entity's canonical symbol in that species) and a links array of external gene pages built for that symbol and species -- RGD, MGI, Alliance, NCBI Gene, Wikidata, UniProt, STRING, GeneMANIA and, per species, GTEx/Open Targets/ Protein Atlas/GWAS Catalog (human) or BioGPS/Allen Brain Atlas (mouse). Two SPARQL queries against WIKIDATA_ENDPOINT (the entity, then its skos:altLabels), kept in an LMDB file, WIKIDATA_CACHE_DIR/gene-aliases.lmdb, for WIKIDATA_TTL_SECONDS (seven days by default). The entity query matches the label case-insensitively, because human genes are labelled SHH and mouse ones Shh, and picks mouse, rat or human deterministically -- ?species=human gets the ortholog whose labels GeneNetwork2's page shows:

    GET /genes/Shh/aliases {"data": {"symbol": "Shh", "wikidata_id": "Q14860079", "species": "mouse", "aliases": ["9530036O11Rik","Dsh","Hhg1","Hx","Hxl3","M100081", "ShhNC","sonic hedgehog"], "fetched_at": 1794315892, "stale": false}}

When Wikidata cannot be reached the cached answer is served with "stale": true and the envelope marked partial -- the last value, not an empty list: an empty alias list and an unreachable upstream mean different things, and only one of them is about the gene. ?refresh=true forces a refetch (an operator's switch). The cache is one file plus LMDB's lock companion, written 0600.

The label is what makes the human-only services reachable from a rodent symbol: ?species=human on a mouse Shh gives label: "SHH", an RGD link for human, and a GTEx link that resolves -- a mouse symbol in GTEx is an empty report. A cached entity written before the label was kept is refreshed rather than served (stale tells you which happened).

Both serve metadata, not values. A token buys the index; the values are per dataset and authorized per dataset, and the next link goes to /datasets/{dataset}/probesets/{name}/values, which checks. There is no endpoint that returns values across datasets, deliberately: that is bulk work and /groups/{group}/traits is the bulk call.

  • GET|POST /baskets, GET|PATCH|DELETE /baskets/{name} - your own column of values, stored in your own SQLite file, and mapped by name. This is how data gets changed without changing the database: take a trait's values, edit one or more of them (or build a covariate column that does not exist), and map that instead. MariaDB is never written to.

sh curl -s "$API/groups/BXD/traits/10001/values" > wbc.json # post the column back as a basket (jq: .data is the values array) curl -s -X POST "$API/baskets" -d '{"name":"wbc-edit","kind":"trait", "base_dataset":"BXD","base_trait":10001,"values":[...]}' curl -s -X PATCH "$API/baskets/wbc-edit" -d '{"values":[...]}' # revision 2 curl -s -X POST "$API/compute/qtl2" -d '{"group":"BXD", "genotype_version":"BXD.4","trait_name":"basket:wbc-edit","basket":"wbc-edit"}'

Choices worth knowing: * The full column is stored, not a diff: the caller has the original beside it and the comparison is theirs, while a mapping needs one complete column. * Text with a parsed copy: raw is what was typed (5.2, NA), value the number or null, status which it is -- so NA and 0 stay different claims. A column containing neither a number nor a missing marker (NA, x, -, ., ?) is refused with the offending samples named, never silently dropped. * An automatic version history: revision 1 is the column as first stored and every PATCH adds one, with its own digest and Unix timestamp — earlier revisions are kept, not overwritten, so GET /baskets/{name}/revisions is the history, ?revision=N reads an old column, and a mapping can pin one: {"basket": "wbc-edit@2"} maps revision 2 even after the basket has moved on. Every reply records which revision and digest ran (Basket 'wbc-edit' revision 2 (digest 9559a70e…)), so two runs on one name can never look alike. * One SQLite file per user, named for them — BASKET_DIR/<user>.sqlite (0600, WAL), where <user> is the token's subject: tokens rotate, subjects do not, so a new token finds the same baskets and an operator can see whose file it is. A subject that is not filename-safe is sanitised and disambiguated with a short digest. Baskets are private: GET /baskets only ever lists yours, and nothing here needs a database the API may write to. * Every compute endpoint takes basket in place of its value field -- /compute/{rqtl,gemma,qtl2,fitqtl} -- and giving both is a 422 rather than a silent choice. PATCH replaces the column, it does not merge: leaving a value out drops it, deliberately (the dropped revision is still in the history).

  • POST /matrices, GET|DELETE /matrices/{name} - upload new traits and covariates as one named table, checked against an existing genotype file. The shape is the one people have: a header row, then one row per individual, the first column naming them.

id,body_weight,glucose,batch BXD1,23.100,9.400,2 BXD2,18.700,8.100,1 ...

  POST /matrices {"name":"my-traits","genotype_version":"BXD.4",
                  "contents":"id,bw,glucose\nBXD1,23.1,9.4\n...",
                  "kinds":{"batch":"covariate"}}

The upload is checked against genotype_version's panel before it is stored: the same number of individuals, and every name one of them. 92 of 93, or a name the panel does not have, is a 422 that says which -- a mapping that silently drops individuals would be worse than an error. kinds marks columns that are covariates; the rest are traits.

Each column becomes a basket named <matrix>.<column>, so a mapping names it the way it names any basket, on any engine, and inherits the revision history and digest:

  POST /compute/qtl2 {"group":"BXD","genotype_version":"BXD.4",
                      "basket":"my-traits.bw"}

Everything lands in the uploader's own SQLite file -- MariaDB is never written to -- and GET /matrices/{name}/values reads the table back (null where a value was missing). A script writes an example table from a genotype file's own header, so the fixture passes the check by construction: scripts/example-matrix.py --genotype <file> --out example-matrix.csv.

  • POST /uploads/genotypes - upload a genotype matrix that updates an existing population, named $USER-GENONAME. Tab-delimited: the header names the individuals from column 4 on, column 1 is the marker, 2 the chromosome, 3 the position in Mb.

marker chr pos BXD1 BXD2 ... rs6269442 1 3.4921 B D

This is an update, not an ingest, and the checks say so: the base version must exist, at most 50 individuals may be new, at least one must overlap (none of the 93 individuals are in the base population: an upload updates a population), and at most 200 000 markers -- beyond that the sysadmin has to organize a file that size. Values must be the same kind as the base's (single letters, whole numbers, or probabilities), because a mapping engine reads the result the way it reads the population it updates; a letters panel full of 0/1 is refused, naming both kinds.

Stored as text in the uploader's own SQLite file, and written out on demand as a GN .geno (Chr, Locus, cM, Mb, individuals) carrying the base's @-metadata: GET /uploads/genotypes/{name}/geno returns the path for the genotype_file parameter of the compute endpoints, which now accepts values by name (samples or a basket) as well as positionally:

POST /compute/qtl2 {"genotype_file":"<path>","samples":[{"name":"BXD1","value":61.4}, …]}

An upload of one chromosome is a poor thing to scan: a leave-one-chromosome- out kinship has nothing to leave out, so the scan comes back flat or empty. scripts/example-genotype-upload.py therefore spreads its markers across chromosomes; use "kinship": "overall" if a file really has only one. GET /uploads/genotypes lists yours, .../markers pages the marker table, DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update (subset of markers, a few individuals dropped or added).

  • GET /groups/{group}/traits - returns every published (phenotype) trait belonging to a group, the listing equivalent of /groups/{group}/traits/{trait_id} (same fields, same join, no trait_id filter). 404 if no group matches. If the group has any published traits, its PublishFreeze row is checked the same way as /traits/{dataset} (see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's own wqflask.api.router.fetch_traits, which defaults to unbounded too. Needs SQL_URI, same as /species above.

The optional abbreviation parameter keeps only the traits with that short name, e.g. /groups/BXD/traits?abbreviation=5HT_MB_M finds trait 17985 (an unknown abbreviation is an empty list).

json { "data": [ {"id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD", "description": "The group these traits belong to"} ] }

  • GET /groups/{group}/traits/{trait_id} - returns one published (phenotype) trait's PublishXRef mapping stats plus its Phenotype description/units and Publication reference, based on the data explored in precompute-publishdata.gmi (e.g. its "More metadata" section, which identifies exactly this join as worth surfacing). trait_id is PublishXRef.Id, scoped by group for the same reason as /groups/{group}/traits/{trait_id}/values. 404 if no group matches, or if the group has no such trait; 403 if the trait's PublishFreeze isn't public and the caller isn't authorized for it (see docs/gn-auth.md).

json { "data": { "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"}, {"href": "/groups/BXD", "description": "The group this trait belongs to"}, {"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"} ] }

dataset is the trait's PublishFreeze, abbreviation its short name (Phenotype.Post_publication_abbreviation, else the pre-publication one) and url its page on GeneNetwork's website. lod is lrs / 4.604.

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute} - one case attribute's values (Sex, Age, Tissue, Batch, ...) for exactly the samples a trait's values cover, in that trait's order, so it lines up with /values without re-sorting -- the covariate source, kept off the values endpoint. values carries the text as recorded (F, Live), null where a sample has none; num_values counts those that have one, and distinct lists each value with its count, commonest first, which is what says whether it is usable as a covariate (a constant one is not). Attributes are per group -- BXD defines Epoch, RRID, SeqCvge, Status and no Sex, while BXD-Longevity and several other BXD-family studies do -- and an unknown one is a 404 whose next is the group's attribute list. Example: /groups/BXD-Longevity/traits/10001/case-attributes/Sex returns 3281 values in AgeAtDeath's sample order, F × 3101, M × 180.

  • GET /groups/{group}/traits/{trait_id}/values - returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2's wqflask.api.router.trait_sample_data (the Publish branch) and gn-guile's gn.data.dataset:get-bxd-publish-name-value-dict, generalized to any group rather than hardcoding BXD. trait_id is the trait's PublishXRef.Id; it alone doesn't identify a trait uniquely across groups (only the pair of group and PublishXRef.Id does), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait's PublishFreeze isn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a #[gnauth(secured)] endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits.

?normalize= / ?winsor_quantile= / ?winsor_fences= apply the same pipeline /compute/* takes (see "Normalize the trait first" below), over exactly the samples returned, and say what ran in notes. The mapping suggestions in actions carry the same fields, so fetching z-scores and then following the suggestion maps the numbers that were shown rather than the raw ones:

sh curl "$API/groups/BXD/traits/10010/values?normalize=zscore" # notes: "Normalized: zscore ... over 28 values; mean 0.000000, sd 1.000000" # actions[*].body.normalize == "zscore"

A pipeline that cannot apply (both winsorising rules at once, a log of a non-positive value, nothing to divide by) is a 422 invalid_normalization.

json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"} ] }

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/datasets - returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2's wqflask.api.router.get_datasets_for_group. 404 if no group matches. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these datasets belong to"} ] }

tissue is what the dataset was measured in (the Tissue its ProbeFreeze points to), also on GET /datasets/{dataset} and GET /species/{species}/datasets.

  • GET /mapping-methods - the mapping tools GeneNetwork offers, the MappingMethod table in id order: qtlreaper, Rqtl, HappyR, PLINK and FastMap. It includes tools no group currently uses (PLINK, FastMap); a group's mapping_method (see GET /groups) names the one offered for it. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 1, "name": "qtlreaper"}, {"id": 2, "name": "Rqtl"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"} ] }

  • GET /accessions/{accession} - resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without the GN prefix. It names the dataset (InfoFiles.InfoPageName), its page title, its category (publish for a phenotype dataset, probeset for an expression dataset, null if the name matches neither) and the group it belongs to. 400 if the text is not a number, 404 if no dataset has the accession. Needs SQL_URI, same as /species above.

json { "data": { "accession": 602, "name": "BXDPublish", "title": "BXD Published Phenotypes", "category": "publish", "group": "BXD" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group the dataset belongs to"}, {"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"} ] }

  • GET /publications/{pubmed_id}/traits - returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as /groups/{group}/traits). 404 if no publication has that id; a publication with no traits is an empty list. Unlike /groups/{group}/traits, the datasets are authorization-checked as in GET /traits/{dataset}: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 17967, "phenotype_id": 16502, "publication_id": 13584, "mean": 6038.655919692095, "locus": "rsm10000012868", "lrs": 10.535408722907198, "lod": 2.2883164037591652, "additive": -647.5889092087912, "nsamples": 34, "pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824, "title": "Evaluation of heritable determinants of blood and brain serotonin homeos...", "authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...", "journal": "Genes, brain, and behavior", "volume": null, "pages": "247-260", "month": "Mar", "year": "2014", "dataset": "BXDPublish", "abbreviation": "5HT_Blood_M", "url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"} ] }

  • GET /datasets/{dataset} - returns every column of the matching row of the ProbeSetFreeze table, matched against Name, Name2, FullName, ShortName or numeric Id (matching GeneNetwork2's wqflask.api.router.get_dataset_info's matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under /groups/{group}. Needs SQL_URI, same as /species above.

json { "data": { "id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M", "name2": "BXDMicroArray_ProbeSet_August03", "full_name": "Brain U74Av2 08/03 MAS5", "short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01", "public": 0, "confidentiality": 0, "data_scale": "log2", "tissue": "Brain mRNA" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"} ] }

  • GET /datasets/{dataset}/probesets - returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of /datasets/{dataset}/probesets/{probeset} (same fields, same join, no probeset filter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset) rather than once per probeset -- see docs/gn-auth.md. Needs SQL_URI, same as /species above.

json { "data": [ {"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"} ] }

  • GET /datasets/{dataset}/probesets/{probeset} - returns one expression (ProbeSet) trait's ProbeSetXRef mapping stats plus its ProbeSet gene annotation, matching GeneNetwork2's wqflask.api.router.get_trait_info's ProbeSet branch. ProbeSetXRef has no single Id column of its own (its primary key is DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)), so a trait here is identified by dataset name plus ProbeSet.Name rather than a single numeric id, unlike /groups/{group}/traits/{trait_id}. 404 if no dataset matches, or if the dataset has no such probeset.

json { "data": { "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"}, {"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"} ] }

Needs SQL_URI, same as /species above.

  • GET /datasets/{dataset}/probesets/{probeset}/values - returns one expression trait's value in each strain in the dataset, matching GeneNetwork2's wqflask.api.router.trait_sample_data's ProbeSet branch, addressed the same way as /datasets/{dataset}/probesets/{probeset}. 404 if no dataset matches, or if the dataset has no such probeset.

json { "data": [ {"name": "B6D2F1", "value": 5.742000102996826}, {"name": "BXD1", "value": 6.414000034332275}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"} ] }

Needs SQL_URI, same as /species above.

/traits -- ProbeSet and Publish data through one URL shape

Everything above addresses ProbeSet (expression) and Publish (phenotype) traits differently: /datasets/{dataset}/probesets/... vs. /groups/{group}/traits/.... A trait is a trait either way -- a dataset, a trait within it, and that trait's per-strain values -- so /traits/... addresses both the same way, (dataset, trait), matching GeneNetwork2's own generic wqflask.api.router.trait_sample_data/get_trait_info. This is additional, not a replacement: the endpoints above are unchanged and keep working exactly as documented.

dataset is resolved against ProbeSetFreeze first, then PublishFreeze (db::resolve_trait_dataset) -- whichever matches decides the category, so a bare trait name/id is never guessed across categories, only the (much smaller, auditable) set of dataset names is. A Publish dataset can be named either way GeneNetwork2 itself accepts: the PublishFreeze's own Name/ShortName/Id (e.g. "BXDPublish"), or its owning group's Name/InbredSetName/ FullName/Id (e.g. "BXD") -- a group normally has only one PublishFreeze, so the two are near-interchangeable in practice.

  • GET /traits/{dataset} - every trait in dataset, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset), same as /datasets/{dataset}/probesets. This is still more than /groups/{group}/traits checks for the same phenotype data: both endpoints require a token now, but only this one also checks PublishFreeze.public/confidentiality and, for a non-public dataset, gn-auth -- /groups/{group}/traits (unchanged) stops at the token.

json { "data": [ {"category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] }

or, for a Publish dataset:

json { "data": [ {"category": "publish", "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] }

  • GET /traits/{dataset}/{trait} - one trait's metadata, tagged by category (ProbeSetInfo's gene annotation or TraitInfo's publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other). trait is ProbeSet.Name for a ProbeSet dataset, or PublishXRef.Id for a Publish dataset (a non-numeric trait against a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. Needs SQL_URI, same as /species above.

json { "data": { "category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"} ] }

  • GET /traits/{dataset}/{trait}/values - that trait's value in each strain, as {name, value} pairs regardless of category (TraitValue and ProbeSetValue are identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as /traits/{dataset}/{trait}.

json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"} ] } - GET /search - free-text search across GN traits (see docs/search-engines.md for switching between the gn3 proxy and the native engine); hits keep gn3's field names and are wrapped in the envelope like every other response. Parameters: q (xapian syntax, e.g. serotonin AND midbrain AND group:BXD), type (phenotype or gene), limit (default 20, at most 100), and optionally sort (lrs or lod, which order identically) with order (desc, the default, or asc), and page (default 1) for the next limit hits.

Three fields are added to every hit: lod (lrs / 4.604) and, for phenotype hits, locus (the peak marker's name) and abbreviation (the trait's short name), both from the database (null for other hits, or when the database is unreachable). pubmed_id is gn3's. With sort, gn3 is asked for its first 1000 matches by relevance, those are ranked, and page/limit are applied to the ranked list, so page=2 continues where page 1 stopped; hits with no lrs come last. Without sort, page is gn3's own. For example, the highest-LOD trait about serotonin in BXD midbrain: /search?q=serotonin AND midbrain AND group:BXD&sort=lod. Tissue and sex are not fields: they appear in each hit's description, so match them there.

Needs a reachable gn3, configured via the GN3_URL config key (default http://127.0.0.1:5003/api/).

json { "data": [ {"name": "17994", "dataset": "BXDPublish", "dataset_fullname": "BXD Published Phenotypes", "group": "BXD", "species": "mouse", "authors": ["Ye R", "Carneiro AM", ...], "year": 2014, "description": "Central nervous system, protein expression: Midbrain serotonin transport...", "mean": 1627.7004466337316, "additive": 233.98946836805567, "lrs": 16.49128337950074, "lod": 3.5819468678324804, "locus": "rsm10000013922", "abbreviation": "SERT_MB_M", "geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD", "pubmed_id": 24405699, "pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."} ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"}, {"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"} ] }

  • POST /plot/rqtl, POST /plot/gemma - the drawing step after a mapping: post the mapping's own response back (the JSON /compute/rqtl or /compute/gemma returned, as-is) and get the SVG the deployment's plotter draws from it (scripts/plot-rqtl.py, scripts/plot-gemma.py; src/plot.rs runs them). Nothing is recomputed -- a plot is one script run, and it can be redrawn from a result you kept. /plot/rqtl also accepts just the data member; /plot/gemma needs the whole response, because its plotter reads meta.columns to find the p_lrt column. The reply is image/svg+xml, not the JSON envelope -- an image is not JSON -- and the plotter's one-line peak summary comes back in x-plot-summary. R/qtl plots LOD with the permutation thresholds as dashed lines when the result carried them. GEMMA plots -log10(p_lrt) with no line unless one is asked for: ?bonferroni=true adds GEMMA's own Bonferroni (computed by the plotter as 0.05 over the analysed markers), and ?thresholds= adds one computed elsewhere, each labelled. The title names the trait: taken from the posted response's own Trait: ... note (which /compute/rqtl and /compute/gemma both write first), or from ?trait= when you post only data; ?trait= wins over the note, and ?title= replaces the whole title. Two traits over the same panel draw two different traces -- the plot is drawn from the rows you posted, and nothing is cached between calls. Configuration: PLOT_PYTHON (default python3), RQTL_PLOTTER, GEMMA_PLOTTER. Both mapping responses suggest these calls in actions.

  • GET /openapi.json - returns an OpenAPI 3.1 document describing the endpoints above; see OpenAPI below.

/compute/gemma

Maps a QTL by running the real gemma-wrapper/GEMMA binaries: a K-matrix step, then an LMM association step, both --loco, over a genotype file, a SNP annotation file and a list of trait values. See src/gemma.rs for the invocation itself and why it's shaped this way (found by running GeneNetwork2's and GeneNetwork3's own gemma code for real against the BXD fixtures -- GeneNetwork3's always errors out; GeneNetwork2's non-LOCO branch does too; only --loco works).

Individuals with no genotypes are left out. A panel can name individuals whose genotype row is empty -- BXD-AE.4 has 136 of 344, BXD-Heart-Metals.4 200 of 481 -- and an empty row makes the kinship matrix singular, which GEMMA reports as eigenvalues close to zero and then a LAPACK LUDecomp ... input domain error, a message that says nothing about the cause. Those individuals are dropped before the fit and counted in notes; they contribute nothing either way, and the panel maps in seconds once they are gone.

Normalize the trait first. normalize changes the values' scale before mapping -- none (default), log2, log10, sqrt, negate, zscore, robust_zscore, or rank -- and winsor_quantile / winsor_fences handle the tails. They run in that fixed order (transform, then winsorise, then standardise), over exactly the individuals that enter the fit, and the response says what ran:

"normalize": "rank", "winsor_fences": 1.5
  • rank is GeneNetwork2's "Quantile" (qnorm): Φ⁻¹((i−0.5)/n) by rank, ties sharing their average rank -- not array quantile normalisation. Accepting qnorm and invert (for negate) is deliberate: those are GN2's own names. zscore uses the population standard deviation, as GN2 does (scipy.stats.mstats.zscore), so the numbers match its trait page; robust_zscore is (x−median)/(1.4826·MAD) and is ours.
  • winsor_quantile: 0.05 clips each tail at the 5th/95th percentile; winsor_fences: 1.5 clips at Q1 − 1.5·IQR / Q3 + 1.5·IQR, the rule GN2 uses to flag outliers -- and a count of those is reported whether or not anything is clipped. The two are mutually exclusive (422).
  • log2/log10 on a non-positive value, sqrt on a negative one, or a standardisation with nothing to divide by is a 422; missing values stay missing and are never imputed. GN2's silent sqrt "add 1 in [0,1]" quirk is deliberately not reproduced.
  • beta, se and additive come out on the transformed scale -- a zscore additive is in standard deviations -- so quote the note with the number.

Name what you are mapping. trait_name is optional -- GEMMA's values are positional, so it needs no name to compute -- but it is what the response reports (Trait: BXD::20010, first in notes), what the PheWAS and R/qtl suggestions carry, and what /plot/gemma titles the picture with. Without it they fall back to <group>::gemma, so two mappings over one panel are distinguishable in the data but identically titled.

No permutations. GEMMA's permutation test is not run here and nperm/permutations/permutate are refused (422 permutations_not_supported), not ignored -- an ignored flag is worse than a refusal, because the caller believes it got thresholds. The reason is cost: gemma-wrapper --permutate shuffles the phenotype file and repeats the entire LOCO association scan per permutation, so a thousand is thousands of full scans -- hours to days on a real panel, against the one slot this endpoint has. R/qtl does have the test, on the same panel and the same values: POST /compute/rqtl with "nperm": 100 returns the suggestive and significant LOD thresholds, and every successful GEMMA response already carries that call, pre-filled, in actions. Judge a GEMMA peak against the smallest p_lrt it reached, never against a genome-wide threshold of its own.

Covariates. covariates fits one or more covariates alongside the trait -- the same request shape /compute/rqtl takes (below), each either a case attribute of the group by name, which the server reads, aligns onto the panel and codes, or explicit per-sample numbers:

"covariates": [
  {"case_attribute": "Sex"},
  {"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}},
  {"name": "age_days", "kind": "numerical",
   "values": [{"name": "F3092", "value": 183.0}, …]}
]

GEMMA wants numbers, positional over the genotype panel, in a file of its own (-c), so src/covariates.rs writes one: one row per individual, no header, no sample names, and a categorical covariate dummy-expanded into k‑1 columns named after their levels (Sex.M) -- GEMMA knows nothing about factors, unlike R/qtl's wrapper, which does that expansion itself. The first level is the reference. A level none of the fitted individuals carries is left out (an all-zero column is singular and GEMMA stops with LUDecomp rather than dropping the term the way R would), and the response says so in notes.

Text with 2–20 distinct values is coded 0..k‑1 in sorted order; numeric text is passed through as numerical; more than 20 levels, one level, or an explicit map that misses a value is a 422 asking for a map. An unknown attribute is a 422 listing the group's own. A missing value is written NA, which GEMMA treats as it does a missing phenotype -- it drops that individual (measured; GeneNetwork2 instead writes -9, which GEMMA fits as a value). Both the coding, how many individuals have a value, and any column left out are reported in notes.

Covariates reach only the association step, never the K-matrix step, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship it was computed against still is not recomputed. Covariates need a group: individuals are aligned by name, which an explicit genotype_file/snps_file request does not carry (422 covariate_unavailable).

Concurrency: large jobs, not all jobs. Gating exists to keep large computations from piling up on this box, and for GEMMA that begins at 50 individuals (compute_gate::GEMMA_LARGE_PANEL) -- i.e. at the smallest real panel, so in practice every mapping request is gated, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. Two reasons, neither about how long a small run takes: every GEMMA run goes through one shared cache directory (GEMMA_CACHE_DIR), where gemma-wrapper extracts per-chromosome kinship archives of hundreds of megabytes by content hash, so two runs at once are two writers over the same archives; and a real BXD WGS run is ~19 s of every core it can get, while UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome.

So GEMMA runs one job at a time, one per caller: a second submission from the same token while its run is running or queued is refused at once with 409 compute_busy naming that run, and a submission from another caller waits for the slot. R/qtl is gated much later (1 000 individuals; see below) because a scanone over a few hundred individuals is seconds of one core and its results are uncached scratch of its own.

Concurrency. A large scan -- 1 000+ individuals (compute_gate::RQTL_LARGE_PANEL), where a scanone starts to be minutes rather than seconds -- takes one of RQTL_MAX_CONCURRENT slots (default 8), one per caller: a second large submission from the same token while its scan is running or queued is refused at once with 409 compute_busy naming that run, and a submission that finds every slot taken waits for one to come free. Eight because a scan is a single-threaded Rscript and this box has cores to spare; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. The count is fixed when the gate is built (at the first large request), so a change needs a restart. Smaller crosses are not gated at all.

Permutations. nperm runs R/qtl's permutation test, and the response carries permutation: {suggestive, significant, values}. A scan without it suggests one with nperm: 100 in actions -- the thresholds a peak needs to be judged against -- and next carries a PheWAS at the peak the run found.

No permutation test, and requesting one is an error. GEMMA reports a p-value per marker; gemma-wrapper --permutate n does exist, but it repeats the whole LOCO scan per permutation, so it is hours of the one slot this endpoint has. nperm, permutations and permutate are therefore refused (422 permutations_not_supported) rather than ignored, and a peak is judged against the smallest p_lrt it reached. The response's actions carries the R/qtl permutation run over the same panel and values, pre-filled, which is where genome-wide thresholds come from here.

Give it a group, not a path. group (an InbredSet name) plus an optional genotype_version makes the server resolve the genotype files itself -- GET /groups/{group}/genotypes lists what is available and which version is current, and src/genotypes.rs applies GeneNetwork2's rule for that. trait_values may then be replaced by named samples, which are matched onto the panel by name; parents and F1s are dropped unless include_parents_f1 is set, as in GeneNetwork2/3. genotype_file + snps_file remain as an explicit override, for files outside that layout.

If you name the files yourself, they are BIMBAM, in GeneNetwork2's own layout: genotype/bimbam/<file>_geno.txt (the genotype matrix) together with <file>_snps.txt (marker/chromosome). The genotype/<file>.geno that R/qtl takes is not a GEMMA input. For BXD's current panel that is bimbam/BXD.8_geno.txt + bimbam/BXD.8_snps.txt -- 235 individuals × 21 056 markers. trait_values must then have one entry per individual, in the genotype file's column order, null where the trait has no value; GEMMA matches by position, so a short or reordered list is a wrong answer, not an error. Expanding a trait's values onto the panel is the caller's job today (the API does not re-index yet); see docs/genotypes.md for the conventions, which version counts as current, and the measured runs -- 17.1 s for the BXD WGS panel (21 055 marker rows) against the 19 s the 7 320-marker fixtures take.

This runs synchronously inside the request -- there is no job queue -- and it runs sequentially, not per-chromosome in parallel. Not feasible past roughly 100,000 markers at this point. The BXD fixtures (7,320 markers, LOCO across 20 chromosomes) already take about 19 seconds; some real GeneNetwork datasets have upwards of 1 million markers, which this endpoint cannot handle yet -- the request would simply run for a very long time inside one HTTP call. Making that feasible needs either parallelising the per-chromosome LOCO jobs (gemma-wrapper --parallel, which gn-guile's own precompute script already uses) or moving the computation out of the request/response cycle entirely (a job queue, or an offline precompute step, the way gn-guile's own scripts/precompute/run-gemma.scm already does it).

{
  "data": [
    [1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, -0.07665315, 0.2178407],
    ...
    [1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, -0.21408755, 0.0003520103]
  ],
  "meta": {
    "state": "complete",
    "took_ms": 18456,
    "request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc",
    "columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "additive", "p_lrt"],
    "num_rows": 7320,
    "num_cols": 12
  },
  "notes": [
    "Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.",
    "Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk",
    "GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9"
  ],
  "next": []
}

(data has one row per marker -- 7,320 rows in this real run against the BXD fixtures, trimmed to two here. Each row is positional, in meta.columns' order, rather than a {"chr": ..., "rs": ..., ...} object per row -- cuts the repeated key names out of a response this large.

The parental design is an endpoint, not a guess, and /groups/{group} carries its headline. That reply's note reads "Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default" — the two strain names answer "which cross is this?" without a second request — and GET /groups/{group}/parents-and-f1s reports a family's parents and F1s from the deployment's parents_and_f1s.json (ported from GeneNetwork2's ParInfo, all 24 groups), plus the @type of its genotype file, the individuals each role names when the metadata names them, and — the part a caller actually needs — which members of the current panel the default rule drops, and which the F1 name heuristic catches. For an unlisted group it says source: "name heuristic" and warns that its parents are unknown, so nothing is assumed. For an RI set it also explains a subtlety: the metadata names the design's F1s (B6D2F1) while the panel carries specific intercross F1 individuals (C57BL/6JxBXD037F1), so the exclusions come from the name rule and excluded_by_name lists precisely which.

Choosing individuals. Parents and F1s are excluded by default, as in GeneNetwork2/3 (include_parents_f1: true turns that off). On top of that rule any request that maps over a group's panel takes a selection:

{"group": "BXD", "include_parents_f1": true,
 "individuals": {"include": ["C57BL/6JxBXD037F1"], "exclude": ["BXD48a"]}}

exclude wins over include, and an unknown name is a 422 invalid_individuals whose message lists the panel — a silently dropped animal changes a kinship matrix without anyone noticing. The reason both exist: the parent/F1 list is data (parents_and_f1s.json, seven groups) that a request cannot reach, while F1 names differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH, C57BL/6JxBXD037F1 for intercross F1s).

Worth knowing before turning it on: including parents/F1s changes the kinship, not the fit. GEMMA computes kinship without reading phenotypes, so they shape the relatedness structure; the association still uses only the individuals that have a trait value. The reply says both numbers — "197 individuals … ; the kinship uses all 197, the association uses the 33 with a trait value" — and a request can therefore legitimately have 234 individuals in the genotype file and 34 in the fit.

additive is the effect size, sitting next to the score: −beta/2, which is what GeneNetwork's own tables and CSV call the additive effect (gn2/wqflask/marker_regression/gemma_mapping.py). Halved because half the difference between the two homozygotes is the usual additive effect; negated because GN2's BIMBAM conversion codes @mat as 1, so GEMMA's beta is per maternal copy while GeneNetwork's published sign is anchored on the other allele. beta and se are left where GEMMA put them -- se is what makes the effect interpretable (beta ± 1.96·se is a 95 % interval) -- and p_lrt stays the last column. These describe this run on these samples and this genotype version; they are not the additive that trait info reports, which came from the published mapping (trait 20010: this run −0.54, published +2.118). The last two notes are the exact gemma-wrapper invocations used (paths shortened here for readability) -- -gk only on the kinship step, -lmm only on the GWA step, never both together; see src/gemma.rs's compute_gwa doc comment for why that distinction matters. lmm defaults to 9, matching GeneNetwork2's and gn-guile's own convention; see the field's own doc comment in src/routes/gemma.rs for what's actually been verified about it.)

One further note is GEMMA's own variance component for the null model, which it writes only when asked (-pve, which this build passes on the GWA step; accepted alongside -lmm 9, verified against GEMMA 0.98.6). Under --loco there is one per chromosome -- twenty different kinship matrices -- so the note carries every value, not a summary:

GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).

The unrounded numbers are also appended to the run log as variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612 lines, so a ?id= run's /compute/gemma/status stream shows them while the run is still in flight; and gemma-wrapper keeps each chromosome's own .log.txt beside its archive (5eb53fa), named by the same hash. pve is the proportion of variance the kinship explains -- the same quantity as /compute/bulklmm's h2, but per chromosome instead of genome-wide -- and it is scale-free, so compare it with h2 rather than comparing vg against σ²_e: a trait measured in thousandths has variances in thousandths. When the estimate sits on a boundary GEMMA prints -nan for se(pve); that is reported as "not available", because a boundary estimate has no standard error, and a pve of exactly 0 or 1 is a direction rather than a proportion.

/compute/gemma/status/{run_id} -- polling a run's progress

Given the endpoint above can genuinely take minutes, pass ?id= on the POST and poll this on a second connection while it's in flight to show progress, the same log-tailing shape GeneNetwork2/3 use for R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the run's id becomes a log file ({cache_dir}/{id}.log); gemma-wrapper's own stderr (its --json-mode progress banner, chromosome list, "Executing: ...", per-step info -- its stdout carries only the one final JSON record, so there's nothing to stream there) is appended to it line by line as both the kinship and GWA steps run; once the whole request finishes, "GN4: computation complete" is appended as a sentinel so a poller can tell "still running" from "done". run_id must be 1-64 ASCII letters/digits/-/_ -- it's a path component, so this is enforced on both this endpoint and the ?id= that names it. id must also be unique: if a log already exists for it (another run still in progress, or a finished one never polled/cleaned up), the POST is refused with 409 rather than silently overwriting it.

POST /compute/gemma?id=my-run-1       (on one connection; blocks until done)
GET  /compute/gemma/status/my-run-1?peek=0       (on another, while the first is in flight)
{
  "data": {
    "run_id": "my-run-1",
    "lines": [
      "gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024",
      "Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022",
      "Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..."
    ],
    "pointer": 612
  },
  "meta": {"state": "complete", "took_ms": 1, "request_id": "..."},
  "notes": [],
  "next": []
}

Pass pointer back as ?peek= on the next poll; lines is empty (not an error) when nothing new has been written since. An unknown run_id (never started, or the main request hasn't written its log yet) is a 404, not an empty response -- genuinely different from "nothing new yet".

/compute/gemma/ravanan -- the same mapping, run per-chromosome via ravanan

Same request/response shape as /compute/gemma -- group (plus genotype_version), an uploaded genotype (genotype_upload), or genotype_file + snps_file; values as samples (named, aligned by name) or trait_values (positional), or a basket -- resolved by the same code as the direct runner (resolve_inputs), so the two cannot drift. Two fields are refused rather than ignored, because this workflow has no step for them: covariates (422 covariates_not_supported, use /compute/gemma) and normalize (422 normalization_not_supported). The same two steps (LOCO kinship, then LOCO GWA), but run by ravanan driving a CWL workflow compiled on every call (via ccwl compile, see below) from the ccwl source at cwl/gemma/gemma-loco.scm, instead of this server's own direct subprocess calls -- an alternative runner for the same computation, kept side by side with /compute/gemma so the two can be compared (correctness, timing).

Ravanan's cache is only as durable as RAVANAN_CACHE_DIR. Ravanan's reuse key is (script, inputs), and both live in its --store under that directory, so the cache survives a container restart if — and only if — the directory is on a host path the container bind-mounts. Measured: 13.6 s cold, then 3.6 s for the identical request after restarting the container, store unchanged. The default is under /tmp, which is neither durable nor visible inside the container; the reply carries a note when it sees that. Same reasoning for GN4_API_TOKENS_DB (./tokens.sqlite by default) and GEMMA_CACHE_DIR. See docs/ravanan-slurm.md.

K is shared across traits. The kinship step is given a constant-valued phenotype file carrying only the trait's missing-data pattern, because gemma -gk uses -p solely to learn which individuals are analysed; gwa gets the real values. Two traits over the same animals therefore reuse all twenty K matrices -- on the BXD fixture a second trait adds ~1 MB of store and 20 GWA jobs rather than ~13 MB and a second kinship scatter.

This runner calls gemma itself, not gemma-wrapper (the direct runner still calls the wrapper, which is what makes the comparison worth making). GEMMA 0.98.6 has -loco <chr> natively for both -gk and -lmm, taking the chromosome to leave out by name and requiring the BIMBAM annotation file (-a) that every genotype shape here supplies -- so gemma-wrapper's LOCO orchestration only duplicates the tool. Its parallel fan-out was never used on this path anyway: ravanan scatters one gemma -gk|-lmm ... -loco <chr> job per chromosome, and the kinship matrix flows from the kinship job to the GWA job as a CWL File output rather than through a wrapper cache. The wrapper's content-hash K cache is the one thing given up; a repeat request recomputes, which is what the per-chromosome scatter costs anyway. This workflow is written in ccwl at cwl/gemma/gemma-loco.scm, not raw CWL; compute_ravanan compiles it to CWL with ccwl compile fresh on every call (cheap -- pure macro expansion, no Guix build involved) into a generated, throwaway file under ravanan_cache_dir, never checked into the repo. compute_ravanan computes the chromosome list from snps_file itself (chromosomes_from) and passes it as the workflow's chromosomes input, since CWL's scatter needs a concrete array at invocation time -- and GEMMA's -loco takes exactly that chromosome name. ravanan_cmd, ccwl_cmd, gemma_ravanan_workflow (the ccwl source, not a CWL file), gemma_ravanan_manifest (ravanan's own required --guix-manifest, a Guix manifest naming gemma, at cwl/gemma/manifest.scm) and ravanan_cache_dir (its own cache directory, separate from gemma_cache_dir, for pheno files, run logs, the compiled workflow, and ravanan's own --store) are configurable the same way as the rest of src/config.rs's layered config. Its genotype and SNP inputs follow the same convention as /compute/gemma -- the bimbam/<file>_geno.txt + <file>_snps.txt pair, not the .geno file (see docs/genotypes.md) -- and the values must likewise line up one for one with that file's individuals.

ravanan's batch system is configurable too: by default each CWL step is a local child process, and RAVANAN_BATCH_SYSTEM=slurm-api submits every step to a Slurm cluster through slurmrestd instead (endpoint, JWT file, partition, nice and scratch directory via RAVANAN_SLURM_* / RAVANAN_SCRATCH). With slurm, ravanan_cache_dir must be on a filesystem the compute nodes can see, since ravanan's --store and scratch live there. See docs/ravanan-slurm.md for the keys, the fail-closed rules and the slurm-side auth requirements, and cwl/README.md for how a request gets from the API to a job on a node (and what the nodes themselves need). GET /compute/gemma/status/{run_id} is shared between both endpoints and checks whichever cache directory the run_id actually logged to.

Verified end to end on 2026-11-12 against the Slurm container from gn-machines (specials/slurm-container.scm): a group: BXD request became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the SNP map's chromosomes), all completed, 21 055 rows. See docs/ravanan-slurm.md for what the host and the nodes must provide -- including the two requirements that bit during that run: ravanan must be a build with the --slurm-api-endpoint fix, and the server's HOME must be writable by the job user inside the cluster.

The reply carries the same variance-component note /compute/gemma does (-pve, every chromosome). Since this workflow runs gemma itself, each job's .log.txt is a first-class CWL output and the note is read straight out of those files -- no archive to unpack and no wrapper-named copy to depend on. See docs/ravanan-slurm.md.

Concurrency. Like /compute/gemma -- same 50-individual threshold, since ravanan writes into the same cache directory -- only a large request claims a slot. One ravanan workflow at a time, one per caller: a second large submission while one runs (or is queued) is refused at once with 409 compute_busy, naming the run and linking /compute/gemma/status/{id} (which ravanan shares with /compute/gemma); a submission from another caller waits for the slot. One slot because ravanan already fans its own per-chromosome steps out -- over the cluster, when the slurm backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from /compute/gemma's: they are different runners of the same computation with their own cache directories.

Exploratory, and genuinely rough around the edges -- three real bugs turned up building this, none of them in /compute/gemma's own direct-subprocess path:

  • ravanan resolves gemma-wrapper by building a Guix profile from --guix-manifest against the ambient Guix process (GUIX_PACKAGE_PATH, no --guix-channels) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package in guix-bioinformatics too new for the pinned base Guix. This needs GUIX_PACKAGE_PATH set in whatever environment starts the genenetwork4 server itself (inherited by the ravanan child process genenetwork4 spawns); it is not something compute_ravanan sets.
  • ravanan's --store needs files/, logs/ and data/ subdirectories to already exist; it does not create them itself (compute_ravanan does now).
  • A step scattered over exactly one input (kinship's chr) needs an explicit scatterMethod even though CWL itself only requires one for multiple scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it.
  • Every chromosome's gwa.cwl run shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in --chromosomes), so the 20 scattered archive outputs all collide under ravanan's optional --output-directory symlink-everything-into-one-flat-dir step; compute_ravanan simply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own --store via its stdout JSON record, never through --output-directory.

A real end-to-end run against the BXD fixtures (same rows as the /compute/gemma example above, confirmed byte-for-byte including rs31443144's exact values) took 16.7s wall-clock scattered across 20 chromosomes this way, vs. /compute/gemma's own ~19s and the earlier sequential (non-scattered) ravanan workflow's ~21s -- a real, if modest, win from letting ravanan's own scheduler fan the chromosomes out instead of going through gemma-wrapper's internal GNU parallel call, on top of no longer needing that path's own PARALLEL_SHELL workaround at all (each scattered job's --no-parallel gemma-wrapper call never invokes GNU parallel in the first place, so the bash-time-keyword bug it worked around does not arise here).

Those 16.7s are with ravanan's default single-machine batch system, where a step is just a local child process. With RAVANAN_BATCH_SYSTEM=slurm-api every step additionally pays Slurm dispatch -- submit, scheduler tick, launch, poll -- which on a small input dwarfs the mapping itself: three chromosomes, measured on the test container (gn-machines/specials/slurm-container.scm, one node, two CPUs), are 2.3s cold / 0.7s warm on /compute/gemma, 4.7s / 3.5s on ravanan single-machine, and 33s on ravanan+Slurm with a 5s scheduler interval (158s with Slurm's defaults), while each job's own gemma-wrapper run took under a second. Same rows in every case. See docs/ravanan-slurm.md for the breakdown and what to tune.

/compute/qtl2

A genome scan with R/qtl2: genotype probabilities from a hidden Markov model, and a linear mixed model with leave-one-chromosome-out kinship. It is the middle ground between /compute/rqtl (Haley-Knott, no kinship) and /compute/gemma (an LMM, but biallelic BIMBAM with mean-imputed genotypes), and it is how multiparent populations are analysed.

{
  "group": "BXD", "genotype_version": "BXD.4",
  "trait_name": "BXD::10001",
  "samples": [{"name": "BXD1", "value": 61.4}, …],
  "kinship": "loco",        // loco (default) | overall | none
  "model": "normal",        // normal (default) | binary
  "cores": 4
}

Rows are marker, chr, pos (Mb), lod — the same shape /compute/rqtl returns, so /plot/rqtl draws it unchanged.

On DOL trait 10003 the three engines tell the same story with different emphasis, which is what modeling relatedness buys: R/qtl's HK scan peaks on chr9 at LOD 5.86 and does not see chr16 at all; GEMMA's LMM prefers chr16 at −log10 p 7.15; R/qtl2's LMM finds chr9 (LOD 4.85), chr19 (4.76) and chr16 (4.24) on one footing.

Autosomes only. The X needs a sex covariate to be fitted properly, and the Y and mitochondrion degenerate the per-chromosome kinship — GEMMA's LOCO run is given the same restriction, for the same reason. The notes say how many markers were left out.

Multiparent crosses. A multiparent population cannot come from a .geno: that file carries two codes per marker (B/D, A/H/B), which is a collapse, not a founder allele. It has to ship its own qtl2 control file with founder genotypes, so the endpoint also takes a cross by name:

{"rqtl2_cross": "do", "trait_name": "DO::WBC",
 "samples": [{"name": "JAXW202.2129", "value": 4.34}, …]}

The name is looked up as genotype/rqtl2/{name}.json, then {name}/{name}.json, then {name}.json under GENOTYPE_FILES; do/ is the one such cross this deployment has (742 individuals, 8 founders A–H, 6 413 markers, WBC and NEUT). The samples are matched to the cross by name and anything outside its population is left out (and counted in notes).

A multiparent scan is a different claim from a biallelic one: every marker is fitted with one coefficient per founder, so the LOD asks whether any founder differs, not which. Reading a multiparent LOD as a strain effect is the usual mistake; per-founder effect sizes need scan1coef, which this endpoint does not return yet.

Measured on the DO (WBC, 8 cores): genotype probabilities 174 s and 1.39 GB, cached per cross and keyed by the control file and its directory; LOCO kinship 15 s, cached; the scan itself 21 s. scan1 is run on one core for a multiparent cross deliberately — its forked path stalls here on allele probabilities (25+ minutes with 8 cores and no CPU consumed, against 21 s serial). The autosomes-only rule applies here too: the DO's X is left out (369 markers) rather than fitted without a sex covariate.

Panel conversion. The cross is generated from this deployment's own .geno file and cached (QTL2_CACHE_DIR), because the shipped genotype/rqtl2/ control files are not usable: they name paths relative to a directory the data is not in, declare no genotype-code map (every genotype reads as missing) and no orientation (individuals read as markers) — four traps that each produce a plausible flat scan rather than an error. The conversion also drops heterozygous calls on a riself panel, as R/qtl's convert2riself does. Configuration: QTL2_SCRIPT_PATH, QTL2_CACHE_DIR.

A gene symbol is useful beyond our own pages, and SKILL.md carries the table of external services that still resolve (checked 2026-11-11): RGD for both mouse and rat, MGI, the Alliance of Genome Resources, NCBI Gene, Wikidata (the QID that /genes/{symbol}/aliases already returns), UniProt, Ensembl, GTEx, STRING, GeneMANIA, Open Targets, the Human Protein Atlas, the GWAS Catalog, BioGPS, the Allen Brain Atlas and GeneCup — with the retired ones named so they are not offered by mistake (HomoloGene, UniGene, Genotation, GeneBridge, PhenoGen).

Two conventions that keep the links honest: quote the species the symbol came from, and prefer RGD/GeneMANIA(rat) for a rat symbol and GTEx/Open Targets/ Protein Atlas for a human one — a mouse symbol in GTEx resolves to an empty report, because GTEx is human-only.

Why R/qtl and R/qtl2 disagree on a riset panel

They read the same file and can end up fitting different predictors, so a peak in one is not a bug in the other. The mechanism, with BXD trait 16627 as the worked case (BXD.8, 35 phenotyped animals, all measured on production):

marker Mb R/qtl R/qtl2 riself R/qtl2 f2
rsm10000000889 5.652 9.59 3.79 9.98
rs33299937 5.653 9.59 3.73 9.68
rs13476332 6.311 9.59 3.70 —

What R/qtl does. scripts/rqtl_wrapper.R reads the cross with genotypes = c(@mat, @het, @pat) — codes 1, 2, 3 — and then, for a riset panel, calls convert2riself, which is adaptive:

gtab <- table(pull.geno(cross))
usethree <- !(gtab["3"] < gtab["2"])   # is code 3 at least as common as code 2?
if (usethree) { dat[!is.na(dat) & dat != 1 & dat != 3] <- NA; dat[dat == 3] <- 2 }
else          { dat[!is.na(dat) & dat != 1 & dat != 2] <- NA }

It keeps the two most numerous codes and drops the third (warning Omitting N genotypes with code==2), then recodes to 1/2. No HMM, no probabilities: scanone fits each marker's own calls.

What R/qtl2 does. riself's state space is AA, BB (two states, no het), so a heterozygous call is not representable and becomes missing; the HMM then re-imputes it from flanking markers, and scan1 fits probabilities, not calls. On BXD.8 the conversion drops 610 684 calls, 12.7 % of the matrix (the reply says so). At a marker with 14 % H, that is a different predictor from R/qtl's, which is why the peak survives in one engine and thins out in the other — not a failure of either.

genotype_model: "f2" keeps the calls as a third genotype (AA, AB, BB) and recovers R/qtl's number (9.68 against 9.59). It is a diagnostic: it fits het as a real genotype and assumes an outbred intercross design, which an inbred line is not. Its value is that it separates "the conversion cost the information" from "the model cannot see the locus".

What the proper model is, and where it stands. qtl2 has general RIL types — genril2 / genail2, from qtl2 0.18: “handle any number of founders; include the number of founders in the cross type… genril assumes the progeny are inbred lines (RIL), while genail assumes the progeny have two random chromosomes (AIL)”, with cross_info of length 1 + n_founders: the first column the number of generations, the rest founder frequencies. Both are supported by the qtl2 0.42 this deployment runs (checked through qtl2:::check_crosstype()), and genril2 is the model that fits a riset panel's residual heterozygosity honestly.

It is not wired yet, and the reason is concrete rather than a matter of effort: a founder-based cross type expects individual genotypes coded as alleles/pairs of founders (as the DO's A–H are), not the 1/2/3 of a two-parent cross; a prototype with numeric codes had every genotype read as missing. The remaining work is therefore exact: build founder_geno from the two parental alleles (for a two-parent cross every marker is homozygous founder 1 vs founder 2 by definition of the coding), write cross_info with the generation count, and code H as the founder pair rather than code 2. Then compare trait 16627 against 9.59/9.68 — and check DOL through the same path, because its H is not residual heterozygosity at all: it is a collapsed DO, where H means "two different founders" and the honest model is crosstype: "do" with founder genotypes the collapse destroyed.

What to do when the engines disagree, in order: .../markers/{marker}/classes?trait= to see the class sizes and means; .../frequencies for the H share at the marker; genotype_model: "f2" to see what keeping the calls would give; and quote which model ran, because a reply says Cross: riself or Cross: f2.

/compute/bulklmm

A genome scan that can be asked for thresholds. GEMMA's LOCO scan is the reference for a biallelic panel, but a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm; the usual stand-in is R/qtl's nperm, whose null shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits one LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition — relatedness included — so a threshold costs seconds:

{"group": "BXD", "genotype_version": "BXD.8", "trait_name": "BXD::16627",
 "samples": [{"name": "BXD1", "value": 1.5}, …], "nperm": 200, "seed": 42}

BXD trait 16627, 35 phenotyped animals, 21 056 markers, 22 s:

peak rsm10000000888 chr2 4.92 Mb  LOD 9.59  beta 0.0027  h2 3.0e-16
permutation thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47

which is the trait's published locus (PublishXRef.Locus rsm10000000888, LOD 9.59) and a genome-wide-significant peak against a threshold this API could not compute before. Rows are marker, chr, pos (Mb), lod, beta, h2.

Thresholds on another engine's plot. /plot/gemma draws no reference line by default — a line on a plot is an annotation, not part of the result, and a reader should be told where each one came from. GEMMA's Bonferroni line is one more opt-in labelling (?bonferroni=true); a threshold computed here travels with the plot, labelled, because the two are different nulls:

POST /plot/gemma?thresholds=3.50,3.65&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
{ the /compute/gemma reply }

Each line is drawn dashed in the series colour with its label and value (bulklmm alpha 0.05: 3.65), and the Bonferroni line only appears when it is asked for — so a plot shows exactly the lines someone chose, each named. /compute/bulklmm's reply already offers that href with its α = 0.05 threshold filled in, so the picture is one link away.

/plot/bulklmm draws two reference lines for it: the genome-wide α = 0.05 and the stricter α = 0.01. The reply also reports α = 0.10, but that is the loosest bar in the set — drawing it invites reading it as the line to clear — so it appears only on request (?all_thresholds=true, or an explicit ?thresholds=3.50&threshold_labels=alpha 0.10). GEMMA's Bonferroni line is opt-in in the same spirit: a reference line is an annotation, not a result.

About cores. BulkLMM's single-trait scan() — what this endpoint calls — is serial in Julia by design: the package threads bulkscan() (many traits) and its helpers, not scan() (checked in its source: Threads.@threads appears in bulkscan.jl and bulkscan_helpers.jl, not in scan.jl). What was parallel before was BLAS, not the scan. The driver therefore splits the markers into independent blocks and scans them on Julia threads (--blocks, default = BULKLMM_THREADS), which is exact — the LODs and permutation thresholds are identical to a serial run, verified bit-for-bit at 1 and 4 blocks — and caches the parsed matrix and the kinship per panel:

cold, 1 block : read 1.3 s, scan 9.0 s, total 20.9 s
cold, 4 blocks: read 0.4 s, scan 7.5 s, total 18.3 s   (same LOD 9.589938)

The gain is modest at this size because the LMM loop is only about 8 s of the run and each block re-estimates the variance components; the rest is Julia startup. Where the cores pay is bulkscan() — a hundred traits in one process, threaded over trait blocks. That is the shape an eQTL endpoint wants, and it is the next thing worth adding here.

Two differences from GEMMA are stated in every reply rather than implied:

  • a global kinship matrix, not leave-one-chromosome-out, so its p-values are not GEMMA's;
  • h2_null says how much relatedness it found. Here it is ~0, so the fit is effectively a fixed-effects model — which is exactly why it agrees with R/qtl on this trait where GEMMA's LOCO scan differs.

See docs/bulklmm.md for the model comparison, the measured core behaviour and the packaging options. The engine is a Julia subprocess (BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT, BULKLMM_SCRIPT, BULKLMM_THREADS, BULKLMM_CACHE_DIR) with the driver in scripts/bulklmm_scan.jl. The Julia depot is prepared once, deliberately, and is small: 79 packages, 139 MB, no GPU/CUDA stack — BulkLMM's closure is CSV, DataFrames, Distributions, Optim and what those need. Every request runs with JULIA_PKG_OFFLINE=true, so a scan can never download anything.

/compute/bnw

A Bayesian network over a set of variables, learned by BNW (the Bayesian Network Webserver): which traits belong together under a causal ordering. The computation runs in BNW's own headless driver — the bnw program in the gn-bnw repository — so this endpoint only prepares a matrix, calls out, and parses the reply (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR).

{
  "group": "BXD", "traits": [10001, 10003, 10038],
  "maxparents": 4, "k": 1, "tiers": 3, "threshold": 0.5
}

or, for a matrix the database does not hold (a basket's columns, a handful of probe sets):

{
  "values": [
    {"name": "Trait1", "samples": [{"name": "BXD1", "value": 1.2}, …]},
    {"name": "Trait2", "samples": [{"name": "BXD1", "value": 0.4}, …]}
  ]
}

Rows are positional with meta.columns:

from, to, posterior, in_structure

One row per candidate edge (variables²−variables of them: self-edges are dropped), strongest posterior first. posterior is the share of the k-best networks that contain the edge; in_structure says whether it survived threshold. The four parameters default to BNW's own (maxparents 4, k 1, tiers 3, threshold 0.5 — from create_tiers_gom_part1.php), and the reply names the values used, because a network at threshold 0.5 and one at 0.9 are different answers.

A posterior is not a p-value. It says how often the search put that arrow there in this sample — a statement about correlation under a causal ordering, not about surprise, and not an effect size. At most 40 variables (the search is exponential in in-degree) and at least 3 samples complete across every variable: BNW models a complete multivariate normal, so a sample missing any value is dropped and the reply says how many survived.

POST /plot/bnw draws it from the posted reply, the same way /plot/gemma does — an SVG back, nothing recomputed, the plotter's one-line summary in x-plot-summary. The default figure is the network: nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge posterior, and the below-threshold edges as faint dotted lines so a reader sees what was considered as well as what survived. ?kind=posterior draws every candidate edge as a cell in a matrix instead, which is the view that says how close a rejected edge came to the threshold. Pure-Python SVG (BNW_PLOTTER, run by PLOT_PYTHON), so no graphviz is needed in the deployment.

From a network to a mappable trait: POST /compute/bnw/features. Every node with at least one parent at the threshold is a module — the node and its parents — scored per sample by the first principal component of the module's standardised members, sign-corrected so the score rises with the node. Without ?module= the reply is one row per module, module, members, size, variance_explained (the share of the module's variance the component explains, which says whether the score is a fair summary). With ?module=NAME it is that module's per-sample scores in the {name, value} shape every mapping endpoint already takes — post them as samples to /compute/rqtl, /compute/gemma or /compute/qtl2 and map the module like any trait. That is the chain BNW exists for in GeneNetwork: structure → module → QTL. A module is a correlation structure, not a QTL: what maps is where its members' shared variation comes from, not which member causes which.

BNW's own worked examples: GET /compute/bnw/examples and GET /compute/bnw/examples/{name}. The five datasets its tutorial and examples pages present (synthetic 2-genotype/6-trait, the scientific 5-node network at 100 and 500 samples, BXD spleen immune genes, the time series) are listed with their sizes, and each one is served as a matrix in the values shape /compute/bnw takes — with a ready-to-post action carrying the body. ?columns= restricts the matrix, which the spleen example needs (14 variables, and a caller may want fewer). genotype_column warns that a column is a genotype code: this driver is continuous-only, so such a column enters the network as a number, as the web application's GeneNetwork path treats it. The files are read from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/example_datasets); a deployment without it answers 404 no_examples rather than pretending.

The driver's contract is documented in gn-bnw/info_files/gn4-integration-plan.md; tests/bnw.rs checks the endpoint against the answer the BNW web application gives for its own six-trait example, so "it works" means the same edges, not merely a 200.

/compute/fitqtl

/compute/rqtl says where a QTL is (a LOD per marker); this says how much, by fitting R/qtl's fitqtl at the markers you name -- normally the peaks a scan already reported, via GET /groups/{group}/genotypes/{version}/markers. It is separate because it is a separate question, asked about a few positions rather than a genome: fitqtl fits one model per marker, so a genome-wide version would be thousands of fits (the cap is 100 markers per request).

{
  "group": "BXD", "genotype_version": "BXD.4",
  "trait_name": "BXD::10001",
  "samples": [{"name": "BXD1", "value": 61.4}, …],
  "markers": ["mCV24409492", "rs8253516"]
}

Rows are positional with meta.columns:

marker, chr, pos, lod, pct_var, additive, se, t, p

additive is fitqtl's estimate for the QTL term. R/qtl codes the QTL genotype ±1, so this is half the difference between the two homozygote classes -- the same convention GeneNetwork's own additive uses, and on BXD trait 10001 it reproduces the published value to six decimals (2.394472 here against GN2's 2.394444, from the same peak region). pct_var is the model's percentage of variance explained and p its p-value; se and t belong to the effect. Markers not in the genotype file, or that fitqtl cannot fit, are absent rather than reported as zero.

Covariates work exactly as on /compute/rqtl -- covariates: [{"case_attribute": "Status"}], inline values, or an explicit map -- and are fitted alongside the QTL, so the effect reported is the one after conditioning. Two details worth knowing: a covariate that is spread across the panel but constant among the individuals a trait covers is left out, with the columns named in notes (R/qtl stops with "system is exactly singular" otherwise), and the LOD then no longer equals the scan's, because it is a different model.

The cross-check that matters: without covariates, fitqtl's lod at a marker equals scanone's LOD at that marker, to every digit the CSV carries -- the test suite pins it. Same alignment rules as /compute/rqtl (values matched onto the panel by name, parents/F1s excluded unless include_parents_f1), the same normalize/winsor_* pipeline (and the effects then come out on that scale), and the same R/qtl gate. Configuration: FITQTL_SCRIPT_PATH (default scripts/qtl_fitqtl.R), RQTL_CACHE_DIR for its scratch files.

/compute/rqtl

Maps a QTL by running GeneNetwork3's own scripts/rqtl_wrapper.R (via Rscript), copied into this repo verbatim: a scanone genome scan, plus permutation thresholds if nperm > 0. Same request/response shape as /compute/gemma -- synchronous, ?id=/GET .../status/{run_id} progress-streaming reusing the same infrastructure (src/streaming.rs) -- see src/rqtl.rs for the invocation itself, its documented scope (no interval mapping or pair-scan scantwo -- just the common single-scan case), and a fix made relative to GeneNetwork3: its own cache-key hashing of the genotype/phenotype files never actually looks at their contents (a key-name mismatch between generate_rqtl_cmd and its real caller), so two different phenotypes mapped with the same other options could silently reuse each other's cached result there; fixed here by always hashing both files' contents.

Covariates. The same covariates list as /compute/gemma (above) -- case attributes by name, or per-sample values -- fitted here with --addcovar and a generated --covarstruct file. R/qtl's wrapper takes categorical covariates as a single column and dummy-expands more than two levels itself, so a level with no observations is its problem, not a singular column: no columns are dropped on this path. A missing value is x (the marker the wrapper calls missing), so R/qtl drops those individuals; the coding and the count of complete cases come back in notes.

Give it a group, not a path. group (an InbredSet name) plus an optional genotype_version resolves the .geno server-side; see GET /groups/{group}/genotypes. samples are then matched onto the panel by name -- unknown names dropped, missing individuals null -- and parents/F1s are excluded unless include_parents_f1 is set. The resolved file, version, panel size and how many values aligned come back in notes, so a run is reproducible without the caller ever naming a path. genotype_file is still accepted as an explicit override.

If you name the file yourself, R/qtl takes the GN genotype file itself -- genotype/<file>.geno, e.g. genotype/BXD.8.geno (235 individuals × 21 057 markers) or genotype/HET3-ITP.12.geno (6 447 individuals × 894 markers) -- and reads its format directly. Which engine a group offers, and which of its genotype files is current, are GeneNetwork2/3 conventions summarised in docs/genotypes.md; InbredSet.MappingMethodId is what decides the engine (BXD = both, HET3-ITP = R/qtl only -- UM-HET3 is a 4-way cross). Measured on this deployment: BXD with the WGS panel, 235 samples of which 34 carry values, 21 056 markers in 7.5 s; UM-HET3 lifespan over 6 447 individuals, 893 markers in 83 s, peak LOD 7.95 at chr 1, 24.04 Mb.

Important: R/qtl's cross file format has no per-sample name matching of its own -- samples must list values in exactly the genotype file's sample-column order, and it must have as many entries as the genotype file has individuals. The second half of that used to go unchecked (as it does in GeneNetwork3): nine values scanned against a six-individual cross came back as a complete, plausible-looking set of LOD scores belonging to nobody. scripts/rqtl_wrapper.R now stops with "the phenotype file has 9 samples but the genotype file has 6 individuals" instead, which surfaces as a 502 rqtl_failed. It still cannot tell whether the order is right -- that is the caller's responsibility, and the reason a trait's values and its genotype file have to come from the same panel.

{
  "data": {
    "markers": [
      ["rs31443144", "1", 3.010274, 0.0923496],
      ["rs6269442", "1", 3.492195, 0.1131095]
    ],
    "permutation": null
  },
  "meta": {
    "state": "complete",
    "took_ms": 420,
    "request_id": "...",
    "columns": ["name", "chr", "pos", "lod"],
    "num_rows": 6,
    "num_cols": 4
  },
  "notes": [
    "markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.",
    "Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename <hash>-output.csv"
  ],
  "next": []
}

(Real output, against the tiny 6-marker/6-sample BXD riset fixture in tests/test_data/, trimmed to two markers. Each marker row is positional, in meta.columns' order, rather than a {"name": ..., "chr": ..., ...} object per row -- same space-saving shape /compute/gemma uses for its own (much larger) rows; a 4-way cross's extra mean/SE effect columns, if present, trail each row and aren't named in meta.columns (see MarkerResult's doc comment in src/rqtl.rs). permutation is null unless nperm was set above 0; when it is, it's {"values": [...], "suggestive": ..., "significant": ...}, the 67th- and 95th-percentile thresholds GeneNetwork3 itself uses -- computed here with numpy.percentile's default linear interpolation, reproduced in src/rqtl.rs's percentile.)

Genotype files

/compute/rqtl and /compute/gemma take paths on the server, so which panel file is current, in which format, and in which sample order is a deployment question rather than an API one. That convention (GN2's per-group genotype/<Group>.json, whose genofile list is newest-first; genotype/bimbam/<file>_geno.txt for GEMMA; the .geno header for the sample order; InbredSet.MappingMethodId for which engine a group offers) is written up in docs/genotypes.md, together with the three mappings measured against this deployment.

OpenAPI

GET /openapi.json serves an OpenAPI 3.1 document covering every route. It's built with utoipa and utoipa-axum:

  • Each handler carries a #[utoipa::path(...)] attribute describing its method, path, parameters and responses; each response/request struct derives utoipa::ToSchema (see VersionInfo, db::Species, db::SpeciesDetail, error::ApiError in src/lib.rs/src/db.rs/ src/error.rs).
  • Routes are registered with utoipa_axum::routes!(handler) instead of plain axum::routing::get(handler), on an OpenApiRouter instead of a plain Router, so each route's metadata is collected as it's added.
  • ApiDoc (a #[derive(utoipa::OpenApi)] struct) supplies the document's info (title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's.
  • OpenApiRouter::split_for_parts() splits the builder back into a plain axum::Router (returned from app()) and the assembled utoipa::openapi::OpenApi value, which the /openapi.json handler then just serializes with Json.
  • ApiDoc's #[openapi(security(("bearer_token" = [])), ...)] makes a token the default for every operation, so an endpoint that needs one doesn't repeat security(...) itself -- only the handful that must NOT require one (/, /version, /get-test-token) override it with an empty security() of their own. error::Unauthorized (a #[derive(utoipa::IntoResponses)] type) is the shared 401 response every gated endpoint references by name in its own responses(...) list (responses(Unauthorized, (status = 200, ...), ...)) instead of writing out (status = 401, description = "...", body = ApiError) by hand each time.

Because the attributes are declared alongside the handlers they describe, the spec can't silently drift out of sync with the code the way a hand-maintained spec file could -- adding or changing a route without updating its #[utoipa::path(...)] is a compile error (the macro checks the response/schema types exist) or an obviously incomplete diff, not something to notice weeks later.

The document is assembled once, when app() runs at process startup (not at cargo build time, and not freshly on every request): the #[utoipa::path]/#[derive(ToSchema)] macros expand to Rust code at compile time, but the actual merging of all routes' metadata into one OpenApi value happens the first time app() executes. Every GET /openapi.json request after that just serializes the same already-built value.

Success responses

Every success response is one JSON envelope (src/response.rs): the data, next to facts about the response, remarks on the data, and suggested follow-up calls. It is the success counterpart of the error shape below.

{
  "data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."},
  "meta": {
    "state": "complete",
    "took_ms": 3,
    "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
  },
  "notes": [],
  "next": [
    {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
    {"href": "/groups?species=mouse", "description": "Groups of this species"}
  ]
}
  • data is the response proper: exactly what each endpoint below documents.
  • meta.state is complete, or partial when the response is usable but something was left out or unavailable -- notes says what. For example /search is partial when the database could not be reached (hits come back without locus and abbreviation) or when a sorted search ranked a full 1000-match window and more matches exist; /publications/{pubmed_id}/traits is partial when traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure.
  • meta.took_ms is the server's time for the request, from when it started handling it; it excludes the network. meta.request_id is the request's id, also in the x-request-id response header and in any error body, so a report can be correlated with logs.
  • x-gn4-user names the caller on every authenticated response: the token's sub (e.g. pjotrp). It is there for the proxy's access log -- user=$upstream_http_x_gn4_user in the gn4 log format -- so the deployment can answer "who is using this, successfully" rather than guessing from addresses and user agents. Unauthenticated responses (/version, /openapi.json, a 401) carry no such header, and the caller only ever sees an identity it already presented.
  • The same identity goes into this server's own access log: one JSON line per request on stderr — {"event":"request","method","path","status","took_ms", "request_id","user"} — with user absent when no token was verified, the path stripped of its query string, and the token itself never logged. GN4_ACCESS_LOG=false turns it off. The event: error lines for 5xx go to the same stream and carry a request_id, so a failure and the request that caused it are one grep apart.
  • notes are server-authored remarks on data, e.g. lod = lrs / 4.604 on responses that carry lod. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's.
  • next lists follow-up calls, each a relative href that resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large). GET /search suggests the next page and the top hit's metadata.
  • actions lists suggested calls that are not plain GETs, so they cannot be next links: each carries method, href, a description, and a body where the server can build one. It is present only when there is something to suggest, so most responses keep the four members above. Today the per-strain values endpoints (/groups/{group}/traits/{id}/values, /datasets/{dataset}/probesets/ {probeset}/values, /traits/{dataset}/{trait}/values) use it to hand back the values they just returned, ready to map:

json "actions": [ {"method": "POST", "href": "/compute/rqtl", "description": "Map 'BXDPublish::10001' with R/qtl (scanone): this body carries the 34 values above as `samples`, in the same order. Replace the `genotype_file` placeholder with a path that exists on the server. …", "body": {"genotype_file": "<path on the server, e.g. a genotype file it already holds>", "trait_name": "BXDPublish::10001", "samples": [{"name": "BXD1", "value": 61.400001525878906}, "…"]}}, {"method": "POST", "href": "/compute/gemma", "description": "…", "body": {"genotype_file": "<…>", "snps_file": "<…>", "trait_values": [61.400001525878906, "…"]}} ]

The file paths stay bracketed placeholders on purpose: they have to exist on the server's own filesystem, which the server cannot know on the caller's behalf (see the /compute sections below).

/ returns the plain text Hello, world! and /openapi.json the OpenAPI document itself; neither is wrapped. In /openapi.json every other success response is an Envelope_* schema (a test enforces this).

Error handling

Every endpoint uses the same ApiError JSON shape on error, modeled on RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see src/error.rs). /species and /species/{species} can actually produce one today (database errors, or a 404 for an unknown species); / and /version document the same 500 response in /openapi.json for consistency, even though neither has a failure path currently (no database access, no I/O):

{
  "type": "/errors/species_not_found",
  "title": "Species not found",
  "status": 404,
  "code": "species_not_found",
  "detail": "No species matching 'nonexistent'.",
  "operation_id": "species_by_name",
  "spec_url": "/openapi.json",
  "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
  "took_ms": 2,
  "next": "/species"
}

next is omitted (null) when there's no obviously related endpoint to suggest, e.g. a hypothetical error from /:

{
  "type": "/errors/internal_error",
  "title": "Internal error",
  "status": 500,
  "code": "internal_error",
  "detail": "example failure for /",
  "operation_id": "hello_world",
  "spec_url": "/openapi.json",
  "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
  "took_ms": 1,
  "next": null
}
  • type and spec_url are relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allows type to be a relative URI reference. spec_url is always /openapi.json, on every error from every endpoint.
  • operation_id is the OpenAPI operation that failed (same name as in /openapi.json).
  • request_id is the request's x-request-id, generated per request (or echoed back if the client already sent one) by SetRequestIdLayer/PropagateRequestIdLayer in src/lib.rs, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token.
  • took_ms is the server's time for the request up to the error, the same measure as a success response's meta.took_ms (see Success responses). RFC 7807 allows extension members like this one, request_id, code and next next to its standard ones.
  • next is an optional hint at a related endpoint to try instead; e.g. a 404 from /species/{species} sets next to /species, so a caller with a wrong species name can discover the valid ones.

Running with cargo

Requires a Rust toolchain with cargo (edition 2024).

cargo run

The server listens on 0.0.0.0:3000 by default. Set GENENETWORK4_PORT to use a different port, or GENENETWORK4_BIND to bind a specific address instead of every interface:

GENENETWORK4_PORT=8080 cargo run
GENENETWORK4_BIND=127.0.0.1 cargo run

This deployment runs two instances: 3000 (production) and 3001 (development)

The reference deployment keeps a development instance next to the live one, on the same box, so work can be tried out without disturbing what the site is serving:

instance port binary run caches proxied by nginx
production 3000 target/release/genenetwork4 defaults (/tmp/gn4-gemma, /tmp/gn4-rqtl, …) yes, at /gn4/ (see docs/nginx.md)
development 3001 target/debug/genenetwork4 /tmp/gn4-dev/{gemma,rqtl,ravanan} no

They differ only in the port, the build profile and those cache directories; both use the same read-only SQL_URI, the same Xapian index and the same auth stub, so a request behaves the same on either. The separate caches matter: a GEMMA or R/qtl run on the development instance must not fill, or be sped up by, production's warm cache.

P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT')   # needs R + gemma-wrapper on PATH

# production
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3000 GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
nohup ./target/release/genenetwork4 > /tmp/gn4-3000.log 2>&1 &   # pid in /tmp/gn4.pid

# development: same, plus the port and its own caches
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3001 GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
GEMMA_CACHE_DIR=/tmp/gn4-dev/gemma RQTL_CACHE_DIR=/tmp/gn4-dev/rqtl \
RAVANAN_CACHE_DIR=/tmp/gn4-dev/ravanan \
nohup ./target/debug/genenetwork4 > /tmp/gn4-dev.log 2>&1 &   # pid in /tmp/gn4-dev.pid

Both need the PATH/R_LIBS_SITE pair (see "Building with cargo, outside a bare toolchain" below) or R/qtl fails with there is no package called 'optparse', and both need the auth stub on 127.0.0.1:8081 to serve /auth/public-jwks (see "Tests").

Anything the server keeps in memory is per instance: the pool, the JWKS and authorisation caches, and any in-process compute limit. So the two instances do not serialise against each other — a mapping on :3001 runs alongside one on :3000 — which is the point of the split, and a good reason not to treat the development instance as a capacity test of what production will do.

Building with cargo, outside a bare toolchain

cargo build in the package's own dependency shell is the supported way to build; a bare cargo build against a distro toolchain fails, and one of the reasons cannot be fixed by installing packages, since it is a Xapian version conflict (see Running with Guix for the pin in guix.scm).

GUIX_PACKAGE_PATH=<path-to-guix-bioinformatics> \
guix shell -D -f guix.scm -- bash -c '
  export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib       # autocxx: libclang 13, not 22
  export DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib      # libduckdb-sys: no duckdb.pc
  export DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include
  cargo build --release'
  • GUIX_PACKAGE_PATH (or -L) is for (gn packages gemma); without it guix.scm fails to load with "no code for module (gn packages gemma)".
  • The three exported variables are exactly what guix.scm's set-duckdb-paths phase exports for the packaged build. Without LIBCLANG_PATH the build stops in xapian-rs's build script with "Unable to find libclang"; without the DUCKDB_* pair libduckdb-sys links a -lduckdb with no search path.
  • The resulting binary resolves every shared library from the shell's store paths on its own (Guix's linker wrapper records them as a RUNPATH), so running it needs no LD_LIBRARY_PATH -- but it does stop working once those store paths are garbage-collected.

Project memory: why this code looks like this

The repository carries an ai-memory store (ai-memory.sqlite, gitignored) that links each commit to the reasoning behind it, plus an optional Xapian topic index over both. It is how a session that starts with an empty context recovers what the last one worked out — measured at 16 ms for a text query and about 100 ms through the topic index:

ai-memory -C . search 'pve'                     # reasoning, full-text
ai-memory -C . why gemma.compute_compute        # commits that touched a symbol, with its reasoning
ai-memory -C . --index ~/.cache/gn4-topics search 'topic:ravanan-slurm'

Operational runbooks live there too — for example the Slurm container from gn-machines, its JWT, the environment /compute/gemma/ravanan needs, and the fastest endpoint test — because the commands are the part a reader cannot reconstruct from a diff. The store is local and per repository: it survives a cleared session, not a lost machine, so every conclusion is also in a commit message and in these docs.

Development

Every endpoint needs a bearer token (see the note under Endpoints), which normally means getting one from gn-auth yourself. For local development, genenetwork4 can hand you one instead: POST /get-test-token calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result. It needs no setting and is not tied to a mode.

cargo run

Get a token, then use it against a real endpoint:

curl -X POST http://127.0.0.1:3000/get-test-token
{
  "data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"},
  "meta": {"state": "complete", "took_ms": 41, "request_id": "..."},
  "notes": [],
  "next": []
}
TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])')
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species
{
  "data": [
    {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"},
    ...
  ],
  "meta": {"state": "complete", "took_ms": 2, "request_id": "..."},
  "notes": [],
  "next": [
    {"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
  ]
}

The endpoint is registered by default and listed in /openapi.json (GENENETWORK4_TEST_TOKEN=false switches it off, see below), and decides per request whether to answer: only a caller on this machine gets a token. That means the TCP connection's peer address is in 127.0.x.x (or is ::1), and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header. Anything else, including a request whose peer address is unknown, is a 403. The peer address comes from the connection itself, never from a header a client can set; the proxy headers are used only to refuse a request that claims to have been forwarded. This is why the check also refuses forwarded requests: a reverse proxy running on the same host connects from 127.0.0.1 for every client, so without it the whole internet would look local. A proxy that forwards without setting any of those headers would defeat the check, so don't put one in front of a server whose gn-auth has the dev user (on a production gn-auth the dev user normally doesn't exist, and the request just fails with a 502).

Behind nginx or in a container

docs/nginx.md is the full deployment recipe: serving genenetwork4 under a path prefix such as https://host/gn4/ from a loopback port, the long proxy_read_timeout /compute/* needs, and how the root-relative links in responses are kept under the prefix. What follows here is only about POST /get-test-token.

The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here):

  • Container on a bridge network, port published. The server sees the bridge gateway (something like 172.17.0.1) as the peer, not 127.0.0.1. Every request through the published port is a 403, whether it comes via nginx or from the host itself. It still works from inside the container, e.g. docker exec <container> curl -X POST http://127.0.0.1:3000/get-test-token.
  • Container sharing the host's network, nginx on the same host. nginx connects from 127.0.0.1, so the peer check passes for every client. The server then relies on refusing requests that carry X-Forwarded-For, Forwarded or X-Real-IP, and a minimal proxy_pass sets none of them. Without one of the measures below, anyone who can reach the site could get a token.

For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third:

  1. Block the path in nginx, independent of genenetwork4:

nginx location = /get-test-token { return 404; }

  1. Make nginx say it is forwarding, so genenetwork4 refuses proxied requests (any one of the three headers is enough):

nginx proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  1. Switch the endpoint off in genenetwork4 itself with GENENETWORK4_TEST_TOKEN=false (an environment variable, or a GENENETWORK4_TEST_TOKEN=false line in GN4_CONF). The route is then not registered at all: POST /get-test-token is a 404 and the endpoint is absent from /openapi.json. The setting fails closed: only true, 1, yes or on (any case) keep the endpoint on, and anything else -- false, an empty value, or a typo such as flase -- turns it off. It is on by default, so a local setup needs nothing.

Also relevant: the token is for gn-auth's dev user (GN_AUTH_DEV_USERNAME). On a production gn-auth that user normally doesn't exist, so a request that does get through fails with a 502 instead of returning a token. That depends on gn-auth's data, not on anything in genenetwork4.

The credentials used (gn-auth's dev client id/secret and dev user email/password, from flask init-dev-clients/gn_auth/wsgi.py) are publicly known and the same for every gn-auth checkout by default -- override GN_AUTH_DEV_CLIENT_ID, GN_AUTH_DEV_CLIENT_SECRET, GN_AUTH_DEV_USERNAME and GN_AUTH_DEV_PASSWORD if your gn-auth instance's dev fixtures differ. Needs a gn-auth reachable at AUTH_SERVER_URL with AUTHLIB_INSECURE_TRANSPORT set (gn-auth itself refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev setting, not something genenetwork4 needs to know about).

The token has no exp claim, and neither genenetwork4 nor gn-auth expires it by time or for lack of use. It stops verifying only when the key that signed it leaves gn-auth's published key set: gn-auth signs with a key for 7 days (JWKS_ROTATION_AGE_DAYS) and deletes it 14 days after its creation (JWKS_DELETION_AGE_DAYS), so a token lasts roughly 7 to 14 days, depending on where in the key's life it was issued. genenetwork4 caches the key set for an hour. When requests start failing with 401 invalid_token, fetch a new token. (exp is optional in genenetwork4's own validation, which is why a token without one like this lives as long as its signing key does -- see "Tokens are stateless" below.)

The test suite uses this same endpoint automatically -- see Tests below.

Configuration is layered: built-in defaults, then GN4_CONF and GN4_SECRETS files of KEY=VALUE lines, then environment variables (which always win). Keys: SQL_URI (MariaDB), SQL_CHECK_HEALTH (see below), GN3_URL (GeneNetwork3 base URL for proxy endpoints), GN3_TIMEOUT_SECS (upstream call timeout, default 30), GENENETWORK4_PORT, GENENETWORK4_BIND, and GENENETWORK4_TEST_TOKEN (false turns POST /get-test-token off; see "Behind nginx or in a container").

Tokens are stateless

Nothing about a bearer token is stored server side: there is no session table, no list of issued tokens, no jti denylist. Verification is signature + iss + exp/nbf, and that is all, so restarting genenetwork4 loses no tokens -- a token minted before a redeploy keeps working after it, as does one minted by a stub that has since been restarted (its key is fixed). What the process does keep is two caches, and restarting only empties them:

  • the signing keys (JWKS_TTL, an hour) -- how long a fetched copy of the auth server's key set is reused before being re-fetched;
  • authorisation answers (AUTHORISATION_TTL, a minute), per (subject, resource).

Neither is a lifetime. The token's own exp is the only thing that ends its life: it is validated on every request (validation.validate_exp in src/auth.rs, with jsonwebtoken's default 60-second leeway), it is never extended, and there is no refresh token. The hour matters in one case only: if the auth server is unreachable, a cached key set keeps verification working for up to JWKS_TTL, and past that every request is 401 invalid_token with a fetching .../auth/public-jwks detail -- the tokens are still valid, the key lookup is not. Run the auth server under the same supervision as genenetwork4 rather than from a shell.

Consequences worth knowing:

  • Restarting is not revocation. Since no state exists, restarting genenetwork4 or the auth server leaves every outstanding token valid. To withdraw one, wait for its exp (hence short lifetimes for tokens you may need to pull, e.g. scripts/gn-auth-stub.py --expires-in). To withdraw all of them at once, change the signing key or AUTH_SERVER_URL -- iss is compared against that string verbatim, so a changed host, scheme or trailing slash invalidates everything with InvalidIssuer.
  • exp is optional. required_spec_claims.clear() means a token without exp is accepted and never expires. gn-auth and scripts/gn-auth-stub.py always set one; a hand-minted token (the test key is in this repository) that omits it is a permanent credential.

Stub-minted tokens behave the same way (see Tests): exp is whatever --expires-in said at mint time, and restarting the stub does not invalidate them -- it always signs with the committed test key.

Request body size

The API accepts bodies up to 64 MiB (BODY_LIMIT_BYTES in src/lib.rs). axum's own default is 2 MiB, which a legitimate request exceeds: /plot/{rqtl,gemma} is posted a whole mapping response (a real GEMMA run on the BXD WGS panel is ~1.9 MB; a genome-sized one is tens of MB), and /compute/* carries one value per individual plus up to eight covariates. Over the limit, axum answers 413 Failed to buffer the request body: length limit exceeded before any handler runs.

Two consequences worth knowing:

  • a proxy in front must allow at least as much (client_max_body_size 64m in etc/nginx-vesuvio.pu.ac.ke.conf), or it refuses first with a 413 the API never sees;
  • a request does not need to be that large. For covariates, prefer {"case_attribute": "Sex"} -- the server reads the attribute and aligns it, so nothing is posted -- over sending a value for every individual in the panel. Only the samples the trait has are used, so posting values for the rest is pure payload.

Database connections

SQL_URI is used through one shared mysql::Pool. Two of the driver's per-checkout round trips are configurable:

  • SQL_CHECK_HEALTH (default true) pings a connection taken from the pool and replaces it when the ping fails. Leave it on unless the database is far away: turning it off saves a round trip, but then a connection the server closed while idle surfaces as 500 database_error on whichever request happens to draw it -- measured here as ten consecutive failures, one per dead pooled socket, until the process was restarted.
  • reset_connection is off unconditionally: a COM_RESET_CONNECTION per checkout, with nothing to reset (the queries are read-only).

Even with the ping on, a socket can die between the ping and the query; when a database error names a connection loss (Broken pipe, Connection reset, server has gone away, ...), the pool is dropped so the next request builds a fresh one (db::is_connection_loss / db::forget_pool, called from ApiError::from_db, the one place database errors pass through). tests/db_recovery.rs kills every pooled session and asserts the following requests still succeed.

Running with Guix

guix build -f guix.scm

builds the package (running both test suites, see below) and prints the resulting store path; the binary is at <store path>/bin/genenetwork4.

The xapian input is pinned to 1.4.29 (xapian-1.4 in guix.scm): the xapian-rs git revision Cargo.toml pins was written against Xapian 1.4, generating bindings for Xapian::InMemory::open, and Xapian 2.x removed it (xapian/dbfactory.h no longer declares the namespace; 2.x spells it as the Xapian::DB_BACKEND_INMEMORY flag). 2.x also needs C++17, where autocxx hands the parser a fixed -std=c++14 and 1.4's headers are happy with it. Guix's own xapian is 2.1.0 now, so without the pin the xapian-rs build script stops with "the 'generate' or 'generate_pod' directive for 'Xapian::InMemory::open' did not result in any code being generated" before compiling anything of this project's. The pin is a plain url-fetch/sha256 for xapian-core-1.4.29.tar.xz with 1.4's own make check-inmemory check-remoteprog check-multi check-glass check-chert test targets; drop it once xapian-rs moves to Xapian 2. The same pin is what guix shell -D and a manual cargo build see (see "Building with cargo, outside a bare toolchain" above).

guix shell -D -f guix.scm

drops you into a shell with all build dependencies available (cargo, rust, python, git), for running cargo build/cargo test/cargo run directly. This shell is not isolated from the network or from your normal environment, which cargo build/cargo run need to reach crates.io.

A fully isolated container (-C) normally needs -N to still reach the network from inside it, but on at least one guix build (1.5.0-6.64d4de2) combining -D with -N causes -D to silently stop providing the package's build dependencies (the shell then has only genenetwork4 itself, not cargo/rust/etc.). If guix shell -C -D -N -f guix.scm fails with "cargo: command not found", drop -C/-N and use the plain guix shell -D -f guix.scm above instead.

Tests

Two test suites exercise the same endpoints:

  • tests/*.rs - Rust integration tests that call the router in-process (no socket), via tower::ServiceExt::oneshot. Run with:

cargo test

Both suites check the envelope on every success response: the Rust helper tests/common::data(&body) and the Python helper server_test_case.envelope_data(body) verify its shape and return data. tests/envelope.rs (and test/python/test_envelope.py) follow every next link the endpoints suggest; tests/envelope_partial.rs checks the partial state without a database; tests/openapi.rs checks that every JSON success response is documented as an envelope.

  • test/python/test_*.py - Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with:

cargo build --release GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python

Either suite can fetch its own token from a live gn-auth automatically (see "Development" above) -- set AUTH_SERVER_URL before running either command above, and nothing further needs setting up by hand. A run without a reachable gn-auth still works, skipping (not failing) every test that needed a token it couldn't get -- set GN4_TEST_AUTH_TOKEN instead if you'd rather provide one manually.

Both suites also run automatically as part of guix build -f guix.scm (the Rust tests in the check phase, the Python tests in a python-rest-test phase after install).

Every test under tests/*.rs/test/python/test_*.py for an endpoint that touches the database (which is most of them -- hello_world, version, openapi and get_test_token's own disabled/wrong-config cases are the exceptions) checks for a live MariaDB on 127.0.0.1:3306 first (see SQL_URI above) and skips with a message rather than failing when none is reachable, since the Guix build sandbox has no network/database access. Likewise, any test for an endpoint that requires a token (every endpoint except those four) checks for one first and skips if none is available -- see "Development mode" above for how the test suite gets one automatically.

The Rust tests in tests/*.rs that hit the database issue their requests through tests/common::get, which times the request and prints a warning to stderr if it takes over a second -- e.g. WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s). Warnings only show with cargo test -- --nocapture (or on a failing test), since cargo captures stdout/stderr for passing tests otherwise.

Against a real database and index

Those skips are the sandbox's, not the code's. Supply the three things the tests look for -- a live MariaDB, a Xapian index, a token -- and the commands above run the whole suite rather than a small subset. In a development environment with all three, cargo test reports 225 tests passed / 0 failed (the handful of skips left are listed below) and the Python suite 94 tests, OK, no skips.

  1. Database: whatever SQL_URI points at, by default mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl. tests/common::db_available() really connects, so wrong credentials skip exactly like an absent server does.
  2. Index: XAPIAN_DB_PATH for a dev index, plus GN4_FULL_INDEX for a full/production one. Only a full index is treated as a count oracle: tests/search_native_db.rs then requires the engine's dataset:bxdpublish match set to equal the database's PublishXRef ids for BXD exactly, id for id -- the invariant a dev snapshot famously fails (it is missing ~1.7k traits and holds ~350 ids the database no longer has). A dev index only has to overlap, and its counts are then printed as reference data, never asserted. When both are set the full one wins.
  3. Token: either a real gn-auth at AUTH_SERVER_URL (see "Development"), or scripts/gn-auth-stub.py, which stands in for one: it serves the JWKS matching the committed test/data/test_rsa_private.pem (the same fixture pair search_native_db spins up in-process), answers /auth/token so POST /get-test-token works, and answers both authorisation endpoints with "no privileges" -- the stranger case the non-public endpoints' 403 tests assert. It is test scaffolding, not a gn-auth replacement: a committed test key, and no knowledge of users or groups. --grant-resource-id ID gives its token group:resource:view-resource on one resource id, for the owner-side tests, which want GN4_TEST_RESOURCE_ID=ID to match.

Left ungranted, a stub token is a public-data-only token, which is useful beyond the tests (handing a caller or an agent something to browse the API with). Only-public is not a property of the token: genenetwork4 decides from the data's own public/confidentiality/create_time columns, and asks this authorisation lookup about anything that is not public (src/auth.rs's check). No privileges therefore means public datasets read normally and every non-public one is a 403 -- the token's scope is never consulted for data access. Dataset metadata (/species/{species}/datasets, /groups/{group}/datasets) stays visible either way: those endpoints are documented as returning names and flags, not trait data, and are not gated on fine-grained authorisation.

--mint-only mints for one subject and exits without listening: how a token is issued for one more user without disturbing the running stub (see docs/www/operator-checklist.md, "Token issuance").

--expires-in sets the token's lifetime (30d, 24h, 60m, or plain seconds; default 24h). A month-long one is fine because the stub's key is fixed and never rotates: nothing invalidates the token but its own exp, and it stays verifiable for as long as anything serving test/data/test_jwks.json is up (genenetwork4 caches the key set for an hour, so a brief gap is survivable):

scripts/gn-auth-stub.py --port 8099 --expires-in 30d --token-file /tmp/public-only.token & AUTH_SERVER_URL=http://127.0.0.1:8099/ GENENETWORK4_PORT=3999 cargo run

Note the port: iss has to match the server's AUTH_SERVER_URL, so a token minted for :8099 is rejected by a server left on the default :8081, and vice versa.

scripts/gn-auth-stub.py --grant-resource-id 424242 --token-file /tmp/gn4-token.txt &
export GN4_TEST_AUTH_TOKEN=$(cat /tmp/gn4-token.txt)
export GN4_TEST_RESOURCE_ID=424242
export GN4_FULL_INDEX=/path/to/full-index XAPIAN_DB_PATH=/path/to/full-index

GUIX_PACKAGE_PATH=<path-to-guix-bioinformatics> \
guix shell -D -f guix.scm -- bash -c '
  export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib \
         DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib \
         DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include
  cargo test --no-fail-fast -- --nocapture'

cargo build --release
GENENETWORK4_BINARY=target/release/genenetwork4 \
  python3 -m unittest discover -v test/python

The stub listens on 127.0.0.1:8081, which is the default AUTH_SERVER_URL -- including its trailing slash, since that string is compared verbatim against the token's iss. --port/--issuer cover the case where AUTH_SERVER_URL says something else.

What still skips with all three in place, and why:

  • one config unit test asserts what an unset XAPIAN_DB_PATH means, so setting the variable at all skips it;
  • src/indexer/cache.rs's RDF-cache test needs an authoritative RDF cache directory, which this does not provide;
  • compute_gemma_ravanan_* needs ravanan, gemma-wrapper and ccwl on PATH;
  • tests/search_native_dev_corpus.rs records a dev corpus, so it needs GN4_DEV_CORPUS_DIR as well as an index.