genenetwork4
A Rust/Axum-based API server for GeneNetwork.
Endpoints
| Method | Path | Description |
|---|---|---|
| GET | / |
Greeting. |
| GET | /version |
Build info (hash, date). |
| GET | /species |
All species. |
| GET | /species/{species} |
One species by name or id. |
| GET | /species/{species}/datasets |
ProbeSet datasets in a species. |
| GET | /groups |
All groups (filterable). |
| GET | /mapping-methods |
Mapping tools offered. |
| GET | /accessions/{accession} |
Resolve a GN accession number. |
| GET | /groups/{group} |
One group, by name or Id. |
| GET | /groups/{group}/parents-and-f1s |
The family's parental design. |
| GET | /groups/{group}/strains |
Strains/samples in a group. |
| GET | /groups/{group}/case-attributes |
Case-attribute names and values. |
| GET | /groups/{group}/traits |
Published (phenotype) traits. |
| GET | /groups/{group}/traits/{trait_id} |
A published trait's metadata. |
| GET | /groups/{group}/traits/{trait_id}/values |
A published trait's values. |
| GET | /groups/{group}/datasets |
ProbeSet datasets in a group. |
| GET | /publications/{pubmed_id}/traits |
Traits a publication reports. |
| GET | /datasets/{dataset} |
One expression dataset. |
| GET | /datasets/{dataset}/probesets |
Expression traits in a dataset. |
| GET | /datasets/{dataset}/probesets/{probeset} |
An expression trait's metadata. |
| GET | /datasets/{dataset}/probesets/{probeset}/values |
An expression trait's values. |
| GET | /traits/{dataset} |
Every trait in a dataset. |
| GET | /traits/{dataset}/{trait} |
One trait's metadata. |
| GET | /traits/{dataset}/{trait}/values |
One trait's values. |
| POST | /compute/correlation |
Correlate a trait (stub; see docs/correlations.md). |
| POST | /compute/gemma |
Map a QTL with GEMMA (see below). |
| GET | /compute/gemma/status/{run_id} |
Poll a gemma run's progress. |
| POST | /compute/gemma/ravanan |
Map a QTL via ravanan (see below). |
| POST | /compute/rqtl |
Map a QTL with R/qtl (see below). |
| POST | /compute/bnw |
Learn a Bayesian network (BNW). |
| POST | /compute/bnw/features |
Score each module as a trait. |
| GET | /compute/bnw/examples[/{name}] |
BNW's worked examples, ready to post. |
| POST | /plot/rqtl |
The R/qtl Manhattan plot, as SVG. |
| POST | /plot/gemma |
The GEMMA Manhattan plot, as SVG. |
| POST | /plot/bnw |
The BNW network, as SVG. |
| GET | /compute/rqtl/status/{run_id} |
Poll an rqtl run's progress. |
| POST | /get-test-token |
Get a test bearer token (local). |
| GET | /tokens |
List short API tokens (local). |
| POST | /tokens |
Mint a short API token (local). |
| DELETE | /tokens/{id} |
Revoke a short API token (local). |
| GET | /tokens/who |
Describe the presented credential (local). |
| GET | /search |
Search traits (free-text). |
| GET | /openapi.json |
OpenAPI 3.1 document. |
Every endpoint above requires a valid gn-auth bearer token
(Authorization: Bearer <token>) except /, /version, /openapi.json
and /get-test-token itself -- see docs/gn-auth.md and the
"Development" section below (POST /get-test-token) for how to get
one for local testing.
Short API tokens. Beside gn-auth's JWT there is a second credential
format for agents and scripts: gn4ai_ + 64-char opaque tokens, stored as
SHA-256 hashes in one SQLite database (tokens.sqlite, GN4_API_TOKENS_DB,
chmod 0600, gitignored). Nothing is stored in the token, so revocation is
deleting a row and takes effect on the next request; a token in a log line
reveals nothing; and if the registry is lost every short token stops working.
Mint, list, revoke and introspect with scripts/gn4-token.py
create|list|revoke|who; the token is printed once and never again.
Administration lives at /tokens, /tokens/{id} and /tokens/who, which
answer only a direct caller on this machine (TCP peer address, plus a
refusal for anything claiming to be proxied) -- minting credentials is not
reachable from the network, and there is no switch to change that. A short
token reads public data only: it works everywhere a token suffices, and on
#[gnauth(secured)] endpoints for public rows; a confidential dataset answers
403 telling the caller to use a JWT, since gn-auth cannot validate an opaque
token. Send it as Authorization: Bearer gn4ai_... and never in a query
string, which would land in nginx's log and in next links.
What a request looks like in the log. One JSON line per request on stderr
(GN4_ACCESS_LOG), carrying the credentials, never the credential:
{"event":"request","ts":"2026-11-14T14:02:11Z","method":"POST",
"path":"/compute/gemma/ravanan","status":200,"took_ms":912,
"request_id":"…","user":"zachs","scheme":"opaque","token":"agent-ravanan-3"}
user is the gn-auth user id, so the line already answers who; scheme and
token say with which credential (absent for a JWT, which has no registry
handle). scripts/gn4-log.py reads such a log and shows it as a table with the
handle resolved against tokens.sqlite — the subject as the registry has it
now, plus the token's status today ([revoked], [expired], [unknown]) and
its note, which is how "an agent was working at 14:02" is told apart from "an
agent was working at 14:02 with a credential revoked since". The path is logged
without its query string, and no secret or hash is ever written.
A missing or invalid token is a 401; which further, resource-level authorization (if any) an endpoint checks once a token is valid is a separate, per-endpoint matter covered in each entry below.
Every success response is wrapped in one envelope (see
Success responses below), except / (a
plain-text greeting) and /openapi.json. The example responses below
are real ones, with long lists and strings abbreviated by ..., and
took_ms and request_id shortened to plausible values.
Details, including example responses, below.
Following the API without memorising it
Every success envelope carries next: the calls that lead on from what you
just read. It is deliberately exhaustive rather than minimal -- a genotype
version leads to its frequencies, its marker names and its scans; a tally
leads to the markers and to one marker's location and tally; a marker leads
to the PheWAS at it; a trait leads to its values, its genotypes and its
group; /version leads to the listings. A caller that knows only /version
can walk to anything.
A scan is the case that matters most: its reply points at the class counts
behind its own peak (.../markers/{peak}/classes?trait={trait_id}), because
"LOD 2.93" reads the same whether it rests on two animals or a hundred, and
/compute/qtl2 says in notes that a riself panel drops heterozygous calls
before the scan while GEMMA keeps them at dosage 0.5.
next is a promise that following a link is a GET that returns 200, which
is what makes blind following safe. So a link that would need a parameter the
server cannot fill is not offered (/groups/{g}/phewas without a locus is a
422, and is left out); a template such as .../markers/{marker}/location
carries a placeholder for the caller to complete; and calls that are not GETs
(the POSTs that run a scan) live in actions, never in next. A response
never suggests itself. tests/links.rs walks the graph from a dozen seeds and
fails if any link does not answer 200.
The convention that follows from this: a reply always offers a next step, and
an assistant relaying it should name one — a single sentence after any job,
taken from that reply's next/actions, unless the user asks for the result
alone. The server's part of that bargain is that the suggestion is always
reachable; the assistant's part is that it is one sentence and it is the
link the reply actually carries. See
docs/www/SKILL.md.
GET /- returnsHello, world!.GET /version- returns a JSON record describing the running build:
json
{
"data": {
"name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0,
"release": 1, "date": "20261107", "git": "9fa0303",
"dirty": false, "source_digest": "2f7f77ed4ace92c1", "branch": "main",
"database": "reachable"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": []
}
date and git are the HEAD commit's date and short hash, embedded at
build time (see build.rs); they describe the commit the binary was
built from, not the time it happened to be compiled. git carries a
-dirty suffix when that build had uncommitted changes -- so a binary
can never claim to be a commit its code does not match -- and
source_digest is a hash over the compiled source files, which tells
two builds of the same commit apart where a commit hash cannot.
branch is the branch HEAD was on, or HEAD when detached.
database is reachable or unreachable, checked at most every ten
seconds (db::health): a service that answers /version while every
data endpoint fails is exactly the state that went unnoticed for
twenty minutes on 2026-11-07, and this field makes it visible. build.rs
declares .git/HEAD, the ref it points at and .git/index as
rerun triggers, so a rebuild after a commit re-stamps rather than
reusing the old hash; committing without rebuilding does not move the
stamp, which is the point.
GET /species- returns a JSON array of all species in theSpeciestable, mirroring GeneNetwork3'sgn3.db.species.get_all_species:
json
{
"data": [
{"name": "human", "menu_name": "Human (Homo sapiens, hg19)",
"family": "Vertebrates"},
{"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)",
"family": "Vertebrates"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
]
}
Needs a live MariaDB connection, configured via the SQL_URI
environment variable (same config key GeneNetwork3 uses), e.g.
mysql://user:password@host/db_webqtl. Defaults to
mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl if unset.
GET /species/{species}- returns every column of the matching row of theSpeciestable, matched againstName,FullNameorSpeciesName(matching GeneNetwork2'swqflask.api.router.get_species_info, which returns the same match logic but only a 4-column subset). 404 if no species matches.
json
{
"data": {
"id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse",
"menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus",
"family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090,
"order_id": 15
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
{"href": "/groups?species=mouse", "description": "Groups of this species"}
]
}
Needs SQL_URI, same as /species above.
GET /species/{species}/datasets- returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of/groups/{group}/datasets. GeneNetwork2's own species-scoped variant ofwqflask.api.router.get_datasets_for_groupstill requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as/groups/{group}/datasets-- this returns dataset metadata (names,public/confidentialityflags), not trait data -- but a token is still required, like every endpoint. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412",
"name2": "B6D2ONCILM_Apr12",
"full_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"short_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"create_time": "2012-05-17", "public": 1, "confidentiality": 0,
"data_scale": "log2", "tissue": "Retina mRNA"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse", "description": "The species these datasets belong to"}
]
}
GET /groups- returns a JSON array of all groups in theInbredSettable, matching the shape of GeneNetwork2'sGET /api/v_pre1/groupsresponse:
json
{
"data": [
{"id": 25, "species_id": 4,
"display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)",
"name": "AD-cases-controls",
"full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)",
"public": 2, "mapping_method_id": null, "genetic_type": null,
"family": null, "species": "human", "mapping_method": null},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species", "description": "The species the groups belong to (filter groups with ?species=<name>)"}
]
}
family is the population category and species the species' short
name as /species lists it. mapping_method is the name of the
mapping tool offered for the group (the MappingMethod row its
mapping_method_id points to, see GET /mapping-methods), null for a
group with none. Three optional filters, combined with
AND and all case-insensitive: family (contains), species (exact
short name) and q (contained in the name, display name or full
name). For example /groups?family=Reference Populations lists the
reference populations, and /groups?species=rat&q=hxb the rat
HXB/BXH sets.
"Group" is GeneNetwork's own alias for InbredSet -- see
docs/genenetwork_datatypes.md for why. Needs SQL_URI, same as
/species above.
GET /groups/{group}- returns every column of the matching row of theInbredSettable, matched againstInbredSetName,Name,FullNameor numericId(matching GeneNetwork2'swqflask.api.router.get_group_info, which matches the same way but not by Id, plus gn-guile'sgn.data.population:population-info, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (seedocs/genenetwork_datatypes.md); only the MariaDB column names themselves stayInbredSet*. 404 if no group matches.
json
{
"data": {
"id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD",
"species_id": 1, "full_name": "BXD Family", "public": 2,
"mapping_method_id": "1", "genetic_type": "riset",
"family": "Reference Populations (replicate average, SE, N)",
"family_order": 1, "menu_order_id": 0.0, "group_code": "BXD",
"description": "<p>...</p>", "mapping_method": "qtlreaper"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/strains", "description": "The group's strains/samples"},
{"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"},
{"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"},
{"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"}
]
}
Needs SQL_URI, same as /species above.
GET /groups/{group}/strains- returns the strains/samples belonging to a group, matching gn-guile'sgn.data.strains:strain-id-names. 404 if no group matches. Takes an optionalused_for_mappingquery parameter to filter to only strains used for mapping (gn-guile'sstrain-id-namestakes the same flag as a keyword argument):
json
{
"data": [
{"id": 1, "name": "B6D2F1", "used_for_mapping": false},
{"id": 2, "name": "C57BL/6J", "used_for_mapping": false},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group these strains belong to"},
{"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}
]
}
GET /groups/BXD/strains?used_for_mapping=true excludes BXD's
parent strains (B6D2F1, C57BL/6J, DBA/2J), which aren't used
for mapping. Needs SQL_URI, same as /species above.
GET /groups/{group}/case-attributes- returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3'sgn3.api.case_attributessplits this into separate.../namesand.../valuesendpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns200with both lists empty, not a 404. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"names": ["Epoch", "RRID", ...],
"values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...]
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/strains", "description": "The group's strains/samples"}
]
}
-
GET /groups/{group}/phewas- a PheWAS around one position: every published trait of the group whose QTL peak (PublishXRef.Locus) is a marker withinwindowbase pairs of it, strongest first. Centre it on a marker (?locus=rs30524230, e.g. a peak from a mapping) or on a position (?chr=9&mb=29.940961); the window is in base pairs and centred, sowindow=10000is 10 kb up- and downstream (default 10000).?genotype_version=picks whose marker table defines the window (default: the group's current one). The answer carries the markers the window covered as well as the traits, so an emptytraitslist is readable as "nothing peaks that close" rather than a failure. Each hit hastrait_id,dataset,abbreviation,description, its peaklocus/chr/mb,lrs,lod(=lrs/ 4.604),mean,additive,nsamples,pubmed_idand the GeneNetwork traiturl. Example:?locus=rs30524230&window=10000returns 4 markers and 10 BXD traits, from LOD 99.9 down to 2.7. -
GET /groups/{group}/genotypes- the genotype files this group can be mapped with, newest first: eachversion(e.g.BXD.8), thefilename (BXD.8.geno), the panel'stitle,individuals,markers, whethergemmaandrqtlcan run on it, and which one iscurrent(what a mapping request gets by default). The order comes from the deployment'sgenotype/<group>.json, whosegenofilelist is newest-first, the same rule GeneNetwork2 uses; groups with no JSON (HET3-ITP) fall back to the highest version number and the listing'snotessays so.POST /compute/rqtl//compute/gemmatakegroupplus an optionalgenotype_versioninstead of paths.
GET /groups/{group}/genotypes/{version} is the metadata of one of
them: the file name, the header's own @-labels (type, mat,
pat, ...), the panel's individuals in file order and the strains
a mapping actually uses (parents/F1s dropped), num_individuals,
num_strains, markers, and per-chromosome marker counts.
GET /groups/{group}/genotypes/{version}/markers?page=&limit=&chr= is
the list of marker names -- one column, paged, filterable by
chromosome -- and a position is one call per marker:
.../markers/{marker}/location returns {"name", "chr", "cM", "Mb"} as
named fields, so a client iterating names never carries a position table
and a position lookup is a lookup. So a caller can see the panel, the
markers and the header it is mapping against; the genotype calls
themselves are never served, only summaries of them. See
docs/genotypes.md.
GET /groups/{group}/genotypes/{version}/frequencies is how the panel
is actually called, which nothing else in the API says:
json
{"data": {"B": 0.4889, "D": 0.492109, "H": 0.018991},
"notes": ["BXD BXD.4: 3811 marker(s), 354423 genotype cell(s)",
"352741 call(s) counted, 1682 missing (0.47 %); …"]}
The map is call -> frequency over the calls that are present, so it
sums to 1; missing codes (U, N, NA, an empty cell) are counted
separately and reported in notes, not as a call. A file whose cells
are probabilities is binned, 0.05 wide by default (twenty bins over
[0, 1], keys are the bin edges: "0.05-0.10"), because 0.4187 and
0.4191 are the same statement about a marker; ?bin=0.1 changes the
width. A file with only a handful of distinct numbers (0, 0.5, 1
for a biallelic dosage) is reported exactly, since binning three values
would hide the thing being looked at.
GET /groups/{group}/genotypes/{version}/markers/{marker}/frequencies
is the same tally for one locus (?marker= on the panel route does the
same thing), which is how a caller checks whether a flat LOD came from
silence or from a nearly-all-missing marker: both look identical in a
scan.
Counting is bounded: cells go through an interner, so a
100-million-cell panel costs one hash lookup per cell and one
allocation per distinct call, and a .geno over 256 MB is refused
(413 genotype_too_large) rather than read -- HSNIH-Palmer_true.geno
is 3.7 GB (134 927 markers x 6 147 individuals), which is not a
diagnosis anyone waits for.
GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait={id}
answers the next question -- what does the trait look like in each genotype
class at this marker -- without serving genotypes: the calls are joined to
the trait's values inside the server and reduced to n, mean, sd, min
and max per class, with that trait's authorization applied. It is how you
tell a peak that rests on two animals from one that rests on a hundred, and
it is what explains why GEMMA and R/qtl2 peak at different markers on DOL
trait 10003:
GET .../markers/JAX00418132/classes?trait=10003
{"A": {"n": 2, "mean": 1.8, "sd": 0.1414, "min": 1.7, "max": 1.9},
"H": {"n": 20, "mean": 2.525, "sd": 1.1652, "min": 1.7, "max": 5.3},
"B": {"n": 171, "mean": 3.9491, "sd": 1.1038, "min": 1.7, "max": 6.7}}
The A class is two animals and they are the trait's two lowest values;
R/qtl2, which drops H on a riself panel (R/qtl's convert2riself),
keeps 173 animals there but only 2 of one class, so the marker is nearly
monomorphic for it. GEMMA keeps the 20 hets at dosage 0.5, which is where
its sharper p-value comes from. Nothing in the payload names an animal.
GET /groups/unlisted/genotypes/{name}/frequencies is the same tally for
a file no group lists (a panel in preparation, a fixture, the DO's do/
cross), and .../markers/{marker}/frequencies for one locus. unlisted
is a namespace, not a group: /groups/unlisted has no record, only its
genotype subtree answers, and {name} is the version, .geno name, or a
path relative to the genotype files directory. One prefix for the whole
genotype surface beats a second spelling for the same tally. The markers
come the same way: /groups/unlisted/genotypes/{name}/markers (names)
and .../markers/{marker}/location (position).
-
GET /probesets/{name}- every dataset that carries a probeset with this name (indexed onProbeSet.Name): one row per dataset with the panel (group,species,platform) and the probeset's mapping statistics there (chr,mb,mean,lrs,lod,locus,additive). Suffixed variants match too --1427571_atalso finds1427571_at_A, the older chips -- and each row'snamesays which one it was, so the reply never leaves you guessing.?group=BXD,?species=mouseand?platform=M430(substring: there is no platform id in this schema) narrow it;?page=and?limit=page it, andnotescarries the count that matters.GET /probesets/1427571_at → 155 rows over 155 datasets (Shh on chr5 28.46 Mb), e.g. ["SA_M2_1104_R","BXD","mouse","Striatum_M430_V2_RMA_November04", "1427571_at","Shh","5",28.4579,6.039,16.318,3.544, "rs52243480",0.129]
This is the lookup a probeset id needed: it is not a Xapian field, so
/search cannot answer it and the only alternative was guessing a dataset
and calling /datasets/{dataset}/probesets/{name}.
-
GET /genes/{symbol}/traits- the same rows, filtered onProbeSet.Symbolinstead: every expression trait of a gene, across datasets, with a real total (unlike/search's capped list), grouped by dataset, and?sort=lod/?min_lod=/?group=/?species=/?platform=so "BXD datasets where Shh is a strong eQTL" is one call.GET /genes/Shh/traits?sort=lod→ 1 225 rows over 17 datasets and 13 probesets, strongest first (HXB_Liver_1208 / 1387584_at, LOD 18.46). -
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=…- the genotypes of the animals at one marker of a confirmed hit, which is the one place this API returns genotypes. It is fenced, because a genotype matrix is the thing GeneNetwork does not hand out: -
the server re-runs the R/qtl scan for the trait (the same machinery and gate as
/compute/rqtl) and takes the peak from that run; - the marker must lie within
?window=Mb of it (default 10) -- a marker elsewhere is422 not_the_peak, quoting the peak's marker, chromosome, Mb and LOD; - the peak must reach
?min_lod=(default 3.0), or the answer is422 not_a_hit; - only animals that have a value for the trait are listed, and only this marker's calls -- nothing else of the matrix leaves.
json
{"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856,
"peak": {"marker": "UNCHS026345", "chr": "9", "pos": 75.7627, "lod": 5.856},
"counts": {"A": 110, "B": 8, "H": 74},
"animals": [{"individual": "TB001", "genotype": "A", "value": 4.3}, …]}
Which is the data behind R/qtl's Fig 4.1, and
POST /plot/genotypes draws it: post that reply's animals (or its
genotypes map) with the trait's values from
/groups/{g}/traits/{id}/values, and get an SVG with one point per animal,
genotype class across, value up, class means marked:
json
{"marker": "UNCHS026345",
"genotypes": {"TB001": "A", "TB002": "A", …},
"values": [{"name": "TB001", "value": 4.3}, …]}
A class with a couple of animals in it is visible at a glance -- the same
warning .../markers/{marker}/classes puts in numbers.
-
GET /genes/{symbol}/aliases- Wikidata's aliases for a gene symbol, the list a trait page shows next to the symbol, plus thelabel(the entity's canonical symbol in that species) and alinksarray of external gene pages built for that symbol and species -- RGD, MGI, Alliance, NCBI Gene, Wikidata, UniProt, STRING, GeneMANIA and, per species, GTEx/Open Targets/ Protein Atlas/GWAS Catalog (human) or BioGPS/Allen Brain Atlas (mouse). Two SPARQL queries againstWIKIDATA_ENDPOINT(the entity, then itsskos:altLabels), kept in an LMDB file,WIKIDATA_CACHE_DIR/gene-aliases.lmdb, forWIKIDATA_TTL_SECONDS(seven days by default). The entity query matches the label case-insensitively, because human genes are labelledSHHand mouse onesShh, and picks mouse, rat or human deterministically --?species=humangets the ortholog whose labels GeneNetwork2's page shows:GET /genes/Shh/aliases {"data": {"symbol": "Shh", "wikidata_id": "Q14860079", "species": "mouse", "aliases": ["9530036O11Rik","Dsh","Hhg1","Hx","Hxl3","M100081", "ShhNC","sonic hedgehog"], "fetched_at": 1794315892, "stale": false}}
When Wikidata cannot be reached the cached answer is served with
"stale": true and the envelope marked partial -- the last value, not an
empty list: an empty alias list and an unreachable upstream mean different
things, and only one of them is about the gene. ?refresh=true forces a
refetch (an operator's switch). The cache is one file plus LMDB's lock
companion, written 0600.
The label is what makes the human-only services reachable from a rodent
symbol: ?species=human on a mouse Shh gives label: "SHH", an RGD link
for human, and a GTEx link that resolves -- a mouse symbol in GTEx is an empty
report. A cached entity written before the label was kept is refreshed rather
than served (stale tells you which happened).
Both serve metadata, not values. A token buys the index; the values
are per dataset and authorized per dataset, and the next link goes to
/datasets/{dataset}/probesets/{name}/values, which checks. There is no
endpoint that returns values across datasets, deliberately: that is bulk
work and /groups/{group}/traits is the bulk call.
GET|POST /baskets,GET|PATCH|DELETE /baskets/{name}- your own column of values, stored in your own SQLite file, and mapped by name. This is how data gets changed without changing the database: take a trait's values, edit one or more of them (or build a covariate column that does not exist), and map that instead. MariaDB is never written to.
sh
curl -s "$API/groups/BXD/traits/10001/values" > wbc.json
# post the column back as a basket (jq: .data is the values array)
curl -s -X POST "$API/baskets" -d '{"name":"wbc-edit","kind":"trait",
"base_dataset":"BXD","base_trait":10001,"values":[...]}'
curl -s -X PATCH "$API/baskets/wbc-edit" -d '{"values":[...]}' # revision 2
curl -s -X POST "$API/compute/qtl2" -d '{"group":"BXD",
"genotype_version":"BXD.4","trait_name":"basket:wbc-edit","basket":"wbc-edit"}'
Choices worth knowing:
* The full column is stored, not a diff: the caller has the original
beside it and the comparison is theirs, while a mapping needs one complete
column.
* Text with a parsed copy: raw is what was typed (5.2, NA), value
the number or null, status which it is -- so NA and 0 stay different
claims. A column containing neither a number nor a missing marker
(NA, x, -, ., ?) is refused with the offending samples named,
never silently dropped.
* An automatic version history: revision 1 is the column as first
stored and every PATCH adds one, with its own digest and Unix
timestamp — earlier revisions are kept, not overwritten, so
GET /baskets/{name}/revisions is the history,
?revision=N reads an old column, and a mapping can pin one:
{"basket": "wbc-edit@2"} maps revision 2 even after the basket has
moved on. Every reply records which revision and digest ran
(Basket 'wbc-edit' revision 2 (digest 9559a70e…)), so two runs on one
name can never look alike.
* One SQLite file per user, named for them —
BASKET_DIR/<user>.sqlite (0600, WAL), where <user> is the token's
subject: tokens rotate, subjects do not, so a new token finds the same
baskets and an operator can see whose file it is. A subject that is not
filename-safe is sanitised and disambiguated with a short digest.
Baskets are private: GET /baskets only ever lists yours, and nothing
here needs a database the API may write to.
* Every compute endpoint takes basket in place of its value field --
/compute/{rqtl,gemma,qtl2,fitqtl} -- and giving both is a 422 rather than
a silent choice. PATCH replaces the column, it does not merge: leaving a
value out drops it, deliberately (the dropped revision is still in the
history).
POST /matrices,GET|DELETE /matrices/{name}- upload new traits and covariates as one named table, checked against an existing genotype file. The shape is the one people have: a header row, then one row per individual, the first column naming them.
id,body_weight,glucose,batch
BXD1,23.100,9.400,2
BXD2,18.700,8.100,1
...
POST /matrices {"name":"my-traits","genotype_version":"BXD.4",
"contents":"id,bw,glucose\nBXD1,23.1,9.4\n...",
"kinds":{"batch":"covariate"}}
The upload is checked against genotype_version's panel before it is
stored: the same number of individuals, and every name one of them. 92 of 93,
or a name the panel does not have, is a 422 that says which -- a mapping that
silently drops individuals would be worse than an error. kinds marks
columns that are covariates; the rest are traits.
Each column becomes a basket named <matrix>.<column>, so a mapping names
it the way it names any basket, on any engine, and inherits the revision
history and digest:
POST /compute/qtl2 {"group":"BXD","genotype_version":"BXD.4",
"basket":"my-traits.bw"}
Everything lands in the uploader's own SQLite file -- MariaDB is never
written to -- and GET /matrices/{name}/values reads the table back
(null where a value was missing). A script writes an example table from a
genotype file's own header, so the fixture passes the check by
construction: scripts/example-matrix.py --genotype <file> --out
example-matrix.csv.
POST /uploads/genotypes- upload a genotype matrix that updates an existing population, named$USER-GENONAME. Tab-delimited: the header names the individuals from column 4 on, column 1 is the marker, 2 the chromosome, 3 the position in Mb.
marker chr pos BXD1 BXD2 ...
rs6269442 1 3.4921 B D
This is an update, not an ingest, and the checks say so: the base version
must exist, at most 50 individuals may be new, at least one must
overlap (none of the 93 individuals are in the base population: an upload
updates a population), and at most 200 000 markers -- beyond that the
sysadmin has to organize a file that size. Values must be the same kind
as the base's (single letters, whole numbers, or probabilities), because a
mapping engine reads the result the way it reads the population it updates; a
letters panel full of 0/1 is refused, naming both kinds.
Stored as text in the uploader's own SQLite file, and written out on demand
as a GN .geno (Chr, Locus, cM, Mb, individuals) carrying the base's
@-metadata: GET /uploads/genotypes/{name}/geno returns the path for the
genotype_file parameter of the compute endpoints, which now accepts values
by name (samples or a basket) as well as positionally:
POST /compute/qtl2 {"genotype_file":"<path>","samples":[{"name":"BXD1","value":61.4}, …]}
An upload of one chromosome is a poor thing to scan: a leave-one-chromosome-
out kinship has nothing to leave out, so the scan comes back flat or empty.
scripts/example-genotype-upload.py therefore spreads its markers across
chromosomes; use "kinship": "overall" if a file really has only one. GET /uploads/genotypes
lists yours, .../markers pages the marker table, DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update
(subset of markers, a few individuals dropped or added).
GET /groups/{group}/traits- returns every published (phenotype) trait belonging to a group, the listing equivalent of/groups/{group}/traits/{trait_id}(same fields, same join, notrait_idfilter). 404 if no group matches. If the group has any published traits, itsPublishFreezerow is checked the same way as/traits/{dataset}(see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's ownwqflask.api.router.fetch_traits, which defaults to unbounded too. NeedsSQL_URI, same as/speciesabove.
The optional abbreviation parameter keeps only the traits with that
short name, e.g. /groups/BXD/traits?abbreviation=5HT_MB_M finds
trait 17985 (an unknown abbreviation is an empty list).
json
{
"data": [
{"id": 10001, "phenotype_id": 4, "publication_id": 116,
"mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/groups/BXD", "description": "The group these traits belong to"}
]
}
GET /groups/{group}/traits/{trait_id}- returns one published (phenotype) trait'sPublishXRefmapping stats plus itsPhenotypedescription/units andPublicationreference, based on the data explored inprecompute-publishdata.gmi(e.g. its "More metadata" section, which identifies exactly this join as worth surfacing).trait_idisPublishXRef.Id, scoped by group for the same reason as/groups/{group}/traits/{trait_id}/values. 404 if no group matches, or if the group has no such trait; 403 if the trait'sPublishFreezeisn't public and the caller isn't authorized for it (see docs/gn-auth.md).
json
{
"data": {
"id": 10001, "phenotype_id": 4, "publication_id": 116,
"mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"},
{"href": "/groups/BXD", "description": "The group this trait belongs to"},
{"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"}
]
}
dataset is the trait's PublishFreeze, abbreviation its short
name (Phenotype.Post_publication_abbreviation, else the
pre-publication one) and url its page on GeneNetwork's website.
lod is lrs / 4.604.
Needs SQL_URI, same as /species above.
-
GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute}- one case attribute's values (Sex,Age,Tissue,Batch, ...) for exactly the samples a trait's values cover, in that trait's order, so it lines up with/valueswithout re-sorting -- the covariate source, kept off the values endpoint.valuescarries the text as recorded (F,Live),nullwhere a sample has none;num_valuescounts those that have one, anddistinctlists each value with its count, commonest first, which is what says whether it is usable as a covariate (a constant one is not). Attributes are per group --BXDdefinesEpoch, RRID, SeqCvge, Statusand noSex, whileBXD-Longevityand several other BXD-family studies do -- and an unknown one is a 404 whosenextis the group's attribute list. Example:/groups/BXD-Longevity/traits/10001/case-attributes/Sexreturns 3281 values inAgeAtDeath's sample order,F× 3101,M× 180. -
GET /groups/{group}/traits/{trait_id}/values- returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2'swqflask.api.router.trait_sample_data(the Publish branch) and gn-guile'sgn.data.dataset:get-bxd-publish-name-value-dict, generalized to any group rather than hardcoding BXD.trait_idis the trait'sPublishXRef.Id; it alone doesn't identify a trait uniquely across groups (only the pair of group andPublishXRef.Iddoes), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait'sPublishFreezeisn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a#[gnauth(secured)]endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits.
?normalize= / ?winsor_quantile= / ?winsor_fences= apply the
same pipeline /compute/* takes (see "Normalize the trait first" below),
over exactly the samples returned, and say what ran in notes. The
mapping suggestions in actions carry the same fields, so fetching
z-scores and then following the suggestion maps the numbers that were
shown rather than the raw ones:
sh
curl "$API/groups/BXD/traits/10010/values?normalize=zscore"
# notes: "Normalized: zscore ... over 28 values; mean 0.000000, sd 1.000000"
# actions[*].body.normalize == "zscore"
A pipeline that cannot apply (both winsorising rules at once, a log of a
non-positive value, nothing to divide by) is a 422
invalid_normalization.
json
{
"data": [
{"name": "BXD1", "value": 61.400001525878906},
{"name": "BXD11", "value": 53.099998474121094},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"}
]
}
Needs SQL_URI, same as /species above.
GET /groups/{group}/datasets- returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2'swqflask.api.router.get_datasets_for_group. 404 if no group matches. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412",
"name2": "B6D2ONCILM_Apr12",
"full_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"short_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"create_time": "2012-05-17", "public": 1, "confidentiality": 0,
"data_scale": "log2", "tissue": "Retina mRNA"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group these datasets belong to"}
]
}
tissue is what the dataset was measured in (the Tissue its
ProbeFreeze points to), also on GET /datasets/{dataset} and
GET /species/{species}/datasets.
GET /mapping-methods- the mapping tools GeneNetwork offers, theMappingMethodtable in id order:qtlreaper,Rqtl,HappyR,PLINKandFastMap. It includes tools no group currently uses (PLINK, FastMap); a group'smapping_method(seeGET /groups) names the one offered for it. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 1, "name": "qtlreaper"},
{"id": 2, "name": "Rqtl"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"}
]
}
GET /accessions/{accession}- resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without theGNprefix. It names the dataset (InfoFiles.InfoPageName), its page title, itscategory(publishfor a phenotype dataset,probesetfor an expression dataset, null if the name matches neither) and thegroupit belongs to. 400 if the text is not a number, 404 if no dataset has the accession. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"accession": 602, "name": "BXDPublish",
"title": "BXD Published Phenotypes", "category": "publish",
"group": "BXD"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group the dataset belongs to"},
{"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"}
]
}
GET /publications/{pubmed_id}/traits- returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as/groups/{group}/traits). 404 if no publication has that id; a publication with no traits is an empty list. Unlike/groups/{group}/traits, the datasets are authorization-checked as inGET /traits/{dataset}: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 17967, "phenotype_id": 16502, "publication_id": 13584,
"mean": 6038.655919692095, "locus": "rsm10000012868",
"lrs": 10.535408722907198, "lod": 2.2883164037591652,
"additive": -647.5889092087912, "nsamples": 34,
"pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...",
"post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...",
"original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824,
"title": "Evaluation of heritable determinants of blood and brain serotonin homeos...",
"authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...",
"journal": "Genes, brain, and behavior", "volume": null,
"pages": "247-260", "month": "Mar", "year": "2014",
"dataset": "BXDPublish", "abbreviation": "5HT_Blood_M",
"url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"}
]
}
GET /datasets/{dataset}- returns every column of the matching row of theProbeSetFreezetable, matched againstName,Name2,FullName,ShortNameor numericId(matching GeneNetwork2'swqflask.api.router.get_dataset_info's matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under/groups/{group}. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M",
"name2": "BXDMicroArray_ProbeSet_August03",
"full_name": "Brain U74Av2 08/03 MAS5",
"short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01",
"public": 0, "confidentiality": 0, "data_scale": "log2",
"tissue": "Brain mRNA"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"}
]
}
GET /datasets/{dataset}/probesets- returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of/datasets/{dataset}/probesets/{probeset}(same fields, same join, noprobesetfilter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset) rather than once per probeset -- see docs/gn-auth.md. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g",
"description": "CD3d antigen, gamma polypeptide", "chr": "9",
"mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619",
"lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269,
"additive": -0.28515625},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"}
]
}
GET /datasets/{dataset}/probesets/{probeset}- returns one expression (ProbeSet) trait'sProbeSetXRefmapping stats plus itsProbeSetgene annotation, matching GeneNetwork2'swqflask.api.router.get_trait_info's ProbeSet branch.ProbeSetXRefhas no single Id column of its own (its primary key isDataId; the natural key is the pair(ProbeSetFreezeId, ProbeSetId)), so a trait here is identified by dataset name plusProbeSet.Namerather than a single numeric id, unlike/groups/{group}/traits/{trait_id}. 404 if no dataset matches, or if the dataset has no such probeset.
json
{
"data": {
"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g",
"description": "CD3d antigen, gamma polypeptide", "chr": "9",
"mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619",
"lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269,
"additive": -0.28515625
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"},
{"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"}
]
}
Needs SQL_URI, same as /species above.
GET /datasets/{dataset}/probesets/{probeset}/values- returns one expression trait's value in each strain in the dataset, matching GeneNetwork2'swqflask.api.router.trait_sample_data's ProbeSet branch, addressed the same way as/datasets/{dataset}/probesets/{probeset}. 404 if no dataset matches, or if the dataset has no such probeset.
json
{
"data": [
{"name": "B6D2F1", "value": 5.742000102996826},
{"name": "BXD1", "value": 6.414000034332275},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"}
]
}
Needs SQL_URI, same as /species above.
/traits -- ProbeSet and Publish data through one URL shape
Everything above addresses ProbeSet (expression) and Publish
(phenotype) traits differently: /datasets/{dataset}/probesets/...
vs. /groups/{group}/traits/.... A trait is a trait either way --
a dataset, a trait within it, and that trait's per-strain values -- so
/traits/... addresses both the same way, (dataset, trait),
matching GeneNetwork2's own generic
wqflask.api.router.trait_sample_data/get_trait_info. This is
additional, not a replacement: the endpoints above are unchanged and
keep working exactly as documented.
dataset is resolved against ProbeSetFreeze first, then
PublishFreeze (db::resolve_trait_dataset) -- whichever matches
decides the category, so a bare trait name/id is never guessed across
categories, only the (much smaller, auditable) set of dataset names
is. A Publish dataset can be named either way GeneNetwork2 itself
accepts: the PublishFreeze's own Name/ShortName/Id (e.g.
"BXDPublish"), or its owning group's Name/InbredSetName/
FullName/Id (e.g. "BXD") -- a group normally has only one
PublishFreeze, so the two are near-interchangeable in practice.
GET /traits/{dataset}- every trait indataset, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset), same as/datasets/{dataset}/probesets. This is still more than/groups/{group}/traitschecks for the same phenotype data: both endpoints require a token now, but only this one also checksPublishFreeze.public/confidentialityand, for a non-public dataset, gn-auth --/groups/{group}/traits(unchanged) stops at the token.
json
{
"data": [
{"category": "probeset", "probeset_id": 1, "name": "100001_at",
"symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide",
"chr": "9", "mb": 44.970689, "mean": 5.48794285714286,
"locus": "rs13480619", "lrs": 12.590069931048,
"lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": []
}
or, for a Publish dataset:
json
{
"data": [
{"category": "publish", "id": 10001, "phenotype_id": 4,
"publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": []
}
GET /traits/{dataset}/{trait}- one trait's metadata, tagged by category (ProbeSetInfo's gene annotation orTraitInfo's publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other).traitisProbeSet.Namefor a ProbeSet dataset, orPublishXRef.Idfor a Publish dataset (a non-numerictraitagainst a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"category": "probeset", "probeset_id": 1, "name": "100001_at",
"symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide",
"chr": "9", "mb": 44.970689, "mean": 5.48794285714286,
"locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697,
"p_value": 0.269, "additive": -0.28515625
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"}
]
}
GET /traits/{dataset}/{trait}/values- that trait's value in each strain, as{name, value}pairs regardless of category (TraitValueandProbeSetValueare identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as/traits/{dataset}/{trait}.
json
{
"data": [
{"name": "BXD1", "value": 61.400001525878906},
{"name": "BXD11", "value": 53.099998474121094},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"}
]
}
- GET /search - free-text search across GN traits (see
docs/search-engines.md for switching between the gn3 proxy and
the native engine); hits keep gn3's field names and are wrapped in
the envelope like every other response. Parameters: q
(xapian syntax, e.g. serotonin AND midbrain AND group:BXD), type
(phenotype or gene), limit (default 20, at most 100), and
optionally sort (lrs or lod, which order identically) with
order (desc, the default, or asc), and page (default 1) for
the next limit hits.
Three fields are added to every hit: lod (lrs / 4.604) and, for
phenotype hits, locus (the peak marker's name) and abbreviation
(the trait's short name), both from the database (null for other
hits, or when the database is unreachable). pubmed_id is gn3's. With
sort, gn3 is asked for its first 1000 matches by relevance, those
are ranked, and page/limit are applied to the ranked list, so
page=2 continues where page 1 stopped; hits with no lrs come
last. Without sort, page is gn3's own. For example, the highest-LOD trait about serotonin in BXD
midbrain: /search?q=serotonin AND midbrain AND group:BXD&sort=lod.
Tissue and sex are not fields: they appear in each hit's description,
so match them there.
Needs a reachable gn3, configured via the GN3_URL config key
(default http://127.0.0.1:5003/api/).
json
{
"data": [
{"name": "17994", "dataset": "BXDPublish",
"dataset_fullname": "BXD Published Phenotypes", "group": "BXD",
"species": "mouse", "authors": ["Ye R", "Carneiro AM", ...],
"year": 2014,
"description": "Central nervous system, protein expression: Midbrain serotonin transport...",
"mean": 1627.7004466337316, "additive": 233.98946836805567,
"lrs": 16.49128337950074, "lod": 3.5819468678324804,
"locus": "rsm10000013922", "abbreviation": "SERT_MB_M",
"geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD",
"pubmed_id": 24405699,
"pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."}
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"},
{"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"}
]
}
-
POST /plot/rqtl,POST /plot/gemma- the drawing step after a mapping: post the mapping's own response back (the JSON/compute/rqtlor/compute/gemmareturned, as-is) and get the SVG the deployment's plotter draws from it (scripts/plot-rqtl.py,scripts/plot-gemma.py;src/plot.rsruns them). Nothing is recomputed -- a plot is one script run, and it can be redrawn from a result you kept./plot/rqtlalso accepts just thedatamember;/plot/gemmaneeds the whole response, because its plotter readsmeta.columnsto find thep_lrtcolumn. The reply isimage/svg+xml, not the JSON envelope -- an image is not JSON -- and the plotter's one-line peak summary comes back inx-plot-summary. R/qtl plots LOD with the permutation thresholds as dashed lines when the result carried them. GEMMA plots-log10(p_lrt)with no line unless one is asked for:?bonferroni=trueadds GEMMA's own Bonferroni (computed by the plotter as 0.05 over the analysed markers), and?thresholds=adds one computed elsewhere, each labelled. The title names the trait: taken from the posted response's ownTrait: ...note (which/compute/rqtland/compute/gemmaboth write first), or from?trait=when you post onlydata;?trait=wins over the note, and?title=replaces the whole title. Two traits over the same panel draw two different traces -- the plot is drawn from the rows you posted, and nothing is cached between calls. Configuration:PLOT_PYTHON(defaultpython3),RQTL_PLOTTER,GEMMA_PLOTTER. Both mapping responses suggest these calls inactions. -
GET /openapi.json- returns an OpenAPI 3.1 document describing the endpoints above; see OpenAPI below.
/compute/gemma
Maps a QTL by running the real gemma-wrapper/GEMMA binaries: a
K-matrix step, then an LMM association step, both --loco, over a
genotype file, a SNP annotation file and a list of trait values. See
src/gemma.rs for the invocation itself and why it's shaped this way
(found by running GeneNetwork2's and GeneNetwork3's own gemma code for
real against the BXD fixtures -- GeneNetwork3's always errors out;
GeneNetwork2's non-LOCO branch does too; only --loco works).
Individuals with no genotypes are left out. A panel can name individuals
whose genotype row is empty -- BXD-AE.4 has 136 of 344, BXD-Heart-Metals.4
200 of 481 -- and an empty row makes the kinship matrix singular, which GEMMA
reports as eigenvalues close to zero and then a LAPACK LUDecomp ... input
domain error, a message that says nothing about the cause. Those individuals
are dropped before the fit and counted in notes; they contribute nothing
either way, and the panel maps in seconds once they are gone.
Normalize the trait first. normalize changes the values' scale before
mapping -- none (default), log2, log10, sqrt, negate, zscore,
robust_zscore, or rank -- and winsor_quantile / winsor_fences
handle the tails. They run in that fixed order (transform, then
winsorise, then standardise), over exactly the individuals that enter the
fit, and the response says what ran:
"normalize": "rank", "winsor_fences": 1.5
rankis GeneNetwork2's "Quantile" (qnorm):Φ⁻¹((i−0.5)/n)by rank, ties sharing their average rank -- not array quantile normalisation. Acceptingqnormandinvert(fornegate) is deliberate: those are GN2's own names.zscoreuses the population standard deviation, as GN2 does (scipy.stats.mstats.zscore), so the numbers match its trait page;robust_zscoreis(x−median)/(1.4826·MAD)and is ours.winsor_quantile: 0.05clips each tail at the 5th/95th percentile;winsor_fences: 1.5clips atQ1 − 1.5·IQR/Q3 + 1.5·IQR, the rule GN2 uses to flag outliers -- and a count of those is reported whether or not anything is clipped. The two are mutually exclusive (422).log2/log10on a non-positive value,sqrton a negative one, or a standardisation with nothing to divide by is a422; missing values stay missing and are never imputed. GN2's silentsqrt"add 1 in [0,1]" quirk is deliberately not reproduced.beta,seandadditivecome out on the transformed scale -- azscoreadditive is in standard deviations -- so quote the note with the number.
Name what you are mapping. trait_name is optional -- GEMMA's values
are positional, so it needs no name to compute -- but it is what the
response reports (Trait: BXD::20010, first in notes), what the PheWAS
and R/qtl suggestions carry, and what /plot/gemma titles the picture
with. Without it they fall back to <group>::gemma, so two mappings over
one panel are distinguishable in the data but identically titled.
No permutations. GEMMA's permutation test is not run here and
nperm/permutations/permutate are refused (422
permutations_not_supported), not ignored -- an ignored flag is worse than
a refusal, because the caller believes it got thresholds. The reason is
cost: gemma-wrapper --permutate shuffles the phenotype file and repeats
the entire LOCO association scan per permutation, so a thousand is
thousands of full scans -- hours to days on a real panel, against the one
slot this endpoint has. R/qtl does have the test, on the same panel and the
same values: POST /compute/rqtl with "nperm": 100 returns the
suggestive and significant LOD thresholds, and every successful GEMMA
response already carries that call, pre-filled, in actions. Judge a GEMMA
peak against the smallest p_lrt it reached, never against a genome-wide
threshold of its own.
Covariates. covariates fits one or more covariates alongside the
trait -- the same request shape /compute/rqtl takes (below), each
either a case attribute of the group by name, which the server reads,
aligns onto the panel and codes, or explicit per-sample numbers:
"covariates": [
{"case_attribute": "Sex"},
{"case_attribute": "Diet", "map": {"HFD": 0, "LFD": 1}},
{"name": "age_days", "kind": "numerical",
"values": [{"name": "F3092", "value": 183.0}, …]}
]
GEMMA wants numbers, positional over the genotype panel, in a file of its
own (-c), so src/covariates.rs writes one: one row per individual,
no header, no sample names, and a categorical covariate dummy-expanded
into k‑1 columns named after their levels (Sex.M) -- GEMMA knows nothing
about factors, unlike R/qtl's wrapper, which does that expansion itself.
The first level is the reference. A level none of the fitted individuals
carries is left out (an all-zero column is singular and GEMMA stops with
LUDecomp rather than dropping the term the way R would), and the
response says so in notes.
Text with 2–20 distinct values is coded 0..k‑1 in sorted order; numeric
text is passed through as numerical; more than 20 levels, one level, or
an explicit map that misses a value is a 422 asking for a map. An
unknown attribute is a 422 listing the group's own. A missing value is
written NA, which GEMMA treats as it does a missing phenotype -- it
drops that individual (measured; GeneNetwork2 instead writes -9, which
GEMMA fits as a value). Both the coding, how many individuals have a
value, and any column left out are reported in notes.
Covariates reach only the association step, never the K-matrix step,
and gemma-wrapper hashes its file arguments, so a different covariate
set is a different cache entry while the kinship it was computed against
still is not recomputed. Covariates need a group: individuals are
aligned by name, which an explicit genotype_file/snps_file request
does not carry (422 covariate_unavailable).
Concurrency: large jobs, not all jobs. Gating exists to keep large
computations from piling up on this box, and for GEMMA that begins at
50 individuals (compute_gate::GEMMA_LARGE_PANEL) -- i.e. at the
smallest real panel, so in practice every mapping request is gated, and
only a handful of individuals (a four-way cross's parents, a probe of the
pipeline) runs free. Two reasons, neither about how long a small run takes:
every GEMMA run goes through one shared cache directory
(GEMMA_CACHE_DIR), where gemma-wrapper extracts per-chromosome kinship
archives of hundreds of megabytes by content hash, so two runs at once are
two writers over the same archives; and a real BXD WGS run is ~19 s of
every core it can get, while UM-HET3's 6 382 individuals write 207 MB of
kinship per chromosome.
So GEMMA runs one job at a time, one per caller: a second submission
from the same token while its run is running or queued is refused at once
with 409 compute_busy naming that run, and a submission from another
caller waits for the slot. R/qtl is gated much later (1 000 individuals;
see below) because a scanone over a few hundred individuals is seconds of
one core and its results are uncached scratch of its own.
Concurrency. A large scan -- 1 000+ individuals
(compute_gate::RQTL_LARGE_PANEL), where a scanone starts to be minutes
rather than seconds -- takes one of RQTL_MAX_CONCURRENT slots (default 8), one per
caller: a second large submission from the same token while its scan is
running or queued is refused at once with 409 compute_busy naming that
run, and a submission that finds every slot taken waits for one to come
free. Eight because a scan is a single-threaded Rscript and this box has
cores to spare; not more, because a scan on a 3 000-individual cross is
minutes of CPU and the same machine serves the database. The count is fixed
when the gate is built (at the first large request), so a change needs a
restart. Smaller crosses are not gated at all.
Permutations. nperm runs R/qtl's permutation test, and the response
carries permutation: {suggestive, significant, values}. A scan without
it suggests one with nperm: 100 in actions -- the thresholds a peak
needs to be judged against -- and next carries a PheWAS at the peak the
run found.
No permutation test, and requesting one is an error. GEMMA reports a
p-value per marker; gemma-wrapper --permutate n does exist, but it
repeats the whole LOCO scan per permutation, so it is hours of the one
slot this endpoint has. nperm, permutations and permutate are
therefore refused (422 permutations_not_supported) rather than ignored,
and a peak is judged against the smallest p_lrt it reached. The
response's actions carries the R/qtl permutation run over the same panel
and values, pre-filled, which is where genome-wide thresholds come from
here.
Give it a group, not a path. group (an InbredSet name) plus an
optional genotype_version makes the server resolve the genotype files
itself -- GET /groups/{group}/genotypes lists what is available and
which version is current, and src/genotypes.rs applies GeneNetwork2's
rule for that. trait_values may then be replaced by named samples,
which are matched onto the panel by name; parents and F1s are dropped
unless include_parents_f1 is set, as in GeneNetwork2/3. genotype_file
+ snps_file remain as an explicit override, for files outside that
layout.
If you name the files yourself, they are BIMBAM, in GeneNetwork2's
own layout:
genotype/bimbam/<file>_geno.txt (the genotype matrix) together with
<file>_snps.txt (marker/chromosome). The genotype/<file>.geno that
R/qtl takes is not a GEMMA input. For BXD's current panel that is
bimbam/BXD.8_geno.txt + bimbam/BXD.8_snps.txt -- 235 individuals ×
21 056 markers. trait_values must then have one entry per
individual, in the genotype file's column order, null where the
trait has no value; GEMMA matches by position, so a short or reordered
list is a wrong answer, not an error. Expanding a trait's values onto
the panel is the caller's job today (the API does not re-index yet);
see docs/genotypes.md for the conventions, which
version counts as current, and the measured runs -- 17.1 s for the BXD
WGS panel (21 055 marker rows) against the 19 s the 7 320-marker
fixtures take.
This runs synchronously inside the request -- there is no job queue --
and it runs sequentially, not per-chromosome in parallel. Not
feasible past roughly 100,000 markers at this point. The BXD
fixtures (7,320 markers, LOCO across 20 chromosomes) already take
about 19 seconds; some real GeneNetwork datasets have upwards of 1
million markers, which this endpoint cannot handle yet -- the request
would simply run for a very long time inside one HTTP call. Making
that feasible needs either parallelising the per-chromosome LOCO jobs
(gemma-wrapper --parallel, which gn-guile's own precompute script
already uses) or moving the computation out of the request/response
cycle entirely (a job queue, or an offline precompute step, the way
gn-guile's own scripts/precompute/run-gemma.scm already does it).
{
"data": [
[1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, -0.07665315, 0.2178407],
...
[1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, -0.21408755, 0.0003520103]
],
"meta": {
"state": "complete",
"took_ms": 18456,
"request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc",
"columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "additive", "p_lrt"],
"num_rows": 7320,
"num_cols": 12
},
"notes": [
"Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.",
"Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk",
"GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9"
],
"next": []
}
(data has one row per marker -- 7,320 rows in this real run against
the BXD fixtures, trimmed to two here. Each row is positional, in
meta.columns' order, rather than a {"chr": ..., "rs": ..., ...}
object per row -- cuts the repeated key names out of a response this
large.
The parental design is an endpoint, not a guess, and /groups/{group}
carries its headline. That reply's note reads "Parental design: C57BL/6J
(maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the
parents and F1s by default" — the two strain names answer "which cross is
this?" without a second request — and GET
/groups/{group}/parents-and-f1s reports a family's parents and F1s from the
deployment's parents_and_f1s.json (ported from GeneNetwork2's ParInfo, all
24 groups), plus the @type of its genotype file, the individuals each role
names when the metadata names them, and — the part a caller actually needs —
which members of the current panel the default rule drops, and which the F1
name heuristic catches. For an unlisted group it says source: "name
heuristic" and warns that its parents are unknown, so nothing is assumed. For
an RI set it also explains a subtlety: the metadata names the design's F1s
(B6D2F1) while the panel carries specific intercross F1 individuals
(C57BL/6JxBXD037F1), so the exclusions come from the name rule and
excluded_by_name lists precisely which.
Choosing individuals. Parents and F1s are excluded by default, as in
GeneNetwork2/3 (include_parents_f1: true turns that off). On top of that rule
any request that maps over a group's panel takes a selection:
{"group": "BXD", "include_parents_f1": true,
"individuals": {"include": ["C57BL/6JxBXD037F1"], "exclude": ["BXD48a"]}}
exclude wins over include, and an unknown name is a 422
invalid_individuals whose message lists the panel — a silently dropped animal
changes a kinship matrix without anyone noticing. The reason both exist: the
parent/F1 list is data (parents_and_f1s.json, seven groups) that a request
cannot reach, while F1 names differ per group (B6D2F1 for BXD, BHF1/HBF1
for BXH, C57BL/6JxBXD037F1 for intercross F1s).
Worth knowing before turning it on: including parents/F1s changes the kinship, not the fit. GEMMA computes kinship without reading phenotypes, so they shape the relatedness structure; the association still uses only the individuals that have a trait value. The reply says both numbers — "197 individuals … ; the kinship uses all 197, the association uses the 33 with a trait value" — and a request can therefore legitimately have 234 individuals in the genotype file and 34 in the fit.
additive is the effect size, sitting next to the score: −beta/2,
which is what GeneNetwork's own tables and CSV call the additive effect
(gn2/wqflask/marker_regression/gemma_mapping.py). Halved because half
the difference between the two homozygotes is the usual additive effect;
negated because GN2's BIMBAM conversion codes @mat as 1, so GEMMA's
beta is per maternal copy while GeneNetwork's published sign is
anchored on the other allele. beta and se are left where GEMMA put
them -- se is what makes the effect interpretable (beta ± 1.96·se is
a 95 % interval) -- and p_lrt stays the last column. These describe
this run on these samples and this genotype version; they are not the
additive that trait info reports, which came from the published
mapping (trait 20010: this run −0.54, published +2.118). The last two notes are the exact gemma-wrapper invocations
used (paths shortened here for readability) -- -gk only on the
kinship step, -lmm only on the GWA step, never both together; see
src/gemma.rs's compute_gwa doc comment for why that distinction
matters. lmm defaults to 9, matching GeneNetwork2's and gn-guile's
own convention; see the field's own doc comment in
src/routes/gemma.rs for what's actually been verified about it.)
One further note is GEMMA's own variance component for the null
model, which it writes only when asked (-pve, which this build passes
on the GWA step; accepted alongside -lmm 9, verified against GEMMA
0.98.6). Under --loco there is one per chromosome -- twenty different
kinship matrices -- so the note carries every value, not a summary:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).
The unrounded numbers are also appended to the run log as variance:
chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612 lines, so a
?id= run's /compute/gemma/status stream shows them while the run is
still in flight; and gemma-wrapper keeps each chromosome's own
.log.txt beside its archive (5eb53fa), named by the same hash. pve
is the proportion of variance the kinship explains -- the same quantity
as /compute/bulklmm's h2, but per chromosome instead of genome-wide --
and it is scale-free, so compare it with h2 rather than comparing vg
against σ²_e: a trait measured in thousandths has variances in
thousandths. When the estimate sits on a boundary GEMMA prints -nan
for se(pve); that is reported as "not available", because a boundary
estimate has no standard error, and a pve of exactly 0 or 1 is a
direction rather than a proportion.
/compute/gemma/status/{run_id} -- polling a run's progress
Given the endpoint above can genuinely take minutes, pass ?id= on
the POST and poll this on a second connection while it's in flight
to show progress, the same log-tailing shape GeneNetwork2/3 use for
R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the
run's id becomes a log file ({cache_dir}/{id}.log); gemma-wrapper's
own stderr (its --json-mode progress banner, chromosome list,
"Executing: ...", per-step info -- its stdout carries only the one
final JSON record, so there's nothing to stream there) is appended to
it line by line as both the kinship and GWA steps run; once the whole
request finishes, "GN4: computation complete" is appended as a
sentinel so a poller can tell "still running" from "done". run_id
must be 1-64 ASCII letters/digits/-/_ -- it's a path component, so
this is enforced on both this endpoint and the ?id= that names it.
id must also be unique: if a log already exists for it (another run
still in progress, or a finished one never polled/cleaned up), the
POST is refused with 409 rather than silently overwriting it.
POST /compute/gemma?id=my-run-1 (on one connection; blocks until done)
GET /compute/gemma/status/my-run-1?peek=0 (on another, while the first is in flight)
{
"data": {
"run_id": "my-run-1",
"lines": [
"gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024",
"Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022",
"Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..."
],
"pointer": 612
},
"meta": {"state": "complete", "took_ms": 1, "request_id": "..."},
"notes": [],
"next": []
}
Pass pointer back as ?peek= on the next poll; lines is empty
(not an error) when nothing new has been written since. An unknown
run_id (never started, or the main request hasn't written its log
yet) is a 404, not an empty response -- genuinely different from
"nothing new yet".
/compute/gemma/ravanan -- the same mapping, run per-chromosome via ravanan
Same request/response shape as /compute/gemma -- group (plus
genotype_version), an uploaded genotype (genotype_upload), or
genotype_file + snps_file; values as samples (named, aligned by
name) or trait_values (positional), or a basket -- resolved by the
same code as the direct runner (resolve_inputs), so the two cannot
drift. Two fields are refused rather than ignored, because this
workflow has no step for them: covariates (422
covariates_not_supported, use /compute/gemma) and normalize (422
normalization_not_supported). The same two steps (LOCO kinship, then
LOCO GWA), but run by ravanan
driving a CWL workflow compiled on every call (via ccwl compile, see
below) from the ccwl source at cwl/gemma/gemma-loco.scm, instead of
this server's own direct subprocess calls -- an alternative runner for
the same computation, kept side by side with /compute/gemma so the
two can be compared (correctness, timing).
Ravanan's cache is only as durable as RAVANAN_CACHE_DIR. Ravanan's reuse
key is (script, inputs), and both live in its --store under that directory,
so the cache survives a container restart if — and only if — the directory is on
a host path the container bind-mounts. Measured: 13.6 s cold, then 3.6 s for the
identical request after restarting the container, store unchanged. The default
is under /tmp, which is neither durable nor visible inside the container; the
reply carries a note when it sees that. Same reasoning for GN4_API_TOKENS_DB
(./tokens.sqlite by default) and GEMMA_CACHE_DIR. See
docs/ravanan-slurm.md.
K is shared across traits. The kinship step is given a constant-valued
phenotype file carrying only the trait's missing-data pattern, because
gemma -gk uses -p solely to learn which individuals are analysed; gwa
gets the real values. Two traits over the same animals therefore reuse all
twenty K matrices -- on the BXD fixture a second trait adds ~1 MB of store and
20 GWA jobs rather than ~13 MB and a second kinship scatter.
This runner calls gemma itself, not gemma-wrapper (the direct
runner still calls the wrapper, which is what makes the comparison
worth making). GEMMA 0.98.6 has -loco <chr> natively for both -gk
and -lmm, taking the chromosome to leave out by name and requiring
the BIMBAM annotation file (-a) that every genotype shape here
supplies -- so gemma-wrapper's LOCO orchestration only duplicates the
tool. Its parallel fan-out was never used on this path anyway: ravanan
scatters one gemma -gk|-lmm ... -loco <chr> job per chromosome, and
the kinship matrix flows from the kinship job to the GWA job as a CWL
File output rather than through a wrapper cache. The wrapper's
content-hash K cache is the one thing given up; a repeat request
recomputes, which is what the per-chromosome scatter costs anyway. This
workflow is written in
ccwl at cwl/gemma/gemma-loco.scm,
not raw CWL; compute_ravanan compiles it to CWL with ccwl compile
fresh on every call (cheap -- pure macro expansion, no Guix build
involved) into a generated, throwaway file under ravanan_cache_dir,
never checked into the repo. compute_ravanan computes the chromosome
list from snps_file itself (chromosomes_from) and passes it as the
workflow's chromosomes input, since CWL's scatter needs a concrete
array at invocation time -- and GEMMA's -loco takes exactly that
chromosome name. ravanan_cmd, ccwl_cmd, gemma_ravanan_workflow
(the ccwl source, not a CWL file), gemma_ravanan_manifest
(ravanan's own required --guix-manifest, a Guix manifest naming
gemma, at cwl/gemma/manifest.scm) and ravanan_cache_dir
(its own cache directory, separate from gemma_cache_dir, for pheno
files, run logs, the compiled workflow, and ravanan's own --store)
are configurable the same way as the rest of src/config.rs's layered
config. Its genotype and SNP inputs follow the same convention as
/compute/gemma -- the bimbam/<file>_geno.txt + <file>_snps.txt
pair, not the .geno file (see docs/genotypes.md) --
and the values must likewise line up one for one with that file's
individuals.
ravanan's batch system is configurable too: by default each CWL step is
a local child process, and RAVANAN_BATCH_SYSTEM=slurm-api submits
every step to a Slurm cluster through slurmrestd instead (endpoint, JWT
file, partition, nice and scratch directory via RAVANAN_SLURM_* /
RAVANAN_SCRATCH). With slurm, ravanan_cache_dir must be on a
filesystem the compute nodes can see, since ravanan's --store and
scratch live there. See docs/ravanan-slurm.md
for the keys, the fail-closed rules and the slurm-side auth
requirements, and cwl/README.md for how a request gets
from the API to a job on a node (and what the nodes themselves need).
GET /compute/gemma/status/{run_id} is shared between both endpoints
and checks whichever cache directory the run_id actually logged to.
Verified end to end on 2026-11-12 against the Slurm container from
gn-machines (specials/slurm-container.scm): a group: BXD request
became 40 Slurm jobs (20 kinship + 20 GWA, scattered over the SNP map's
chromosomes), all completed, 21 055 rows. See
docs/ravanan-slurm.md for what the host and the
nodes must provide -- including the two requirements that bit during
that run: ravanan must be a build with the --slurm-api-endpoint
fix, and the server's HOME must be writable by the job user inside
the cluster.
The reply carries the same variance-component note /compute/gemma does
(-pve, every chromosome). Since this workflow runs gemma itself,
each job's .log.txt is a first-class CWL output and the note is read
straight out of those files -- no archive to unpack and no wrapper-named
copy to depend on. See docs/ravanan-slurm.md.
Concurrency. Like /compute/gemma -- same 50-individual threshold,
since ravanan writes into the same cache directory -- only a large request
claims a slot. One ravanan workflow at a time, one
per caller: a second large submission while one runs (or is queued) is
refused at once with 409 compute_busy, naming the run and linking
/compute/gemma/status/{id} (which ravanan shares with /compute/gemma);
a submission from another caller waits for the slot. One slot because
ravanan already fans its own
per-chromosome steps out -- over the cluster, when the slurm backend is
selected -- so a second request would only queue behind the same scratch
directory. Its gate is separate from /compute/gemma's: they are
different runners of the same computation with their own cache
directories.
Exploratory, and genuinely rough around the edges -- three real bugs
turned up building this, none of them in /compute/gemma's own
direct-subprocess path:
- ravanan resolves
gemma-wrapperby building a Guix profile from--guix-manifestagainst the ambient Guix process (GUIX_PACKAGE_PATH, no--guix-channels) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package inguix-bioinformaticstoo new for the pinned base Guix. This needsGUIX_PACKAGE_PATHset in whatever environment starts the genenetwork4 server itself (inherited by theravananchild process genenetwork4 spawns); it is not somethingcompute_ravanansets. - ravanan's
--storeneedsfiles/,logs/anddata/subdirectories to already exist; it does not create them itself (compute_ravanandoes now). - A step scattered over exactly one input (
kinship'schr) needs an explicitscatterMethodeven though CWL itself only requires one for multiple scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it. - Every chromosome's
gwa.cwlrun shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in--chromosomes), so the 20 scatteredarchiveoutputs all collide under ravanan's optional--output-directorysymlink-everything-into-one-flat-dir step;compute_ravanansimply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own--storevia its stdout JSON record, never through--output-directory.
A real end-to-end run against the BXD fixtures (same rows as the
/compute/gemma example above, confirmed byte-for-byte including
rs31443144's exact values) took 16.7s wall-clock scattered across
20 chromosomes this way, vs. /compute/gemma's own ~19s and the
earlier sequential (non-scattered) ravanan workflow's ~21s -- a real,
if modest, win from letting ravanan's own scheduler fan the
chromosomes out instead of going through gemma-wrapper's internal GNU
parallel call, on top of no longer needing that path's own
PARALLEL_SHELL workaround at all (each scattered job's
--no-parallel gemma-wrapper call never invokes GNU parallel in the
first place, so the bash-time-keyword bug it worked around does not
arise here).
Those 16.7s are with ravanan's default single-machine batch system,
where a step is just a local child process. With
RAVANAN_BATCH_SYSTEM=slurm-api every step additionally pays Slurm
dispatch -- submit, scheduler tick, launch, poll -- which on a small
input dwarfs the mapping itself: three chromosomes, measured on the test
container (gn-machines/specials/slurm-container.scm, one node, two
CPUs), are 2.3s cold / 0.7s warm on /compute/gemma,
4.7s / 3.5s on ravanan single-machine, and 33s on
ravanan+Slurm with a 5s scheduler interval (158s with Slurm's
defaults), while each job's own gemma-wrapper run took under a second.
Same rows in every case. See docs/ravanan-slurm.md
for the breakdown and what to tune.
/compute/qtl2
A genome scan with R/qtl2: genotype probabilities from a hidden Markov
model, and a linear mixed model with leave-one-chromosome-out kinship. It is
the middle ground between /compute/rqtl (Haley-Knott, no kinship) and
/compute/gemma (an LMM, but biallelic BIMBAM with mean-imputed genotypes),
and it is how multiparent populations are analysed.
{
"group": "BXD", "genotype_version": "BXD.4",
"trait_name": "BXD::10001",
"samples": [{"name": "BXD1", "value": 61.4}, …],
"kinship": "loco", // loco (default) | overall | none
"model": "normal", // normal (default) | binary
"cores": 4
}
Rows are marker, chr, pos (Mb), lod — the same shape /compute/rqtl
returns, so /plot/rqtl draws it unchanged.
On DOL trait 10003 the three engines tell the same story with different emphasis, which is what modeling relatedness buys: R/qtl's HK scan peaks on chr9 at LOD 5.86 and does not see chr16 at all; GEMMA's LMM prefers chr16 at −log10 p 7.15; R/qtl2's LMM finds chr9 (LOD 4.85), chr19 (4.76) and chr16 (4.24) on one footing.
Autosomes only. The X needs a sex covariate to be fitted properly, and
the Y and mitochondrion degenerate the per-chromosome kinship — GEMMA's LOCO
run is given the same restriction, for the same reason. The notes say how
many markers were left out.
Multiparent crosses. A multiparent population cannot come from a
.geno: that file carries two codes per marker (B/D, A/H/B), which
is a collapse, not a founder allele. It has to ship its own qtl2 control file
with founder genotypes, so the endpoint also takes a cross by name:
{"rqtl2_cross": "do", "trait_name": "DO::WBC",
"samples": [{"name": "JAXW202.2129", "value": 4.34}, …]}
The name is looked up as genotype/rqtl2/{name}.json, then
{name}/{name}.json, then {name}.json under GENOTYPE_FILES; do/ is the
one such cross this deployment has (742 individuals, 8 founders A–H, 6 413
markers, WBC and NEUT). The samples are matched to the cross by name and
anything outside its population is left out (and counted in notes).
A multiparent scan is a different claim from a biallelic one: every marker
is fitted with one coefficient per founder, so the LOD asks whether any
founder differs, not which. Reading a multiparent LOD as a strain effect is the
usual mistake; per-founder effect sizes need scan1coef, which this endpoint
does not return yet.
Measured on the DO (WBC, 8 cores): genotype probabilities 174 s and 1.39 GB,
cached per cross and keyed by the control file and its directory; LOCO
kinship 15 s, cached; the scan itself 21 s. scan1 is run on one core for a
multiparent cross deliberately — its forked path stalls here on allele
probabilities (25+ minutes with 8 cores and no CPU consumed, against 21 s
serial). The autosomes-only rule applies here too: the DO's X is left out (369
markers) rather than fitted without a sex covariate.
Panel conversion. The cross is generated from this deployment's own
.geno file and cached (QTL2_CACHE_DIR), because the shipped
genotype/rqtl2/ control files are not usable: they name paths relative to a
directory the data is not in, declare no genotype-code map (every genotype
reads as missing) and no orientation (individuals read as markers) — four
traps that each produce a plausible flat scan rather than an error. The
conversion also drops heterozygous calls on a riself panel, as R/qtl's
convert2riself does. Configuration: QTL2_SCRIPT_PATH,
QTL2_CACHE_DIR.
Gene link-outs
A gene symbol is useful beyond our own pages, and SKILL.md carries the table of
external services that still resolve (checked 2026-11-11): RGD for both mouse
and rat, MGI, the Alliance of Genome Resources, NCBI Gene, Wikidata (the QID
that /genes/{symbol}/aliases already returns), UniProt, Ensembl, GTEx,
STRING, GeneMANIA, Open Targets, the Human Protein Atlas, the GWAS Catalog,
BioGPS, the Allen Brain Atlas and GeneCup — with the retired ones named so they
are not offered by mistake (HomoloGene, UniGene, Genotation, GeneBridge,
PhenoGen).
Two conventions that keep the links honest: quote the species the symbol came from, and prefer RGD/GeneMANIA(rat) for a rat symbol and GTEx/Open Targets/ Protein Atlas for a human one — a mouse symbol in GTEx resolves to an empty report, because GTEx is human-only.
Why R/qtl and R/qtl2 disagree on a riset panel
They read the same file and can end up fitting different predictors, so a peak in one is not a bug in the other. The mechanism, with BXD trait 16627 as the worked case (BXD.8, 35 phenotyped animals, all measured on production):
| marker | Mb | R/qtl | R/qtl2 riself |
R/qtl2 f2 |
|---|---|---|---|---|
rsm10000000889 |
5.652 | 9.59 | 3.79 | 9.98 |
rs33299937 |
5.653 | 9.59 | 3.73 | 9.68 |
rs13476332 |
6.311 | 9.59 | 3.70 | — |
What R/qtl does. scripts/rqtl_wrapper.R reads the cross with
genotypes = c(@mat, @het, @pat) — codes 1, 2, 3 — and then, for a riset
panel, calls convert2riself, which is adaptive:
gtab <- table(pull.geno(cross))
usethree <- !(gtab["3"] < gtab["2"]) # is code 3 at least as common as code 2?
if (usethree) { dat[!is.na(dat) & dat != 1 & dat != 3] <- NA; dat[dat == 3] <- 2 }
else { dat[!is.na(dat) & dat != 1 & dat != 2] <- NA }
It keeps the two most numerous codes and drops the third (warning
Omitting N genotypes with code==2), then recodes to 1/2. No HMM, no
probabilities: scanone fits each marker's own calls.
What R/qtl2 does. riself's state space is AA, BB (two states, no het),
so a heterozygous call is not representable and becomes missing; the HMM then
re-imputes it from flanking markers, and scan1 fits probabilities, not
calls. On BXD.8 the conversion drops 610 684 calls, 12.7 % of the matrix
(the reply says so). At a marker with 14 % H, that is a different predictor
from R/qtl's, which is why the peak survives in one engine and thins out in the
other — not a failure of either.
genotype_model: "f2" keeps the calls as a third genotype (AA, AB, BB) and
recovers R/qtl's number (9.68 against 9.59). It is a diagnostic: it fits
het as a real genotype and assumes an outbred intercross design, which an
inbred line is not. Its value is that it separates "the conversion cost the
information" from "the model cannot see the locus".
What the proper model is, and where it stands. qtl2 has general RIL types —
genril2 / genail2, from qtl2 0.18: “handle any number of founders; include
the number of founders in the cross type… genril assumes the progeny are
inbred lines (RIL), while genail assumes the progeny have two random
chromosomes (AIL)”, with cross_info of length 1 + n_founders: the first
column the number of generations, the rest founder frequencies. Both are
supported by the qtl2 0.42 this deployment runs (checked through
qtl2:::check_crosstype()), and genril2 is the model that fits a riset
panel's residual heterozygosity honestly.
It is not wired yet, and the reason is concrete rather than a matter of
effort: a founder-based cross type expects individual genotypes coded as
alleles/pairs of founders (as the DO's A–H are), not the 1/2/3 of a
two-parent cross; a prototype with numeric codes had every genotype read as
missing. The remaining work is therefore exact: build founder_geno from the
two parental alleles (for a two-parent cross every marker is homozygous
founder 1 vs founder 2 by definition of the coding), write cross_info with
the generation count, and code H as the founder pair rather than code 2.
Then compare trait 16627 against 9.59/9.68 — and check DOL through the same
path, because its H is not residual heterozygosity at all: it is a collapsed
DO, where H means "two different founders" and the honest model is
crosstype: "do" with founder genotypes the collapse destroyed.
What to do when the engines disagree, in order: .../markers/{marker}/classes?trait=
to see the class sizes and means; .../frequencies for the H share at the
marker; genotype_model: "f2" to see what keeping the calls would give; and
quote which model ran, because a reply says Cross: riself or Cross: f2.
/compute/bulklmm
A genome scan that can be asked for thresholds. GEMMA's LOCO scan is the
reference for a biallelic panel, but a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm; the usual stand-in is
R/qtl's nperm, whose null shuffles phenotypes as if the individuals were
unrelated. BulkLMM.jl fits one LMM
per marker with a global kinship matrix and permutes by rotating the data
through the kinship's eigen decomposition — relatedness included — so a
threshold costs seconds:
{"group": "BXD", "genotype_version": "BXD.8", "trait_name": "BXD::16627",
"samples": [{"name": "BXD1", "value": 1.5}, …], "nperm": 200, "seed": 42}
BXD trait 16627, 35 phenotyped animals, 21 056 markers, 22 s:
peak rsm10000000888 chr2 4.92 Mb LOD 9.59 beta 0.0027 h2 3.0e-16
permutation thresholds (200 rotations, seed 42): alpha 0.10 → 3.51, 0.05 → 3.81, 0.01 → 4.47
which is the trait's published locus (PublishXRef.Locus rsm10000000888, LOD
9.59) and a genome-wide-significant peak against a threshold this API could not
compute before. Rows are marker, chr, pos (Mb), lod, beta, h2.
Thresholds on another engine's plot. /plot/gemma draws no reference
line by default — a line on a plot is an annotation, not part of the result,
and a reader should be told where each one came from. GEMMA's Bonferroni line is
one more opt-in labelling (?bonferroni=true); a threshold computed here
travels with the plot, labelled, because the two are different nulls:
POST /plot/gemma?thresholds=3.50,3.65&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
{ the /compute/gemma reply }
Each line is drawn dashed in the series colour with its label and value
(bulklmm alpha 0.05: 3.65), and the Bonferroni line only appears when it is
asked for — so a plot shows exactly the lines someone chose, each named.
/compute/bulklmm's reply already offers that href with its α = 0.05 threshold
filled in, so the picture is one link away.
/plot/bulklmm draws two reference lines for it: the genome-wide α = 0.05 and
the stricter α = 0.01. The reply also reports α = 0.10, but that is the loosest
bar in the set — drawing it invites reading it as the line to clear — so it
appears only on request (?all_thresholds=true, or an explicit
?thresholds=3.50&threshold_labels=alpha 0.10). GEMMA's Bonferroni line is
opt-in in the same spirit: a reference line is an annotation, not a result.
About cores. BulkLMM's single-trait scan() — what this endpoint calls —
is serial in Julia by design: the package threads bulkscan() (many traits)
and its helpers, not scan() (checked in its source: Threads.@threads appears
in bulkscan.jl and bulkscan_helpers.jl, not in scan.jl). What was
parallel before was BLAS, not the scan. The driver therefore splits the markers
into independent blocks and scans them on Julia threads (--blocks, default =
BULKLMM_THREADS), which is exact — the LODs and permutation thresholds are
identical to a serial run, verified bit-for-bit at 1 and 4 blocks — and caches
the parsed matrix and the kinship per panel:
cold, 1 block : read 1.3 s, scan 9.0 s, total 20.9 s
cold, 4 blocks: read 0.4 s, scan 7.5 s, total 18.3 s (same LOD 9.589938)
The gain is modest at this size because the LMM loop is only about 8 s of the
run and each block re-estimates the variance components; the rest is Julia
startup. Where the cores pay is bulkscan() — a hundred traits in one
process, threaded over trait blocks. That is the shape an eQTL endpoint wants,
and it is the next thing worth adding here.
Two differences from GEMMA are stated in every reply rather than implied:
- a global kinship matrix, not leave-one-chromosome-out, so its p-values are not GEMMA's;
h2_nullsays how much relatedness it found. Here it is ~0, so the fit is effectively a fixed-effects model — which is exactly why it agrees with R/qtl on this trait where GEMMA's LOCO scan differs.
See docs/bulklmm.md for the model comparison, the measured
core behaviour and the packaging options. The engine is a Julia subprocess
(BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT,
BULKLMM_SCRIPT, BULKLMM_THREADS, BULKLMM_CACHE_DIR) with the driver in
scripts/bulklmm_scan.jl. The Julia depot is prepared once, deliberately, and
is small: 79 packages, 139 MB, no GPU/CUDA stack — BulkLMM's closure is CSV,
DataFrames, Distributions, Optim and what those need. Every request runs with
JULIA_PKG_OFFLINE=true, so a scan can never download anything.
/compute/bnw
A Bayesian network over a set of variables, learned by BNW (the Bayesian
Network Webserver): which traits belong together under a causal ordering. The
computation runs in BNW's own headless driver — the bnw program in the
gn-bnw repository — so this endpoint only prepares a matrix, calls out, and
parses the reply (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR).
{
"group": "BXD", "traits": [10001, 10003, 10038],
"maxparents": 4, "k": 1, "tiers": 3, "threshold": 0.5
}
or, for a matrix the database does not hold (a basket's columns, a handful of probe sets):
{
"values": [
{"name": "Trait1", "samples": [{"name": "BXD1", "value": 1.2}, …]},
{"name": "Trait2", "samples": [{"name": "BXD1", "value": 0.4}, …]}
]
}
Rows are positional with meta.columns:
from, to, posterior, in_structure
One row per candidate edge (variables²−variables of them: self-edges are
dropped), strongest posterior first. posterior is the share of the k-best
networks that contain the edge; in_structure says whether it survived
threshold. The four parameters default to BNW's own (maxparents 4, k 1,
tiers 3, threshold 0.5 — from create_tiers_gom_part1.php), and the reply
names the values used, because a network at threshold 0.5 and one at 0.9 are
different answers.
A posterior is not a p-value. It says how often the search put that arrow there in this sample — a statement about correlation under a causal ordering, not about surprise, and not an effect size. At most 40 variables (the search is exponential in in-degree) and at least 3 samples complete across every variable: BNW models a complete multivariate normal, so a sample missing any value is dropped and the reply says how many survived.
POST /plot/bnw draws it from the posted reply, the same way /plot/gemma
does — an SVG back, nothing recomputed, the plotter's one-line summary in
x-plot-summary. The default figure is the network: nodes on a circle, one
arrow per edge that survived the threshold, arrow width and label = the edge
posterior, and the below-threshold edges as faint dotted lines so a reader sees
what was considered as well as what survived. ?kind=posterior draws every
candidate edge as a cell in a matrix instead, which is the view that says how
close a rejected edge came to the threshold. Pure-Python SVG (BNW_PLOTTER,
run by PLOT_PYTHON), so no graphviz is needed in the deployment.
From a network to a mappable trait: POST /compute/bnw/features. Every
node with at least one parent at the threshold is a module — the node and its
parents — scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module, module, members, size,
variance_explained (the share of the module's variance the component explains,
which says whether the score is a fair summary). With ?module=NAME it is that
module's per-sample scores in the {name, value} shape every mapping endpoint
already takes — post them as samples to /compute/rqtl, /compute/gemma or
/compute/qtl2 and map the module like any trait. That is the chain BNW exists
for in GeneNetwork: structure → module → QTL. A module is a correlation
structure, not a QTL: what maps is where its members' shared variation comes
from, not which member causes which.
BNW's own worked examples: GET /compute/bnw/examples and
GET /compute/bnw/examples/{name}. The five datasets its tutorial and examples
pages present (synthetic 2-genotype/6-trait, the scientific 5-node network at
100 and 500 samples, BXD spleen immune genes, the time series) are listed with
their sizes, and each one is served as a matrix in the values shape
/compute/bnw takes — with a ready-to-post action carrying the body. ?columns=
restricts the matrix, which the spleen example needs (14 variables, and a caller
may want fewer). genotype_column warns that a column is a genotype code: this
driver is continuous-only, so such a column enters the network as a number, as
the web application's GeneNetwork path treats it. The files are read from
BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/example_datasets); a
deployment without it answers 404 no_examples rather than pretending.
The driver's contract is documented in
gn-bnw/info_files/gn4-integration-plan.md;
tests/bnw.rs checks the endpoint against the answer the BNW web
application gives for its own six-trait example, so "it works" means the same
edges, not merely a 200.
/compute/fitqtl
/compute/rqtl says where a QTL is (a LOD per marker); this says how
much, by fitting R/qtl's fitqtl at the markers you name -- normally the
peaks a scan already reported, via GET
/groups/{group}/genotypes/{version}/markers. It is separate because it is a
separate question, asked about a few positions rather than a genome: fitqtl
fits one model per marker, so a genome-wide version would be thousands of
fits (the cap is 100 markers per request).
{
"group": "BXD", "genotype_version": "BXD.4",
"trait_name": "BXD::10001",
"samples": [{"name": "BXD1", "value": 61.4}, …],
"markers": ["mCV24409492", "rs8253516"]
}
Rows are positional with meta.columns:
marker, chr, pos, lod, pct_var, additive, se, t, p
additive is fitqtl's estimate for the QTL term. R/qtl codes the QTL
genotype ±1, so this is half the difference between the two homozygote
classes -- the same convention GeneNetwork's own additive uses, and on
BXD trait 10001 it reproduces the published value to six decimals
(2.394472 here against GN2's 2.394444, from the same peak region).
pct_var is the model's percentage of variance explained and p its
p-value; se and t belong to the effect. Markers not in the genotype
file, or that fitqtl cannot fit, are absent rather than reported as zero.
Covariates work exactly as on /compute/rqtl -- covariates:
[{"case_attribute": "Status"}], inline values, or an explicit map -- and
are fitted alongside the QTL, so the effect reported is the one after
conditioning. Two details worth knowing: a covariate that is spread across
the panel but constant among the individuals a trait covers is left out, with
the columns named in notes (R/qtl stops with "system is exactly singular"
otherwise), and the LOD then no longer equals the scan's, because it is a
different model.
The cross-check that matters: without covariates, fitqtl's lod at a
marker equals scanone's LOD at that marker, to every digit the CSV
carries -- the test suite pins it. Same alignment rules as /compute/rqtl
(values matched onto the panel by name, parents/F1s excluded unless
include_parents_f1), the same normalize/winsor_* pipeline (and the
effects then come out on that scale), and the same R/qtl gate. Configuration: FITQTL_SCRIPT_PATH
(default scripts/qtl_fitqtl.R), RQTL_CACHE_DIR for its scratch files.
/compute/rqtl
Maps a QTL by running GeneNetwork3's own scripts/rqtl_wrapper.R (via
Rscript), copied into this repo verbatim: a scanone genome scan,
plus permutation thresholds if nperm > 0. Same request/response shape
as /compute/gemma -- synchronous, ?id=/GET .../status/{run_id}
progress-streaming reusing the same infrastructure (src/streaming.rs)
-- see src/rqtl.rs for the invocation itself, its documented scope
(no interval mapping or pair-scan scantwo -- just the common
single-scan case), and a fix made relative to GeneNetwork3:
its own cache-key hashing of the genotype/phenotype files never
actually looks at their contents (a key-name mismatch between
generate_rqtl_cmd and its real caller), so two different phenotypes
mapped with the same other options could silently reuse each other's
cached result there; fixed here by always hashing both files' contents.
Covariates. The same covariates list as /compute/gemma (above)
-- case attributes by name, or per-sample values -- fitted here with
--addcovar and a generated --covarstruct file. R/qtl's wrapper takes
categorical covariates as a single column and dummy-expands more than two
levels itself, so a level with no observations is its problem, not a
singular column: no columns are dropped on this path. A missing value is
x (the marker the wrapper calls missing), so R/qtl drops those
individuals; the coding and the count of complete cases come back in
notes.
Give it a group, not a path. group (an InbredSet name) plus an
optional genotype_version resolves the .geno server-side; see
GET /groups/{group}/genotypes. samples are then matched onto the
panel by name -- unknown names dropped, missing individuals null --
and parents/F1s are excluded unless include_parents_f1 is set. The
resolved file, version, panel size and how many values aligned come back
in notes, so a run is reproducible without the caller ever naming a
path. genotype_file is still accepted as an explicit override.
If you name the file yourself, R/qtl takes the GN genotype file
itself -- genotype/<file>.geno, e.g. genotype/BXD.8.geno (235
individuals × 21 057 markers) or genotype/HET3-ITP.12.geno (6 447
individuals × 894 markers) -- and reads its format directly. Which
engine a group offers, and which of its genotype files is current, are
GeneNetwork2/3 conventions summarised in
docs/genotypes.md; InbredSet.MappingMethodId is
what decides the engine (BXD = both, HET3-ITP = R/qtl only -- UM-HET3
is a 4-way cross). Measured on this deployment: BXD with the WGS panel,
235 samples of which 34 carry values, 21 056 markers in 7.5 s; UM-HET3
lifespan over 6 447 individuals, 893 markers in 83 s, peak LOD 7.95 at
chr 1, 24.04 Mb.
Important: R/qtl's cross file format has no per-sample name
matching of its own -- samples must list values in exactly the
genotype file's sample-column order, and it must have as many entries
as the genotype file has individuals. The second half of that used to go
unchecked (as it does in GeneNetwork3): nine values scanned against a
six-individual cross came back as a complete, plausible-looking set of
LOD scores belonging to nobody. scripts/rqtl_wrapper.R now stops with
"the phenotype file has 9 samples but the genotype file has 6
individuals" instead, which surfaces as a 502 rqtl_failed. It still
cannot tell whether the order is right -- that is the caller's
responsibility, and the reason a trait's values and its genotype file
have to come from the same panel.
{
"data": {
"markers": [
["rs31443144", "1", 3.010274, 0.0923496],
["rs6269442", "1", 3.492195, 0.1131095]
],
"permutation": null
},
"meta": {
"state": "complete",
"took_ms": 420,
"request_id": "...",
"columns": ["name", "chr", "pos", "lod"],
"num_rows": 6,
"num_cols": 4
},
"notes": [
"markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.",
"Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename <hash>-output.csv"
],
"next": []
}
(Real output, against the tiny 6-marker/6-sample BXD riset fixture in
tests/test_data/, trimmed to two markers. Each marker row is
positional, in meta.columns' order, rather than a
{"name": ..., "chr": ..., ...} object per row -- same
space-saving shape /compute/gemma uses for its own (much larger)
rows; a 4-way cross's extra mean/SE effect columns, if present, trail
each row and aren't named in meta.columns (see MarkerResult's doc
comment in src/rqtl.rs). permutation is null unless nperm was
set above 0; when it is, it's
{"values": [...], "suggestive": ..., "significant": ...}, the 67th-
and 95th-percentile thresholds GeneNetwork3 itself uses -- computed
here with numpy.percentile's default linear interpolation,
reproduced in src/rqtl.rs's percentile.)
Genotype files
/compute/rqtl and /compute/gemma take paths on the server, so
which panel file is current, in which format, and in which sample order
is a deployment question rather than an API one. That convention (GN2's
per-group genotype/<Group>.json, whose genofile list is newest-first;
genotype/bimbam/<file>_geno.txt for GEMMA; the .geno header for the
sample order; InbredSet.MappingMethodId for which engine a group
offers) is written up in docs/genotypes.md,
together with the three mappings measured against this deployment.
OpenAPI
GET /openapi.json serves an OpenAPI 3.1 document covering every route.
It's built with utoipa and
utoipa-axum:
- Each handler carries a
#[utoipa::path(...)]attribute describing its method, path, parameters and responses; each response/request struct derivesutoipa::ToSchema(seeVersionInfo,db::Species,db::SpeciesDetail,error::ApiErrorinsrc/lib.rs/src/db.rs/src/error.rs). - Routes are registered with
utoipa_axum::routes!(handler)instead of plainaxum::routing::get(handler), on anOpenApiRouterinstead of a plainRouter, so each route's metadata is collected as it's added. ApiDoc(a#[derive(utoipa::OpenApi)]struct) supplies the document'sinfo(title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's.OpenApiRouter::split_for_parts()splits the builder back into a plainaxum::Router(returned fromapp()) and the assembledutoipa::openapi::OpenApivalue, which the/openapi.jsonhandler then just serializes withJson.ApiDoc's#[openapi(security(("bearer_token" = [])), ...)]makes a token the default for every operation, so an endpoint that needs one doesn't repeatsecurity(...)itself -- only the handful that must NOT require one (/,/version,/get-test-token) override it with an emptysecurity()of their own.error::Unauthorized(a#[derive(utoipa::IntoResponses)]type) is the shared 401 response every gated endpoint references by name in its ownresponses(...)list (responses(Unauthorized, (status = 200, ...), ...)) instead of writing out(status = 401, description = "...", body = ApiError)by hand each time.
Because the attributes are declared alongside the handlers they
describe, the spec can't silently drift out of sync with the code the
way a hand-maintained spec file could -- adding or changing a route
without updating its #[utoipa::path(...)] is a compile error (the
macro checks the response/schema types exist) or an obviously incomplete
diff, not something to notice weeks later.
The document is assembled once, when app() runs at process startup
(not at cargo build time, and not freshly on every request): the
#[utoipa::path]/#[derive(ToSchema)] macros expand to Rust code at
compile time, but the actual merging of all routes' metadata into one
OpenApi value happens the first time app() executes. Every
GET /openapi.json request after that just serializes the same
already-built value.
Success responses
Every success response is one JSON envelope (src/response.rs): the
data, next to facts about the response, remarks on the data, and
suggested follow-up calls. It is the success counterpart of the error
shape below.
{
"data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
{"href": "/groups?species=mouse", "description": "Groups of this species"}
]
}
datais the response proper: exactly what each endpoint below documents.meta.stateiscomplete, orpartialwhen the response is usable but something was left out or unavailable --notessays what. For example/searchispartialwhen the database could not be reached (hits come back withoutlocusandabbreviation) or when a sorted search ranked a full 1000-match window and more matches exist;/publications/{pubmed_id}/traitsispartialwhen traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure.meta.took_msis the server's time for the request, from when it started handling it; it excludes the network.meta.request_idis the request's id, also in thex-request-idresponse header and in any error body, so a report can be correlated with logs.x-gn4-usernames the caller on every authenticated response: the token'ssub(e.g.pjotrp). It is there for the proxy's access log --user=$upstream_http_x_gn4_userin thegn4log format -- so the deployment can answer "who is using this, successfully" rather than guessing from addresses and user agents. Unauthenticated responses (/version,/openapi.json, a401) carry no such header, and the caller only ever sees an identity it already presented.- The same identity goes into this server's own access log: one JSON line per
request on stderr —
{"event":"request","method","path","status","took_ms", "request_id","user"}— withuserabsent when no token was verified, the path stripped of its query string, and the token itself never logged.GN4_ACCESS_LOG=falseturns it off. Theevent: errorlines for 5xx go to the same stream and carry arequest_id, so a failure and the request that caused it are onegrepapart. notesare server-authored remarks ondata, e.g.lod = lrs / 4.604on responses that carrylod. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's.nextlists follow-up calls, each a relativehrefthat resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large).GET /searchsuggests the next page and the top hit's metadata.actionslists suggested calls that are not plainGETs, so they cannot benextlinks: each carriesmethod,href, adescription, and abodywhere the server can build one. It is present only when there is something to suggest, so most responses keep the four members above. Today the per-strain values endpoints (/groups/{group}/traits/{id}/values,/datasets/{dataset}/probesets/ {probeset}/values,/traits/{dataset}/{trait}/values) use it to hand back the values they just returned, ready to map:
json
"actions": [
{"method": "POST", "href": "/compute/rqtl",
"description": "Map 'BXDPublish::10001' with R/qtl (scanone): this body carries the 34 values above as `samples`, in the same order. Replace the `genotype_file` placeholder with a path that exists on the server. …",
"body": {"genotype_file": "<path on the server, e.g. a genotype file it already holds>",
"trait_name": "BXDPublish::10001",
"samples": [{"name": "BXD1", "value": 61.400001525878906}, "…"]}},
{"method": "POST", "href": "/compute/gemma", "description": "…",
"body": {"genotype_file": "<…>", "snps_file": "<…>",
"trait_values": [61.400001525878906, "…"]}}
]
The file paths stay bracketed placeholders on purpose: they have to
exist on the server's own filesystem, which the server cannot know on
the caller's behalf (see the /compute sections below).
/ returns the plain text Hello, world! and /openapi.json the
OpenAPI document itself; neither is wrapped. In /openapi.json every
other success response is an Envelope_* schema (a test enforces this).
Error handling
Every endpoint uses the same ApiError JSON shape on error, modeled on
RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see
src/error.rs). /species and /species/{species} can actually produce
one today (database errors, or a 404 for an unknown species); / and
/version document the same 500 response in /openapi.json for
consistency, even though neither has a failure path currently (no
database access, no I/O):
{
"type": "/errors/species_not_found",
"title": "Species not found",
"status": 404,
"code": "species_not_found",
"detail": "No species matching 'nonexistent'.",
"operation_id": "species_by_name",
"spec_url": "/openapi.json",
"request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
"took_ms": 2,
"next": "/species"
}
next is omitted (null) when there's no obviously related endpoint to
suggest, e.g. a hypothetical error from /:
{
"type": "/errors/internal_error",
"title": "Internal error",
"status": 500,
"code": "internal_error",
"detail": "example failure for /",
"operation_id": "hello_world",
"spec_url": "/openapi.json",
"request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
"took_ms": 1,
"next": null
}
typeandspec_urlare relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allowstypeto be a relative URI reference.spec_urlis always/openapi.json, on every error from every endpoint.operation_idis the OpenAPI operation that failed (same name as in/openapi.json).request_idis the request'sx-request-id, generated per request (or echoed back if the client already sent one) bySetRequestIdLayer/PropagateRequestIdLayerinsrc/lib.rs, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token.took_msis the server's time for the request up to the error, the same measure as a success response'smeta.took_ms(see Success responses). RFC 7807 allows extension members like this one,request_id,codeandnextnext to its standard ones.nextis an optional hint at a related endpoint to try instead; e.g. a 404 from/species/{species}setsnextto/species, so a caller with a wrong species name can discover the valid ones.
Running with cargo
Requires a Rust toolchain with cargo (edition 2024).
cargo run
The server listens on 0.0.0.0:3000 by default. Set GENENETWORK4_PORT
to use a different port, or GENENETWORK4_BIND to bind a specific
address instead of every interface:
GENENETWORK4_PORT=8080 cargo run
GENENETWORK4_BIND=127.0.0.1 cargo run
This deployment runs two instances: 3000 (production) and 3001 (development)
The reference deployment keeps a development instance next to the live one, on the same box, so work can be tried out without disturbing what the site is serving:
| instance | port | binary | run caches | proxied by nginx |
|---|---|---|---|---|
| production | 3000 |
target/release/genenetwork4 |
defaults (/tmp/gn4-gemma, /tmp/gn4-rqtl, …) |
yes, at /gn4/ (see docs/nginx.md) |
| development | 3001 |
target/debug/genenetwork4 |
/tmp/gn4-dev/{gemma,rqtl,ravanan} |
no |
They differ only in the port, the build profile and those cache
directories; both use the same read-only SQL_URI, the same Xapian index
and the same auth stub, so a request behaves the same on either. The
separate caches matter: a GEMMA or R/qtl run on the development instance
must not fill, or be sped up by, production's warm cache.
P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT') # needs R + gemma-wrapper on PATH
# production
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3000 GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
nohup ./target/release/genenetwork4 > /tmp/gn4-3000.log 2>&1 & # pid in /tmp/gn4.pid
# development: same, plus the port and its own caches
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ \
GENENETWORK4_BIND=127.0.0.1 GENENETWORK4_PORT=3001 GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/home/wrk/tmp/ai-root/factory/gn4-index SEARCH_ENGINE=native \
GEMMA_CACHE_DIR=/tmp/gn4-dev/gemma RQTL_CACHE_DIR=/tmp/gn4-dev/rqtl \
RAVANAN_CACHE_DIR=/tmp/gn4-dev/ravanan \
nohup ./target/debug/genenetwork4 > /tmp/gn4-dev.log 2>&1 & # pid in /tmp/gn4-dev.pid
Both need the PATH/R_LIBS_SITE pair (see "Building with cargo, outside
a bare toolchain" below) or R/qtl fails with there is no package called
'optparse', and both need the auth stub on 127.0.0.1:8081 to serve
/auth/public-jwks (see "Tests").
Anything the server keeps in memory is per instance: the pool, the JWKS
and authorisation caches, and any in-process compute limit. So the two
instances do not serialise against each other — a mapping on :3001 runs
alongside one on :3000 — which is the point of the split, and a good
reason not to treat the development instance as a capacity test of what
production will do.
Building with cargo, outside a bare toolchain
cargo build in the package's own dependency shell is the supported
way to build; a bare cargo build against a distro toolchain fails,
and one of the reasons cannot be fixed by installing packages, since it
is a Xapian version conflict (see Running with Guix
for the pin in guix.scm).
GUIX_PACKAGE_PATH=<path-to-guix-bioinformatics> \
guix shell -D -f guix.scm -- bash -c '
export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib # autocxx: libclang 13, not 22
export DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib # libduckdb-sys: no duckdb.pc
export DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include
cargo build --release'
GUIX_PACKAGE_PATH(or-L) is for(gn packages gemma); without itguix.scmfails to load with "no code for module (gn packages gemma)".- The three exported variables are exactly what
guix.scm'sset-duckdb-pathsphase exports for the packaged build. WithoutLIBCLANG_PATHthe build stops inxapian-rs's build script with "Unable to find libclang"; without theDUCKDB_*pairlibduckdb-syslinks a-lduckdbwith no search path. - The resulting binary resolves every shared library from the shell's
store paths on its own (Guix's linker wrapper records them as a
RUNPATH), so running it needs no
LD_LIBRARY_PATH-- but it does stop working once those store paths are garbage-collected.
Project memory: why this code looks like this
The repository carries an ai-memory store (ai-memory.sqlite, gitignored)
that links each commit to the reasoning behind it, plus an optional Xapian
topic index over both. It is how a session that starts with an empty context
recovers what the last one worked out — measured at 16 ms for a text query and
about 100 ms through the topic index:
ai-memory -C . search 'pve' # reasoning, full-text
ai-memory -C . why gemma.compute_compute # commits that touched a symbol, with its reasoning
ai-memory -C . --index ~/.cache/gn4-topics search 'topic:ravanan-slurm'
Operational runbooks live there too — for example the Slurm container from
gn-machines, its JWT, the environment /compute/gemma/ravanan needs, and
the fastest endpoint test — because the commands are the part a reader cannot
reconstruct from a diff. The store is local and per repository: it survives a
cleared session, not a lost machine, so every conclusion is also in a commit
message and in these docs.
Development
Every endpoint needs a bearer token (see the note under
Endpoints), which normally means getting one from
gn-auth yourself. For local development, genenetwork4 can hand you one
instead: POST /get-test-token calls gn-auth's own password grant
(POST /auth/token) for gn-auth's well-known development user and
returns the result. It needs no setting and is not tied to a mode.
cargo run
Get a token, then use it against a real endpoint:
curl -X POST http://127.0.0.1:3000/get-test-token
{
"data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"},
"meta": {"state": "complete", "took_ms": 41, "request_id": "..."},
"notes": [],
"next": []
}
TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])')
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species
{
"data": [
{"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"},
...
],
"meta": {"state": "complete", "took_ms": 2, "request_id": "..."},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
]
}
The endpoint is registered by default and listed in /openapi.json
(GENENETWORK4_TEST_TOKEN=false switches it off, see below), and decides
per request whether to answer: only a caller on this machine gets a
token. That means the TCP connection's peer address is in
127.0.x.x (or is ::1), and the request carries no X-Forwarded-For,
Forwarded or X-Real-IP header. Anything else, including a request
whose peer address is unknown, is a 403. The peer address comes from
the connection itself, never from a header a client can set; the proxy
headers are used only to refuse a request that claims to have been
forwarded. This is why the check also refuses forwarded requests: a
reverse proxy running on the same host connects from 127.0.0.1 for
every client, so without it the whole internet would look local. A proxy
that forwards without setting any of those headers would defeat the
check, so don't put one in front of a server whose gn-auth has the dev
user (on a production gn-auth the dev user normally doesn't exist, and
the request just fails with a 502).
Behind nginx or in a container
docs/nginx.md is the full deployment recipe: serving
genenetwork4 under a path prefix such as https://host/gn4/ from a
loopback port, the long proxy_read_timeout /compute/* needs, and how
the root-relative links in responses are kept under the prefix. What
follows here is only about POST /get-test-token.
The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here):
- Container on a bridge network, port published. The server sees
the bridge gateway (something like
172.17.0.1) as the peer, not127.0.0.1. Every request through the published port is a403, whether it comes via nginx or from the host itself. It still works from inside the container, e.g.docker exec <container> curl -X POST http://127.0.0.1:3000/get-test-token. - Container sharing the host's network, nginx on the same host.
nginx connects from
127.0.0.1, so the peer check passes for every client. The server then relies on refusing requests that carryX-Forwarded-For,ForwardedorX-Real-IP, and a minimalproxy_passsets none of them. Without one of the measures below, anyone who can reach the site could get a token.
For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third:
- Block the path in nginx, independent of genenetwork4:
nginx
location = /get-test-token { return 404; }
- Make nginx say it is forwarding, so genenetwork4 refuses proxied requests (any one of the three headers is enough):
nginx
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- Switch the endpoint off in genenetwork4 itself with
GENENETWORK4_TEST_TOKEN=false(an environment variable, or aGENENETWORK4_TEST_TOKEN=falseline inGN4_CONF). The route is then not registered at all:POST /get-test-tokenis a404and the endpoint is absent from/openapi.json. The setting fails closed: onlytrue,1,yesoron(any case) keep the endpoint on, and anything else --false, an empty value, or a typo such asflase-- turns it off. It is on by default, so a local setup needs nothing.
Also relevant: the token is for gn-auth's dev user
(GN_AUTH_DEV_USERNAME). On a production gn-auth that user normally
doesn't exist, so a request that does get through fails with a 502
instead of returning a token. That depends on gn-auth's data, not on
anything in genenetwork4.
The credentials used (gn-auth's dev client id/secret and dev user
email/password, from flask init-dev-clients/gn_auth/wsgi.py) are
publicly known and the same for every gn-auth checkout by default --
override GN_AUTH_DEV_CLIENT_ID, GN_AUTH_DEV_CLIENT_SECRET,
GN_AUTH_DEV_USERNAME and GN_AUTH_DEV_PASSWORD if your gn-auth
instance's dev fixtures differ. Needs a gn-auth reachable at
AUTH_SERVER_URL with AUTHLIB_INSECURE_TRANSPORT set (gn-auth itself
refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev
setting, not something genenetwork4 needs to know about).
The token has no exp claim, and neither genenetwork4 nor gn-auth
expires it by time or for lack of use. It stops verifying only when the
key that signed it leaves gn-auth's published key set: gn-auth signs with
a key for 7 days (JWKS_ROTATION_AGE_DAYS) and deletes it 14 days after
its creation (JWKS_DELETION_AGE_DAYS), so a token lasts roughly 7 to
14 days, depending on where in the key's life it was issued.
genenetwork4 caches the key set for an hour. When requests start
failing with 401 invalid_token, fetch a new token. (exp is optional
in genenetwork4's own validation, which is why a token without one like
this lives as long as its signing key does -- see "Tokens are
stateless" below.)
The test suite uses this same endpoint automatically -- see Tests below.
Configuration is layered: built-in defaults, then GN4_CONF and
GN4_SECRETS files of KEY=VALUE lines, then environment variables
(which always win). Keys: SQL_URI (MariaDB), SQL_CHECK_HEALTH (see
below), GN3_URL (GeneNetwork3
base URL for proxy endpoints), GN3_TIMEOUT_SECS (upstream call
timeout, default 30), GENENETWORK4_PORT, GENENETWORK4_BIND, and
GENENETWORK4_TEST_TOKEN (false turns POST /get-test-token off; see
"Behind nginx or in a container").
Tokens are stateless
Nothing about a bearer token is stored server side: there is no session
table, no list of issued tokens, no jti denylist. Verification is
signature + iss + exp/nbf, and that is all, so restarting
genenetwork4 loses no tokens -- a token minted before a redeploy keeps
working after it, as does one minted by a stub that has since been
restarted (its key is fixed). What the process does keep is two caches,
and restarting only empties them:
- the signing keys (
JWKS_TTL, an hour) -- how long a fetched copy of the auth server's key set is reused before being re-fetched; - authorisation answers (
AUTHORISATION_TTL, a minute), per(subject, resource).
Neither is a lifetime. The token's own exp is the only thing that ends
its life: it is validated on every request (validation.validate_exp in
src/auth.rs, with jsonwebtoken's default 60-second leeway), it is never
extended, and there is no refresh token. The hour matters in one case
only: if the auth server is unreachable, a cached key set keeps
verification working for up to JWKS_TTL, and past that every request
is 401 invalid_token with a fetching .../auth/public-jwks detail --
the tokens are still valid, the key lookup is not. Run the auth server
under the same supervision as genenetwork4 rather than from a shell.
Consequences worth knowing:
- Restarting is not revocation. Since no state exists, restarting
genenetwork4 or the auth server leaves every outstanding token valid.
To withdraw one, wait for its
exp(hence short lifetimes for tokens you may need to pull, e.g.scripts/gn-auth-stub.py --expires-in). To withdraw all of them at once, change the signing key orAUTH_SERVER_URL--issis compared against that string verbatim, so a changed host, scheme or trailing slash invalidates everything withInvalidIssuer. expis optional.required_spec_claims.clear()means a token withoutexpis accepted and never expires. gn-auth andscripts/gn-auth-stub.pyalways set one; a hand-minted token (the test key is in this repository) that omits it is a permanent credential.
Stub-minted tokens behave the same way (see Tests): exp is
whatever --expires-in said at mint time, and restarting the stub does
not invalidate them -- it always signs with the committed test key.
Request body size
The API accepts bodies up to 64 MiB (BODY_LIMIT_BYTES in
src/lib.rs). axum's own default is 2 MiB, which a legitimate request
exceeds: /plot/{rqtl,gemma} is posted a whole mapping response (a
real GEMMA run on the BXD WGS panel is ~1.9 MB; a genome-sized one is
tens of MB), and /compute/* carries one value per individual plus up to
eight covariates. Over the limit, axum answers 413 Failed to buffer the
request body: length limit exceeded before any handler runs.
Two consequences worth knowing:
- a proxy in front must allow at least as much
(
client_max_body_size 64minetc/nginx-vesuvio.pu.ac.ke.conf), or it refuses first with a 413 the API never sees; - a request does not need to be that large. For covariates, prefer
{"case_attribute": "Sex"}-- the server reads the attribute and aligns it, so nothing is posted -- over sending a value for every individual in the panel. Only the samples the trait has are used, so posting values for the rest is pure payload.
Database connections
SQL_URI is used through one shared mysql::Pool. Two of the driver's
per-checkout round trips are configurable:
SQL_CHECK_HEALTH(defaulttrue) pings a connection taken from the pool and replaces it when the ping fails. Leave it on unless the database is far away: turning it off saves a round trip, but then a connection the server closed while idle surfaces as500 database_erroron whichever request happens to draw it -- measured here as ten consecutive failures, one per dead pooled socket, until the process was restarted.reset_connectionis off unconditionally: aCOM_RESET_CONNECTIONper checkout, with nothing to reset (the queries are read-only).
Even with the ping on, a socket can die between the ping and the query;
when a database error names a connection loss (Broken pipe, Connection
reset, server has gone away, ...), the pool is dropped so the next
request builds a fresh one (db::is_connection_loss / db::forget_pool,
called from ApiError::from_db, the one place database errors pass
through). tests/db_recovery.rs kills every pooled session and asserts
the following requests still succeed.
Running with Guix
guix build -f guix.scm
builds the package (running both test suites, see below) and prints the
resulting store path; the binary is at <store path>/bin/genenetwork4.
The xapian input is pinned to 1.4.29 (xapian-1.4 in guix.scm): the
xapian-rs git revision Cargo.toml pins was written against Xapian
1.4, generating bindings for Xapian::InMemory::open, and Xapian 2.x
removed it (xapian/dbfactory.h no longer declares the namespace; 2.x
spells it as the Xapian::DB_BACKEND_INMEMORY flag). 2.x also needs
C++17, where autocxx hands the parser a fixed -std=c++14 and 1.4's
headers are happy with it. Guix's own xapian is 2.1.0 now, so without
the pin the xapian-rs build script stops with "the 'generate' or
'generate_pod' directive for 'Xapian::InMemory::open' did not result in
any code being generated" before compiling anything of this project's.
The pin is a plain url-fetch/sha256 for xapian-core-1.4.29.tar.xz
with 1.4's own make check-inmemory check-remoteprog check-multi
check-glass check-chert test targets; drop it once xapian-rs moves to
Xapian 2. The same pin is what guix shell -D and a manual cargo
build see (see "Building with cargo, outside a bare toolchain" above).
guix shell -D -f guix.scm
drops you into a shell with all build dependencies available (cargo,
rust, python, git), for running cargo build/cargo test/cargo run
directly. This shell is not isolated from the network or from your
normal environment, which cargo build/cargo run need to reach
crates.io.
A fully isolated container (-C) normally needs -N to still reach the
network from inside it, but on at least one guix build
(1.5.0-6.64d4de2) combining -D with -N causes -D to silently stop
providing the package's build dependencies (the shell then has only
genenetwork4 itself, not cargo/rust/etc.). If guix shell -C -D -N -f
guix.scm fails with "cargo: command not found", drop -C/-N and use
the plain guix shell -D -f guix.scm above instead.
Tests
Two test suites exercise the same endpoints:
tests/*.rs- Rust integration tests that call the router in-process (no socket), viatower::ServiceExt::oneshot. Run with:
cargo test
Both suites check the envelope on every success response: the Rust
helper tests/common::data(&body) and the Python helper
server_test_case.envelope_data(body) verify its shape and return
data. tests/envelope.rs (and test/python/test_envelope.py)
follow every next link the endpoints suggest;
tests/envelope_partial.rs checks the partial state without a
database; tests/openapi.rs checks that every JSON success response
is documented as an envelope.
test/python/test_*.py- Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with:
cargo build --release
GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python
Either suite can fetch its own token from a live gn-auth automatically
(see "Development" above) -- set AUTH_SERVER_URL before running
either command above, and nothing further needs setting up by hand. A
run without a reachable gn-auth still works, skipping (not failing)
every test that needed a token it couldn't get -- set
GN4_TEST_AUTH_TOKEN instead if you'd rather provide one manually.
Both suites also run automatically as part of guix build -f guix.scm
(the Rust tests in the check phase, the Python tests in a
python-rest-test phase after install).
Every test under tests/*.rs/test/python/test_*.py for an endpoint
that touches the database (which is most of them -- hello_world,
version, openapi and get_test_token's own disabled/wrong-config
cases are the exceptions) checks for a live MariaDB on 127.0.0.1:3306
first (see SQL_URI above) and skips with a message rather than
failing when none is reachable, since the Guix build sandbox has no
network/database access. Likewise, any test for an endpoint that
requires a token (every endpoint except those four) checks for one
first and skips if none is available -- see "Development mode" above
for how the test suite gets one automatically.
The Rust tests in tests/*.rs that hit the database issue their
requests through tests/common::get, which times the request and
prints a warning to stderr if it takes over a second -- e.g.
WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s). Warnings only
show with cargo test -- --nocapture (or on a failing test), since
cargo captures stdout/stderr for passing tests otherwise.
Against a real database and index
Those skips are the sandbox's, not the code's. Supply the three things
the tests look for -- a live MariaDB, a Xapian index, a token -- and the
commands above run the whole suite rather than a small subset. In a
development environment with all three, cargo test reports 225 tests
passed / 0 failed (the handful of skips left are listed below) and the
Python suite 94 tests, OK, no skips.
- Database: whatever
SQL_URIpoints at, by defaultmysql://webqtlout:webqtlout@127.0.0.1/db_webqtl.tests/common::db_available()really connects, so wrong credentials skip exactly like an absent server does. - Index:
XAPIAN_DB_PATHfor a dev index, plusGN4_FULL_INDEXfor a full/production one. Only a full index is treated as a count oracle:tests/search_native_db.rsthen requires the engine'sdataset:bxdpublishmatch set to equal the database'sPublishXRefids for BXD exactly, id for id -- the invariant a dev snapshot famously fails (it is missing ~1.7k traits and holds ~350 ids the database no longer has). A dev index only has to overlap, and its counts are then printed as reference data, never asserted. When both are set the full one wins. - Token: either a real gn-auth at
AUTH_SERVER_URL(see "Development"), orscripts/gn-auth-stub.py, which stands in for one: it serves the JWKS matching the committedtest/data/test_rsa_private.pem(the same fixture pairsearch_native_dbspins up in-process), answers/auth/tokensoPOST /get-test-tokenworks, and answers both authorisation endpoints with "no privileges" -- the stranger case the non-public endpoints' 403 tests assert. It is test scaffolding, not a gn-auth replacement: a committed test key, and no knowledge of users or groups.--grant-resource-id IDgives its tokengroup:resource:view-resourceon one resource id, for the owner-side tests, which wantGN4_TEST_RESOURCE_ID=IDto match.
Left ungranted, a stub token is a public-data-only token, which is
useful beyond the tests (handing a caller or an agent something to
browse the API with). Only-public is not a property of the token:
genenetwork4 decides from the data's own
public/confidentiality/create_time columns, and asks this
authorisation lookup about anything that is not public (src/auth.rs's
check). No privileges therefore means public datasets read normally
and every non-public one is a 403 -- the token's scope is never
consulted for data access. Dataset metadata
(/species/{species}/datasets, /groups/{group}/datasets) stays
visible either way: those endpoints are documented as returning names
and flags, not trait data, and are not gated on fine-grained
authorisation.
--mint-only mints for one subject and exits without listening: how a
token is issued for one more user without disturbing the running stub
(see docs/www/operator-checklist.md, "Token issuance").
--expires-in sets the token's lifetime (30d, 24h, 60m, or
plain seconds; default 24h). A month-long one is fine because the
stub's key is fixed and never rotates: nothing invalidates the token
but its own exp, and it stays verifiable for as long as anything
serving test/data/test_jwks.json is up (genenetwork4 caches the key
set for an hour, so a brief gap is survivable):
scripts/gn-auth-stub.py --port 8099 --expires-in 30d --token-file /tmp/public-only.token &
AUTH_SERVER_URL=http://127.0.0.1:8099/ GENENETWORK4_PORT=3999 cargo run
Note the port: iss has to match the server's AUTH_SERVER_URL, so a
token minted for :8099 is rejected by a server left on the default
:8081, and vice versa.
scripts/gn-auth-stub.py --grant-resource-id 424242 --token-file /tmp/gn4-token.txt &
export GN4_TEST_AUTH_TOKEN=$(cat /tmp/gn4-token.txt)
export GN4_TEST_RESOURCE_ID=424242
export GN4_FULL_INDEX=/path/to/full-index XAPIAN_DB_PATH=/path/to/full-index
GUIX_PACKAGE_PATH=<path-to-guix-bioinformatics> \
guix shell -D -f guix.scm -- bash -c '
export LIBCLANG_PATH=$GUIX_ENVIRONMENT/lib \
DUCKDB_LIB_DIR=$GUIX_ENVIRONMENT/lib \
DUCKDB_INCLUDE_DIR=$GUIX_ENVIRONMENT/include
cargo test --no-fail-fast -- --nocapture'
cargo build --release
GENENETWORK4_BINARY=target/release/genenetwork4 \
python3 -m unittest discover -v test/python
The stub listens on 127.0.0.1:8081, which is the default
AUTH_SERVER_URL -- including its trailing slash, since that string is
compared verbatim against the token's iss. --port/--issuer cover
the case where AUTH_SERVER_URL says something else.
What still skips with all three in place, and why:
- one
configunit test asserts what an unsetXAPIAN_DB_PATHmeans, so setting the variable at all skips it; src/indexer/cache.rs's RDF-cache test needs an authoritative RDF cache directory, which this does not provide;compute_gemma_ravanan_*needs ravanan, gemma-wrapper and ccwl onPATH;tests/search_native_dev_corpus.rsrecords a dev corpus, so it needsGN4_DEV_CORPUS_DIRas well as an index.
