genenetwork4
A Rust/Axum-based API server for GeneNetwork.
Endpoints
| Method | Path | Description |
|---|---|---|
| GET | / |
Greeting. |
| GET | /version |
Build info (hash, date). |
| GET | /species |
All species. |
| GET | /species/{species} |
One species by name or id. |
| GET | /species/{species}/datasets |
ProbeSet datasets in a species. |
| GET | /groups |
All groups (filterable). |
| GET | /mapping-methods |
Mapping tools offered. |
| GET | /accessions/{accession} |
Resolve a GN accession number. |
| GET | /groups/{group} |
One group, by name or Id. |
| GET | /groups/{group}/strains |
Strains/samples in a group. |
| GET | /groups/{group}/case-attributes |
Case-attribute names and values. |
| GET | /groups/{group}/traits |
Published (phenotype) traits. |
| GET | /groups/{group}/traits/{trait_id} |
A published trait's metadata. |
| GET | /groups/{group}/traits/{trait_id}/values |
A published trait's values. |
| GET | /groups/{group}/datasets |
ProbeSet datasets in a group. |
| GET | /publications/{pubmed_id}/traits |
Traits a publication reports. |
| GET | /datasets/{dataset} |
One expression dataset. |
| GET | /datasets/{dataset}/probesets |
Expression traits in a dataset. |
| GET | /datasets/{dataset}/probesets/{probeset} |
An expression trait's metadata. |
| GET | /datasets/{dataset}/probesets/{probeset}/values |
An expression trait's values. |
| GET | /traits/{dataset} |
Every trait in a dataset. |
| GET | /traits/{dataset}/{trait} |
One trait's metadata. |
| GET | /traits/{dataset}/{trait}/values |
One trait's values. |
| POST | /compute/correlation |
Correlate a trait (stub). |
| POST | /compute/gemma |
Map a QTL with GEMMA (see below). |
| GET | /compute/gemma/status/{run_id} |
Poll a gemma run's progress. |
| POST | /compute/gemma/ravanan |
Map a QTL via ravanan (see below). |
| POST | /compute/rqtl |
Map a QTL with R/qtl (see below). |
| GET | /compute/rqtl/status/{run_id} |
Poll an rqtl run's progress. |
| POST | /get-test-token |
Get a test bearer token (local). |
| GET | /search |
Search traits (free-text). |
| GET | /openapi.json |
OpenAPI 3.1 document. |
Every endpoint above requires a valid gn-auth bearer token
(Authorization: Bearer <token>) except /, /version, /openapi.json
and /get-test-token itself -- see docs/gn-auth.md and the
"Development" section below (POST /get-test-token) for how to get
one for local testing. A missing or invalid token is a 401; which
further, resource-level authorization (if any) an endpoint checks once
a token is valid is a separate, per-endpoint matter covered in each
entry below.
Every success response is wrapped in one envelope (see
Success responses below), except / (a
plain-text greeting) and /openapi.json. The example responses below
are real ones, with long lists and strings abbreviated by ..., and
took_ms and request_id shortened to plausible values.
Details, including example responses, below.
GET /- returnsHello, world!.GET /version- returns a JSON record describing the running build:
json
{
"data": {
"name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0,
"release": 1, "date": "20261008", "git": "208eeac"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": []
}
date and git are the HEAD commit's date and short hash, embedded at
build time (see build.rs); they describe the commit the binary was
built from, not the time it happened to be compiled.
GET /species- returns a JSON array of all species in theSpeciestable, mirroring GeneNetwork3'sgn3.db.species.get_all_species:
json
{
"data": [
{"name": "human", "menu_name": "Human (Homo sapiens, hg19)",
"family": "Vertebrates"},
{"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)",
"family": "Vertebrates"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
]
}
Needs a live MariaDB connection, configured via the SQL_URI
environment variable (same config key GeneNetwork3 uses), e.g.
mysql://user:password@host/db_webqtl. Defaults to
mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl if unset.
GET /species/{species}- returns every column of the matching row of theSpeciestable, matched againstName,FullNameorSpeciesName(matching GeneNetwork2'swqflask.api.router.get_species_info, which returns the same match logic but only a 4-column subset). 404 if no species matches.
json
{
"data": {
"id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse",
"menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus",
"family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090,
"order_id": 15
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
{"href": "/groups?species=mouse", "description": "Groups of this species"}
]
}
Needs SQL_URI, same as /species above.
GET /species/{species}/datasets- returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of/groups/{group}/datasets. GeneNetwork2's own species-scoped variant ofwqflask.api.router.get_datasets_for_groupstill requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as/groups/{group}/datasets-- this returns dataset metadata (names,public/confidentialityflags), not trait data -- but a token is still required, like every endpoint. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412",
"name2": "B6D2ONCILM_Apr12",
"full_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"short_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"create_time": "2012-05-17", "public": 1, "confidentiality": 0,
"data_scale": "log2", "tissue": "Retina mRNA"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse", "description": "The species these datasets belong to"}
]
}
GET /groups- returns a JSON array of all groups in theInbredSettable, matching the shape of GeneNetwork2'sGET /api/v_pre1/groupsresponse:
json
{
"data": [
{"id": 25, "species_id": 4,
"display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)",
"name": "AD-cases-controls",
"full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)",
"public": 2, "mapping_method_id": null, "genetic_type": null,
"family": null, "species": "human", "mapping_method": null},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species", "description": "The species the groups belong to (filter groups with ?species=<name>)"}
]
}
family is the population category and species the species' short
name as /species lists it. mapping_method is the name of the
mapping tool offered for the group (the MappingMethod row its
mapping_method_id points to, see GET /mapping-methods), null for a
group with none. Three optional filters, combined with
AND and all case-insensitive: family (contains), species (exact
short name) and q (contained in the name, display name or full
name). For example /groups?family=Reference Populations lists the
reference populations, and /groups?species=rat&q=hxb the rat
HXB/BXH sets.
"Group" is GeneNetwork's own alias for InbredSet -- see
docs/genenetwork_datatypes.md for why. Needs SQL_URI, same as
/species above.
GET /groups/{group}- returns every column of the matching row of theInbredSettable, matched againstInbredSetName,Name,FullNameor numericId(matching GeneNetwork2'swqflask.api.router.get_group_info, which matches the same way but not by Id, plus gn-guile'sgn.data.population:population-info, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (seedocs/genenetwork_datatypes.md); only the MariaDB column names themselves stayInbredSet*. 404 if no group matches.
json
{
"data": {
"id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD",
"species_id": 1, "full_name": "BXD Family", "public": 2,
"mapping_method_id": "1", "genetic_type": "riset",
"family": "Reference Populations (replicate average, SE, N)",
"family_order": 1, "menu_order_id": 0.0, "group_code": "BXD",
"description": "<p>...</p>", "mapping_method": "qtlreaper"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/strains", "description": "The group's strains/samples"},
{"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"},
{"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"},
{"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"}
]
}
Needs SQL_URI, same as /species above.
GET /groups/{group}/strains- returns the strains/samples belonging to a group, matching gn-guile'sgn.data.strains:strain-id-names. 404 if no group matches. Takes an optionalused_for_mappingquery parameter to filter to only strains used for mapping (gn-guile'sstrain-id-namestakes the same flag as a keyword argument):
json
{
"data": [
{"id": 1, "name": "B6D2F1", "used_for_mapping": false},
{"id": 2, "name": "C57BL/6J", "used_for_mapping": false},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group these strains belong to"},
{"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}
]
}
GET /groups/BXD/strains?used_for_mapping=true excludes BXD's
parent strains (B6D2F1, C57BL/6J, DBA/2J), which aren't used
for mapping. Needs SQL_URI, same as /species above.
GET /groups/{group}/case-attributes- returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3'sgn3.api.case_attributessplits this into separate.../namesand.../valuesendpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns200with both lists empty, not a 404. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"names": ["Epoch", "RRID", ...],
"values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...]
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/strains", "description": "The group's strains/samples"}
]
}
GET /groups/{group}/traits- returns every published (phenotype) trait belonging to a group, the listing equivalent of/groups/{group}/traits/{trait_id}(same fields, same join, notrait_idfilter). 404 if no group matches. If the group has any published traits, itsPublishFreezerow is checked the same way as/traits/{dataset}(see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's ownwqflask.api.router.fetch_traits, which defaults to unbounded too. NeedsSQL_URI, same as/speciesabove.
The optional abbreviation parameter keeps only the traits with that
short name, e.g. /groups/BXD/traits?abbreviation=5HT_MB_M finds
trait 17985 (an unknown abbreviation is an empty list).
json
{
"data": [
{"id": 10001, "phenotype_id": 4, "publication_id": 116,
"mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/groups/BXD", "description": "The group these traits belong to"}
]
}
GET /groups/{group}/traits/{trait_id}- returns one published (phenotype) trait'sPublishXRefmapping stats plus itsPhenotypedescription/units andPublicationreference, based on the data explored inprecompute-publishdata.gmi(e.g. its "More metadata" section, which identifies exactly this join as worth surfacing).trait_idisPublishXRef.Id, scoped by group for the same reason as/groups/{group}/traits/{trait_id}/values. 404 if no group matches, or if the group has no such trait; 403 if the trait'sPublishFreezeisn't public and the caller isn't authorized for it (see docs/gn-auth.md).
json
{
"data": {
"id": 10001, "phenotype_id": 4, "publication_id": 116,
"mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"},
{"href": "/groups/BXD", "description": "The group this trait belongs to"},
{"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"}
]
}
dataset is the trait's PublishFreeze, abbreviation its short
name (Phenotype.Post_publication_abbreviation, else the
pre-publication one) and url its page on GeneNetwork's website.
lod is lrs / 4.604.
Needs SQL_URI, same as /species above.
GET /groups/{group}/traits/{trait_id}/values- returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2'swqflask.api.router.trait_sample_data(the Publish branch) and gn-guile'sgn.data.dataset:get-bxd-publish-name-value-dict, generalized to any group rather than hardcoding BXD.trait_idis the trait'sPublishXRef.Id; it alone doesn't identify a trait uniquely across groups (only the pair of group andPublishXRef.Iddoes), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait'sPublishFreezeisn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a#[gnauth(secured)]endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits.
json
{
"data": [
{"name": "BXD1", "value": 61.400001525878906},
{"name": "BXD11", "value": 53.099998474121094},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"}
]
}
Needs SQL_URI, same as /species above.
GET /groups/{group}/datasets- returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2'swqflask.api.router.get_datasets_for_group. 404 if no group matches. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412",
"name2": "B6D2ONCILM_Apr12",
"full_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"short_name": "HEI ONC Retina B6D2 (Apr12) RankInv",
"create_time": "2012-05-17", "public": 1, "confidentiality": 0,
"data_scale": "log2", "tissue": "Retina mRNA"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group these datasets belong to"}
]
}
tissue is what the dataset was measured in (the Tissue its
ProbeFreeze points to), also on GET /datasets/{dataset} and
GET /species/{species}/datasets.
GET /mapping-methods- the mapping tools GeneNetwork offers, theMappingMethodtable in id order:qtlreaper,Rqtl,HappyR,PLINKandFastMap. It includes tools no group currently uses (PLINK, FastMap); a group'smapping_method(seeGET /groups) names the one offered for it. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 1, "name": "qtlreaper"},
{"id": 2, "name": "Rqtl"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"}
]
}
GET /accessions/{accession}- resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without theGNprefix. It names the dataset (InfoFiles.InfoPageName), its page title, itscategory(publishfor a phenotype dataset,probesetfor an expression dataset, null if the name matches neither) and thegroupit belongs to. 400 if the text is not a number, 404 if no dataset has the accession. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"accession": 602, "name": "BXDPublish",
"title": "BXD Published Phenotypes", "category": "publish",
"group": "BXD"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/groups/BXD", "description": "The group the dataset belongs to"},
{"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"}
]
}
GET /publications/{pubmed_id}/traits- returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as/groups/{group}/traits). 404 if no publication has that id; a publication with no traits is an empty list. Unlike/groups/{group}/traits, the datasets are authorization-checked as inGET /traits/{dataset}: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"id": 17967, "phenotype_id": 16502, "publication_id": 13584,
"mean": 6038.655919692095, "locus": "rsm10000012868",
"lrs": 10.535408722907198, "lod": 2.2883164037591652,
"additive": -647.5889092087912, "nsamples": 34,
"pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...",
"post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...",
"original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824,
"title": "Evaluation of heritable determinants of blood and brain serotonin homeos...",
"authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...",
"journal": "Genes, brain, and behavior", "volume": null,
"pages": "247-260", "month": "Mar", "year": "2014",
"dataset": "BXDPublish", "abbreviation": "5HT_Blood_M",
"url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"}
]
}
GET /datasets/{dataset}- returns every column of the matching row of theProbeSetFreezetable, matched againstName,Name2,FullName,ShortNameor numericId(matching GeneNetwork2'swqflask.api.router.get_dataset_info's matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under/groups/{group}. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M",
"name2": "BXDMicroArray_ProbeSet_August03",
"full_name": "Brain U74Av2 08/03 MAS5",
"short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01",
"public": 0, "confidentiality": 0, "data_scale": "log2",
"tissue": "Brain mRNA"
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"}
]
}
GET /datasets/{dataset}/probesets- returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of/datasets/{dataset}/probesets/{probeset}(same fields, same join, noprobesetfilter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset) rather than once per probeset -- see docs/gn-auth.md. NeedsSQL_URI, same as/speciesabove.
json
{
"data": [
{"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g",
"description": "CD3d antigen, gamma polypeptide", "chr": "9",
"mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619",
"lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269,
"additive": -0.28515625},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"}
]
}
GET /datasets/{dataset}/probesets/{probeset}- returns one expression (ProbeSet) trait'sProbeSetXRefmapping stats plus itsProbeSetgene annotation, matching GeneNetwork2'swqflask.api.router.get_trait_info's ProbeSet branch.ProbeSetXRefhas no single Id column of its own (its primary key isDataId; the natural key is the pair(ProbeSetFreezeId, ProbeSetId)), so a trait here is identified by dataset name plusProbeSet.Namerather than a single numeric id, unlike/groups/{group}/traits/{trait_id}. 404 if no dataset matches, or if the dataset has no such probeset.
json
{
"data": {
"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g",
"description": "CD3d antigen, gamma polypeptide", "chr": "9",
"mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619",
"lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269,
"additive": -0.28515625
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"},
{"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"}
]
}
Needs SQL_URI, same as /species above.
GET /datasets/{dataset}/probesets/{probeset}/values- returns one expression trait's value in each strain in the dataset, matching GeneNetwork2'swqflask.api.router.trait_sample_data's ProbeSet branch, addressed the same way as/datasets/{dataset}/probesets/{probeset}. 404 if no dataset matches, or if the dataset has no such probeset.
json
{
"data": [
{"name": "B6D2F1", "value": 5.742000102996826},
{"name": "BXD1", "value": 6.414000034332275},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"}
]
}
Needs SQL_URI, same as /species above.
/traits -- ProbeSet and Publish data through one URL shape
Everything above addresses ProbeSet (expression) and Publish
(phenotype) traits differently: /datasets/{dataset}/probesets/...
vs. /groups/{group}/traits/.... A trait is a trait either way --
a dataset, a trait within it, and that trait's per-strain values -- so
/traits/... addresses both the same way, (dataset, trait),
matching GeneNetwork2's own generic
wqflask.api.router.trait_sample_data/get_trait_info. This is
additional, not a replacement: the endpoints above are unchanged and
keep working exactly as documented.
dataset is resolved against ProbeSetFreeze first, then
PublishFreeze (db::resolve_trait_dataset) -- whichever matches
decides the category, so a bare trait name/id is never guessed across
categories, only the (much smaller, auditable) set of dataset names
is. A Publish dataset can be named either way GeneNetwork2 itself
accepts: the PublishFreeze's own Name/ShortName/Id (e.g.
"BXDPublish"), or its owning group's Name/InbredSetName/
FullName/Id (e.g. "BXD") -- a group normally has only one
PublishFreeze, so the two are near-interchangeable in practice.
GET /traits/{dataset}- every trait indataset, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset), same as/datasets/{dataset}/probesets. This is still more than/groups/{group}/traitschecks for the same phenotype data: both endpoints require a token now, but only this one also checksPublishFreeze.public/confidentialityand, for a non-public dataset, gn-auth --/groups/{group}/traits(unchanged) stops at the token.
json
{
"data": [
{"category": "probeset", "probeset_id": 1, "name": "100001_at",
"symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide",
"chr": "9", "mb": 44.970689, "mean": 5.48794285714286,
"locus": "rs13480619", "lrs": 12.590069931048,
"lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": []
}
or, for a Publish dataset:
json
{
"data": [
{"category": "publish", "id": 10001, "phenotype_id": 4,
"publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159",
"lrs": 13.497491147108706, "lod": 2.931687912056626,
"additive": 2.3944443506944437, "nsamples": 34,
"pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...",
"original_description": "Original post publication description: Cerebellum weight [mg]",
"units": "mg", "pubmed_id": 11438585,
"title": "Genetic control of the mouse cerebellum: identification of quantitative...",
"authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience",
"volume": "21", "pages": "5099-5109", "month": null, "year": "2001",
"dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight",
"url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": []
}
GET /traits/{dataset}/{trait}- one trait's metadata, tagged by category (ProbeSetInfo's gene annotation orTraitInfo's publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other).traitisProbeSet.Namefor a ProbeSet dataset, orPublishXRef.Idfor a Publish dataset (a non-numerictraitagainst a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. NeedsSQL_URI, same as/speciesabove.
json
{
"data": {
"category": "probeset", "probeset_id": 1, "name": "100001_at",
"symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide",
"chr": "9", "mb": 44.970689, "mean": 5.48794285714286,
"locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697,
"p_value": 0.269, "additive": -0.28515625
},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"}
]
}
GET /traits/{dataset}/{trait}/values- that trait's value in each strain, as{name, value}pairs regardless of category (TraitValueandProbeSetValueare identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as/traits/{dataset}/{trait}.
json
{
"data": [
{"name": "BXD1", "value": 61.400001525878906},
{"name": "BXD11", "value": 53.099998474121094},
...
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"}
]
}
- GET /search - free-text search across GN traits (see
docs/search-engines.md for switching between the gn3 proxy and
the native engine); hits keep gn3's field names and are wrapped in
the envelope like every other response. Parameters: q
(xapian syntax, e.g. serotonin AND midbrain AND group:BXD), type
(phenotype or gene), limit (default 20, at most 100), and
optionally sort (lrs or lod, which order identically) with
order (desc, the default, or asc), and page (default 1) for
the next limit hits.
Three fields are added to every hit: lod (lrs / 4.604) and, for
phenotype hits, locus (the peak marker's name) and abbreviation
(the trait's short name), both from the database (null for other
hits, or when the database is unreachable). pubmed_id is gn3's. With
sort, gn3 is asked for its first 1000 matches by relevance, those
are ranked, and page/limit are applied to the ranked list, so
page=2 continues where page 1 stopped; hits with no lrs come
last. Without sort, page is gn3's own. For example, the highest-LOD trait about serotonin in BXD
midbrain: /search?q=serotonin AND midbrain AND group:BXD&sort=lod.
Tissue and sex are not fields: they appear in each hit's description,
so match them there.
Needs a reachable gn3, configured via the GN3_URL config key
(default http://127.0.0.1:5003/api/).
json
{
"data": [
{"name": "17994", "dataset": "BXDPublish",
"dataset_fullname": "BXD Published Phenotypes", "group": "BXD",
"species": "mouse", "authors": ["Ye R", "Carneiro AM", ...],
"year": 2014,
"description": "Central nervous system, protein expression: Midbrain serotonin transport...",
"mean": 1627.7004466337316, "additive": 233.98946836805567,
"lrs": 16.49128337950074, "lod": 3.5819468678324804,
"locus": "rsm10000013922", "abbreviation": "SERT_MB_M",
"geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD",
"pubmed_id": 24405699,
"pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."}
],
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": ["lod = lrs / 4.604"],
"next": [
{"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"},
{"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"}
]
}
GET /openapi.json- returns an OpenAPI 3.1 document describing the endpoints above; see OpenAPI below.
/compute/gemma
Maps a QTL by running the real gemma-wrapper/GEMMA binaries: a
K-matrix step, then an LMM association step, both --loco, over a
genotype file, a SNP annotation file and a list of trait values. See
src/gemma.rs for the invocation itself and why it's shaped this way
(found by running GeneNetwork2's and GeneNetwork3's own gemma code for
real against the BXD fixtures -- GeneNetwork3's always errors out;
GeneNetwork2's non-LOCO branch does too; only --loco works).
This runs synchronously inside the request -- there is no job queue --
and it runs sequentially, not per-chromosome in parallel. Not
feasible past roughly 100,000 markers at this point. The BXD
fixtures (7,320 markers, LOCO across 20 chromosomes) already take
about 19 seconds; some real GeneNetwork datasets have upwards of 1
million markers, which this endpoint cannot handle yet -- the request
would simply run for a very long time inside one HTTP call. Making
that feasible needs either parallelising the per-chromosome LOCO jobs
(gemma-wrapper --parallel, which gn-guile's own precompute script
already uses) or moving the computation out of the request/response
cycle entirely (a job queue, or an offline precompute step, the way
gn-guile's own scripts/precompute/run-gemma.scm already does it).
{
"data": [
[1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, 0.2178407],
...
[1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, 0.0003520103]
],
"meta": {
"state": "complete",
"took_ms": 18456,
"request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc",
"columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "p_lrt"],
"num_rows": 7320,
"num_cols": 11
},
"notes": [
"Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.",
"Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk",
"GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9"
],
"next": []
}
(data has one row per marker -- 7,320 rows in this real run against
the BXD fixtures, trimmed to two here. Each row is positional, in
meta.columns' order, rather than a {"chr": ..., "rs": ..., ...}
object per row -- cuts the repeated key names out of a response this
large. The last two notes are the exact gemma-wrapper invocations
used (paths shortened here for readability) -- -gk only on the
kinship step, -lmm only on the GWA step, never both together; see
src/gemma.rs's compute_gwa doc comment for why that distinction
matters. lmm defaults to 9, matching GeneNetwork2's and gn-guile's
own convention; see the field's own doc comment in
src/routes/gemma.rs for what's actually been verified about it.)
/compute/gemma/status/{run_id} -- polling a run's progress
Given the endpoint above can genuinely take minutes, pass ?id= on
the POST and poll this on a second connection while it's in flight
to show progress, the same log-tailing shape GeneNetwork2/3 use for
R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the
run's id becomes a log file ({cache_dir}/{id}.log); gemma-wrapper's
own stderr (its --json-mode progress banner, chromosome list,
"Executing: ...", per-step info -- its stdout carries only the one
final JSON record, so there's nothing to stream there) is appended to
it line by line as both the kinship and GWA steps run; once the whole
request finishes, "GN4: computation complete" is appended as a
sentinel so a poller can tell "still running" from "done". run_id
must be 1-64 ASCII letters/digits/-/_ -- it's a path component, so
this is enforced on both this endpoint and the ?id= that names it.
id must also be unique: if a log already exists for it (another run
still in progress, or a finished one never polled/cleaned up), the
POST is refused with 409 rather than silently overwriting it.
POST /compute/gemma?id=my-run-1 (on one connection; blocks until done)
GET /compute/gemma/status/my-run-1?peek=0 (on another, while the first is in flight)
{
"data": {
"run_id": "my-run-1",
"lines": [
"gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024",
"Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022",
"Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..."
],
"pointer": 612
},
"meta": {"state": "complete", "took_ms": 1, "request_id": "..."},
"notes": [],
"next": []
}
Pass pointer back as ?peek= on the next poll; lines is empty
(not an error) when nothing new has been written since. An unknown
run_id (never started, or the main request hasn't written its log
yet) is a 404, not an empty response -- genuinely different from
"nothing new yet".
/compute/gemma/ravanan -- the same mapping, run per-chromosome via ravanan
Same request/response shape as /compute/gemma, same two gemma-wrapper
steps (kinship, then GWA, both --loco), but driven by
ravanan running a CWL workflow
compiled on every call (via ccwl compile, see below) from the ccwl
source at cwl/gemma/gemma-loco.scm, instead of this server's own
direct subprocess calls -- an alternative runner for the same
computation, kept side by side with /compute/gemma so the two can be
compared (correctness, timing). Unlike gemma-wrapper's own --loco
mode, which fans its per-chromosome kinship/GWA jobs out internally via
GNU parallel, this workflow has ravanan itself do that fan-out: it
scatters one gemma-wrapper --chromosomes <chr> --no-parallel ...
invocation per chromosome, each restricted to exactly the chromosome
it was given (gemma-wrapper itself still does the correct "exclude
this chromosome from kinship, test only its markers" LOCO logic,
against the full genotype/SNP files -- --chromosomes only
overrides which chromosomes gemma-wrapper loops over, not the
per-chromosome math itself). This workflow is written in
ccwl at cwl/gemma/gemma-loco.scm,
not raw CWL; compute_ravanan compiles it to CWL with ccwl compile
fresh on every call (cheap -- pure macro expansion, no Guix build
involved) into a generated, throwaway file under ravanan_cache_dir,
never checked into the repo. compute_ravanan computes the chromosome
list from snps_file itself (gemma-wrapper's own get_chromosomes,
reproduced as chromosomes_in) and passes it as the workflow's
chromosomes input, since CWL's scatter needs a concrete array at
invocation time. ravanan_cmd, ccwl_cmd, gemma_ravanan_workflow
(the ccwl source, not a CWL file), gemma_ravanan_manifest
(ravanan's own required --guix-manifest, a Guix manifest naming
gemma-wrapper, at cwl/gemma/manifest.scm) and ravanan_cache_dir
(its own cache directory, separate from gemma_cache_dir, for pheno
files, run logs, the compiled workflow, and ravanan's own --store)
are configurable the same way as the rest of src/config.rs's layered
config.
GET /compute/gemma/status/{run_id} is shared between both endpoints
and checks whichever cache directory the run_id actually logged to.
Exploratory, and genuinely rough around the edges -- three real bugs
turned up building this, none of them in /compute/gemma's own
direct-subprocess path:
- ravanan resolves
gemma-wrapperby building a Guix profile from--guix-manifestagainst the ambient Guix process (GUIX_PACKAGE_PATH, no--guix-channels) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package inguix-bioinformaticstoo new for the pinned base Guix. This needsGUIX_PACKAGE_PATHset in whatever environment starts the genenetwork4 server itself (inherited by theravananchild process genenetwork4 spawns); it is not somethingcompute_ravanansets. - ravanan's
--storeneedsfiles/,logs/anddata/subdirectories to already exist; it does not create them itself (compute_ravanandoes now). - A step scattered over exactly one input (
kinship'schr) needs an explicitscatterMethodeven though CWL itself only requires one for multiple scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it. - Every chromosome's
gwa.cwlrun shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in--chromosomes), so the 20 scatteredarchiveoutputs all collide under ravanan's optional--output-directorysymlink-everything-into-one-flat-dir step;compute_ravanansimply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own--storevia its stdout JSON record, never through--output-directory.
A real end-to-end run against the BXD fixtures (same rows as the
/compute/gemma example above, confirmed byte-for-byte including
rs31443144's exact values) took 16.7s wall-clock scattered across
20 chromosomes this way, vs. /compute/gemma's own ~19s and the
earlier sequential (non-scattered) ravanan workflow's ~21s -- a real,
if modest, win from letting ravanan's own scheduler fan the
chromosomes out instead of going through gemma-wrapper's internal GNU
parallel call, on top of no longer needing that path's own
PARALLEL_SHELL workaround at all (each scattered job's
--no-parallel gemma-wrapper call never invokes GNU parallel in the
first place, so the bash-time-keyword bug it worked around does not
arise here).
/compute/rqtl
Maps a QTL by running GeneNetwork3's own scripts/rqtl_wrapper.R (via
Rscript), copied into this repo verbatim: a scanone genome scan,
plus permutation thresholds if nperm > 0. Same request/response shape
as /compute/gemma -- synchronous, ?id=/GET .../status/{run_id}
progress-streaming reusing the same infrastructure (src/streaming.rs)
-- see src/rqtl.rs for the invocation itself, its documented scope
(no covariates, interval mapping or pair-scan scantwo -- just the
common single-scan case), and a fix made relative to GeneNetwork3:
its own cache-key hashing of the genotype/phenotype files never
actually looks at their contents (a key-name mismatch between
generate_rqtl_cmd and its real caller), so two different phenotypes
mapped with the same other options could silently reuse each other's
cached result there; fixed here by always hashing both files' contents.
Important: R/qtl's cross file format has no per-sample name
matching of its own -- samples must list values in exactly the
genotype file's sample-column order, or results are silently
misassigned to the wrong individual. This wrapper can't check that for
you.
{
"data": {
"markers": [
["rs31443144", "1", 3.010274, 0.0923496],
["rs6269442", "1", 3.492195, 0.1131095]
],
"permutation": null
},
"meta": {
"state": "complete",
"took_ms": 420,
"request_id": "...",
"columns": ["name", "chr", "pos", "lod"],
"num_rows": 6,
"num_cols": 4
},
"notes": [
"markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.",
"Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename <hash>-output.csv"
],
"next": []
}
(Real output, against the tiny 6-marker/6-sample BXD riset fixture in
tests/test_data/, trimmed to two markers. Each marker row is
positional, in meta.columns' order, rather than a
{"name": ..., "chr": ..., ...} object per row -- same
space-saving shape /compute/gemma uses for its own (much larger)
rows; a 4-way cross's extra mean/SE effect columns, if present, trail
each row and aren't named in meta.columns (see MarkerResult's doc
comment in src/rqtl.rs). permutation is null unless nperm was
set above 0; when it is, it's
{"values": [...], "suggestive": ..., "significant": ...}, the 67th-
and 95th-percentile thresholds GeneNetwork3 itself uses -- computed
here with numpy.percentile's default linear interpolation,
reproduced in src/rqtl.rs's percentile.)
OpenAPI
GET /openapi.json serves an OpenAPI 3.1 document covering every route.
It's built with utoipa and
utoipa-axum:
- Each handler carries a
#[utoipa::path(...)]attribute describing its method, path, parameters and responses; each response/request struct derivesutoipa::ToSchema(seeVersionInfo,db::Species,db::SpeciesDetail,error::ApiErrorinsrc/lib.rs/src/db.rs/src/error.rs). - Routes are registered with
utoipa_axum::routes!(handler)instead of plainaxum::routing::get(handler), on anOpenApiRouterinstead of a plainRouter, so each route's metadata is collected as it's added. ApiDoc(a#[derive(utoipa::OpenApi)]struct) supplies the document'sinfo(title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's.OpenApiRouter::split_for_parts()splits the builder back into a plainaxum::Router(returned fromapp()) and the assembledutoipa::openapi::OpenApivalue, which the/openapi.jsonhandler then just serializes withJson.ApiDoc's#[openapi(security(("bearer_token" = [])), ...)]makes a token the default for every operation, so an endpoint that needs one doesn't repeatsecurity(...)itself -- only the handful that must NOT require one (/,/version,/get-test-token) override it with an emptysecurity()of their own.error::Unauthorized(a#[derive(utoipa::IntoResponses)]type) is the shared 401 response every gated endpoint references by name in its ownresponses(...)list (responses(Unauthorized, (status = 200, ...), ...)) instead of writing out(status = 401, description = "...", body = ApiError)by hand each time.
Because the attributes are declared alongside the handlers they
describe, the spec can't silently drift out of sync with the code the
way a hand-maintained spec file could -- adding or changing a route
without updating its #[utoipa::path(...)] is a compile error (the
macro checks the response/schema types exist) or an obviously incomplete
diff, not something to notice weeks later.
The document is assembled once, when app() runs at process startup
(not at cargo build time, and not freshly on every request): the
#[utoipa::path]/#[derive(ToSchema)] macros expand to Rust code at
compile time, but the actual merging of all routes' metadata into one
OpenApi value happens the first time app() executes. Every
GET /openapi.json request after that just serializes the same
already-built value.
Success responses
Every success response is one JSON envelope (src/response.rs): the
data, next to facts about the response, remarks on the data, and
suggested follow-up calls. It is the success counterpart of the error
shape below.
{
"data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."},
"meta": {
"state": "complete",
"took_ms": 3,
"request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
},
"notes": [],
"next": [
{"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
{"href": "/groups?species=mouse", "description": "Groups of this species"}
]
}
datais the response proper: exactly what each endpoint below documents.meta.stateiscomplete, orpartialwhen the response is usable but something was left out or unavailable --notessays what. For example/searchispartialwhen the database could not be reached (hits come back withoutlocusandabbreviation) or when a sorted search ranked a full 1000-match window and more matches exist;/publications/{pubmed_id}/traitsispartialwhen traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure.meta.took_msis the server's time for the request, from when it started handling it; it excludes the network.meta.request_idis the request's id, also in thex-request-idresponse header and in any error body, so a report can be correlated with logs.notesare server-authored remarks ondata, e.g.lod = lrs / 4.604on responses that carrylod. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's.nextlists follow-up calls, each a relativehrefthat resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large).GET /searchsuggests the next page and the top hit's metadata.
/ returns the plain text Hello, world! and /openapi.json the
OpenAPI document itself; neither is wrapped. In /openapi.json every
other success response is an Envelope_* schema (a test enforces this).
Error handling
Every endpoint uses the same ApiError JSON shape on error, modeled on
RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see
src/error.rs). /species and /species/{species} can actually produce
one today (database errors, or a 404 for an unknown species); / and
/version document the same 500 response in /openapi.json for
consistency, even though neither has a failure path currently (no
database access, no I/O):
{
"type": "/errors/species_not_found",
"title": "Species not found",
"status": 404,
"code": "species_not_found",
"detail": "No species matching 'nonexistent'.",
"operation_id": "species_by_name",
"spec_url": "/openapi.json",
"request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
"took_ms": 2,
"next": "/species"
}
next is omitted (null) when there's no obviously related endpoint to
suggest, e.g. a hypothetical error from /:
{
"type": "/errors/internal_error",
"title": "Internal error",
"status": 500,
"code": "internal_error",
"detail": "example failure for /",
"operation_id": "hello_world",
"spec_url": "/openapi.json",
"request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
"took_ms": 1,
"next": null
}
typeandspec_urlare relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allowstypeto be a relative URI reference.spec_urlis always/openapi.json, on every error from every endpoint.operation_idis the OpenAPI operation that failed (same name as in/openapi.json).request_idis the request'sx-request-id, generated per request (or echoed back if the client already sent one) bySetRequestIdLayer/PropagateRequestIdLayerinsrc/lib.rs, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token.took_msis the server's time for the request up to the error, the same measure as a success response'smeta.took_ms(see Success responses). RFC 7807 allows extension members like this one,request_id,codeandnextnext to its standard ones.nextis an optional hint at a related endpoint to try instead; e.g. a 404 from/species/{species}setsnextto/species, so a caller with a wrong species name can discover the valid ones.
Running with cargo
Requires a Rust toolchain with cargo (edition 2024).
cargo run
The server listens on 0.0.0.0:3000 by default. Set GENENETWORK4_PORT
to use a different port, or GENENETWORK4_BIND to bind a specific
address instead of every interface:
GENENETWORK4_PORT=8080 cargo run
GENENETWORK4_BIND=127.0.0.1 cargo run
Development
Every endpoint needs a bearer token (see the note under
Endpoints), which normally means getting one from
gn-auth yourself. For local development, genenetwork4 can hand you one
instead: POST /get-test-token calls gn-auth's own password grant
(POST /auth/token) for gn-auth's well-known development user and
returns the result. It needs no setting and is not tied to a mode.
cargo run
Get a token, then use it against a real endpoint:
curl -X POST http://127.0.0.1:3000/get-test-token
{
"data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"},
"meta": {"state": "complete", "took_ms": 41, "request_id": "..."},
"notes": [],
"next": []
}
TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])')
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species
{
"data": [
{"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"},
...
],
"meta": {"state": "complete", "took_ms": 2, "request_id": "..."},
"notes": [],
"next": [
{"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
]
}
The endpoint is registered by default and listed in /openapi.json
(GENENETWORK4_TEST_TOKEN=false switches it off, see below), and decides
per request whether to answer: only a caller on this machine gets a
token. That means the TCP connection's peer address is in
127.0.x.x (or is ::1), and the request carries no X-Forwarded-For,
Forwarded or X-Real-IP header. Anything else, including a request
whose peer address is unknown, is a 403. The peer address comes from
the connection itself, never from a header a client can set; the proxy
headers are used only to refuse a request that claims to have been
forwarded. This is why the check also refuses forwarded requests: a
reverse proxy running on the same host connects from 127.0.0.1 for
every client, so without it the whole internet would look local. A proxy
that forwards without setting any of those headers would defeat the
check, so don't put one in front of a server whose gn-auth has the dev
user (on a production gn-auth the dev user normally doesn't exist, and
the request just fails with a 502).
Behind nginx or in a container
The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here):
- Container on a bridge network, port published. The server sees
the bridge gateway (something like
172.17.0.1) as the peer, not127.0.0.1. Every request through the published port is a403, whether it comes via nginx or from the host itself. It still works from inside the container, e.g.docker exec <container> curl -X POST http://127.0.0.1:3000/get-test-token. - Container sharing the host's network, nginx on the same host.
nginx connects from
127.0.0.1, so the peer check passes for every client. The server then relies on refusing requests that carryX-Forwarded-For,ForwardedorX-Real-IP, and a minimalproxy_passsets none of them. Without one of the measures below, anyone who can reach the site could get a token.
For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third:
- Block the path in nginx, independent of genenetwork4:
nginx
location = /get-test-token { return 404; }
- Make nginx say it is forwarding, so genenetwork4 refuses proxied requests (any one of the three headers is enough):
nginx
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- Switch the endpoint off in genenetwork4 itself with
GENENETWORK4_TEST_TOKEN=false(an environment variable, or aGENENETWORK4_TEST_TOKEN=falseline inGN4_CONF). The route is then not registered at all:POST /get-test-tokenis a404and the endpoint is absent from/openapi.json. The setting fails closed: onlytrue,1,yesoron(any case) keep the endpoint on, and anything else --false, an empty value, or a typo such asflase-- turns it off. It is on by default, so a local setup needs nothing.
Also relevant: the token is for gn-auth's dev user
(GN_AUTH_DEV_USERNAME). On a production gn-auth that user normally
doesn't exist, so a request that does get through fails with a 502
instead of returning a token. That depends on gn-auth's data, not on
anything in genenetwork4.
The credentials used (gn-auth's dev client id/secret and dev user
email/password, from flask init-dev-clients/gn_auth/wsgi.py) are
publicly known and the same for every gn-auth checkout by default --
override GN_AUTH_DEV_CLIENT_ID, GN_AUTH_DEV_CLIENT_SECRET,
GN_AUTH_DEV_USERNAME and GN_AUTH_DEV_PASSWORD if your gn-auth
instance's dev fixtures differ. Needs a gn-auth reachable at
AUTH_SERVER_URL with AUTHLIB_INSECURE_TRANSPORT set (gn-auth itself
refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev
setting, not something genenetwork4 needs to know about).
The token has no exp claim, and neither genenetwork4 nor gn-auth
expires it by time or for lack of use. It stops verifying only when the
key that signed it leaves gn-auth's published key set: gn-auth signs with
a key for 7 days (JWKS_ROTATION_AGE_DAYS) and deletes it 14 days after
its creation (JWKS_DELETION_AGE_DAYS), so a token lasts roughly 7 to
14 days, depending on where in the key's life it was issued.
genenetwork4 caches the key set for an hour. When requests start
failing with 401 invalid_token, fetch a new token.
The test suite uses this same endpoint automatically -- see
Tests below.
Configuration is layered: built-in defaults, then GN4_CONF and
GN4_SECRETS files of KEY=VALUE lines, then environment variables
(which always win). Keys: SQL_URI (MariaDB), GN3_URL (GeneNetwork3
base URL for proxy endpoints), GN3_TIMEOUT_SECS (upstream call
timeout, default 30), GENENETWORK4_PORT, GENENETWORK4_BIND, and
GENENETWORK4_TEST_TOKEN (false turns POST /get-test-token off; see
"Behind nginx or in a container").
Running with Guix
guix build -f guix.scm
builds the package (running both test suites, see below) and prints the
resulting store path; the binary is at <store path>/bin/genenetwork4.
guix shell -D -f guix.scm
drops you into a shell with all build dependencies available (cargo,
rust, python, git), for running cargo build/cargo test/cargo run
directly. This shell is not isolated from the network or from your
normal environment, which cargo build/cargo run need to reach
crates.io.
A fully isolated container (-C) normally needs -N to still reach the
network from inside it, but on at least one guix build
(1.5.0-6.64d4de2) combining -D with -N causes -D to silently stop
providing the package's build dependencies (the shell then has only
genenetwork4 itself, not cargo/rust/etc.). If guix shell -C -D -N -f
guix.scm fails with "cargo: command not found", drop -C/-N and use
the plain guix shell -D -f guix.scm above instead.
Tests
Two test suites exercise the same endpoints:
tests/*.rs- Rust integration tests that call the router in-process (no socket), viatower::ServiceExt::oneshot. Run with:
cargo test
Both suites check the envelope on every success response: the Rust
helper tests/common::data(&body) and the Python helper
server_test_case.envelope_data(body) verify its shape and return
data. tests/envelope.rs (and test/python/test_envelope.py)
follow every next link the endpoints suggest;
tests/envelope_partial.rs checks the partial state without a
database; tests/openapi.rs checks that every JSON success response
is documented as an envelope.
test/python/test_*.py- Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with:
cargo build --release
GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python
Either suite can fetch its own token from a live gn-auth automatically
(see "Development" above) -- set AUTH_SERVER_URL before running
either command above, and nothing further needs setting up by hand. A
run without a reachable gn-auth still works, skipping (not failing)
every test that needed a token it couldn't get -- set
GN4_TEST_AUTH_TOKEN instead if you'd rather provide one manually.
Both suites also run automatically as part of guix build -f guix.scm
(the Rust tests in the check phase, the Python tests in a
python-rest-test phase after install).
Every test under tests/*.rs/test/python/test_*.py for an endpoint
that touches the database (which is most of them -- hello_world,
version, openapi and get_test_token's own disabled/wrong-config
cases are the exceptions) checks for a live MariaDB on 127.0.0.1:3306
first (see SQL_URI above) and skips with a message rather than
failing when none is reachable, since the Guix build sandbox has no
network/database access. Likewise, any test for an endpoint that
requires a token (every endpoint except those four) checks for one
first and skips if none is available -- see "Development mode" above
for how the test suite gets one automatically.
The Rust tests in tests/*.rs that hit the database issue their
requests through tests/common::get, which times the request and
prints a warning to stderr if it takes over a second -- e.g.
WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s). Warnings only
show with cargo test -- --nocapture (or on a failing test), since
cargo captures stdout/stderr for passing tests otherwise.
