about summary refs log tree commit diff

genenetwork4

A Rust/Axum-based API server for GeneNetwork.

Endpoints

Method Path Description
GET / Greeting.
GET /version Build info (hash, date).
GET /species All species.
GET /species/{species} One species by name or id.
GET /species/{species}/datasets ProbeSet datasets in a species.
GET /groups All groups (filterable).
GET /mapping-methods Mapping tools offered.
GET /accessions/{accession} Resolve a GN accession number.
GET /groups/{group} One group, by name or Id.
GET /groups/{group}/strains Strains/samples in a group.
GET /groups/{group}/case-attributes Case-attribute names and values.
GET /groups/{group}/traits Published (phenotype) traits.
GET /groups/{group}/traits/{trait_id} A published trait's metadata.
GET /groups/{group}/traits/{trait_id}/values A published trait's values.
GET /groups/{group}/datasets ProbeSet datasets in a group.
GET /publications/{pubmed_id}/traits Traits a publication reports.
GET /datasets/{dataset} One expression dataset.
GET /datasets/{dataset}/probesets Expression traits in a dataset.
GET /datasets/{dataset}/probesets/{probeset} An expression trait's metadata.
GET /datasets/{dataset}/probesets/{probeset}/values An expression trait's values.
GET /traits/{dataset} Every trait in a dataset.
GET /traits/{dataset}/{trait} One trait's metadata.
GET /traits/{dataset}/{trait}/values One trait's values.
POST /compute/correlation Correlate a trait (stub).
POST /compute/gemma Map a QTL with GEMMA (see below).
GET /compute/gemma/status/{run_id} Poll a gemma run's progress.
POST /compute/gemma/ravanan Map a QTL via ravanan (see below).
POST /compute/rqtl Map a QTL with R/qtl (see below).
GET /compute/rqtl/status/{run_id} Poll an rqtl run's progress.
POST /get-test-token Get a test bearer token (local).
GET /search Search traits (free-text).
GET /openapi.json OpenAPI 3.1 document.

Every endpoint above requires a valid gn-auth bearer token (Authorization: Bearer <token>) except /, /version, /openapi.json and /get-test-token itself -- see docs/gn-auth.md and the "Development" section below (POST /get-test-token) for how to get one for local testing. A missing or invalid token is a 401; which further, resource-level authorization (if any) an endpoint checks once a token is valid is a separate, per-endpoint matter covered in each entry below.

Every success response is wrapped in one envelope (see Success responses below), except / (a plain-text greeting) and /openapi.json. The example responses below are real ones, with long lists and strings abbreviated by ..., and took_ms and request_id shortened to plausible values.

Details, including example responses, below.

  • GET / - returns Hello, world!.
  • GET /version - returns a JSON record describing the running build:

json { "data": { "name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0, "release": 1, "date": "20261008", "git": "208eeac" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [] }

date and git are the HEAD commit's date and short hash, embedded at build time (see build.rs); they describe the commit the binary was built from, not the time it happened to be compiled.

  • GET /species - returns a JSON array of all species in the Species table, mirroring GeneNetwork3's gn3.db.species.get_all_species:

json { "data": [ {"name": "human", "menu_name": "Human (Homo sapiens, hg19)", "family": "Vertebrates"}, {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"} ] }

Needs a live MariaDB connection, configured via the SQL_URI environment variable (same config key GeneNetwork3 uses), e.g. mysql://user:password@host/db_webqtl. Defaults to mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl if unset.

  • GET /species/{species} - returns every column of the matching row of the Species table, matched against Name, FullName or SpeciesName (matching GeneNetwork2's wqflask.api.router.get_species_info, which returns the same match logic but only a 4-column subset). 404 if no species matches.

json { "data": { "id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus", "family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090, "order_id": 15 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] }

Needs SQL_URI, same as /species above.

  • GET /species/{species}/datasets - returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of /groups/{group}/datasets. GeneNetwork2's own species-scoped variant of wqflask.api.router.get_datasets_for_group still requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as /groups/{group}/datasets -- this returns dataset metadata (names, public/confidentiality flags), not trait data -- but a token is still required, like every endpoint. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse", "description": "The species these datasets belong to"} ] }

  • GET /groups - returns a JSON array of all groups in the InbredSet table, matching the shape of GeneNetwork2's GET /api/v_pre1/groups response:

json { "data": [ {"id": 25, "species_id": 4, "display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "name": "AD-cases-controls", "full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "public": 2, "mapping_method_id": null, "genetic_type": null, "family": null, "species": "human", "mapping_method": null}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species", "description": "The species the groups belong to (filter groups with ?species=<name>)"} ] }

family is the population category and species the species' short name as /species lists it. mapping_method is the name of the mapping tool offered for the group (the MappingMethod row its mapping_method_id points to, see GET /mapping-methods), null for a group with none. Three optional filters, combined with AND and all case-insensitive: family (contains), species (exact short name) and q (contained in the name, display name or full name). For example /groups?family=Reference Populations lists the reference populations, and /groups?species=rat&q=hxb the rat HXB/BXH sets.

"Group" is GeneNetwork's own alias for InbredSet -- see docs/genenetwork_datatypes.md for why. Needs SQL_URI, same as /species above.

  • GET /groups/{group} - returns every column of the matching row of the InbredSet table, matched against InbredSetName, Name, FullName or numeric Id (matching GeneNetwork2's wqflask.api.router.get_group_info, which matches the same way but not by Id, plus gn-guile's gn.data.population:population-info, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (see docs/genenetwork_datatypes.md); only the MariaDB column names themselves stay InbredSet*. 404 if no group matches.

json { "data": { "id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD", "species_id": 1, "full_name": "BXD Family", "public": 2, "mapping_method_id": "1", "genetic_type": "riset", "family": "Reference Populations (replicate average, SE, N)", "family_order": 1, "menu_order_id": 0.0, "group_code": "BXD", "description": "<p>...</p>", "mapping_method": "qtlreaper" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}, {"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"}, {"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"} ] }

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/strains - returns the strains/samples belonging to a group, matching gn-guile's gn.data.strains:strain-id-names. 404 if no group matches. Takes an optional used_for_mapping query parameter to filter to only strains used for mapping (gn-guile's strain-id-names takes the same flag as a keyword argument):

json { "data": [ {"id": 1, "name": "B6D2F1", "used_for_mapping": false}, {"id": 2, "name": "C57BL/6J", "used_for_mapping": false}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these strains belong to"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"} ] }

GET /groups/BXD/strains?used_for_mapping=true excludes BXD's parent strains (B6D2F1, C57BL/6J, DBA/2J), which aren't used for mapping. Needs SQL_URI, same as /species above.

  • GET /groups/{group}/case-attributes - returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3's gn3.api.case_attributes splits this into separate .../names and .../values endpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns 200 with both lists empty, not a 404. Needs SQL_URI, same as /species above.

json { "data": { "names": ["Epoch", "RRID", ...], "values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...] }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"} ] }

  • GET /groups/{group}/traits - returns every published (phenotype) trait belonging to a group, the listing equivalent of /groups/{group}/traits/{trait_id} (same fields, same join, no trait_id filter). 404 if no group matches. If the group has any published traits, its PublishFreeze row is checked the same way as /traits/{dataset} (see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's own wqflask.api.router.fetch_traits, which defaults to unbounded too. Needs SQL_URI, same as /species above.

The optional abbreviation parameter keeps only the traits with that short name, e.g. /groups/BXD/traits?abbreviation=5HT_MB_M finds trait 17985 (an unknown abbreviation is an empty list).

json { "data": [ {"id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD", "description": "The group these traits belong to"} ] }

  • GET /groups/{group}/traits/{trait_id} - returns one published (phenotype) trait's PublishXRef mapping stats plus its Phenotype description/units and Publication reference, based on the data explored in precompute-publishdata.gmi (e.g. its "More metadata" section, which identifies exactly this join as worth surfacing). trait_id is PublishXRef.Id, scoped by group for the same reason as /groups/{group}/traits/{trait_id}/values. 404 if no group matches, or if the group has no such trait; 403 if the trait's PublishFreeze isn't public and the caller isn't authorized for it (see docs/gn-auth.md).

json { "data": { "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"}, {"href": "/groups/BXD", "description": "The group this trait belongs to"}, {"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"} ] }

dataset is the trait's PublishFreeze, abbreviation its short name (Phenotype.Post_publication_abbreviation, else the pre-publication one) and url its page on GeneNetwork's website. lod is lrs / 4.604.

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/traits/{trait_id}/values - returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2's wqflask.api.router.trait_sample_data (the Publish branch) and gn-guile's gn.data.dataset:get-bxd-publish-name-value-dict, generalized to any group rather than hardcoding BXD. trait_id is the trait's PublishXRef.Id; it alone doesn't identify a trait uniquely across groups (only the pair of group and PublishXRef.Id does), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait's PublishFreeze isn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a #[gnauth(secured)] endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits.

json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"} ] }

Needs SQL_URI, same as /species above.

  • GET /groups/{group}/datasets - returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2's wqflask.api.router.get_datasets_for_group. 404 if no group matches. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these datasets belong to"} ] }

tissue is what the dataset was measured in (the Tissue its ProbeFreeze points to), also on GET /datasets/{dataset} and GET /species/{species}/datasets.

  • GET /mapping-methods - the mapping tools GeneNetwork offers, the MappingMethod table in id order: qtlreaper, Rqtl, HappyR, PLINK and FastMap. It includes tools no group currently uses (PLINK, FastMap); a group's mapping_method (see GET /groups) names the one offered for it. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 1, "name": "qtlreaper"}, {"id": 2, "name": "Rqtl"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"} ] }

  • GET /accessions/{accession} - resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without the GN prefix. It names the dataset (InfoFiles.InfoPageName), its page title, its category (publish for a phenotype dataset, probeset for an expression dataset, null if the name matches neither) and the group it belongs to. 400 if the text is not a number, 404 if no dataset has the accession. Needs SQL_URI, same as /species above.

json { "data": { "accession": 602, "name": "BXDPublish", "title": "BXD Published Phenotypes", "category": "publish", "group": "BXD" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group the dataset belongs to"}, {"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"} ] }

  • GET /publications/{pubmed_id}/traits - returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as /groups/{group}/traits). 404 if no publication has that id; a publication with no traits is an empty list. Unlike /groups/{group}/traits, the datasets are authorization-checked as in GET /traits/{dataset}: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. Needs SQL_URI, same as /species above.

json { "data": [ {"id": 17967, "phenotype_id": 16502, "publication_id": 13584, "mean": 6038.655919692095, "locus": "rsm10000012868", "lrs": 10.535408722907198, "lod": 2.2883164037591652, "additive": -647.5889092087912, "nsamples": 34, "pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824, "title": "Evaluation of heritable determinants of blood and brain serotonin homeos...", "authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...", "journal": "Genes, brain, and behavior", "volume": null, "pages": "247-260", "month": "Mar", "year": "2014", "dataset": "BXDPublish", "abbreviation": "5HT_Blood_M", "url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"} ] }

  • GET /datasets/{dataset} - returns every column of the matching row of the ProbeSetFreeze table, matched against Name, Name2, FullName, ShortName or numeric Id (matching GeneNetwork2's wqflask.api.router.get_dataset_info's matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under /groups/{group}. Needs SQL_URI, same as /species above.

json { "data": { "id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M", "name2": "BXDMicroArray_ProbeSet_August03", "full_name": "Brain U74Av2 08/03 MAS5", "short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01", "public": 0, "confidentiality": 0, "data_scale": "log2", "tissue": "Brain mRNA" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"} ] }

  • GET /datasets/{dataset}/probesets - returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of /datasets/{dataset}/probesets/{probeset} (same fields, same join, no probeset filter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset) rather than once per probeset -- see docs/gn-auth.md. Needs SQL_URI, same as /species above.

json { "data": [ {"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"} ] }

  • GET /datasets/{dataset}/probesets/{probeset} - returns one expression (ProbeSet) trait's ProbeSetXRef mapping stats plus its ProbeSet gene annotation, matching GeneNetwork2's wqflask.api.router.get_trait_info's ProbeSet branch. ProbeSetXRef has no single Id column of its own (its primary key is DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)), so a trait here is identified by dataset name plus ProbeSet.Name rather than a single numeric id, unlike /groups/{group}/traits/{trait_id}. 404 if no dataset matches, or if the dataset has no such probeset.

json { "data": { "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"}, {"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"} ] }

Needs SQL_URI, same as /species above.

  • GET /datasets/{dataset}/probesets/{probeset}/values - returns one expression trait's value in each strain in the dataset, matching GeneNetwork2's wqflask.api.router.trait_sample_data's ProbeSet branch, addressed the same way as /datasets/{dataset}/probesets/{probeset}. 404 if no dataset matches, or if the dataset has no such probeset.

json { "data": [ {"name": "B6D2F1", "value": 5.742000102996826}, {"name": "BXD1", "value": 6.414000034332275}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"} ] }

Needs SQL_URI, same as /species above.

/traits -- ProbeSet and Publish data through one URL shape

Everything above addresses ProbeSet (expression) and Publish (phenotype) traits differently: /datasets/{dataset}/probesets/... vs. /groups/{group}/traits/.... A trait is a trait either way -- a dataset, a trait within it, and that trait's per-strain values -- so /traits/... addresses both the same way, (dataset, trait), matching GeneNetwork2's own generic wqflask.api.router.trait_sample_data/get_trait_info. This is additional, not a replacement: the endpoints above are unchanged and keep working exactly as documented.

dataset is resolved against ProbeSetFreeze first, then PublishFreeze (db::resolve_trait_dataset) -- whichever matches decides the category, so a bare trait name/id is never guessed across categories, only the (much smaller, auditable) set of dataset names is. A Publish dataset can be named either way GeneNetwork2 itself accepts: the PublishFreeze's own Name/ShortName/Id (e.g. "BXDPublish"), or its owning group's Name/InbredSetName/ FullName/Id (e.g. "BXD") -- a group normally has only one PublishFreeze, so the two are near-interchangeable in practice.

  • GET /traits/{dataset} - every trait in dataset, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (auth::authorise_dataset), same as /datasets/{dataset}/probesets. This is still more than /groups/{group}/traits checks for the same phenotype data: both endpoints require a token now, but only this one also checks PublishFreeze.public/confidentiality and, for a non-public dataset, gn-auth -- /groups/{group}/traits (unchanged) stops at the token.

json { "data": [ {"category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] }

or, for a Publish dataset:

json { "data": [ {"category": "publish", "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] }

  • GET /traits/{dataset}/{trait} - one trait's metadata, tagged by category (ProbeSetInfo's gene annotation or TraitInfo's publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other). trait is ProbeSet.Name for a ProbeSet dataset, or PublishXRef.Id for a Publish dataset (a non-numeric trait against a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. Needs SQL_URI, same as /species above.

json { "data": { "category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"} ] }

  • GET /traits/{dataset}/{trait}/values - that trait's value in each strain, as {name, value} pairs regardless of category (TraitValue and ProbeSetValue are identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as /traits/{dataset}/{trait}.

json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"} ] } - GET /search - free-text search across GN traits (see docs/search-engines.md for switching between the gn3 proxy and the native engine); hits keep gn3's field names and are wrapped in the envelope like every other response. Parameters: q (xapian syntax, e.g. serotonin AND midbrain AND group:BXD), type (phenotype or gene), limit (default 20, at most 100), and optionally sort (lrs or lod, which order identically) with order (desc, the default, or asc), and page (default 1) for the next limit hits.

Three fields are added to every hit: lod (lrs / 4.604) and, for phenotype hits, locus (the peak marker's name) and abbreviation (the trait's short name), both from the database (null for other hits, or when the database is unreachable). pubmed_id is gn3's. With sort, gn3 is asked for its first 1000 matches by relevance, those are ranked, and page/limit are applied to the ranked list, so page=2 continues where page 1 stopped; hits with no lrs come last. Without sort, page is gn3's own. For example, the highest-LOD trait about serotonin in BXD midbrain: /search?q=serotonin AND midbrain AND group:BXD&sort=lod. Tissue and sex are not fields: they appear in each hit's description, so match them there.

Needs a reachable gn3, configured via the GN3_URL config key (default http://127.0.0.1:5003/api/).

json { "data": [ {"name": "17994", "dataset": "BXDPublish", "dataset_fullname": "BXD Published Phenotypes", "group": "BXD", "species": "mouse", "authors": ["Ye R", "Carneiro AM", ...], "year": 2014, "description": "Central nervous system, protein expression: Midbrain serotonin transport...", "mean": 1627.7004466337316, "additive": 233.98946836805567, "lrs": 16.49128337950074, "lod": 3.5819468678324804, "locus": "rsm10000013922", "abbreviation": "SERT_MB_M", "geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD", "pubmed_id": 24405699, "pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."} ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"}, {"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"} ] }

  • GET /openapi.json - returns an OpenAPI 3.1 document describing the endpoints above; see OpenAPI below.

/compute/gemma

Maps a QTL by running the real gemma-wrapper/GEMMA binaries: a K-matrix step, then an LMM association step, both --loco, over a genotype file, a SNP annotation file and a list of trait values. See src/gemma.rs for the invocation itself and why it's shaped this way (found by running GeneNetwork2's and GeneNetwork3's own gemma code for real against the BXD fixtures -- GeneNetwork3's always errors out; GeneNetwork2's non-LOCO branch does too; only --loco works).

This runs synchronously inside the request -- there is no job queue -- and it runs sequentially, not per-chromosome in parallel. Not feasible past roughly 100,000 markers at this point. The BXD fixtures (7,320 markers, LOCO across 20 chromosomes) already take about 19 seconds; some real GeneNetwork datasets have upwards of 1 million markers, which this endpoint cannot handle yet -- the request would simply run for a very long time inside one HTTP call. Making that feasible needs either parallelising the per-chromosome LOCO jobs (gemma-wrapper --parallel, which gn-guile's own precompute script already uses) or moving the computation out of the request/response cycle entirely (a job queue, or an offline precompute step, the way gn-guile's own scripts/precompute/run-gemma.scm already does it).

{
  "data": [
    [1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, 0.2178407],
    ...
    [1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, 0.0003520103]
  ],
  "meta": {
    "state": "complete",
    "took_ms": 18456,
    "request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc",
    "columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "p_lrt"],
    "num_rows": 7320,
    "num_cols": 11
  },
  "notes": [
    "Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.",
    "Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk",
    "GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9"
  ],
  "next": []
}

(data has one row per marker -- 7,320 rows in this real run against the BXD fixtures, trimmed to two here. Each row is positional, in meta.columns' order, rather than a {"chr": ..., "rs": ..., ...} object per row -- cuts the repeated key names out of a response this large. The last two notes are the exact gemma-wrapper invocations used (paths shortened here for readability) -- -gk only on the kinship step, -lmm only on the GWA step, never both together; see src/gemma.rs's compute_gwa doc comment for why that distinction matters. lmm defaults to 9, matching GeneNetwork2's and gn-guile's own convention; see the field's own doc comment in src/routes/gemma.rs for what's actually been verified about it.)

/compute/gemma/status/{run_id} -- polling a run's progress

Given the endpoint above can genuinely take minutes, pass ?id= on the POST and poll this on a second connection while it's in flight to show progress, the same log-tailing shape GeneNetwork2/3 use for R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the run's id becomes a log file ({cache_dir}/{id}.log); gemma-wrapper's own stderr (its --json-mode progress banner, chromosome list, "Executing: ...", per-step info -- its stdout carries only the one final JSON record, so there's nothing to stream there) is appended to it line by line as both the kinship and GWA steps run; once the whole request finishes, "GN4: computation complete" is appended as a sentinel so a poller can tell "still running" from "done". run_id must be 1-64 ASCII letters/digits/-/_ -- it's a path component, so this is enforced on both this endpoint and the ?id= that names it. id must also be unique: if a log already exists for it (another run still in progress, or a finished one never polled/cleaned up), the POST is refused with 409 rather than silently overwriting it.

POST /compute/gemma?id=my-run-1       (on one connection; blocks until done)
GET  /compute/gemma/status/my-run-1?peek=0       (on another, while the first is in flight)
{
  "data": {
    "run_id": "my-run-1",
    "lines": [
      "gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024",
      "Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022",
      "Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..."
    ],
    "pointer": 612
  },
  "meta": {"state": "complete", "took_ms": 1, "request_id": "..."},
  "notes": [],
  "next": []
}

Pass pointer back as ?peek= on the next poll; lines is empty (not an error) when nothing new has been written since. An unknown run_id (never started, or the main request hasn't written its log yet) is a 404, not an empty response -- genuinely different from "nothing new yet".

/compute/gemma/ravanan -- the same mapping, run per-chromosome via ravanan

Same request/response shape as /compute/gemma, same two gemma-wrapper steps (kinship, then GWA, both --loco), but driven by ravanan running a CWL workflow compiled on every call (via ccwl compile, see below) from the ccwl source at cwl/gemma/gemma-loco.scm, instead of this server's own direct subprocess calls -- an alternative runner for the same computation, kept side by side with /compute/gemma so the two can be compared (correctness, timing). Unlike gemma-wrapper's own --loco mode, which fans its per-chromosome kinship/GWA jobs out internally via GNU parallel, this workflow has ravanan itself do that fan-out: it scatters one gemma-wrapper --chromosomes <chr> --no-parallel ... invocation per chromosome, each restricted to exactly the chromosome it was given (gemma-wrapper itself still does the correct "exclude this chromosome from kinship, test only its markers" LOCO logic, against the full genotype/SNP files -- --chromosomes only overrides which chromosomes gemma-wrapper loops over, not the per-chromosome math itself). This workflow is written in ccwl at cwl/gemma/gemma-loco.scm, not raw CWL; compute_ravanan compiles it to CWL with ccwl compile fresh on every call (cheap -- pure macro expansion, no Guix build involved) into a generated, throwaway file under ravanan_cache_dir, never checked into the repo. compute_ravanan computes the chromosome list from snps_file itself (gemma-wrapper's own get_chromosomes, reproduced as chromosomes_in) and passes it as the workflow's chromosomes input, since CWL's scatter needs a concrete array at invocation time. ravanan_cmd, ccwl_cmd, gemma_ravanan_workflow (the ccwl source, not a CWL file), gemma_ravanan_manifest (ravanan's own required --guix-manifest, a Guix manifest naming gemma-wrapper, at cwl/gemma/manifest.scm) and ravanan_cache_dir (its own cache directory, separate from gemma_cache_dir, for pheno files, run logs, the compiled workflow, and ravanan's own --store) are configurable the same way as the rest of src/config.rs's layered config. GET /compute/gemma/status/{run_id} is shared between both endpoints and checks whichever cache directory the run_id actually logged to.

Exploratory, and genuinely rough around the edges -- three real bugs turned up building this, none of them in /compute/gemma's own direct-subprocess path:

  • ravanan resolves gemma-wrapper by building a Guix profile from --guix-manifest against the ambient Guix process (GUIX_PACKAGE_PATH, no --guix-channels) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package in guix-bioinformatics too new for the pinned base Guix. This needs GUIX_PACKAGE_PATH set in whatever environment starts the genenetwork4 server itself (inherited by the ravanan child process genenetwork4 spawns); it is not something compute_ravanan sets.
  • ravanan's --store needs files/, logs/ and data/ subdirectories to already exist; it does not create them itself (compute_ravanan does now).
  • A step scattered over exactly one input (kinship's chr) needs an explicit scatterMethod even though CWL itself only requires one for multiple scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it.
  • Every chromosome's gwa.cwl run shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in --chromosomes), so the 20 scattered archive outputs all collide under ravanan's optional --output-directory symlink-everything-into-one-flat-dir step; compute_ravanan simply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own --store via its stdout JSON record, never through --output-directory.

A real end-to-end run against the BXD fixtures (same rows as the /compute/gemma example above, confirmed byte-for-byte including rs31443144's exact values) took 16.7s wall-clock scattered across 20 chromosomes this way, vs. /compute/gemma's own ~19s and the earlier sequential (non-scattered) ravanan workflow's ~21s -- a real, if modest, win from letting ravanan's own scheduler fan the chromosomes out instead of going through gemma-wrapper's internal GNU parallel call, on top of no longer needing that path's own PARALLEL_SHELL workaround at all (each scattered job's --no-parallel gemma-wrapper call never invokes GNU parallel in the first place, so the bash-time-keyword bug it worked around does not arise here).

/compute/rqtl

Maps a QTL by running GeneNetwork3's own scripts/rqtl_wrapper.R (via Rscript), copied into this repo verbatim: a scanone genome scan, plus permutation thresholds if nperm > 0. Same request/response shape as /compute/gemma -- synchronous, ?id=/GET .../status/{run_id} progress-streaming reusing the same infrastructure (src/streaming.rs) -- see src/rqtl.rs for the invocation itself, its documented scope (no covariates, interval mapping or pair-scan scantwo -- just the common single-scan case), and a fix made relative to GeneNetwork3: its own cache-key hashing of the genotype/phenotype files never actually looks at their contents (a key-name mismatch between generate_rqtl_cmd and its real caller), so two different phenotypes mapped with the same other options could silently reuse each other's cached result there; fixed here by always hashing both files' contents.

Important: R/qtl's cross file format has no per-sample name matching of its own -- samples must list values in exactly the genotype file's sample-column order, or results are silently misassigned to the wrong individual. This wrapper can't check that for you.

{
  "data": {
    "markers": [
      ["rs31443144", "1", 3.010274, 0.0923496],
      ["rs6269442", "1", 3.492195, 0.1131095]
    ],
    "permutation": null
  },
  "meta": {
    "state": "complete",
    "took_ms": 420,
    "request_id": "...",
    "columns": ["name", "chr", "pos", "lod"],
    "num_rows": 6,
    "num_cols": 4
  },
  "notes": [
    "markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.",
    "Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename <hash>-output.csv"
  ],
  "next": []
}

(Real output, against the tiny 6-marker/6-sample BXD riset fixture in tests/test_data/, trimmed to two markers. Each marker row is positional, in meta.columns' order, rather than a {"name": ..., "chr": ..., ...} object per row -- same space-saving shape /compute/gemma uses for its own (much larger) rows; a 4-way cross's extra mean/SE effect columns, if present, trail each row and aren't named in meta.columns (see MarkerResult's doc comment in src/rqtl.rs). permutation is null unless nperm was set above 0; when it is, it's {"values": [...], "suggestive": ..., "significant": ...}, the 67th- and 95th-percentile thresholds GeneNetwork3 itself uses -- computed here with numpy.percentile's default linear interpolation, reproduced in src/rqtl.rs's percentile.)

OpenAPI

GET /openapi.json serves an OpenAPI 3.1 document covering every route. It's built with utoipa and utoipa-axum:

  • Each handler carries a #[utoipa::path(...)] attribute describing its method, path, parameters and responses; each response/request struct derives utoipa::ToSchema (see VersionInfo, db::Species, db::SpeciesDetail, error::ApiError in src/lib.rs/src/db.rs/ src/error.rs).
  • Routes are registered with utoipa_axum::routes!(handler) instead of plain axum::routing::get(handler), on an OpenApiRouter instead of a plain Router, so each route's metadata is collected as it's added.
  • ApiDoc (a #[derive(utoipa::OpenApi)] struct) supplies the document's info (title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's.
  • OpenApiRouter::split_for_parts() splits the builder back into a plain axum::Router (returned from app()) and the assembled utoipa::openapi::OpenApi value, which the /openapi.json handler then just serializes with Json.
  • ApiDoc's #[openapi(security(("bearer_token" = [])), ...)] makes a token the default for every operation, so an endpoint that needs one doesn't repeat security(...) itself -- only the handful that must NOT require one (/, /version, /get-test-token) override it with an empty security() of their own. error::Unauthorized (a #[derive(utoipa::IntoResponses)] type) is the shared 401 response every gated endpoint references by name in its own responses(...) list (responses(Unauthorized, (status = 200, ...), ...)) instead of writing out (status = 401, description = "...", body = ApiError) by hand each time.

Because the attributes are declared alongside the handlers they describe, the spec can't silently drift out of sync with the code the way a hand-maintained spec file could -- adding or changing a route without updating its #[utoipa::path(...)] is a compile error (the macro checks the response/schema types exist) or an obviously incomplete diff, not something to notice weeks later.

The document is assembled once, when app() runs at process startup (not at cargo build time, and not freshly on every request): the #[utoipa::path]/#[derive(ToSchema)] macros expand to Rust code at compile time, but the actual merging of all routes' metadata into one OpenApi value happens the first time app() executes. Every GET /openapi.json request after that just serializes the same already-built value.

Success responses

Every success response is one JSON envelope (src/response.rs): the data, next to facts about the response, remarks on the data, and suggested follow-up calls. It is the success counterpart of the error shape below.

{
  "data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."},
  "meta": {
    "state": "complete",
    "took_ms": 3,
    "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5"
  },
  "notes": [],
  "next": [
    {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"},
    {"href": "/groups?species=mouse", "description": "Groups of this species"}
  ]
}
  • data is the response proper: exactly what each endpoint below documents.
  • meta.state is complete, or partial when the response is usable but something was left out or unavailable -- notes says what. For example /search is partial when the database could not be reached (hits come back without locus and abbreviation) or when a sorted search ranked a full 1000-match window and more matches exist; /publications/{pubmed_id}/traits is partial when traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure.
  • meta.took_ms is the server's time for the request, from when it started handling it; it excludes the network. meta.request_id is the request's id, also in the x-request-id response header and in any error body, so a report can be correlated with logs.
  • notes are server-authored remarks on data, e.g. lod = lrs / 4.604 on responses that carry lod. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's.
  • next lists follow-up calls, each a relative href that resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large). GET /search suggests the next page and the top hit's metadata.

/ returns the plain text Hello, world! and /openapi.json the OpenAPI document itself; neither is wrapped. In /openapi.json every other success response is an Envelope_* schema (a test enforces this).

Error handling

Every endpoint uses the same ApiError JSON shape on error, modeled on RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see src/error.rs). /species and /species/{species} can actually produce one today (database errors, or a 404 for an unknown species); / and /version document the same 500 response in /openapi.json for consistency, even though neither has a failure path currently (no database access, no I/O):

{
  "type": "/errors/species_not_found",
  "title": "Species not found",
  "status": 404,
  "code": "species_not_found",
  "detail": "No species matching 'nonexistent'.",
  "operation_id": "species_by_name",
  "spec_url": "/openapi.json",
  "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
  "took_ms": 2,
  "next": "/species"
}

next is omitted (null) when there's no obviously related endpoint to suggest, e.g. a hypothetical error from /:

{
  "type": "/errors/internal_error",
  "title": "Internal error",
  "status": 500,
  "code": "internal_error",
  "detail": "example failure for /",
  "operation_id": "hello_world",
  "spec_url": "/openapi.json",
  "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f",
  "took_ms": 1,
  "next": null
}
  • type and spec_url are relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allows type to be a relative URI reference. spec_url is always /openapi.json, on every error from every endpoint.
  • operation_id is the OpenAPI operation that failed (same name as in /openapi.json).
  • request_id is the request's x-request-id, generated per request (or echoed back if the client already sent one) by SetRequestIdLayer/PropagateRequestIdLayer in src/lib.rs, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token.
  • took_ms is the server's time for the request up to the error, the same measure as a success response's meta.took_ms (see Success responses). RFC 7807 allows extension members like this one, request_id, code and next next to its standard ones.
  • next is an optional hint at a related endpoint to try instead; e.g. a 404 from /species/{species} sets next to /species, so a caller with a wrong species name can discover the valid ones.

Running with cargo

Requires a Rust toolchain with cargo (edition 2024).

cargo run

The server listens on 0.0.0.0:3000 by default. Set GENENETWORK4_PORT to use a different port, or GENENETWORK4_BIND to bind a specific address instead of every interface:

GENENETWORK4_PORT=8080 cargo run
GENENETWORK4_BIND=127.0.0.1 cargo run

Development

Every endpoint needs a bearer token (see the note under Endpoints), which normally means getting one from gn-auth yourself. For local development, genenetwork4 can hand you one instead: POST /get-test-token calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result. It needs no setting and is not tied to a mode.

cargo run

Get a token, then use it against a real endpoint:

curl -X POST http://127.0.0.1:3000/get-test-token
{
  "data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"},
  "meta": {"state": "complete", "took_ms": 41, "request_id": "..."},
  "notes": [],
  "next": []
}
TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])')
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species
{
  "data": [
    {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"},
    ...
  ],
  "meta": {"state": "complete", "took_ms": 2, "request_id": "..."},
  "notes": [],
  "next": [
    {"href": "/groups", "description": "Groups (populations); add ?species=<name> to list one species' groups"}
  ]
}

The endpoint is registered by default and listed in /openapi.json (GENENETWORK4_TEST_TOKEN=false switches it off, see below), and decides per request whether to answer: only a caller on this machine gets a token. That means the TCP connection's peer address is in 127.0.x.x (or is ::1), and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header. Anything else, including a request whose peer address is unknown, is a 403. The peer address comes from the connection itself, never from a header a client can set; the proxy headers are used only to refuse a request that claims to have been forwarded. This is why the check also refuses forwarded requests: a reverse proxy running on the same host connects from 127.0.0.1 for every client, so without it the whole internet would look local. A proxy that forwards without setting any of those headers would defeat the check, so don't put one in front of a server whose gn-auth has the dev user (on a production gn-auth the dev user normally doesn't exist, and the request just fails with a 502).

Behind nginx or in a container

The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here):

  • Container on a bridge network, port published. The server sees the bridge gateway (something like 172.17.0.1) as the peer, not 127.0.0.1. Every request through the published port is a 403, whether it comes via nginx or from the host itself. It still works from inside the container, e.g. docker exec <container> curl -X POST http://127.0.0.1:3000/get-test-token.
  • Container sharing the host's network, nginx on the same host. nginx connects from 127.0.0.1, so the peer check passes for every client. The server then relies on refusing requests that carry X-Forwarded-For, Forwarded or X-Real-IP, and a minimal proxy_pass sets none of them. Without one of the measures below, anyone who can reach the site could get a token.

For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third:

  1. Block the path in nginx, independent of genenetwork4:

nginx location = /get-test-token { return 404; }

  1. Make nginx say it is forwarding, so genenetwork4 refuses proxied requests (any one of the three headers is enough):

nginx proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  1. Switch the endpoint off in genenetwork4 itself with GENENETWORK4_TEST_TOKEN=false (an environment variable, or a GENENETWORK4_TEST_TOKEN=false line in GN4_CONF). The route is then not registered at all: POST /get-test-token is a 404 and the endpoint is absent from /openapi.json. The setting fails closed: only true, 1, yes or on (any case) keep the endpoint on, and anything else -- false, an empty value, or a typo such as flase -- turns it off. It is on by default, so a local setup needs nothing.

Also relevant: the token is for gn-auth's dev user (GN_AUTH_DEV_USERNAME). On a production gn-auth that user normally doesn't exist, so a request that does get through fails with a 502 instead of returning a token. That depends on gn-auth's data, not on anything in genenetwork4.

The credentials used (gn-auth's dev client id/secret and dev user email/password, from flask init-dev-clients/gn_auth/wsgi.py) are publicly known and the same for every gn-auth checkout by default -- override GN_AUTH_DEV_CLIENT_ID, GN_AUTH_DEV_CLIENT_SECRET, GN_AUTH_DEV_USERNAME and GN_AUTH_DEV_PASSWORD if your gn-auth instance's dev fixtures differ. Needs a gn-auth reachable at AUTH_SERVER_URL with AUTHLIB_INSECURE_TRANSPORT set (gn-auth itself refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev setting, not something genenetwork4 needs to know about).

The token has no exp claim, and neither genenetwork4 nor gn-auth expires it by time or for lack of use. It stops verifying only when the key that signed it leaves gn-auth's published key set: gn-auth signs with a key for 7 days (JWKS_ROTATION_AGE_DAYS) and deletes it 14 days after its creation (JWKS_DELETION_AGE_DAYS), so a token lasts roughly 7 to 14 days, depending on where in the key's life it was issued. genenetwork4 caches the key set for an hour. When requests start failing with 401 invalid_token, fetch a new token.

The test suite uses this same endpoint automatically -- see Tests below. Configuration is layered: built-in defaults, then GN4_CONF and GN4_SECRETS files of KEY=VALUE lines, then environment variables (which always win). Keys: SQL_URI (MariaDB), GN3_URL (GeneNetwork3 base URL for proxy endpoints), GN3_TIMEOUT_SECS (upstream call timeout, default 30), GENENETWORK4_PORT, GENENETWORK4_BIND, and GENENETWORK4_TEST_TOKEN (false turns POST /get-test-token off; see "Behind nginx or in a container").

Running with Guix

guix build -f guix.scm

builds the package (running both test suites, see below) and prints the resulting store path; the binary is at <store path>/bin/genenetwork4.

guix shell -D -f guix.scm

drops you into a shell with all build dependencies available (cargo, rust, python, git), for running cargo build/cargo test/cargo run directly. This shell is not isolated from the network or from your normal environment, which cargo build/cargo run need to reach crates.io.

A fully isolated container (-C) normally needs -N to still reach the network from inside it, but on at least one guix build (1.5.0-6.64d4de2) combining -D with -N causes -D to silently stop providing the package's build dependencies (the shell then has only genenetwork4 itself, not cargo/rust/etc.). If guix shell -C -D -N -f guix.scm fails with "cargo: command not found", drop -C/-N and use the plain guix shell -D -f guix.scm above instead.

Tests

Two test suites exercise the same endpoints:

  • tests/*.rs - Rust integration tests that call the router in-process (no socket), via tower::ServiceExt::oneshot. Run with:

cargo test

Both suites check the envelope on every success response: the Rust helper tests/common::data(&body) and the Python helper server_test_case.envelope_data(body) verify its shape and return data. tests/envelope.rs (and test/python/test_envelope.py) follow every next link the endpoints suggest; tests/envelope_partial.rs checks the partial state without a database; tests/openapi.rs checks that every JSON success response is documented as an envelope.

  • test/python/test_*.py - Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with:

cargo build --release GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python

Either suite can fetch its own token from a live gn-auth automatically (see "Development" above) -- set AUTH_SERVER_URL before running either command above, and nothing further needs setting up by hand. A run without a reachable gn-auth still works, skipping (not failing) every test that needed a token it couldn't get -- set GN4_TEST_AUTH_TOKEN instead if you'd rather provide one manually.

Both suites also run automatically as part of guix build -f guix.scm (the Rust tests in the check phase, the Python tests in a python-rest-test phase after install).

Every test under tests/*.rs/test/python/test_*.py for an endpoint that touches the database (which is most of them -- hello_world, version, openapi and get_test_token's own disabled/wrong-config cases are the exceptions) checks for a live MariaDB on 127.0.0.1:3306 first (see SQL_URI above) and skips with a message rather than failing when none is reachable, since the Guix build sandbox has no network/database access. Likewise, any test for an endpoint that requires a token (every endpoint except those four) checks for one first and skips if none is available -- see "Development mode" above for how the test suite gets one automatically.

The Rust tests in tests/*.rs that hit the database issue their requests through tests/common::get, which times the request and prints a warning to stderr if it takes over a second -- e.g. WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s). Warnings only show with cargo test -- --nocapture (or on a failing test), since cargo captures stdout/stderr for passing tests otherwise.