# genenetwork4 A Rust/Axum-based API server for GeneNetwork. ## Endpoints | Method | Path | Description | |--------|---------------------------------------------------|-----------------------------------| | GET | `/` | Greeting. | | GET | `/version` | Build info (hash, date). | | GET | `/species` | All species. | | GET | `/species/{species}` | One species by name or id. | | GET | `/species/{species}/datasets` | ProbeSet datasets in a species. | | GET | `/groups` | All groups (filterable). | | GET | `/mapping-methods` | Mapping tools offered. | | GET | `/accessions/{accession}` | Resolve a GN accession number. | | GET | `/groups/{group}` | One group, by name or Id. | | GET | `/groups/{group}/strains` | Strains/samples in a group. | | GET | `/groups/{group}/case-attributes` | Case-attribute names and values. | | GET | `/groups/{group}/traits` | Published (phenotype) traits. | | GET | `/groups/{group}/traits/{trait_id}` | A published trait's metadata. | | GET | `/groups/{group}/traits/{trait_id}/values` | A published trait's values. | | GET | `/groups/{group}/datasets` | ProbeSet datasets in a group. | | GET | `/publications/{pubmed_id}/traits` | Traits a publication reports. | | GET | `/datasets/{dataset}` | One expression dataset. | | GET | `/datasets/{dataset}/probesets` | Expression traits in a dataset. | | GET | `/datasets/{dataset}/probesets/{probeset}` | An expression trait's metadata. | | GET | `/datasets/{dataset}/probesets/{probeset}/values` | An expression trait's values. | | GET | `/traits/{dataset}` | Every trait in a dataset. | | GET | `/traits/{dataset}/{trait}` | One trait's metadata. | | GET | `/traits/{dataset}/{trait}/values` | One trait's values. | | POST | `/compute/correlation` | Correlate a trait (stub). | | POST | `/compute/gemma` | Map a QTL with GEMMA (see below). | | GET | `/compute/gemma/status/{run_id}` | Poll a gemma run's progress. | | POST | `/compute/gemma/ravanan` | Map a QTL via ravanan (see below).| | POST | `/compute/rqtl` | Map a QTL with R/qtl (see below). | | GET | `/compute/rqtl/status/{run_id}` | Poll an rqtl run's progress. | | POST | `/get-test-token` | Get a test bearer token (local). | | GET | `/search` | Search traits (free-text). | | GET | `/openapi.json` | OpenAPI 3.1 document. | Every endpoint above requires a valid gn-auth bearer token (`Authorization: Bearer `) except `/`, `/version`, `/openapi.json` and `/get-test-token` itself -- see docs/gn-auth.md and the "Development" section below (`POST /get-test-token`) for how to get one for local testing. A missing or invalid token is a 401; which further, resource-level authorization (if any) an endpoint checks once a token is valid is a separate, per-endpoint matter covered in each entry below. Every success response is wrapped in one envelope (see [Success responses](#success-responses) below), except `/` (a plain-text greeting) and `/openapi.json`. The example responses below are real ones, with long lists and strings abbreviated by `...`, and `took_ms` and `request_id` shortened to plausible values. Details, including example responses, below. - `GET /` - returns `Hello, world!`. - `GET /version` - returns a JSON record describing the running build: ```json { "data": { "name": "GeneNetwork4", "short_name": "GN4AI", "version": 4.0, "release": 1, "date": "20261008", "git": "208eeac" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [] } ``` `date` and `git` are the HEAD commit's date and short hash, embedded at build time (see `build.rs`); they describe the commit the binary was built from, not the time it happened to be compiled. - `GET /species` - returns a JSON array of all species in the `Species` table, mirroring GeneNetwork3's `gn3.db.species.get_all_species`: ```json { "data": [ {"name": "human", "menu_name": "Human (Homo sapiens, hg19)", "family": "Vertebrates"}, {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species= to list one species' groups"} ] } ``` Needs a live MariaDB connection, configured via the `SQL_URI` environment variable (same config key GeneNetwork3 uses), e.g. `mysql://user:password@host/db_webqtl`. Defaults to `mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl` if unset. - `GET /species/{species}` - returns every column of the matching row of the `Species` table, matched against `Name`, `FullName` or `SpeciesName` (matching GeneNetwork2's `wqflask.api.router.get_species_info`, which returns the same match logic but only a 4-column subset). 404 if no species matches. ```json { "data": { "id": 1, "species_id": 1, "species_name": "Mouse", "name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "full_name": "Mus musculus", "family": "Vertebrates", "family_order_id": 1, "taxonomy_id": 10090, "order_id": 15 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /species/{species}/datasets` - returns the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of `/groups/{group}/datasets`. GeneNetwork2's own species-scoped variant of `wqflask.api.router.get_datasets_for_group` still requires a group name (species only disambiguates it); this doesn't. 404 if no species matches. Not gated on fine-grained gn-auth authorization, same as `/groups/{group}/datasets` -- this returns dataset metadata (names, `public`/`confidentiality` flags), not trait data -- but a token is still required, like every endpoint. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse", "description": "The species these datasets belong to"} ] } ``` - `GET /groups` - returns a JSON array of all groups in the `InbredSet` table, matching the shape of GeneNetwork2's `GET /api/v_pre1/groups` response: ```json { "data": [ {"id": 25, "species_id": 4, "display_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "name": "AD-cases-controls", "full_name": "Brain, Aging: AD, Normal Gene Expression (Liang)", "public": 2, "mapping_method_id": null, "genetic_type": null, "family": null, "species": "human", "mapping_method": null}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species", "description": "The species the groups belong to (filter groups with ?species=)"} ] } ``` `family` is the population category and `species` the species' short name as `/species` lists it. `mapping_method` is the name of the mapping tool offered for the group (the `MappingMethod` row its `mapping_method_id` points to, see `GET /mapping-methods`), null for a group with none. Three optional filters, combined with AND and all case-insensitive: `family` (contains), `species` (exact short name) and `q` (contained in the name, display name or full name). For example `/groups?family=Reference Populations` lists the reference populations, and `/groups?species=rat&q=hxb` the rat HXB/BXH sets. "Group" is GeneNetwork's own alias for `InbredSet` -- see `docs/genenetwork_datatypes.md` for why. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}` - returns every column of the matching row of the `InbredSet` table, matched against `InbredSetName`, `Name`, `FullName` or numeric `Id` (matching GeneNetwork2's `wqflask.api.router.get_group_info`, which matches the same way but not by Id, plus gn-guile's `gn.data.population:population-info`, which only matches by Id). The response uses the "group" alias instead of "inbred_set" in its field names (see `docs/genenetwork_datatypes.md`); only the MariaDB column names themselves stay `InbredSet*`. 404 if no group matches. ```json { "data": { "id": 1, "group_id": 1, "display_name": "BXD", "name": "BXD", "species_id": 1, "full_name": "BXD Family", "public": 2, "mapping_method_id": "1", "genetic_type": "riset", "family": "Reference Populations (replicate average, SE, N)", "family_order": 1, "menu_order_id": 0.0, "group_code": "BXD", "description": "

...

", "mapping_method": "qtlreaper" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"}, {"href": "/groups/BXD/datasets", "description": "Expression (ProbeSet) datasets of the group"}, {"href": "/groups/BXD/traits", "description": "Published (phenotype) traits of the group (can be very large)"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/strains` - returns the strains/samples belonging to a group, matching gn-guile's `gn.data.strains:strain-id-names`. 404 if no group matches. Takes an optional `used_for_mapping` query parameter to filter to only strains used for mapping (gn-guile's `strain-id-names` takes the same flag as a keyword argument): ```json { "data": [ {"id": 1, "name": "B6D2F1", "used_for_mapping": false}, {"id": 2, "name": "C57BL/6J", "used_for_mapping": false}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these strains belong to"}, {"href": "/groups/BXD/case-attributes", "description": "Case-attribute names and per-strain values"} ] } ``` `GET /groups/BXD/strains?used_for_mapping=true` excludes BXD's parent strains (`B6D2F1`, `C57BL/6J`, `DBA/2J`), which aren't used for mapping. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/case-attributes` - returns a group's case attributes: extra per-strain metadata beyond genotype/phenotype data (e.g. age, sex, treatment) -- every attribute name defined for the group, and every strain's values for them, combined into one response (GeneNetwork3's `gn3.api.case_attributes` splits this into separate `.../names` and `.../values` endpoints; neither list is large enough on its own to need splitting). This is the only one of the three sibling GeneNetwork codebases that has case attributes at all -- GeneNetwork2 delegates to GeneNetwork3 for them rather than storing them itself, and gn-guile doesn't have the concept. 404 if no group matches; a group with no case attributes defined returns `200` with both lists empty, not a 404. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "names": ["Epoch", "RRID", ...], "values": [{"strain": "B6D2F1", "case_attributes": {"Status": "Live", "SeqCvge": "", "RRID": "JAX:100006", "Epoch": "0 "}}, ...] }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/strains", "description": "The group's strains/samples"} ] } ``` - `GET /groups/{group}/traits` - returns every published (phenotype) trait belonging to a group, the listing equivalent of `/groups/{group}/traits/{trait_id}` (same fields, same join, no `trait_id` filter). 404 if no group matches. If the group has any published traits, its `PublishFreeze` row is checked the same way as `/traits/{dataset}` (see docs/gn-auth.md): 403 if it isn't public and the caller isn't authorized for it; an empty list (most groups have no phenotype dataset at all) skips the check, since there's nothing in it to protect. A large group (e.g. BXD, with 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; there's no pagination or limit yet, matching GeneNetwork2's own `wqflask.api.router.fetch_traits`, which defaults to unbounded too. Needs `SQL_URI`, same as `/species` above. The optional `abbreviation` parameter keeps only the traits with that short name, e.g. `/groups/BXD/traits?abbreviation=5HT_MB_M` finds trait 17985 (an unknown abbreviation is an empty list). ```json { "data": [ {"id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD", "description": "The group these traits belong to"} ] } ``` - `GET /groups/{group}/traits/{trait_id}` - returns one published (phenotype) trait's `PublishXRef` mapping stats plus its `Phenotype` description/units and `Publication` reference, based on the data explored in `precompute-publishdata.gmi` (e.g. its "More metadata" section, which identifies exactly this join as worth surfacing). `trait_id` is `PublishXRef.Id`, scoped by group for the same reason as `/groups/{group}/traits/{trait_id}/values`. 404 if no group matches, or if the group has no such trait; 403 if the trait's `PublishFreeze` isn't public and the caller isn't authorized for it (see docs/gn-auth.md). ```json { "data": { "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/groups/BXD/traits/10001/values", "description": "The trait's value in each strain"}, {"href": "/groups/BXD", "description": "The group this trait belongs to"}, {"href": "/publications/11438585/traits", "description": "Every trait reported by this trait's publication"} ] } ``` `dataset` is the trait's `PublishFreeze`, `abbreviation` its short name (`Phenotype.Post_publication_abbreviation`, else the pre-publication one) and `url` its page on GeneNetwork's website. `lod` is `lrs` / 4.604. Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/traits/{trait_id}/values` - returns one published (phenotype) trait's value in each of the group's strains, matching GeneNetwork2's `wqflask.api.router.trait_sample_data` (the Publish branch) and gn-guile's `gn.data.dataset:get-bxd-publish-name-value-dict`, generalized to any group rather than hardcoding BXD. `trait_id` is the trait's `PublishXRef.Id`; it alone doesn't identify a trait uniquely across groups (only the pair of group and `PublishXRef.Id` does), hence the group in the path. 404 if no group matches, or if the group has no such trait; 403 if the trait's `PublishFreeze` isn't public and the caller isn't authorized for it (see docs/gn-auth.md -- this is a `#[gnauth(secured)]` endpoint, since it shares actual trait values). Only Publish (phenotype) traits are supported so far, not ProbeSet (expression) or Geno (genotype) traits. ```json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD/traits/10001", "description": "The trait's metadata"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /groups/{group}/datasets` - returns the expression (ProbeSet) datasets belonging to a group, matching GeneNetwork2's `wqflask.api.router.get_datasets_for_group`. 404 if no group matches. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 386, "probe_freeze_id": 156, "name": "B6D2ONCILM_0412", "name2": "B6D2ONCILM_Apr12", "full_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "short_name": "HEI ONC Retina B6D2 (Apr12) RankInv", "create_time": "2012-05-17", "public": 1, "confidentiality": 0, "data_scale": "log2", "tissue": "Retina mRNA"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group these datasets belong to"} ] } ``` `tissue` is what the dataset was measured in (the `Tissue` its `ProbeFreeze` points to), also on `GET /datasets/{dataset}` and `GET /species/{species}/datasets`. - `GET /mapping-methods` - the mapping tools GeneNetwork offers, the `MappingMethod` table in id order: `qtlreaper`, `Rqtl`, `HappyR`, `PLINK` and `FastMap`. It includes tools no group currently uses (PLINK, FastMap); a group's `mapping_method` (see `GET /groups`) names the one offered for it. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 1, "name": "qtlreaper"}, {"id": 2, "name": "Rqtl"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups", "description": "Groups, each with the mapping tool offered for it (`mapping_method`)"} ] } ``` - `GET /accessions/{accession}` - resolves a GeneNetwork accession number, the "GN602" of a dataset's page, with or without the `GN` prefix. It names the dataset (`InfoFiles.InfoPageName`), its page title, its `category` (`publish` for a phenotype dataset, `probeset` for an expression dataset, null if the name matches neither) and the `group` it belongs to. 400 if the text is not a number, 404 if no dataset has the accession. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "accession": 602, "name": "BXDPublish", "title": "BXD Published Phenotypes", "category": "publish", "group": "BXD" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/groups/BXD", "description": "The group the dataset belongs to"}, {"href": "/groups/BXD/traits", "description": "The dataset's published traits (can be very large)"} ] } ``` - `GET /publications/{pubmed_id}/traits` - returns every published (phenotype) trait, across groups, whose publication has this PubMed id (the same fields as `/groups/{group}/traits`). 404 if no publication has that id; a publication with no traits is an empty list. Unlike `/groups/{group}/traits`, the datasets are authorization-checked as in `GET /traits/{dataset}`: traits of a dataset the caller may not view are left out. A paper can report thousands of traits (PubMed 31634382 reports 2700+ microbiome ones), and there is no pagination yet. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"id": 17967, "phenotype_id": 16502, "publication_id": 13584, "mean": 6038.655919692095, "locus": "rsm10000012868", "lrs": 10.535408722907198, "lod": 2.2883164037591652, "additive": -647.5889092087912, "nsamples": 34, "pre_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "post_publication_description": "Blood chemistry: Blood 5-hydroxytryptamine (5-HT, serotonin) level in ad...", "original_description": null, "units": "[pg/ul]", "pubmed_id": 24102824, "title": "Evaluation of heritable determinants of blood and brain serotonin homeos...", "authors": "Ye R, Carneiro AM, Airey D, Sanders-Bush E, Williams RW, Lu L, Wang J, Z...", "journal": "Genes, brain, and behavior", "volume": null, "pages": "247-260", "month": "Mar", "year": "2014", "dataset": "BXDPublish", "abbreviation": "5HT_Blood_M", "url": "https://genenetwork.org/show_trait?trait_id=17967&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/BXDPublish/17967", "description": "Metadata of the first trait"} ] } ``` - `GET /datasets/{dataset}` - returns every column of the matching row of the `ProbeSetFreeze` table, matched against `Name`, `Name2`, `FullName`, `ShortName` or numeric `Id` (matching GeneNetwork2's `wqflask.api.router.get_dataset_info`'s matching logic for its ProbeSet branch). 404 if no dataset matches. A dataset name is globally unique on its own (unlike a group's traits), so this endpoint isn't scoped under `/groups/{group}`. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "id": 1, "probe_freeze_id": 10, "name": "Br_U_0803_M", "name2": "BXDMicroArray_ProbeSet_August03", "full_name": "Brain U74Av2 08/03 MAS5", "short_name": "Brain U74Av2 08/03 MAS5", "create_time": "2003-08-01", "public": 0, "confidentiality": 0, "data_scale": "log2", "tissue": "Brain mRNA" }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets", "description": "The expression traits (probesets) of this dataset (can be very large)"} ] } ``` - `GET /datasets/{dataset}/probesets` - returns every expression (ProbeSet) trait belonging to a dataset, the listing equivalent of `/datasets/{dataset}/probesets/{probeset}` (same fields, same join, no `probeset` filter). 404 if no dataset matches. Gated the same way as its singular sibling -- a valid token is always required, and a non-public dataset is checked against gn-auth once for the dataset as a whole (`auth::authorise_dataset`) rather than once per probeset -- see docs/gn-auth.md. Needs `SQL_URI`, same as `/species` above. ```json { "data": [ {"probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M", "description": "The dataset these probesets belong to"} ] } ``` - `GET /datasets/{dataset}/probesets/{probeset}` - returns one expression (ProbeSet) trait's `ProbeSetXRef` mapping stats plus its `ProbeSet` gene annotation, matching GeneNetwork2's `wqflask.api.router.get_trait_info`'s ProbeSet branch. `ProbeSetXRef` has no single Id column of its own (its primary key is `DataId`; the natural key is the pair `(ProbeSetFreezeId, ProbeSetId)`), so a trait here is identified by dataset name plus `ProbeSet.Name` rather than a single numeric id, unlike `/groups/{group}/traits/{trait_id}`. 404 if no dataset matches, or if the dataset has no such probeset. ```json { "data": { "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at/values", "description": "The probeset's value in each strain"}, {"href": "/datasets/Br_U_0803_M", "description": "The dataset this probeset belongs to"} ] } ``` Needs `SQL_URI`, same as `/species` above. - `GET /datasets/{dataset}/probesets/{probeset}/values` - returns one expression trait's value in each strain in the dataset, matching GeneNetwork2's `wqflask.api.router.trait_sample_data`'s ProbeSet branch, addressed the same way as `/datasets/{dataset}/probesets/{probeset}`. 404 if no dataset matches, or if the dataset has no such probeset. ```json { "data": [ {"name": "B6D2F1", "value": 5.742000102996826}, {"name": "BXD1", "value": 6.414000034332275}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/datasets/Br_U_0803_M/probesets/100001_at", "description": "The probeset's metadata"} ] } ``` Needs `SQL_URI`, same as `/species` above. ### `/traits` -- ProbeSet and Publish data through one URL shape Everything above addresses ProbeSet (expression) and Publish (phenotype) traits differently: `/datasets/{dataset}/probesets/...` vs. `/groups/{group}/traits/...`. A trait is a trait either way -- a dataset, a trait within it, and that trait's per-strain values -- so `/traits/...` addresses both the same way, `(dataset, trait)`, matching GeneNetwork2's own generic `wqflask.api.router.trait_sample_data`/`get_trait_info`. This is additional, not a replacement: the endpoints above are unchanged and keep working exactly as documented. `dataset` is resolved against `ProbeSetFreeze` first, then `PublishFreeze` (`db::resolve_trait_dataset`) -- whichever matches decides the category, so a bare trait name/id is never guessed across categories, only the (much smaller, auditable) set of dataset names is. A Publish dataset can be named either way GeneNetwork2 itself accepts: the `PublishFreeze`'s own `Name`/`ShortName`/`Id` (e.g. `"BXDPublish"`), or its owning group's `Name`/`InbredSetName`/ `FullName`/`Id` (e.g. `"BXD"`) -- a group normally has only one `PublishFreeze`, so the two are near-interchangeable in practice. - `GET /traits/{dataset}` - every trait in `dataset`, tagged by category. 404 if no dataset matches. A non-public dataset is checked against gn-auth once for the dataset as a whole (`auth::authorise_dataset`), same as `/datasets/{dataset}/probesets`. This is still more than `/groups/{group}/traits` checks for the same phenotype data: both endpoints require a token now, but only this one also checks `PublishFreeze.public`/`confidentiality` and, for a non-public dataset, gn-auth -- `/groups/{group}/traits` (unchanged) stops at the token. ```json { "data": [ {"category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] } ``` or, for a Publish dataset: ```json { "data": [ {"category": "publish", "id": 10001, "phenotype_id": 4, "publication_id": 116, "mean": 52.13529418496525, "locus": "rs48756159", "lrs": 13.497491147108706, "lod": 2.931687912056626, "additive": 2.3944443506944437, "nsamples": 34, "pre_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "post_publication_description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral...", "original_description": "Original post publication description: Cerebellum weight [mg]", "units": "mg", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative...", "authors": "Airey DC, Lu L, Williams RW", "journal": "J Neuroscience", "volume": "21", "pages": "5099-5109", "month": null, "year": "2001", "dataset": "BXDPublish", "abbreviation": "Cerebellum_Weight", "url": "https://genenetwork.org/show_trait?trait_id=10001&dataset=BXDPublish"}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [] } ``` - `GET /traits/{dataset}/{trait}` - one trait's metadata, tagged by category (`ProbeSetInfo`'s gene annotation or `TraitInfo`'s publication reference -- the two share little beyond mean/locus/lrs/additive, so neither is collapsed into the other). `trait` is `ProbeSet.Name` for a ProbeSet dataset, or `PublishXRef.Id` for a Publish dataset (a non-numeric `trait` against a Publish dataset is a 404, not a 500 -- it just can't match). 404 if no dataset matches, or no such trait in it; 403 if the dataset isn't public and the caller isn't authorized; 502 if gn-auth itself couldn't be reached. Needs `SQL_URI`, same as `/species` above. ```json { "data": { "category": "probeset", "probeset_id": 1, "name": "100001_at", "symbol": "Cd3g", "description": "CD3d antigen, gamma polypeptide", "chr": "9", "mb": 44.970689, "mean": 5.48794285714286, "locus": "rs13480619", "lrs": 12.590069931048, "lod": 2.7345938164743697, "p_value": 0.269, "additive": -0.28515625 }, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/traits/Br_U_0803_M/100001_at/values", "description": "The trait's value in each strain"} ] } ``` - `GET /traits/{dataset}/{trait}/values` - that trait's value in each strain, as `{name, value}` pairs regardless of category (`TraitValue` and `ProbeSetValue` are identical in shape, so this is the one piece that unifies with zero loss). Same 404/403/502 cases as `/traits/{dataset}/{trait}`. ```json { "data": [ {"name": "BXD1", "value": 61.400001525878906}, {"name": "BXD11", "value": 53.099998474121094}, ... ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/traits/BXDPublish/10001", "description": "The trait's metadata"} ] } ``` - `GET /search` - free-text search across GN traits (see `docs/search-engines.md` for switching between the gn3 proxy and the native engine); hits keep gn3's field names and are wrapped in the envelope like every other response. Parameters: `q` (xapian syntax, e.g. `serotonin AND midbrain AND group:BXD`), `type` (`phenotype` or `gene`), `limit` (default 20, at most 100), and optionally `sort` (`lrs` or `lod`, which order identically) with `order` (`desc`, the default, or `asc`), and `page` (default 1) for the next `limit` hits. Three fields are added to every hit: `lod` (`lrs` / 4.604) and, for phenotype hits, `locus` (the peak marker's name) and `abbreviation` (the trait's short name), both from the database (null for other hits, or when the database is unreachable). `pubmed_id` is gn3's. With `sort`, gn3 is asked for its first 1000 matches by relevance, those are ranked, and `page`/`limit` are applied to the ranked list, so `page=2` continues where page 1 stopped; hits with no `lrs` come last. Without `sort`, `page` is gn3's own. For example, the highest-LOD trait about serotonin in BXD midbrain: `/search?q=serotonin AND midbrain AND group:BXD&sort=lod`. Tissue and sex are not fields: they appear in each hit's description, so match them there. Needs a reachable gn3, configured via the `GN3_URL` config key (default `http://127.0.0.1:5003/api/`). ```json { "data": [ {"name": "17994", "dataset": "BXDPublish", "dataset_fullname": "BXD Published Phenotypes", "group": "BXD", "species": "mouse", "authors": ["Ye R", "Carneiro AM", ...], "year": 2014, "description": "Central nervous system, protein expression: Midbrain serotonin transport...", "mean": 1627.7004466337316, "additive": 233.98946836805567, "lrs": 16.49128337950074, "lod": 3.5819468678324804, "locus": "rsm10000013922", "abbreviation": "SERT_MB_M", "geno_chr": "19", "geno_mb": 58.673324, "inbredsetcode": "BXD", "pubmed_id": 24405699, "pubmed_link": "http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&lis..."} ], "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": ["lod = lrs / 4.604"], "next": [ {"href": "/search?q=serotonin+AND+midbrain+AND+group%3ABXD&limit=1&page=2", "description": "The next page of hits"}, {"href": "/traits/BXDPublish/17994", "description": "Metadata of the top hit"} ] } ``` - `GET /openapi.json` - returns an OpenAPI 3.1 document describing the endpoints above; see [OpenAPI](#openapi) below. ### `/compute/gemma` Maps a QTL by running the real `gemma-wrapper`/GEMMA binaries: a K-matrix step, then an LMM association step, both `--loco`, over a genotype file, a SNP annotation file and a list of trait values. See `src/gemma.rs` for the invocation itself and why it's shaped this way (found by running GeneNetwork2's and GeneNetwork3's own gemma code for real against the BXD fixtures -- GeneNetwork3's always errors out; GeneNetwork2's non-LOCO branch does too; only `--loco` works). This runs synchronously inside the request -- there is no job queue -- and it runs sequentially, not per-chromosome in parallel. **Not feasible past roughly 100,000 markers at this point.** The BXD fixtures (7,320 markers, LOCO across 20 chromosomes) already take about 19 seconds; some real GeneNetwork datasets have upwards of 1 million markers, which this endpoint cannot handle yet -- the request would simply run for a very long time inside one HTTP call. Making that feasible needs either parallelising the per-chromosome LOCO jobs (`gemma-wrapper --parallel`, which gn-guile's own precompute script already uses) or moving the computation out of the request/response cycle entirely (a job queue, or an offline precompute step, the way gn-guile's own `scripts/precompute/run-gemma.scm` already does it). ```json { "data": [ [1.0, "rs31443144", 3010274.0, 0.0, "X", "Y", 0.276, 0.1533063, 0.1255921, 0.00001, 0.2178407], ... [1.0, "rs8253327", 191908118.0, 0.0, "X", "Y", 0.284, 0.4281751, 0.1158946, 0.00001, 0.0003520103] ], "meta": { "state": "complete", "took_ms": 18456, "request_id": "ade51981-4417-47cd-93d7-70dbd9da1ecc", "columns": ["chr", "rs", "ps", "n_miss", "allele1", "allele0", "af", "beta", "se", "l_mle", "p_lrt"], "num_rows": 7320, "num_cols": 11 }, "notes": [ "Rows come from GEMMA's own .assoc.txt output, positional (see meta.columns for names); column names depend on the lmm value used.", "Kinship: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -gk", "GWA: gemma-wrapper --json --cache-dir /tmp/gn4-gemma --loco --input /tmp/gn4-gemma/492514-...-k.json -- -g .../BXD_geno.txt.gz -p /tmp/gn4-gemma/492514-...-pheno.txt -a .../BXD_snps.txt -lmm 9" ], "next": [] } ``` (`data` has one row per marker -- 7,320 rows in this real run against the BXD fixtures, trimmed to two here. Each row is positional, in `meta.columns`' order, rather than a `{"chr": ..., "rs": ..., ...}` object per row -- cuts the repeated key names out of a response this large. The last two `notes` are the exact gemma-wrapper invocations used (paths shortened here for readability) -- `-gk` only on the kinship step, `-lmm` only on the GWA step, never both together; see `src/gemma.rs`'s `compute_gwa` doc comment for why that distinction matters. `lmm` defaults to 9, matching GeneNetwork2's and gn-guile's own convention; see the field's own doc comment in `src/routes/gemma.rs` for what's actually been verified about it.) #### `/compute/gemma/status/{run_id}` -- polling a run's progress Given the endpoint above can genuinely take minutes, pass `?id=` on the `POST` and poll this on a second connection while it's in flight to show progress, the same log-tailing shape GeneNetwork2/3 use for R/qtl (`gn3/computations/streaming.py`, `gn3/api/streaming.py`): the run's `id` becomes a log file (`{cache_dir}/{id}.log`); gemma-wrapper's own stderr (its `--json`-mode progress banner, chromosome list, "Executing: ...", per-step info -- its stdout carries only the one final JSON record, so there's nothing to stream there) is appended to it line by line as both the kinship and GWA steps run; once the whole request finishes, `"GN4: computation complete"` is appended as a sentinel so a poller can tell "still running" from "done". `run_id` must be 1-64 ASCII letters/digits/`-`/`_` -- it's a path component, so this is enforced on both this endpoint and the `?id=` that names it. `id` must also be unique: if a log already exists for it (another run still in progress, or a finished one never polled/cleaned up), the `POST` is refused with 409 rather than silently overwriting it. ``` POST /compute/gemma?id=my-run-1 (on one connection; blocks until done) GET /compute/gemma/status/my-run-1?peek=0 (on another, while the first is in flight) ``` ```json { "data": { "run_id": "my-run-1", "lines": [ "gemma-wrapper 1.00-pre1 (Ruby 3.3.9) by Pjotr Prins 2017-2024", "Using GEMMA 0.98.6 (2022-08-05) by Xiang Zhou, Pjotr Prins and team (C) 2012-2022", "Executing: time -v .../gemma -o ... -g ... -p ... -a ... -gk -outdir ..." ], "pointer": 612 }, "meta": {"state": "complete", "took_ms": 1, "request_id": "..."}, "notes": [], "next": [] } ``` Pass `pointer` back as `?peek=` on the next poll; `lines` is empty (not an error) when nothing new has been written since. An unknown `run_id` (never started, or the main request hasn't written its log yet) is a 404, not an empty response -- genuinely different from "nothing new yet". #### `/compute/gemma/ravanan` -- the same mapping, run per-chromosome via ravanan Same request/response shape as `/compute/gemma`, same two gemma-wrapper steps (kinship, then GWA, both `--loco`), but driven by [ravanan](https://github.com/vepiteski/ravanan) running a CWL workflow compiled on every call (via `ccwl compile`, see below) from the ccwl source at `cwl/gemma/gemma-loco.scm`, instead of this server's own direct subprocess calls -- an alternative runner for the same computation, kept side by side with `/compute/gemma` so the two can be compared (correctness, timing). Unlike gemma-wrapper's own `--loco` mode, which fans its per-chromosome kinship/GWA jobs out internally via GNU `parallel`, this workflow has ravanan itself do that fan-out: it scatters one `gemma-wrapper --chromosomes --no-parallel ...` invocation per chromosome, each restricted to exactly the chromosome it was given (gemma-wrapper itself still does the correct "exclude this chromosome from kinship, test only its markers" LOCO logic, against the *full* genotype/SNP files -- `--chromosomes` only overrides which chromosomes gemma-wrapper loops over, not the per-chromosome math itself). This workflow is written in [ccwl](https://ccwl.systemreboot.net/) at `cwl/gemma/gemma-loco.scm`, not raw CWL; `compute_ravanan` compiles it to CWL with `ccwl compile` fresh on every call (cheap -- pure macro expansion, no Guix build involved) into a generated, throwaway file under `ravanan_cache_dir`, never checked into the repo. `compute_ravanan` computes the chromosome list from `snps_file` itself (gemma-wrapper's own `get_chromosomes`, reproduced as `chromosomes_in`) and passes it as the workflow's `chromosomes` input, since CWL's `scatter` needs a concrete array at invocation time. `ravanan_cmd`, `ccwl_cmd`, `gemma_ravanan_workflow` (the ccwl source, not a CWL file), `gemma_ravanan_manifest` (ravanan's own required `--guix-manifest`, a Guix manifest naming `gemma-wrapper`, at `cwl/gemma/manifest.scm`) and `ravanan_cache_dir` (its own cache directory, separate from `gemma_cache_dir`, for pheno files, run logs, the compiled workflow, and ravanan's own `--store`) are configurable the same way as the rest of `src/config.rs`'s layered config. `GET /compute/gemma/status/{run_id}` is shared between both endpoints and checks whichever cache directory the `run_id` actually logged to. Exploratory, and genuinely rough around the edges -- three real bugs turned up building this, none of them in `/compute/gemma`'s own direct-subprocess path: - ravanan resolves `gemma-wrapper` by building a Guix profile from `--guix-manifest` against the *ambient* Guix process (`GUIX_PACKAGE_PATH`, no `--guix-channels`) rather than spinning up a separately pinned inferior -- the latter forces a whole-channel package-cache rebuild that breaks on any unrelated package in `guix-bioinformatics` too new for the pinned base Guix. This needs `GUIX_PACKAGE_PATH` set in whatever environment starts the genenetwork4 server itself (inherited by the `ravanan` child process genenetwork4 spawns); it is not something `compute_ravanan` sets. - ravanan's `--store` needs `files/`, `logs/` and `data/` subdirectories to already exist; it does not create them itself (`compute_ravanan` does now). - A step scattered over exactly one input (`kinship`'s `chr`) needs an explicit `scatterMethod` even though CWL itself only requires one for *multiple* scattered inputs -- ravanan's scheduler has no fallback case for a missing one and silently misbehaves without it. - Every chromosome's `gwa.cwl` run shares gemma-wrapper's own content-hash basename for its output archive (the hash doesn't factor in `--chromosomes`), so the 20 scattered `archive` outputs all collide under ravanan's optional `--output-directory` symlink-everything-into-one-flat-dir step; `compute_ravanan` simply doesn't pass that flag, since it was already reading every output's path straight out of ravanan's own `--store` via its stdout JSON record, never through `--output-directory`. A real end-to-end run against the BXD fixtures (same rows as the `/compute/gemma` example above, confirmed byte-for-byte including `rs31443144`'s exact values) took **16.7s** wall-clock scattered across 20 chromosomes this way, vs. `/compute/gemma`'s own ~19s and the earlier sequential (non-scattered) ravanan workflow's ~21s -- a real, if modest, win from letting ravanan's own scheduler fan the chromosomes out instead of going through gemma-wrapper's internal GNU `parallel` call, on top of no longer needing that path's own `PARALLEL_SHELL` workaround at all (each scattered job's `--no-parallel` gemma-wrapper call never invokes GNU `parallel` in the first place, so the bash-`time`-keyword bug it worked around does not arise here). ### `/compute/rqtl` Maps a QTL by running GeneNetwork3's own `scripts/rqtl_wrapper.R` (via `Rscript`), copied into this repo verbatim: a `scanone` genome scan, plus permutation thresholds if `nperm > 0`. Same request/response shape as `/compute/gemma` -- synchronous, `?id=`/`GET .../status/{run_id}` progress-streaming reusing the same infrastructure (`src/streaming.rs`) -- see `src/rqtl.rs` for the invocation itself, its documented scope (no covariates, interval mapping or pair-scan `scantwo` -- just the common single-scan case), and a fix made relative to GeneNetwork3: its own cache-key hashing of the genotype/phenotype files never actually looks at their contents (a key-name mismatch between `generate_rqtl_cmd` and its real caller), so two different phenotypes mapped with the same other options could silently reuse each other's cached result there; fixed here by always hashing both files' contents. **Important:** R/qtl's cross file format has no per-sample name matching of its own -- `samples` must list values in exactly the genotype file's sample-column order, or results are silently misassigned to the wrong individual. This wrapper can't check that for you. ```json { "data": { "markers": [ ["rs31443144", "1", 3.010274, 0.0923496], ["rs6269442", "1", 3.492195, 0.1131095] ], "permutation": null }, "meta": { "state": "complete", "took_ms": 420, "request_id": "...", "columns": ["name", "chr", "pos", "lod"], "num_rows": 6, "num_cols": 4 }, "notes": [ "markers come from R/qtl's scanone output, positional (see meta.columns for names); permutation is null unless nperm > 0 was requested.", "Rscript scripts/rqtl_wrapper.R --geno .../rqtl_sample1.geno --pheno .../pheno.txt --model normal --method hk --nperm 0 --scale mb --outdir /tmp/gn4-rqtl --filename -output.csv" ], "next": [] } ``` (Real output, against the tiny 6-marker/6-sample BXD riset fixture in `tests/test_data/`, trimmed to two markers. Each marker row is positional, in `meta.columns`' order, rather than a `{"name": ..., "chr": ..., ...}` object per row -- same space-saving shape `/compute/gemma` uses for its own (much larger) rows; a 4-way cross's extra mean/SE effect columns, if present, trail each row and aren't named in `meta.columns` (see `MarkerResult`'s doc comment in `src/rqtl.rs`). `permutation` is `null` unless `nperm` was set above 0; when it is, it's `{"values": [...], "suggestive": ..., "significant": ...}`, the 67th- and 95th-percentile thresholds GeneNetwork3 itself uses -- computed here with `numpy.percentile`'s default linear interpolation, reproduced in `src/rqtl.rs`'s `percentile`.) ## OpenAPI `GET /openapi.json` serves an OpenAPI 3.1 document covering every route. It's built with [utoipa](https://docs.rs/utoipa) and [utoipa-axum](https://docs.rs/utoipa-axum): - Each handler carries a `#[utoipa::path(...)]` attribute describing its method, path, parameters and responses; each response/request struct derives `utoipa::ToSchema` (see `VersionInfo`, `db::Species`, `db::SpeciesDetail`, `error::ApiError` in `src/lib.rs`/`src/db.rs`/ `src/error.rs`). - Routes are registered with `utoipa_axum::routes!(handler)` instead of plain `axum::routing::get(handler)`, on an `OpenApiRouter` instead of a plain `Router`, so each route's metadata is collected as it's added. - `ApiDoc` (a `#[derive(utoipa::OpenApi)]` struct) supplies the document's `info` (title, description, license); without it the generated spec would carry utoipa-axum's own placeholder metadata instead of this API's. - `OpenApiRouter::split_for_parts()` splits the builder back into a plain `axum::Router` (returned from `app()`) and the assembled `utoipa::openapi::OpenApi` value, which the `/openapi.json` handler then just serializes with `Json`. - `ApiDoc`'s `#[openapi(security(("bearer_token" = [])), ...)]` makes a token the default for every operation, so an endpoint that needs one doesn't repeat `security(...)` itself -- only the handful that must NOT require one (`/`, `/version`, `/get-test-token`) override it with an empty `security()` of their own. `error::Unauthorized` (a `#[derive(utoipa::IntoResponses)]` type) is the shared 401 response every gated endpoint references by name in its own `responses(...)` list (`responses(Unauthorized, (status = 200, ...), ...)`) instead of writing out `(status = 401, description = "...", body = ApiError)` by hand each time. Because the attributes are declared alongside the handlers they describe, the spec can't silently drift out of sync with the code the way a hand-maintained spec file could -- adding or changing a route without updating its `#[utoipa::path(...)]` is a compile error (the macro checks the response/schema types exist) or an obviously incomplete diff, not something to notice weeks later. The document is assembled once, when `app()` runs at process startup (not at `cargo build` time, and not freshly on every request): the `#[utoipa::path]`/`#[derive(ToSchema)]` macros expand to Rust code at compile time, but the actual merging of all routes' metadata into one `OpenApi` value happens the first time `app()` executes. Every `GET /openapi.json` request after that just serializes the same already-built value. ## Success responses Every success response is one JSON envelope (`src/response.rs`): the data, next to facts about the response, remarks on the data, and suggested follow-up calls. It is the success counterpart of the error shape below. ```json { "data": {"name": "mouse", "full_name": "Mus musculus", "...": "..."}, "meta": { "state": "complete", "took_ms": 3, "request_id": "d4f1f9a0-62d7-4a52-9d51-62b0c8bfa5f5" }, "notes": [], "next": [ {"href": "/species/mouse/datasets", "description": "Expression datasets of this species"}, {"href": "/groups?species=mouse", "description": "Groups of this species"} ] } ``` - `data` is the response proper: exactly what each endpoint below documents. - `meta.state` is `complete`, or `partial` when the response is usable but something was left out or unavailable -- `notes` says what. For example `/search` is `partial` when the database could not be reached (hits come back without `locus` and `abbreviation`) or when a sorted search ranked a full 1000-match window and more matches exist; `/publications/{pubmed_id}/traits` is `partial` when traits of datasets the caller may not view were left out. The HTTP status stays 200: the state is about completeness, not failure. - `meta.took_ms` is the server's time for the request, from when it started handling it; it excludes the network. `meta.request_id` is the request's id, also in the `x-request-id` response header and in any error body, so a report can be correlated with logs. - `notes` are server-authored remarks on `data`, e.g. `lod = lrs / 4.604` on responses that carry `lod`. They are never derived from database content or request input, so a model reading them is not reading anyone's text but this server's. - `next` lists follow-up calls, each a relative `href` that resolves with a 200 for the response it comes from, and what it gives you (the test suite follows them, except the two listings of thousands of rows -- a group's traits and a dataset's probesets -- whose descriptions say they can be very large). `GET /search` suggests the next page and the top hit's metadata. `/` returns the plain text `Hello, world!` and `/openapi.json` the OpenAPI document itself; neither is wrapped. In `/openapi.json` every other success response is an `Envelope_*` schema (a test enforces this). ## Error handling Every endpoint uses the same `ApiError` JSON shape on error, modeled on RFC 7807 (Problem Details for HTTP APIs) plus a few extra fields (see `src/error.rs`). `/species` and `/species/{species}` can actually produce one today (database errors, or a 404 for an unknown species); `/` and `/version` document the same 500 response in `/openapi.json` for consistency, even though neither has a failure path currently (no database access, no I/O): ```json { "type": "/errors/species_not_found", "title": "Species not found", "status": 404, "code": "species_not_found", "detail": "No species matching 'nonexistent'.", "operation_id": "species_by_name", "spec_url": "/openapi.json", "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f", "took_ms": 2, "next": "/species" } ``` `next` is omitted (`null`) when there's no obviously related endpoint to suggest, e.g. a hypothetical error from `/`: ```json { "type": "/errors/internal_error", "title": "Internal error", "status": 500, "code": "internal_error", "detail": "example failure for /", "operation_id": "hello_world", "spec_url": "/openapi.json", "request_id": "20974d91-52c0-41fe-ba2b-4df399a4cc4f", "took_ms": 1, "next": null } ``` - `type` and `spec_url` are relative paths, not absolute URLs, since this service has no fixed public domain; RFC 7807 explicitly allows `type` to be a relative URI reference. `spec_url` is always `/openapi.json`, on every error from every endpoint. - `operation_id` is the OpenAPI operation that failed (same name as in `/openapi.json`). - `request_id` is the request's `x-request-id`, generated per request (or echoed back if the client already sent one) by `SetRequestIdLayer`/`PropagateRequestIdLayer` in `src/lib.rs`, and included on the response too, for correlating a failure with logs. Every error has one, including the 401 for a missing or invalid token. - `took_ms` is the server's time for the request up to the error, the same measure as a success response's `meta.took_ms` (see [Success responses](#success-responses)). RFC 7807 allows extension members like this one, `request_id`, `code` and `next` next to its standard ones. - `next` is an optional hint at a related endpoint to try instead; e.g. a 404 from `/species/{species}` sets `next` to `/species`, so a caller with a wrong species name can discover the valid ones. ## Running with cargo Requires a Rust toolchain with cargo (edition 2024). ``` cargo run ``` The server listens on `0.0.0.0:3000` by default. Set `GENENETWORK4_PORT` to use a different port, or `GENENETWORK4_BIND` to bind a specific address instead of every interface: ``` GENENETWORK4_PORT=8080 cargo run GENENETWORK4_BIND=127.0.0.1 cargo run ``` ## Development Every endpoint needs a bearer token (see the note under [Endpoints](#endpoints)), which normally means getting one from gn-auth yourself. For local development, genenetwork4 can hand you one instead: `POST /get-test-token` calls gn-auth's own password grant (`POST /auth/token`) for gn-auth's well-known development user and returns the result. It needs no setting and is not tied to a mode. ``` cargo run ``` Get a token, then use it against a real endpoint: ``` curl -X POST http://127.0.0.1:3000/get-test-token ``` ```json { "data": {"access_token": "eyJhbGc...", "token_type": "Bearer", "scope": "profile group resource"}, "meta": {"state": "complete", "took_ms": 41, "request_id": "..."}, "notes": [], "next": [] } ``` ``` TOKEN=$(curl -s -X POST http://127.0.0.1:3000/get-test-token | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["access_token"])') curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/species ``` ```json { "data": [ {"name": "mouse", "menu_name": "Mouse (Mus musculus, mm10)", "family": "Vertebrates"}, ... ], "meta": {"state": "complete", "took_ms": 2, "request_id": "..."}, "notes": [], "next": [ {"href": "/groups", "description": "Groups (populations); add ?species= to list one species' groups"} ] } ``` The endpoint is registered by default and listed in `/openapi.json` (`GENENETWORK4_TEST_TOKEN=false` switches it off, see below), and decides per request whether to answer: only a caller on this machine gets a token. That means the TCP connection's peer address is in `127.0.x.x` (or is `::1`), and the request carries no `X-Forwarded-For`, `Forwarded` or `X-Real-IP` header. Anything else, including a request whose peer address is unknown, is a `403`. The peer address comes from the connection itself, never from a header a client can set; the proxy headers are used only to refuse a request that claims to have been forwarded. This is why the check also refuses forwarded requests: a reverse proxy running on the same host connects from `127.0.0.1` for every client, so without it the whole internet would look local. A proxy that forwards without setting any of those headers would defeat the check, so don't put one in front of a server whose gn-auth has the dev user (on a production gn-auth the dev user normally doesn't exist, and the request just fails with a `502`). ### Behind nginx or in a container The check looks at who connected to genenetwork4, so what it sees depends on the deployment (the Docker behavior below is how Docker networking works; it was not tested here): - **Container on a bridge network, port published.** The server sees the bridge gateway (something like `172.17.0.1`) as the peer, not `127.0.0.1`. Every request through the published port is a `403`, whether it comes via nginx or from the host itself. It still works from inside the container, e.g. `docker exec curl -X POST http://127.0.0.1:3000/get-test-token`. - **Container sharing the host's network, nginx on the same host.** nginx connects from `127.0.0.1`, so the peer check passes for every client. The server then relies on refusing requests that carry `X-Forwarded-For`, `Forwarded` or `X-Real-IP`, and a minimal `proxy_pass` sets none of them. Without one of the measures below, anyone who can reach the site could get a token. For a deployed instance, don't rely on the check alone. Any one of these is enough, and a production deployment should use the third: 1. **Block the path in nginx**, independent of genenetwork4: ```nginx location = /get-test-token { return 404; } ``` 2. **Make nginx say it is forwarding**, so genenetwork4 refuses proxied requests (any one of the three headers is enough): ```nginx proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ``` 3. **Switch the endpoint off in genenetwork4 itself** with `GENENETWORK4_TEST_TOKEN=false` (an environment variable, or a `GENENETWORK4_TEST_TOKEN=false` line in `GN4_CONF`). The route is then not registered at all: `POST /get-test-token` is a `404` and the endpoint is absent from `/openapi.json`. The setting fails closed: only `true`, `1`, `yes` or `on` (any case) keep the endpoint on, and anything else -- `false`, an empty value, or a typo such as `flase` -- turns it off. It is on by default, so a local setup needs nothing. Also relevant: the token is for gn-auth's dev user (`GN_AUTH_DEV_USERNAME`). On a production gn-auth that user normally doesn't exist, so a request that does get through fails with a `502` instead of returning a token. That depends on gn-auth's data, not on anything in genenetwork4. The credentials used (gn-auth's dev client id/secret and dev user email/password, from `flask init-dev-clients`/`gn_auth/wsgi.py`) are publicly known and the same for every gn-auth checkout by default -- override `GN_AUTH_DEV_CLIENT_ID`, `GN_AUTH_DEV_CLIENT_SECRET`, `GN_AUTH_DEV_USERNAME` and `GN_AUTH_DEV_PASSWORD` if your gn-auth instance's dev fixtures differ. Needs a gn-auth reachable at `AUTH_SERVER_URL` with `AUTHLIB_INSECURE_TRANSPORT` set (gn-auth itself refuses OAuth2 grants over plain HTTP otherwise -- a gn-auth-side dev setting, not something genenetwork4 needs to know about). The token has no `exp` claim, and neither genenetwork4 nor gn-auth expires it by time or for lack of use. It stops verifying only when the key that signed it leaves gn-auth's published key set: gn-auth signs with a key for 7 days (`JWKS_ROTATION_AGE_DAYS`) and deletes it 14 days after its creation (`JWKS_DELETION_AGE_DAYS`), so a token lasts roughly 7 to 14 days, depending on where in the key's life it was issued. genenetwork4 caches the key set for an hour. When requests start failing with `401 invalid_token`, fetch a new token. The test suite uses this same endpoint automatically -- see [Tests](#tests) below. Configuration is layered: built-in defaults, then `GN4_CONF` and `GN4_SECRETS` files of `KEY=VALUE` lines, then environment variables (which always win). Keys: `SQL_URI` (MariaDB), `GN3_URL` (GeneNetwork3 base URL for proxy endpoints), `GN3_TIMEOUT_SECS` (upstream call timeout, default 30), `GENENETWORK4_PORT`, `GENENETWORK4_BIND`, and `GENENETWORK4_TEST_TOKEN` (`false` turns `POST /get-test-token` off; see "Behind nginx or in a container"). ## Running with Guix ``` guix build -f guix.scm ``` builds the package (running both test suites, see below) and prints the resulting store path; the binary is at `/bin/genenetwork4`. ``` guix shell -D -f guix.scm ``` drops you into a shell with all build dependencies available (cargo, rust, python, git), for running `cargo build`/`cargo test`/`cargo run` directly. This shell is not isolated from the network or from your normal environment, which `cargo build`/`cargo run` need to reach crates.io. A fully isolated container (`-C`) normally needs `-N` to still reach the network from inside it, but on at least one guix build (1.5.0-6.64d4de2) combining `-D` with `-N` causes `-D` to silently stop providing the package's build dependencies (the shell then has only `genenetwork4` itself, not cargo/rust/etc.). If `guix shell -C -D -N -f guix.scm` fails with "cargo: command not found", drop `-C`/`-N` and use the plain `guix shell -D -f guix.scm` above instead. ## Tests Two test suites exercise the same endpoints: - `tests/*.rs` - Rust integration tests that call the router in-process (no socket), via `tower::ServiceExt::oneshot`. Run with: ``` cargo test ``` Both suites check the envelope on every success response: the Rust helper `tests/common::data(&body)` and the Python helper `server_test_case.envelope_data(body)` verify its shape and return `data`. `tests/envelope.rs` (and `test/python/test_envelope.py`) follow every `next` link the endpoints suggest; `tests/envelope_partial.rs` checks the `partial` state without a database; `tests/openapi.rs` checks that every JSON success response is documented as an envelope. - `test/python/test_*.py` - Python tests that spawn the actual compiled binary and talk to it over real HTTP, since Python can only reach the server that way. Run with: ``` cargo build --release GENENETWORK4_BINARY=target/release/genenetwork4 python3 -m unittest discover -v test/python ``` Either suite can fetch its own token from a live gn-auth automatically (see "Development" above) -- set `AUTH_SERVER_URL` before running either command above, and nothing further needs setting up by hand. A run without a reachable gn-auth still works, skipping (not failing) every test that needed a token it couldn't get -- set `GN4_TEST_AUTH_TOKEN` instead if you'd rather provide one manually. Both suites also run automatically as part of `guix build -f guix.scm` (the Rust tests in the `check` phase, the Python tests in a `python-rest-test` phase after install). Every test under `tests/*.rs`/`test/python/test_*.py` for an endpoint that touches the database (which is most of them -- `hello_world`, `version`, `openapi` and `get_test_token`'s own disabled/wrong-config cases are the exceptions) checks for a live MariaDB on `127.0.0.1:3306` first (see `SQL_URI` above) and skips with a message rather than failing when none is reachable, since the Guix build sandbox has no network/database access. Likewise, any test for an endpoint that requires a token (every endpoint except those four) checks for one first and skips if none is available -- see "Development mode" above for how the test suite gets one automatically. The Rust tests in `tests/*.rs` that hit the database issue their requests through `tests/common::get`, which times the request and prints a warning to stderr if it takes over a second -- e.g. `WARNING: GET /groups/BXD/traits/10001 took 7.1s (> 1s)`. Warnings only show with `cargo test -- --nocapture` (or on a failing test), since cargo captures stdout/stderr for passing tests otherwise.