# Release Notes ## Unreleased (success envelope) ### Changed (breaking) - Every success response is now wrapped in one envelope, the success counterpart of the RFC 7807 error shape: `{"data": ..., "meta": {"state", "took_ms", "request_id"}, "notes": [...], "next": [{"href", "description"}]}`. Clients that read a bare payload must now read `data`. `meta.state` is `complete` or `partial` (a usable response with something left out or unavailable, explained in `notes`); `took_ms` is server time; `next` suggests follow-up calls. Only `/` (plain text) and `/openapi.json` are not wrapped. See the README, "Success responses". `/search` already used `data` and `next`; it gains `meta` and `notes`. - `lod` is `lrs / 4.604` (was 4.61), the divisor the benchmark's RDF data uses. - `POST /get-test-token` is no longer tied to dev mode: the `GENENETWORK4_DEV_MODE` setting is removed and the route is always registered and documented (`GENENETWORK4_BIND` no longer affects it). It answers a request only when it comes from this machine: the TCP peer is in 127.0.x.x (or `::1`) and the request has no `X-Forwarded-For`, `Forwarded` or `X-Real-IP` header; anything else, including an unknown peer, is a 403. The server now provides the peer address (`into_make_service_with_connect_info`, `src/main.rs`). - New `GENENETWORK4_TEST_TOKEN` setting (on by default) leaves `POST /get-test-token` out entirely when it is not a clearly affirmative value (`true`, `1`, `yes`, `on`): the route is then a 404 and absent from `/openapi.json`. A typo such as `flase` turns it off. Recommended for deployments behind a proxy or in a container; see the README, "Behind nginx or in a container". - Errors gain `took_ms` (an RFC 7807 extension member: the server's time before the error, as `meta.took_ms` on a success). The 401 for a missing or invalid token now also carries its `request_id`, which it never did. ### Added - Groups gain `mapping_method` (the name of the mapping tool offered for them: the `MappingMethod` row `mapping_method_id` points to), and datasets gain `tissue` (the `Tissue` their `ProbeFreeze` points to). - `GET /mapping-methods` lists the mapping tools (the `MappingMethod` table); `GET /accessions/{accession}` resolves a GeneNetwork accession number such as GN602 to its dataset, category and group. - `src/response.rs`: `Envelope`, `Meta`, `State` and the `Ctx` extractor handlers use to build a response (`ctx.envelope(data)`). - `GET /publications/{pubmed_id}/traits`: the published traits a publication reports; traits of datasets the caller may not view are left out and the response is `partial`. - Published-trait metadata gains `dataset`, `abbreviation` (the trait's short name) and `url` (its GeneNetwork page), and `GET /groups/{group}/traits?abbreviation=` looks a trait up by short name. - `GET /groups` gains `family` and `species` on every group and the `family`, `species` and `q` filters. - `GET /search` gains `lod`, `locus` and `abbreviation` on hits, `sort` (`lrs`/`lod`) and `order`, `page`, and a declared `pubmed_id`. - Tests: `tests/envelope.rs` and `test/python/test_envelope.py` (the envelope, and every `next` link resolves), `tests/envelope_partial.rs` (a `partial` search without a database), an OpenAPI check that every JSON success response is an envelope, and `tests/benchmark.rs` (36 of the 39 questions of aisearch's `small_benchmark.csv`). ## Unreleased (`gn-auth` branch) Securing endpoints with per-user tokens issued by gn-auth. Every endpoint now requires one (see "Every endpoint now requires a valid token" below); which further, resource-level authorization (if any) an endpoint also checks is a separate, per-endpoint matter -- see `docs/gn-auth.md`. ### Added - `docs/gn-auth.md`: how gn-auth works from a client's perspective -- tokens are RS256 JWTs verified locally via `GET /auth/public-jwks` (no per-request call to gn-auth needed), and resource-scoped authorization (e.g. "can this user do X on InbredSet Y") via `GET /auth/resource/populations/resource-id/...` followed by `POST /auth/resource/authorisation`, mirroring `genenetwork3/gn3/api/case_attributes.py`. - `src/auth.rs`: `AuthToken`, an Axum extractor that verifies a bearer token's RS256 signature and standard time claims against a cached, periodically-refreshed JWKS; `is_publicly_readable(public, confidentiality)`, matching GeneNetwork2's own threshold for whether a dataset needs authorization at all; and `authorise(...)`, wrapping gn-auth's resource-authorization check with a short-lived per-(user, resource) cache. `tests/auth.rs` runs the whole flow (including the cache) against a real gn-auth instance. - `jsonwebtoken` (RS256 verification) and `ureq` (gn-auth's HTTP API) as new dependencies; `AUTH_SERVER_URL` as a new `Config` key, matching GeneNetwork3's own config key of the same name. - `is_publicly_readable` also treats a dataset older than 5 years as public, regardless of its `public`/`confidentiality` flags. - `auth::authorise_trait`, wrapping `POST /auth/data/authorisation` (the `"Dataset::Trait"`-keyed equivalent of `authorise`), for data addressed by a `(dataset, probeset)` pair rather than a resource id. - `GET /datasets/{dataset}/probesets/{probeset}` and its `/values` sibling now require a valid token unconditionally, and -- for a dataset that isn't publicly readable -- check `group:resource:view-resource` via `authorise_trait`, returning 403 (not 404) when denied and 502 if gn-auth itself couldn't be reached. See docs/gn-auth.md. - `GET /groups/{group}/traits`: list the published (phenotype) traits in a group, the listing equivalent of `/groups/{group}/traits/{trait_id}`. - `GET /datasets/{dataset}/probesets`: list the expression (ProbeSet) traits in a dataset. Gated the same way as `.../probesets/{probeset}` -- via the new `auth::authorise_dataset`, which checks a whole dataset at once rather than once per probeset. - `GET /species/{species}/datasets`: list the expression (ProbeSet) datasets in a species, across all its groups -- the species-scoped equivalent of `/groups/{group}/datasets`. - `GET /traits/{dataset}`, `GET /traits/{dataset}/{trait}` and its `/values` sibling (`src/routes/traits.rs`): a new, additional way to reach both ProbeSet and Publish trait data through one `(dataset, trait)` shape, matching GeneNetwork2's own generic `trait_sample_data`/`get_trait_info` rather than the two category-specific shapes above (which are unchanged and keep working). `db::resolve_trait_dataset` resolves `dataset` against `ProbeSetFreeze` then `PublishFreeze` so a bare trait name/id is never guessed across categories. Responses are tagged (`{"category": "probeset"|"publish", ...}`) rather than collapsed to a lossy common schema. Gated the same way as `/datasets/{dataset}/probesets...`, extended to resource-level authorization for Publish data too, which `/groups/{group}/traits` itself still doesn't check (see "Every endpoint now requires a valid token" below for the separate, blanket token requirement that does cover it) -- `authorise_trait`/`authorise_dataset` already works unchanged for Publish datasets, since gn-auth's own `/auth/data/authorisation` already dispatches `*Publish`-suffixed dataset names to its phenotype resource resolver. - `POST /get-test-token` (`src/routes/dev.rs`; named `/dev/token` very briefly before this): development-only convenience endpoint that calls gn-auth's password grant for its well-known dev user and returns the token, so local testing no longer needs a manual curl/gn-auth round trip. Only registered -- not just rejected, genuinely absent from the router and from `/openapi.json` -- when two new `Config` fields are both set: `dev_mode` (`GENENETWORK4_DEV_MODE`) and a loopback `bind` (`GENENETWORK4_BIND`, new -- `src/main.rs` now binds to this instead of a hardcoded `"0.0.0.0"`). `GN_AUTH_DEV_CLIENT_ID`/ `_CLIENT_SECRET`/`_USERNAME`/`_PASSWORD` default to gn-auth's own published dev fixtures, so this works unconfigured against an unmodified `flask init-dev-clients` gn-auth instance. - `GET /search` now requires a valid bearer token (`AuthToken`), matching the blanket requirement below; it previously proxied to gn3 unauthenticated. - **Every endpoint now requires a valid token** -- `/`, `/version`, `/openapi.json` and `/get-test-token` itself are the only exceptions. This closes what the previous entries above left open: the `dataset_probeset*`/`traits*` family already checked resource-level authorization for non-public data, but `/species*`, `/groups*` and `/datasets/{dataset}` previously didn't even require a token. `ApiDoc`'s new `#[openapi(security(("bearer_token" = [])), ...)]` makes a token the document-wide default so individual handlers don't each repeat `security(...)`; the three exceptions override it with their own empty `security()`. `error::Unauthorized`, a `#[derive(utoipa::IntoResponses)]` type, is the shared 401 every gated handler references by name in its own `responses(...)` instead of writing out the same tuple by hand each time. - Rust (`tests/common::auth_token()`) and Python (`test/python/auth_helpers.auth_token()`) test helpers now fetch their own token from `POST /get-test-token` first, falling back to `GN4_TEST_AUTH_TOKEN` -- the whole test suite (both languages) now runs with zero manually-provisioned tokens, given `GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1` for the test run (and a reachable gn-auth with `AUTHLIB_INSECURE_TRANSPORT` set). - `GET /groups/{group}/case-attributes`: a group's case attributes -- extra per-strain metadata beyond genotype/phenotype data -- combining GeneNetwork3's separate `.../names` and `.../values` endpoints (`gn3.api.case_attributes`) into one response. The only one of GeneNetwork2/3/gn-guile that has this concept at all is GeneNetwork3 (GeneNetwork2 delegates to it rather than storing case attributes itself; gn-guile doesn't have the concept). Found and fixed a bug along the way: GeneNetwork3's own query joins `CaseAttribute` to `CaseAttributeXRefNew` on `CaseAttributeId` alone, but `CaseAttributeId` is not unique across groups (it's a shared vocabulary -- e.g. id 37 means "Epoch" for dozens of unrelated groups at once; the real key is `(InbredSetId, CaseAttributeId)`) -- confirmed live against this database: without also joining on `InbredSetId`, BXD's result included a spurious `"Strain": "Live"` attribute belonging to an unrelated group. `db::get_group_case_attribute_values` adds the missing join condition. ### Not yet done - Which gn-auth privilege a non-token-related endpoint's resource-level authorization should check is still only decided for the `dataset_probeset*`/`traits*` family (`group:resource:view-resource` via `authorise_trait`/`authorise_dataset`). `/species*`, `/groups*` (outside `/traits`) and `/datasets/{dataset}` itself now require a token but don't check any further resource-level authorization -- deciding whether/how they should is separate from requiring a token at all, which is now resolved. - The Publish branch of `/traits/...`'s gating was exercised against gn-auth's "unlinked dataset" deny path and the already-proven ProbeSet branch, but not against a genuinely non-public `PublishFreeze` row -- none exists in the database this was tested against. ### Found and fixed upstream, in gn-auth While standing up a from-scratch gn-auth instance to test against (no prior JWKs/users/sysadmins), four bugs surfaced that blocked this integration; see gn-auth's own `RELEASE_NOTES.md` for details: - A freshly-started gn-auth could reject every token for up to 6 hours (stale/empty JWKS cache). - gn-auth's own `flask init-dev-clients` dev client could not authenticate at all (unhashed client secret). - `POST /auth/resource/authorisation` -- the endpoint this integration depends on -- crashed with a 500 on every fresh deployment. - The same endpoint also crashed (instead of degrading to the anonymous view, as it does for a missing token) on an invalid bearer token. All four are fixed on gn-auth's `fix-fresh-deployment-bugs` branch.