1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
|
# Release Notes
## Unreleased (success envelope)
### Changed (breaking)
- Every success response is now wrapped in one envelope, the success
counterpart of the RFC 7807 error shape: `{"data": ..., "meta":
{"state", "took_ms", "request_id"}, "notes": [...], "next":
[{"href", "description"}]}`. Clients that read a bare payload must now
read `data`. `meta.state` is `complete` or `partial` (a usable
response with something left out or unavailable, explained in
`notes`); `took_ms` is server time; `next` suggests follow-up calls.
Only `/` (plain text) and `/openapi.json` are not wrapped. See the
README, "Success responses". `/search` already used `data` and `next`;
it gains `meta` and `notes`.
- `lod` is `lrs / 4.604` (was 4.61), the divisor the benchmark's RDF
data uses.
- `POST /get-test-token` is no longer tied to dev mode: the
`GENENETWORK4_DEV_MODE` setting is removed and the route is always
registered and documented (`GENENETWORK4_BIND` no longer affects it).
It answers a request only when it comes from this machine: the TCP
peer is in 127.0.x.x (or `::1`) and the request has no
`X-Forwarded-For`, `Forwarded` or `X-Real-IP` header; anything else,
including an unknown peer, is a 403. The server now provides the peer
address (`into_make_service_with_connect_info`, `src/main.rs`).
- New `GENENETWORK4_TEST_TOKEN` setting (on by default) leaves
`POST /get-test-token` out entirely when it is not a clearly
affirmative value (`true`, `1`, `yes`, `on`): the route is then a 404
and absent from `/openapi.json`. A typo such as `flase` turns it off.
Recommended for deployments behind a proxy or in a container; see the
README, "Behind nginx or in a container".
- Errors gain `took_ms` (an RFC 7807 extension member: the server's
time before the error, as `meta.took_ms` on a success). The 401 for a
missing or invalid token now also carries its `request_id`, which it
never did.
### Added
- Groups gain `mapping_method` (the name of the mapping tool offered
for them: the `MappingMethod` row `mapping_method_id` points to), and
datasets gain `tissue` (the `Tissue` their `ProbeFreeze` points to).
- `GET /mapping-methods` lists the mapping tools (the `MappingMethod`
table); `GET /accessions/{accession}` resolves a GeneNetwork
accession number such as GN602 to its dataset, category and group.
- `src/response.rs`: `Envelope`, `Meta`, `State` and the `Ctx`
extractor handlers use to build a response (`ctx.envelope(data)`).
- `GET /publications/{pubmed_id}/traits`: the published traits a
publication reports; traits of datasets the caller may not view are
left out and the response is `partial`.
- Published-trait metadata gains `dataset`, `abbreviation` (the trait's
short name) and `url` (its GeneNetwork page), and
`GET /groups/{group}/traits?abbreviation=` looks a trait up by short
name.
- `GET /groups` gains `family` and `species` on every group and the
`family`, `species` and `q` filters.
- `GET /search` gains `lod`, `locus` and `abbreviation` on hits,
`sort` (`lrs`/`lod`) and `order`, `page`, and a declared `pubmed_id`.
- Tests: `tests/envelope.rs` and `test/python/test_envelope.py` (the
envelope, and every `next` link resolves), `tests/envelope_partial.rs`
(a `partial` search without a database), an OpenAPI check that every
JSON success response is an envelope, and `tests/benchmark.rs`
(36 of the 39 questions of aisearch's `small_benchmark.csv`).
## Unreleased (`gn-auth` branch)
Securing endpoints with per-user tokens issued by gn-auth. Every
endpoint now requires one (see "Every endpoint now requires a valid
token" below); which further, resource-level authorization (if any) an
endpoint also checks is a separate, per-endpoint matter -- see
`docs/gn-auth.md`.
### Added
- `docs/gn-auth.md`: how gn-auth works from a client's perspective --
tokens are RS256 JWTs verified locally via `GET /auth/public-jwks`
(no per-request call to gn-auth needed), and resource-scoped
authorization (e.g. "can this user do X on InbredSet Y") via
`GET /auth/resource/populations/resource-id/...` followed by
`POST /auth/resource/authorisation`, mirroring
`genenetwork3/gn3/api/case_attributes.py`.
- `src/auth.rs`: `AuthToken`, an Axum extractor that verifies a bearer
token's RS256 signature and standard time claims against a cached,
periodically-refreshed JWKS; `is_publicly_readable(public,
confidentiality)`, matching GeneNetwork2's own threshold for whether
a dataset needs authorization at all; and `authorise(...)`, wrapping
gn-auth's resource-authorization check with a short-lived
per-(user, resource) cache. `tests/auth.rs` runs the whole flow
(including the cache) against a real gn-auth instance.
- `jsonwebtoken` (RS256 verification) and `ureq` (gn-auth's HTTP API)
as new dependencies; `AUTH_SERVER_URL` as a new `Config` key,
matching GeneNetwork3's own config key of the same name.
- `is_publicly_readable` also treats a dataset older than 5 years as
public, regardless of its `public`/`confidentiality` flags.
- `auth::authorise_trait`, wrapping `POST /auth/data/authorisation`
(the `"Dataset::Trait"`-keyed equivalent of `authorise`), for data
addressed by a `(dataset, probeset)` pair rather than a resource id.
- `GET /datasets/{dataset}/probesets/{probeset}` and its `/values`
sibling now require a valid token unconditionally, and -- for a
dataset that isn't publicly readable -- check
`group:resource:view-resource` via `authorise_trait`, returning 403
(not 404) when denied and 502 if gn-auth itself couldn't be reached.
See docs/gn-auth.md.
- `GET /groups/{group}/traits`: list the published (phenotype) traits
in a group, the listing equivalent of
`/groups/{group}/traits/{trait_id}`.
- `GET /datasets/{dataset}/probesets`: list the expression (ProbeSet)
traits in a dataset. Gated the same way as `.../probesets/{probeset}`
-- via the new `auth::authorise_dataset`, which checks a whole
dataset at once rather than once per probeset.
- `GET /species/{species}/datasets`: list the expression (ProbeSet)
datasets in a species, across all its groups -- the species-scoped
equivalent of `/groups/{group}/datasets`.
- `GET /traits/{dataset}`, `GET /traits/{dataset}/{trait}` and its
`/values` sibling (`src/routes/traits.rs`): a new, additional way to
reach both ProbeSet and Publish trait data through one `(dataset,
trait)` shape, matching GeneNetwork2's own generic
`trait_sample_data`/`get_trait_info` rather than the two
category-specific shapes above (which are unchanged and keep
working). `db::resolve_trait_dataset` resolves `dataset` against
`ProbeSetFreeze` then `PublishFreeze` so a bare trait name/id is
never guessed across categories. Responses are tagged
(`{"category": "probeset"|"publish", ...}`) rather than collapsed to
a lossy common schema. Gated the same way as
`/datasets/{dataset}/probesets...`, extended to resource-level
authorization for Publish data too, which `/groups/{group}/traits`
itself still doesn't check (see "Every endpoint now requires a valid
token" below for the separate, blanket token requirement that does
cover it) -- `authorise_trait`/`authorise_dataset` already works
unchanged for Publish datasets, since gn-auth's own
`/auth/data/authorisation` already dispatches `*Publish`-suffixed
dataset names to its phenotype resource resolver.
- `POST /get-test-token` (`src/routes/dev.rs`; named `/dev/token` very
briefly before this): development-only convenience endpoint that
calls gn-auth's password grant for its well-known dev user and
returns the token, so local testing no longer needs a manual
curl/gn-auth round trip. Only registered -- not just rejected,
genuinely absent from the router and from `/openapi.json` -- when two
new `Config` fields are both set: `dev_mode`
(`GENENETWORK4_DEV_MODE`) and a loopback `bind`
(`GENENETWORK4_BIND`, new -- `src/main.rs` now binds to this instead
of a hardcoded `"0.0.0.0"`). `GN_AUTH_DEV_CLIENT_ID`/
`_CLIENT_SECRET`/`_USERNAME`/`_PASSWORD` default to gn-auth's own
published dev fixtures, so this works unconfigured against an
unmodified `flask init-dev-clients` gn-auth instance.
- `GET /search` now requires a valid bearer token (`AuthToken`),
matching the blanket requirement below; it previously proxied to gn3
unauthenticated.
- **Every endpoint now requires a valid token** -- `/`, `/version`,
`/openapi.json` and `/get-test-token` itself are the only exceptions.
This closes what the previous entries above left open: the
`dataset_probeset*`/`traits*` family already checked resource-level
authorization for non-public data, but `/species*`, `/groups*` and
`/datasets/{dataset}` previously didn't even require a token.
`ApiDoc`'s new `#[openapi(security(("bearer_token" = [])), ...)]`
makes a token the document-wide default so individual handlers don't
each repeat `security(...)`; the three exceptions override it with
their own empty `security()`. `error::Unauthorized`, a
`#[derive(utoipa::IntoResponses)]` type, is the shared 401 every
gated handler references by name in its own `responses(...)` instead
of writing out the same tuple by hand each time.
- Rust (`tests/common::auth_token()`) and Python
(`test/python/auth_helpers.auth_token()`) test helpers now fetch
their own token from `POST /get-test-token` first, falling back to
`GN4_TEST_AUTH_TOKEN` -- the whole test suite (both languages) now
runs with zero manually-provisioned tokens, given
`GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1` for the test
run (and a reachable gn-auth with `AUTHLIB_INSECURE_TRANSPORT` set).
- `GET /groups/{group}/case-attributes`: a group's case attributes --
extra per-strain metadata beyond genotype/phenotype data -- combining
GeneNetwork3's separate `.../names` and `.../values` endpoints
(`gn3.api.case_attributes`) into one response. The only one of
GeneNetwork2/3/gn-guile that has this concept at all is GeneNetwork3
(GeneNetwork2 delegates to it rather than storing case attributes
itself; gn-guile doesn't have the concept). Found and fixed a bug
along the way: GeneNetwork3's own query joins `CaseAttribute` to
`CaseAttributeXRefNew` on `CaseAttributeId` alone, but
`CaseAttributeId` is not unique across groups (it's a shared
vocabulary -- e.g. id 37 means "Epoch" for dozens of unrelated groups
at once; the real key is `(InbredSetId, CaseAttributeId)`) --
confirmed live against this database: without also joining on
`InbredSetId`, BXD's result included a spurious `"Strain": "Live"`
attribute belonging to an unrelated group. `db::get_group_case_attribute_values`
adds the missing join condition.
### Not yet done
- Which gn-auth privilege a non-token-related endpoint's resource-level
authorization should check is still only decided for the
`dataset_probeset*`/`traits*` family (`group:resource:view-resource`
via `authorise_trait`/`authorise_dataset`). `/species*`, `/groups*`
(outside `/traits`) and `/datasets/{dataset}` itself now require a
token but don't check any further resource-level authorization --
deciding whether/how they should is separate from requiring a token
at all, which is now resolved.
- The Publish branch of `/traits/...`'s gating was exercised against
gn-auth's "unlinked dataset" deny path and the already-proven
ProbeSet branch, but not against a genuinely non-public
`PublishFreeze` row -- none exists in the database this was tested
against.
### Found and fixed upstream, in gn-auth
While standing up a from-scratch gn-auth instance to test against (no
prior JWKs/users/sysadmins), four bugs surfaced that blocked this
integration; see gn-auth's own `RELEASE_NOTES.md` for details:
- A freshly-started gn-auth could reject every token for up to 6 hours
(stale/empty JWKS cache).
- gn-auth's own `flask init-dev-clients` dev client could not
authenticate at all (unhashed client secret).
- `POST /auth/resource/authorisation` -- the endpoint this integration
depends on -- crashed with a 500 on every fresh deployment.
- The same endpoint also crashed (instead of degrading to the
anonymous view, as it does for a missing token) on an invalid bearer
token.
All four are fixed on gn-auth's `fix-fresh-deployment-bugs` branch.
|