about summary refs log tree commit diff
path: root/RELEASE_NOTES.md
blob: 9eb654b888df3dc4abc24c27f3410fcf49bd76fa (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
# Release Notes

## Unreleased (success envelope)

### Changed (breaking)

- Every success response is now wrapped in one envelope, the success
  counterpart of the RFC 7807 error shape: `{"data": ..., "meta":
  {"state", "took_ms", "request_id"}, "notes": [...], "next":
  [{"href", "description"}]}`. Clients that read a bare payload must now
  read `data`. `meta.state` is `complete` or `partial` (a usable
  response with something left out or unavailable, explained in
  `notes`); `took_ms` is server time; `next` suggests follow-up calls.
  Only `/` (plain text) and `/openapi.json` are not wrapped. See the
  README, "Success responses". `/search` already used `data` and `next`;
  it gains `meta` and `notes`.
- `lod` is `lrs / 4.604` (was 4.61), the divisor the benchmark's RDF
  data uses.
- `POST /get-test-token` is no longer tied to dev mode: the
  `GENENETWORK4_DEV_MODE` setting is removed and the route is always
  registered and documented (`GENENETWORK4_BIND` no longer affects it).
  It answers a request only when it comes from this machine: the TCP
  peer is in 127.0.x.x (or `::1`) and the request has no
  `X-Forwarded-For`, `Forwarded` or `X-Real-IP` header; anything else,
  including an unknown peer, is a 403. The server now provides the peer
  address (`into_make_service_with_connect_info`, `src/main.rs`).
- New `GENENETWORK4_TEST_TOKEN` setting (on by default) leaves
  `POST /get-test-token` out entirely when it is not a clearly
  affirmative value (`true`, `1`, `yes`, `on`): the route is then a 404
  and absent from `/openapi.json`. A typo such as `flase` turns it off.
  Recommended for deployments behind a proxy or in a container; see the
  README, "Behind nginx or in a container".
- Errors gain `took_ms` (an RFC 7807 extension member: the server's
  time before the error, as `meta.took_ms` on a success). The 401 for a
  missing or invalid token now also carries its `request_id`, which it
  never did.

### Added

- Groups gain `mapping_method` (the name of the mapping tool offered
  for them: the `MappingMethod` row `mapping_method_id` points to), and
  datasets gain `tissue` (the `Tissue` their `ProbeFreeze` points to).
- `GET /mapping-methods` lists the mapping tools (the `MappingMethod`
  table); `GET /accessions/{accession}` resolves a GeneNetwork
  accession number such as GN602 to its dataset, category and group.
- `src/response.rs`: `Envelope`, `Meta`, `State` and the `Ctx`
  extractor handlers use to build a response (`ctx.envelope(data)`).
- `GET /publications/{pubmed_id}/traits`: the published traits a
  publication reports; traits of datasets the caller may not view are
  left out and the response is `partial`.
- Published-trait metadata gains `dataset`, `abbreviation` (the trait's
  short name) and `url` (its GeneNetwork page), and
  `GET /groups/{group}/traits?abbreviation=` looks a trait up by short
  name.
- `GET /groups` gains `family` and `species` on every group and the
  `family`, `species` and `q` filters.
- `GET /search` gains `lod`, `locus` and `abbreviation` on hits,
  `sort` (`lrs`/`lod`) and `order`, `page`, and a declared `pubmed_id`.
- Tests: `tests/envelope.rs` and `test/python/test_envelope.py` (the
  envelope, and every `next` link resolves), `tests/envelope_partial.rs`
  (a `partial` search without a database), an OpenAPI check that every
  JSON success response is an envelope, and `tests/benchmark.rs`
  (36 of the 39 questions of aisearch's `small_benchmark.csv`).

## Unreleased (`gn-auth` branch)

Securing endpoints with per-user tokens issued by gn-auth. Every
endpoint now requires one (see "Every endpoint now requires a valid
token" below); which further, resource-level authorization (if any) an
endpoint also checks is a separate, per-endpoint matter -- see
`docs/gn-auth.md`.

### Added

- `docs/gn-auth.md`: how gn-auth works from a client's perspective --
  tokens are RS256 JWTs verified locally via `GET /auth/public-jwks`
  (no per-request call to gn-auth needed), and resource-scoped
  authorization (e.g. "can this user do X on InbredSet Y") via
  `GET /auth/resource/populations/resource-id/...` followed by
  `POST /auth/resource/authorisation`, mirroring
  `genenetwork3/gn3/api/case_attributes.py`.
- `src/auth.rs`: `AuthToken`, an Axum extractor that verifies a bearer
  token's RS256 signature and standard time claims against a cached,
  periodically-refreshed JWKS; `is_publicly_readable(public,
  confidentiality)`, matching GeneNetwork2's own threshold for whether
  a dataset needs authorization at all; and `authorise(...)`, wrapping
  gn-auth's resource-authorization check with a short-lived
  per-(user, resource) cache. `tests/auth.rs` runs the whole flow
  (including the cache) against a real gn-auth instance.
- `jsonwebtoken` (RS256 verification) and `ureq` (gn-auth's HTTP API)
  as new dependencies; `AUTH_SERVER_URL` as a new `Config` key,
  matching GeneNetwork3's own config key of the same name.
- `is_publicly_readable` also treats a dataset older than 5 years as
  public, regardless of its `public`/`confidentiality` flags.
- `auth::authorise_trait`, wrapping `POST /auth/data/authorisation`
  (the `"Dataset::Trait"`-keyed equivalent of `authorise`), for data
  addressed by a `(dataset, probeset)` pair rather than a resource id.
- `GET /datasets/{dataset}/probesets/{probeset}` and its `/values`
  sibling now require a valid token unconditionally, and -- for a
  dataset that isn't publicly readable -- check
  `group:resource:view-resource` via `authorise_trait`, returning 403
  (not 404) when denied and 502 if gn-auth itself couldn't be reached.
  See docs/gn-auth.md.
- `GET /groups/{group}/traits`: list the published (phenotype) traits
  in a group, the listing equivalent of
  `/groups/{group}/traits/{trait_id}`.
- `GET /datasets/{dataset}/probesets`: list the expression (ProbeSet)
  traits in a dataset. Gated the same way as `.../probesets/{probeset}`
  -- via the new `auth::authorise_dataset`, which checks a whole
  dataset at once rather than once per probeset.
- `GET /species/{species}/datasets`: list the expression (ProbeSet)
  datasets in a species, across all its groups -- the species-scoped
  equivalent of `/groups/{group}/datasets`.
- `GET /traits/{dataset}`, `GET /traits/{dataset}/{trait}` and its
  `/values` sibling (`src/routes/traits.rs`): a new, additional way to
  reach both ProbeSet and Publish trait data through one `(dataset,
  trait)` shape, matching GeneNetwork2's own generic
  `trait_sample_data`/`get_trait_info` rather than the two
  category-specific shapes above (which are unchanged and keep
  working). `db::resolve_trait_dataset` resolves `dataset` against
  `ProbeSetFreeze` then `PublishFreeze` so a bare trait name/id is
  never guessed across categories. Responses are tagged
  (`{"category": "probeset"|"publish", ...}`) rather than collapsed to
  a lossy common schema. Gated the same way as
  `/datasets/{dataset}/probesets...`, extended to resource-level
  authorization for Publish data too, which `/groups/{group}/traits`
  itself still doesn't check (see "Every endpoint now requires a valid
  token" below for the separate, blanket token requirement that does
  cover it) -- `authorise_trait`/`authorise_dataset` already works
  unchanged for Publish datasets, since gn-auth's own
  `/auth/data/authorisation` already dispatches `*Publish`-suffixed
  dataset names to its phenotype resource resolver.
- `POST /get-test-token` (`src/routes/dev.rs`; named `/dev/token` very
  briefly before this): development-only convenience endpoint that
  calls gn-auth's password grant for its well-known dev user and
  returns the token, so local testing no longer needs a manual
  curl/gn-auth round trip. Only registered -- not just rejected,
  genuinely absent from the router and from `/openapi.json` -- when two
  new `Config` fields are both set: `dev_mode`
  (`GENENETWORK4_DEV_MODE`) and a loopback `bind`
  (`GENENETWORK4_BIND`, new -- `src/main.rs` now binds to this instead
  of a hardcoded `"0.0.0.0"`). `GN_AUTH_DEV_CLIENT_ID`/
  `_CLIENT_SECRET`/`_USERNAME`/`_PASSWORD` default to gn-auth's own
  published dev fixtures, so this works unconfigured against an
  unmodified `flask init-dev-clients` gn-auth instance.
- `GET /search` now requires a valid bearer token (`AuthToken`),
  matching the blanket requirement below; it previously proxied to gn3
  unauthenticated.
- **Every endpoint now requires a valid token** -- `/`, `/version`,
  `/openapi.json` and `/get-test-token` itself are the only exceptions.
  This closes what the previous entries above left open: the
  `dataset_probeset*`/`traits*` family already checked resource-level
  authorization for non-public data, but `/species*`, `/groups*` and
  `/datasets/{dataset}` previously didn't even require a token.
  `ApiDoc`'s new `#[openapi(security(("bearer_token" = [])), ...)]`
  makes a token the document-wide default so individual handlers don't
  each repeat `security(...)`; the three exceptions override it with
  their own empty `security()`. `error::Unauthorized`, a
  `#[derive(utoipa::IntoResponses)]` type, is the shared 401 every
  gated handler references by name in its own `responses(...)` instead
  of writing out the same tuple by hand each time.
- Rust (`tests/common::auth_token()`) and Python
  (`test/python/auth_helpers.auth_token()`) test helpers now fetch
  their own token from `POST /get-test-token` first, falling back to
  `GN4_TEST_AUTH_TOKEN` -- the whole test suite (both languages) now
  runs with zero manually-provisioned tokens, given
  `GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1` for the test
  run (and a reachable gn-auth with `AUTHLIB_INSECURE_TRANSPORT` set).
- `GET /groups/{group}/case-attributes`: a group's case attributes --
  extra per-strain metadata beyond genotype/phenotype data -- combining
  GeneNetwork3's separate `.../names` and `.../values` endpoints
  (`gn3.api.case_attributes`) into one response. The only one of
  GeneNetwork2/3/gn-guile that has this concept at all is GeneNetwork3
  (GeneNetwork2 delegates to it rather than storing case attributes
  itself; gn-guile doesn't have the concept). Found and fixed a bug
  along the way: GeneNetwork3's own query joins `CaseAttribute` to
  `CaseAttributeXRefNew` on `CaseAttributeId` alone, but
  `CaseAttributeId` is not unique across groups (it's a shared
  vocabulary -- e.g. id 37 means "Epoch" for dozens of unrelated groups
  at once; the real key is `(InbredSetId, CaseAttributeId)`) --
  confirmed live against this database: without also joining on
  `InbredSetId`, BXD's result included a spurious `"Strain": "Live"`
  attribute belonging to an unrelated group. `db::get_group_case_attribute_values`
  adds the missing join condition.

### Not yet done

- Which gn-auth privilege a non-token-related endpoint's resource-level
  authorization should check is still only decided for the
  `dataset_probeset*`/`traits*` family (`group:resource:view-resource`
  via `authorise_trait`/`authorise_dataset`). `/species*`, `/groups*`
  (outside `/traits`) and `/datasets/{dataset}` itself now require a
  token but don't check any further resource-level authorization --
  deciding whether/how they should is separate from requiring a token
  at all, which is now resolved.
- The Publish branch of `/traits/...`'s gating was exercised against
  gn-auth's "unlinked dataset" deny path and the already-proven
  ProbeSet branch, but not against a genuinely non-public
  `PublishFreeze` row -- none exists in the database this was tested
  against.

### Found and fixed upstream, in gn-auth

While standing up a from-scratch gn-auth instance to test against (no
prior JWKs/users/sysadmins), four bugs surfaced that blocked this
integration; see gn-auth's own `RELEASE_NOTES.md` for details:

- A freshly-started gn-auth could reject every token for up to 6 hours
  (stale/empty JWKS cache).
- gn-auth's own `flask init-dev-clients` dev client could not
  authenticate at all (unhashed client secret).
- `POST /auth/resource/authorisation` -- the endpoint this integration
  depends on -- crashed with a 500 on every fresh deployment.
- The same endpoint also crashed (instead of degrading to the
  anonymous view, as it does for a missing token) on an invalid bearer
  token.

All four are fixed on gn-auth's `fix-fresh-deployment-bugs` branch.