about summary refs log tree commit diff
path: root/docs/gn-auth.md
blob: 87d0c70b442cd5908ed9e8609577309e579d33f1 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
# gn-auth

gn-auth (`factory/code/gn-auth`, package `gn_auth`) is GeneNetwork's
shared OAuth2 authentication/authorisation server (Flask +
[authlib](https://authlib.org/)). It issues tokens to end users and
client applications, and answers "does the bearer of this token have
privilege X on resource Y" questions for the rest of the GeneNetwork
suite. GeneNetwork4 doesn't run its own user/token database -- it
calls gn-auth over HTTP for both of those things, the same way
GeneNetwork3 (gn3) already does.

Everything below is about **how to call gn-auth as a client**, not how
gn-auth itself is built. In dev, gn-auth listens on
`http://127.0.0.1:8081/`, every route below is under an `/auth` prefix
(`gn_auth/__init__.py`, `gn_auth/auth/views.py`).

## Tokens are JWTs, not opaque strings

Every grant type gn-auth actually wires up --
`password`, `urn:ietf:params:oauth:grant-type:jwt-bearer` and
`refresh_token` -- shares one token generator
(`gn_auth/auth/authentication/oauth2/server.py`) that produces an
RS256-signed JWT as the access token, plus a 42-character opaque
refresh token. The JWT's claims include `scope`, `client_id`, `sub`
(the user id), `iat`, `exp` and `jti`
(`gn_auth/auth/authentication/oauth2/grants/jwt_bearer_grant.py`).

This matters because it means a consuming service can **verify a token
itself**, locally, by checking its signature against gn-auth's public
key -- it doesn't have to ask gn-auth "is this token valid" on every
request.

## Verifying a token: fetch the JWKS, check the signature locally

`GET /auth/public-jwks` (no auth required) returns gn-auth's current
RSA public keys, oldest first:

```json
{
  "documentation": "The keys are listed in order of creation, from the oldest (first) to the newest (last).",
  "jwks": [{"kty": "RSA", "kid": "...", "n": "...", "e": "...", ...}, ...]
}
```

gn-auth rotates in a new key every `JWKS_ROTATION_AGE_DAYS` (default
7) and keeps old ones around for `JWKS_DELETION_AGE_DAYS` (default 14,
`gn_auth/settings.py`) -- so more than one key can be valid at once;
try each one rather than assuming the first or last.

**This is exactly what gn3 already does** (so genenetwork4 should
mirror gn3, not invent a new pattern):

```python
# genenetwork3/gn3/oauth2/jwks.py
def fetch_jwks(authserveruri, path="auth/public-jwks") -> KeySet:
    response = requests.get(urljoin(authserveruri, path), timeout=300)
    return KeySet([JsonWebKey.import_key(key) for key in response.json()["jwks"]])

def validate_token(token: str, keys: KeySet) -> dict:
    for key in keys.keys:
        try:
            return JsonWebToken(["RS256"]).decode(token, key=key)
        except BadSignatureError:
            pass
    raise TokenValidationError("No key was found for validation.")
```

`genenetwork3/gn3/oauth2/authorisation.py`'s `require_token` decorator
then just reads the `Authorization: Bearer <token>` header and calls
the above.

Two things worth doing better than gn3's version, in a genenetwork4
implementation:

- **gn3 refetches the JWKS on every single incoming request** (no
  caching). gn-auth's own internal validator caches the key set and
  only refreshes it every `jwt_refresh_frequency_hours` (default 6,
  `gn_auth/auth/authentication/oauth2/resource_server.py`) -- that's
  the better model: fetch once, cache, refresh periodically (e.g.
  hourly), not per-request.
- **gn3's `validate_token` only checks the signature.** It never calls
  `.validate()` to check `exp`/`nbf`/`iss`, so an expired token with a
  valid signature still passes. genenetwork4 should check the
  signature *and* the standard time-based claims.

## Checking authorization for a specific resource (e.g. a group)

Knowing a token is validly signed only proves *who* the caller is, not
*what they're allowed to do*. For that, gn-auth models permissions as:

- a **resource**: a UUID, a name, a category (`mrna`, `phenotype`,
  `genotype`, `system`, `group`, `inbredset-group`, ...), and whether
  it's public.
- a **privilege**: an id string shaped `<scope>:<category>:<action>`,
  e.g. `group:resource:view-resource`, `system:data:link-to-group`,
  `system:user:masquerade` (real examples from gn-auth's migrations).
- **roles**, which bundle privileges and get assigned to users on
  specific resources.

A GeneNetwork group/`InbredSet` is one such resource. To check "can
this token's user do X on InbredSet Y", **mirror
`genenetwork3/gn3/api/case_attributes.py`'s `required_access`
function** exactly -- two calls:

```python
# 1. Resolve (SpeciesId, InbredSetId) to a gn-auth resource id. No auth needed.
GET {AUTH_SERVER_URL}/auth/resource/populations/resource-id/{species_id}/{inbredset_id}
-> {"resource-id": "<uuid>"}

# 2. Ask what the bearer of this token can do on that resource.
POST {AUTH_SERVER_URL}/auth/resource/authorisation
Authorization: Bearer <token>
{"resource-ids": ["<uuid>"]}
-> {
     "<uuid>": {
       "public-read": false,
       "roles": [
         {"role_id": "...", "role_name": "...",
          "privileges": [{"privilege_id": "group:resource:view-resource", ...}, ...]}
       ]
     }
   }
```

Collect every `privilege_id` across every role in the response and
check the ones you need are in there (`case_attributes.py` does
`all(level in privileges for level in required_levels)`).

There's a dataset/trait-oriented equivalent, `POST
/auth/data/authorisation`, keyed by `"Dataset::Trait"` strings instead
of resource ids (e.g. `"BXDPublish::10001"`) -- useful if genenetwork4
ever needs to authorize a specific trait rather than a whole group;
see `gn_auth/auth/authorisation/data/views.py` for its exact
request/response shape.

Both of these routes are deliberately **not** hard-gated by gn-auth:
call them with no `Authorization` header and you get back only the
public-resource view; add a valid bearer token and you get the full,
user-specific answer. That's by design, not a bug to route around.

## Endpoints reference

This table runs wider than 100 chars: its longest path,
`/auth/resource/populations/resource-id/<speciesid>/<inbredsetid>`, is
66 chars by itself, leaving no room for the Auth required/Purpose
columns if kept under that width.

| Method | Path                                                               | Auth required                                        | Purpose                                                                                          |
|--------|--------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------------------------------------------|
| GET    | `/auth/public-jwks`                                                | none                                                 | RSA public keys for verifying JWTs locally.                                                      |
| POST   | `/auth/token`                                                      | client credentials                                   | Obtain a token (any grant type).                                                                 |
| POST   | `/auth/revoke`                                                     | client credentials                                   | Revoke a token.                                                                                  |
| POST   | `/auth/introspect`                                                 | bearer token, `introspect` scope                     | RFC 7662 introspection -- see Known issues, do not rely on this yet.                             |
| GET    | `/auth/resource/populations/resource-id/<speciesid>/<inbredsetid>` | none                                                 | Map a `(SpeciesId, InbredSetId)` pair to a gn-auth `resource-id`.                                |
| POST   | `/auth/resource/authorisation`                                     | bearer token (falls back to public-only without one) | Given `{"resource-ids": [...]}`, return the caller's roles/privileges on each resource.          |
| GET    | `/auth/resource/authorisation/<name>`                              | bearer token                                         | Roles/privileges for one resource looked up by name.                                             |
| POST   | `/auth/data/authorisation`                                         | bearer token (falls back to public-only without one) | Same as resource/authorisation, but keyed by `"Dataset::Trait"` strings instead of resource ids. |
| GET    | `/auth/data/species`                                               | none                                                 | List species known to gn-auth.                                                                   |

These are the routes relevant to a service consuming gn-auth. gn-auth
has many more (`/auth/role/...`, `/auth/user/...`,
`/auth/system/administration/...`) for managing the auth system itself
-- not needed here.

## Known issues -- don't build on these yet

- **`/auth/introspect` doesn't currently work.** Three independent
  problems: the config key that allow-lists which clients may call it,
  `OAUTH2_CLIENTS_WITH_INTROSPECTION_PRIVILEGE`, is never set by
  default; no client-creation tooling in the repo ever creates a
  client with the `client_type: "internal"` that `check_permission`
  requires (`gn_auth/auth/authentication/oauth2/endpoints/introspection.py`);
  and tokens issued via the password/jwt-bearer/refresh_token grants
  are never written to the `oauth2_tokens` table in the first place
  (`server.py`'s `save_token` is wired to a no-op for JWT tokens), so
  there's nothing in the database for introspection to find even if
  the first two problems were fixed. Use local JWT verification
  (above) instead.
- The introspection response's `sub` claim is dead code -- a helper
  function to compute it exists but its result is never actually
  included in the returned JSON.
- The `password` grant is marked broken by gn-auth's own test suite
  (skipped, with a comment that JWTs broke the original
  opaque-token-shaped test). Prefer the `jwt-bearer` grant, or treat
  token issuance as already-solved by whatever front end mints the
  token for a user and focus genenetwork4's side on verification.

## What this means for genenetwork4

genenetwork4 doesn't need to talk to gn-auth for every request -- only
for the two things above: verifying a presented token's signature
(cache the JWKS, refresh periodically, don't refetch per-request), and
-- for data that isn't public -- asking gn-auth what that token's
holder is allowed to do on a specific resource. Every endpoint needs
the first (see "Every endpoint requires a token" below); only some
need the second.

Whether data "is public" (and so skips the second, resource-level
check) is answered by GeneNetwork's own schema, not by asking gn-auth:
GeneNetwork2 treats a dataset (`ProbeSetFreeze`, `PublishFreeze`,
`GenoFreeze`) as public exactly when `public > 0 AND confidentiality <
1` (`wqflask/api/router.py`, used consistently throughout GN2/GN3).
`InbredSet`/group rows carry a `public` column too, but GN2 never
actually filters on it anywhere -- it's informational only there -- so
groups/species endpoints need no *resource-level* gating, matching
GN2's own behavior; they still need a valid token like every endpoint
does. For dataset/trait data that *is* public by the threshold above,
likewise still require a valid token (don't skip verification
entirely), but skip the resource-authorization call to gn-auth -- one
less network round trip for what is, in practice, most of the data.

A dataset older than 5 years is treated as public too, regardless of
its `public`/`confidentiality` flags -- a confidentiality embargo isn't
meant to last forever. This isn't a GN2 convention, just a rule applied
here, based on `CreateTime`.

### Resolving a probeset/trait to a gn-auth resource

A group/`InbredSet` resource's id is resolved via
`GET /auth/resource/populations/resource-id/<speciesid>/<inbredsetid>`
(above). A probeset/trait doesn't have an equivalent "resolve this id
to a resource id" endpoint -- instead, call
`POST /auth/data/authorisation` directly with a `"Dataset::Trait"`
string (e.g. `{"traits": ["HC_M2_0606_P::1442370_at"]}`); gn-auth
resolves the resource itself, internally, via a join on
`linked_mrna_data.dataset_name` (`gn_auth/auth/authorisation/data/
mrna.py`'s `resources_by_datasets_and_traits`). If the trait's dataset
was never linked to a gn-auth resource at all (the common case for
data nobody has ever restricted), gn-auth answers with a 404, which
genenetwork4 treats the same as "no privileges" -- not a request
failure.

### Implemented, in `src/auth.rs`

- `AuthToken`, an Axum extractor (`impl FromRequestParts`): verifies a
  bearer token's RS256 signature and standard time claims (`exp`/`nbf`,
  checked when present -- gn-auth itself treats them as optional,
  confirmed against its own validator) against a cached, periodically
  refreshed JWKS. Add it as a handler parameter to require a valid
  token; omit it for endpoints that don't need one.
- `is_publicly_readable(public, confidentiality, create_time)`: the GN2
  threshold above, plus the 5-year embargo-age rule, as a plain
  function a handler calls on its own fetched row.
- `authorise(auth_server_url, token, resource_id, privilege)`: the two
  gn-auth calls (resource-id resolution is the caller's job --
  `GET /auth/resource/populations/resource-id/...` -- `authorise`
  itself only wraps `POST /auth/resource/authorisation`), with the
  per-(user, resource) answer cached briefly so repeated requests for
  the same resource by the same user don't each cost a round trip.
- `authorise_trait(auth_server_url, token, dataset_name, trait_name,
  privilege)`: the trait-keyed equivalent of `authorise`, wrapping
  `POST /auth/data/authorisation` -- for data addressed by a
  `(dataset, probeset)` pair rather than a resource id directly. Same
  caching as `authorise`.
- `authorise_dataset(auth_server_url, token, dataset_name, privilege)`:
  `authorise_trait` with an empty trait name, for checking a whole
  dataset at once (e.g. before listing every probeset in it) -- gn-auth
  resolves a trait's resource purely from its dataset name, so the
  trait half of the `"Dataset::Trait"` string never affects the answer.
- `check(auth_server_url, token, dataset_name, trait_name, public,
  confidentiality, create_time)`: combines `is_publicly_readable` and
  `authorise_trait`/`authorise_dataset` (always with
  `"group:resource:view-resource"`, the only privilege any route here
  checks) into the one branch every gated handler below actually
  wants, returning `Option<Authorized>` -- the capability a handler
  can require to construct a "found and visible" result, rather than
  writing the branch out by hand. `is_authorised` is its boolean form,
  for a caller that filters a list instead of rejecting one request
  (`publication_traits`).
- `tests/auth.rs` runs all of this against a real gn-auth instance
  (set `GN4_TEST_AUTH_TOKEN`/`GN4_TEST_RESOURCE_ID`; skips otherwise).

### Wired into routes: the `/datasets/{dataset}/probesets...` family

Three endpoints are gated this way so far, all the same way:
`GET /datasets/{dataset}/probesets/{probeset}`, its `/values` sibling,
and the `GET /datasets/{dataset}/probesets` listing (checked once for
the dataset as a whole via `authorise_dataset`, rather than once per
probeset in the list):

- An `AuthToken` is required unconditionally -- every caller needs a
  valid token, even for public data.
- If the dataset is publicly readable (`is_publicly_readable`), the
  data is returned with no further gn-auth call.
- Otherwise, `authorise_trait`/`authorise_dataset` with
  `"group:resource:view-resource"` is called; a `false` result
  (including the "never linked, so no privileges" 404 case above) is a
  403, not a 404 -- the dataset (and probeset, where there is one)
  exist, the caller just isn't allowed to see them.
- A failure to reach/parse gn-auth's answer is a 502, distinct from a
  403 (denied) or a 500 (this service's own database error) -- see
  `tests/dataset_probeset_info.rs`, `tests/dataset_probeset_values.rs`
  and `tests/dataset_probesets.rs` for all of these cases exercised
  against a real gn-auth instance.

`GET /groups/{group}/traits` and its singular siblings (`/traits/
{trait_id}` and its `/values` sibling) are gated the same way as
`/traits/{dataset}/...` below: resolve the group's own `PublishFreeze`
row (by the group's name -- `PublishXRef`/`PublishFreeze` aren't tied
by a foreign key, so this is a name lookup, not a join) and run the
same `is_publicly_readable`/`authorise_trait`/`authorise_dataset`
check against it. If a group has published traits but no matching
`PublishFreeze` row to check, that's treated as a deny rather than as
ungated/public data. `/traits` (the listing) skips the round trip only
when the group has zero published traits -- there's nothing to
protect in an empty list, so no reason to spend a gn-auth call (or
risk denying) on it.

All three endpoints, and the two above them
(`/datasets/{dataset}/probesets/{probeset}/values` and `/traits/
{dataset}/{trait}/values`), carry `#[gnauth(secured)]`
(`gnauth-macros`): a compile-time tag, checked by that attribute
macro, for handlers that can share actual trait/probeset *values*.
It's a textual check only -- it can't see across function calls -- so
it verifies the handler takes `AuthToken` and mentions
`auth::Authorized`, the capability type only `auth::check` can
produce (its field is private, so it can't be fabricated outside
`src/auth.rs`). Metadata-only endpoints (`/traits/{dataset}`,
`/datasets/{dataset}/probesets`, `/groups/{group}/traits` and their
singular siblings) are gated the same way but don't carry the tag --
they don't share values, just whether a trait/dataset is visible at
all.

`gnauth-macros` also has the opposite tag, `#[gnauth(disabled)]`, for
the handful of endpoints that intentionally take no `AuthToken` at
all -- `hello_world` and `version` (`src/lib.rs`) carry it, marking
"no token needed" as a deliberate decision rather than an oversight.
It checks the converse of `secured`: the signature must *not* take
`AuthToken`. (`routes::dev::get_test_token` also takes no token, but
isn't tagged -- it's a conditionally-registered test utility, a
different category from a normal public endpoint.)

### Wired into routes: `/traits/{dataset}/...`, spanning both categories

`/traits/{dataset}`, `/traits/{dataset}/{trait}` and its `/values`
sibling (`src/routes/traits.rs`) answer the same question for *either*
category through one URL shape -- see the README's `/traits` section
for the addressing scheme. Gating is identical to the
`/datasets/{dataset}/probesets...` family above, just resolved against
whichever table the dataset belongs to: `PublishFreeze.public`/
`confidentiality` feed `is_publicly_readable` for a Publish dataset the
same way `ProbeSetFreeze.public`/`confidentiality` already do for a
ProbeSet one, and `authorise_trait`/`authorise_dataset` is called with
that dataset's own name either way -- gn-auth's
`/auth/data/authorisation` already dispatches "Publish"-suffixed
dataset names to `gn_auth/auth/authorisation/data/phenotypes.py`'s
`resources_by_datasets_and_traits`, the exact mRNA mechanism this
integration already relies on, so no gn-auth-side change was needed to
extend gating to phenotype data through this endpoint.

No non-public `PublishFreeze` exists in the database this was tested
against, so the deny path for the Publish branch specifically wasn't
exercised against real data -- only against the already-proven
ProbeSet branch and gn-auth's own "unlinked dataset" 404-to-deny
behavior, which this new branch calls unchanged. The same caveat
applies to `/groups/{group}/traits...` above, which resolves to the
same `PublishFreeze` check.

### Every endpoint requires a token

Everything above was about *authorization* (is this specific resource
public, does this user have a privilege on it) for the handful of
endpoints that check it. Separately, and more broadly: every endpoint
in this API now requires a valid token (`AuthToken`, checked by local
JWKS signature verification as described above) just to be called at
all -- `/`, `/version`, `/openapi.json` and `/get-test-token` itself
are the only exceptions (a health check, a version check, the spec
describing all this, and the endpoint that hands out a token in the
first place would be circular if it needed one). This is a blanket
requirement independent of whether an endpoint also checks
resource-level authorization: `/species`, `/groups/{group}/strains`,
`/datasets/{dataset}` and the like need a token now but don't check
anything beyond that it's valid -- same as a `is_publicly_readable`
dataset in the `dataset_probeset*`/`traits*` family skips the
authorization call but still needs a valid token.

`ApiDoc`'s `#[openapi(security(("bearer_token" = [])), ...)]`
(`src/lib.rs`) makes this the OpenAPI document's default security
requirement, so individual handlers don't each repeat `security(...)`;
the three exceptions besides `/get-test-token` (`/`, `/version`)
override it with their own empty `security()`. `error::Unauthorized` (a
`#[derive(utoipa::IntoResponses)]` type wrapping `ApiError`) is the
shared 401 response every gated handler references by name in its own
`responses(...)` list instead of writing out the same tuple by hand.

### Getting a token for local testing: `POST /get-test-token`

Every live test in this repo (`tests/*.rs`, `test/python/test_*.py`)
needs a real bearer token, obtained by hand so far (e.g.
`test-token2.txt` in the setup notes). genenetwork4 can hand one out
itself now: `POST /get-test-token` (`src/routes/dev.rs`) calls gn-auth's own
password grant for gn-auth's well-known development user
(`test@development.user`, from `flask init-dev-clients`/
`gn_auth/wsgi.py`) and returns the result as the `data` of the usual
success envelope (README, "Success responses"). See the README's
"Development" section for who it answers (only a caller on this
machine; nothing to enable) and gn-auth's own
`AUTHLIB_INSECURE_TRANSPORT` requirement for accepting the grant over
plain HTTP at all.

This doesn't change anything about how genenetwork4 verifies or
authorizes tokens (src/auth.rs) -- it is purely a convenience for
obtaining one of the tokens that mechanism already accepts, limited to
callers on the same machine by a per-request check of the connection's
peer address (127.0.x.x or ::1, and no proxy header). That check sees a
reverse proxy or a container's network gateway as the peer, so a
deployed instance needs more than this -- ideally
`GENENETWORK4_TEST_TOKEN=false`, which leaves the route out entirely;
see the README's "Behind nginx or in a container".

The test suite uses it automatically: `tests/common::auth_token()`
(Rust) and `test/python/auth_helpers.auth_token()` (Python) both try
`POST /get-test-token` first, falling back to a `GN4_TEST_AUTH_TOKEN`
env var only if that fails. With a reachable gn-auth
(`AUTHLIB_INSECURE_TRANSPORT` set) the whole suite -- every endpoint now
needing a token -- runs with no token manually provisioned at all.

### Found wrong in gn-auth while building this

Four bugs in gn-auth itself blocked this integration before they were
fixed (see gn-auth's own `RELEASE_NOTES.md` and its
`fix-fresh-deployment-bugs` branch): a JWKS cache that could stay empty
for hours after a fresh deployment's first token was issued, a dev
client whose secret was never hashed so it could never authenticate,
`public_resources()` crashing on every fresh deployment, and
`POST /auth/resource/authorisation` crashing (instead of degrading to
the anonymous view) on an invalid rather than missing bearer token.