diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-07 14:40:00 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-07 14:40:00 +0000 |
| commit | f346dc273ef38cdb268fd68bb5697a8ee30e5fde (patch) | |
| tree | fa68a18aefe6f530b4dab5bbaa42c557a3cb4127 /docs | |
| parent | afcb0b3f48d7ecf61ea41ce99618eef70b1030d1 (diff) | |
| download | genenetwork4-f346dc273ef38cdb268fd68bb5697a8ee30e5fde.tar.gz | |
docs/www: rewrite the AI instructions for the live deployment
ai-instruction.md was written before anything was deployed: it took a
placeholder base URL, did not say where the API is mounted, and buried
the operator's deployment notes in an appendix labelled "not for the AI
reader" -- awkward to hand to an assistant, and confusing when it is.
So: the AI-facing page now names the deployment
(https://vesuvio.pu.ac.ke/gn4) and is self-contained, and the operator
material moves to operator-checklist.md. Everything in the page was
exercised against the running deployment.
Content changes, beyond the move:
* `next` hrefs arrive already prefixed (`/gn4/groups/BXD/strains`), so
they resolve against the origin -- joining them to the current URL's
directory double-prefixes them. Said plainly, since it is the one
thing the prefix breaks.
* says what cannot work and should not be retried: get-test-token is a
404, /compute/* is a 403 from the front end, and neither is a
permissions problem to work around.
* `/publications/{pubmed_id}/traits` wants the PubMed id, not the
internal `publication_id` in the same trait record, which 404s.
* states the current rate-limit position (none configured) instead of
implying a 429 policy that does not exist yet.
* docs/www/ai-instruction.md: rewritten, paste-ready.
* docs/www/operator-checklist.md: new file, the former appendix, updated
with what is now enforced at the proxy and a record of the verified
request matrix.
* README.md, docs/nginx.md, etc/nginx-vesuvio.pu.ac.ke.conf: point at
the operator checklist by its new name.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/nginx.md | 2 | ||||
| -rw-r--r-- | docs/www/ai-instruction.md | 360 | ||||
| -rw-r--r-- | docs/www/operator-checklist.md | 175 |
3 files changed, 298 insertions, 239 deletions
diff --git a/docs/nginx.md b/docs/nginx.md index 6b7c8e9..414d0e4 100644 --- a/docs/nginx.md +++ b/docs/nginx.md @@ -142,7 +142,7 @@ the same thing as a complete, drop-in file -- TLS, log format, the `/gn4` redirect, and the `/compute/` and `/get-test-token` refusals. Note that it refuses `/compute/*`: those endpoints take server-side file paths, so they belong on the internal listener only (see the operator checklist in -[`docs/www/ai-instruction.md`](www/ai-instruction.md)); drop the two +[`docs/www/operator-checklist.md`](www/operator-checklist.md)); drop the two `location` lines only if every token holder may read the server's files. `http2 on;` needs nginx >= 1.25.1; on older versions use `listen 443 diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index f613f26..f514577 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -1,71 +1,54 @@ # Using the GeneNetwork API from an AI assistant -Minimal instructions for fetching public GeneNetwork data over HTTP. The -full endpoint-by-endpoint description is in -[`../../README.md`](../../README.md) and, machine-readably, at -`GET /openapi.json`; this page is only about *getting in* and staying -inside the public data. +Read-only, public-data HTTP API. The base URL for every request is ``` -1. You are given a token → by the operator, out of band -2. Send it on every request → Authorization: Bearer <token> -3. Read JSON, check meta → {"data": ..., "meta": ..., "notes": [...], "next": [...]} +https://vesuvio.pu.ac.ke/gn4 ``` -## 1. Get a token +so `/gn4/species` below means `https://vesuvio.pu.ac.ke/gn4/species`. +`/gn4` without the trailing slash redirects there; everything outside +`/gn4/` is a 404 from the front end, not from the API. -**You cannot mint a token yourself.** Tokens are issued by hand, on the -server, one per user, and handed to you out of band. If you do not -already have a token in your task context, stop and ask for one — do not -go looking for a token URL, an API key or a login form on the public -host; there is none. +Machine-readable description of every endpoint: `GET /gn4/openapi.json`. +Endpoint-by-endpoint prose: [`../../README.md`](../../README.md). -```sh -# what the operator runs, on the server itself, once per user: -scripts/gn-auth-stub.py --mint-only --subject alice \ - --expires-in 30d --token-file /run/gn4/alice.token +``` +1. You are given a token → by the operator, out of band +2. Send it on every request → Authorization: Bearer <token> +3. Read JSON, check meta → {"data": ..., "meta": ..., "notes": [...], "next": [...]} ``` -At this stage, no gn-auth is involved: the deployment runs -`scripts/gn-auth-stub.py` as its whole auth server, and `--mint-only` -signs a JWT with that stub's key and issuer, prints it (and writes -`--token-file`) and exits without listening — so issuing a token -disturbs nothing already running, and the token verifies immediately. -`--subject` is the name that becomes the token's `sub`: that is what -makes a call attributable to one user rather than to the server as a -whole. Installs that do have gn-auth mint from -`POST /get-test-token` instead; either way, minting happens on the box, -never through the public host. +## 1. Get a token -Paste that token into your environment and use it as a bearer token: +**You cannot mint a token yourself.** Tokens are issued by a human, on +the server, one per user. If you have not been given one in your task +context, stop and ask — do not go looking for a login form, an API key, +or a token endpoint on the public host; there is none +(`POST /gn4/get-test-token` is a deliberate 404). ```sh -API=https://<deployment> # the public base URL -TOKEN=… # the JWT you were given +TOKEN=… # the JWT you were given +API=https://vesuvio.pu.ac.ke/gn4 curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/traits?limit=1" ``` -The token is a **JWT**, valid until its `exp` — whatever `--expires-in` -said when it was minted (30 days in the command above). Nothing renews -or extends it, and there is no way to revoke one early, so treat the -token you were given as your only credential for its whole life. It -identifies you (`sub`); it does not by itself grant access to anything -that is not public (see §4). +The token is a **JWT** whose `exp` is set at mint time (typically 30 +days). Nothing renews it and nothing revokes it early, so: -Rules that keep you working: - -* one token for the whole session; use it on every request; -* never put the token in a URL, in a commit, or in anything you keep or - print for others; -* on `401` the token is missing, expired or not signed by the deployment's - auth server. Requesting another one is **not** something you can do: - say so, and ask the operator for a fresh token (then retry once); -* if the deployment rate-limits you (`429`), honour `Retry-After` and back - off exponentially. +* use it on every request, and keep it for the whole session; +* never put it in a URL, a commit, a log line, or anything you show to + someone else; +* on `401` the token is missing, expired, or was not issued for this + deployment. You cannot fix that yourself: say so, ask for a fresh + token, retry once; +* it carries no rights of its own — it gets you past the door, and what + you may read is decided per dataset (see §4). Anything non-public + answers `403` no matter which token you hold. ## 2. Discover the API, don't guess -Two routes need no token and are always safe to call first: +Two routes are safe to call first and need no token: ```sh curl -s "$API/openapi.json" # every endpoint, parameter, schema @@ -79,238 +62,139 @@ Every response is one envelope: | `data` | the payload | | `meta` | `state` (`complete` or `partial`), `took_ms`, `request_id`, plus `columns`/`num_rows` on tabular payloads | | `notes` | server-authored remarks; **read them**, they explain partial answers | -| `next` | relative links to related endpoints — following them is the cheapest way to walk the data | +| `next` | links to related endpoints — following them is the cheapest way to walk the data | + +`meta.state: partial` means the answer is usable but something is +missing, with the reason in `notes`. Do not read `null` enrichment +fields as "no such value in GeneNetwork"; read it as "ask again later". -`meta.state: partial` means the answer is usable but something is missing -(the reason is in `notes`) — typically the database was unavailable, so -enrichment fields are `null`. Do not treat `null` as "no value in -GeneNetwork"; treat it as "ask again later". +**How to use `next`.** Each `href` is already an absolute path +including the mount point, e.g. `/gn4/groups/BXD/strains`. Resolve it +against the origin, not against the URL you just fetched: + +```sh +curl -s -H "Authorization: Bearer $TOKEN" "https://vesuvio.pu.ac.ke$href" +``` + +Joining it to the current URL's directory instead (`…/gn4/gn4/…`) +double-prefixes it. ## 3. What you can read -All of these are read-only GETs on public data, all need the bearer token: +All of these are read-only `GET`s on public data; all need the token. +Paths are relative to `https://vesuvio.pu.ac.ke/gn4`. | endpoint | what it gives you | |---|---| -| `/species`, `/species/{species}`, `/species/{species}/datasets` | species and their expression datasets | +| `/species`, `/species/{species}` | species records | +| `/species/{species}/datasets` | expression datasets of a species, across its groups | | `/groups?species={name}`, `/groups/{group}` | populations/InbredSets (e.g. `BXD`) | -| `/groups/{group}/strains` | the samples/strains of a group | +| `/groups/{group}/strains` | samples/strains of a group | | `/groups/{group}/datasets` | expression datasets of a group | -| `/groups/{group}/case-attributes` | case-attribute names and values | -| `/groups/{group}/traits` | **every published (phenotype) trait of the group** — the list you want for bulk work | +| `/groups/{group}/case-attributes` | case-attribute names and per-strain values | +| `/groups/{group}/traits` | **every published (phenotype) trait of the group** — the list for bulk work | | `/groups/{group}/traits/{trait_id}` | one trait's metadata | | `/groups/{group}/traits/{trait_id}/values` | one trait's value per strain | | `/datasets/{dataset}`, `/datasets/{dataset}/probesets` | expression dataset and its probesets | | `/datasets/{dataset}/probesets/{probeset}` (+`/values`) | one expression trait, its values | | `/traits/{dataset}`, `/traits/{dataset}/{trait}` (+`/values`) | the same, by dataset | -| `/publications/{pubmed_id}/traits` | traits a paper reports, across groups | +| `/publications/{pubmed_id}/traits` | traits a paper reports, across groups. `{pubmed_id}` is the **PubMed** id (`2760876`), which a trait record carries as `pubmed_id` — not the internal `publication_id` (`116`) beside it, which will 404 there | | `/accessions/{accession}` | resolve a GN accession (`GN600`) to its dataset | | `/mapping-methods` | QTL methods offered | | `/search?q={query}` | full-text search over published phenotypes | -Search takes gn3's Xapian grammar: free text, `AND`/`OR`, phrases, -`group:BXD`, `dataset:BXDPublish`, `author:smith`, `species:mouse`, -`chr:1`, and numeric ranges (`year:2020..2024`). Boolean prefix values -are case-insensitive. `limit` defaults to 20 and caps at **100**; `page` -pages through hits. There is **no total** in the response: to count, page -until a short page — and note that each page costs time proportional to -the number of remaining matches, so paging through 15 000 hits takes about -half a minute. For a bulk list, use `/groups/{group}/traits` instead. +### Search + +`q` takes the Xapian query grammar: free text, `AND`/`OR`/`NOT`, +`"quoted phrases"`, and field prefixes — `group:BXD`, +`dataset:BXDPublish`, `author:smith`, `species:mouse`, `symbol:insr`, +`rif:insulin`, `chr:1`, `peakchr:1`, plus numeric ranges on +`mean`, `peak`, `peakmb`, `additive` and `year` written +`field:lo..hi` (either end may be omitted). Boolean prefix values are +case-insensitive. `type` selects `phenotype` (default) or `gene`. + +`limit` defaults to 20 and caps at **100**; `page` pages through hits. +There is **no total** in the response — to count, page until a short +page, and note that each page costs time proportional to the number of +matches remaining, so paging through ~15 000 hits takes about half a +minute and hammers the server. For bulk work use +`/groups/{group}/traits`, not search. ## 4. What is public — and what you will not get A dataset is readable without further authorization when * `public > 0` **and** `confidentiality < 1`, **or** -* it is older than five years (confidentiality embargoes expire). +* it is older than five years (embargoes expire). -That is the only rule; it matches GeneNetwork2/3. Everything else answers -`403 forbidden`. +That is the only rule, and it matches GeneNetwork2/3. Everything else +answers `403 forbidden`. * Metadata (names, descriptions, dataset records, trait metadata) is - visible for datasets that are not readable; **values** are not. -* The search index holds public documents only, so search will not reveal - private traits — it will simply not return them. -* There are **no write endpoints** and **no server-side compute endpoints** - in the public surface: `POST /compute/*` (GEMMA, R/qtl, ravanan) runs - jobs on the server's filesystem and is deliberately not exposed. Do not - attempt to use it; it will be refused. -* Token minting is entirely operator-side (`scripts/gn-auth-stub.py - --mint-only`, see §1): it runs on the server, never through the public - host. `/get-test-token` — the equivalent route on gn-auth-backed - installs — is not reachable from here either, and you do not need it: - your token comes from a human. - -If your task needs private data or server-side computation, stop and ask a -human: it needs an account and a gn-auth grant, not a different URL. + visible even for datasets whose **values** you may not read. +* The search index holds public documents only, so search never reveals + private traits — it simply does not return them. +* There are **no write endpoints**. +* Server-side computation is **not exposed**: `POST /compute/*` (GEMMA, + R/qtl, ravanan) runs jobs against the server's own filesystem and is + refused with `403` by the front end. Do not attempt it; the `403` is + the answer, not a permissions problem to work around. + +If your task needs private data or server-side computation, stop and ask +a human: that needs an account and a grant, not a different URL. ## 5. Errors and what to do Errors are RFC 7807-style JSON (`type`, `title`, `status`, `code`, -`detail`, `operation_id`, `request_id`, sometimes `next`): +`detail`, `operation_id`, `spec_url`, `request_id`, sometimes `next`). +`type`, `spec_url` and `next` are absolute paths under `/gn4/`. | status | code (typical) | do | |---|---|---| -| 401 | `invalid_token` | expired/invalid token: you cannot mint one — ask the operator for a fresh token, then retry once | -| 403 | `forbidden` | the resource is not public; do not retry | -| 404 | `not_found` | check the name; `next` often lists valid values | -| 422 | `invalid_query`, `invalid_lmm` | fix the query/parameter; do not retry | -| 429 | — | back off, honour `Retry-After` | -| 502 / 503 | `backend_unreachable`, `db_unavailable` | transient; retry with exponential backoff (max ~5 tries) | +| 401 | `invalid_token` | expired/invalid token (the `detail` distinguishes `ExpiredSignature` from `InvalidIssuer`); you cannot mint one — ask the operator, retry once | +| 403 | `forbidden` | the data is not public: do not retry | +| 404 | `not_found` | check the name; `next` often lists the valid values | +| 422 | `invalid_query`, `invalid_lmm` | fix the query or parameter; do not retry | +| 429 | — | back off, honour `Retry-After` (not currently emitted, but written into the API's contract) | +| 502 / 503 | `backend_unreachable`, `db_unavailable` | transient: retry with exponential backoff, max ~5 tries | + +Quote `request_id` when reporting a problem — it is in the body, in the +`x-request-id` header, and in the server's access log. -`request_id` is the handle for reporting a problem; quote it. +No rate limit is configured on this deployment today. Be considerate +anyway: a few concurrent requests, no tight polling loops, and prefer +one bulk call to a thousand small ones. ## 6. A worked example ```sh -API=https://<deployment> -TOKEN=... # from §1 - -curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD" # group record -curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/traits" \ - | jq '.data | length' # 15060 published traits -curl -s -H "Authorization: Bearer $TOKEN" "$API/groups/BXD/traits/10001/values" \ - | jq '.data.strains[0:3]' -curl -s -H "Authorization: Bearer $TOKEN" \ - --get --data-urlencode 'q=cerebellum AND group:BXD' --data 'limit=5' "$API/search" +API=https://vesuvio.pu.ac.ke/gn4 +TOKEN=… # from §1 +AUTH="Authorization: Bearer $TOKEN" + +curl -s -H "$AUTH" "$API/groups/BXD" # group record +curl -s -H "$AUTH" "$API/groups/BXD/traits" \ + | jq '.data | length' # 15060 published traits +curl -s -H "$AUTH" "$API/groups/BXD/traits/10001" # one trait's metadata +curl -s -H "$AUTH" "$API/groups/BXD/traits/10001/values" \ + | jq '.data.strains[0:3]' # its values +curl -s -H "$AUTH" --get --data-urlencode 'q=cerebellum AND group:BXD' \ + --data 'limit=5' "$API/search" +curl -s -H "$AUTH" "$API/datasets/Br_U_0803_M/probesets/100001_at" +curl -s -H "$AUTH" "$API/publications/2760876/traits" | jq '.data | length' ``` +Watch the two id spaces: a trait record carries both `publication_id` +(internal, e.g. `116`) and `pubmed_id` (e.g. `2760876`); only the +PubMed one works in `/publications/{pubmed_id}/traits`. + For a bulk job: fetch `/groups/{group}/traits` once (it is the whole -`PublishXRef` list for the group, with `id`, `phenotype_id`, `dataset`, -`abbreviation`, `nsamples`), then fetch `…/traits/{id}/values` per trait, -politely (a few concurrent requests, backoff on 429). +`PublishXRef` list for the group — `id`, `phenotype_id`, `dataset`, +`abbreviation`, `nsamples`), then fetch `…/traits/{id}/values` per +trait, a few at a time. --- -## Appendix: operator checklist (not for the AI reader) - -Deploying this on the internet without leaking is mostly about *not* -exposing things that were built for a trusted caller, and about keeping -both the token-minting path and the auth stub off the public surface. - -**Must be off / not routed** - -* `POST /get-test-token` — irrelevant while there is no gn-auth (the - deployment's tokens come from `scripts/gn-auth-stub.py --mint-only`, - see "Token issuance"), but the moment gn-auth is behind it, this is a - minting route, and the only thing standing between it and the internet - is a loopback peer check plus a refusal of requests that carry proxy - headers (`src/routes/dev.rs`). Behind nginx the peer *is* loopback, so - one omitting `X-Forwarded-For` makes it open to the world: **do not - proxy it**, keep GN4 bound to `127.0.0.1`, and set - `GENENETWORK4_TEST_TOKEN=false` on any instance that is not the - operator's. -* `POST /compute/*` (`/compute/gemma`, `/compute/gemma/ravanan`, - `/compute/rqtl`, `/compute/correlation`) — these take **server-side file - paths** (`genotype_file`, `snps_file`) and run gemma/ravanan/R on them; - with a token they are an arbitrary-read/execution surface. They are - `token_only`, not resource-checked, and must stay on the internal - listener. Same for `/compute/*/status/{id}`, which tails run logs by - name. -* Any dev-build leftovers: `target/debug` binaries, the `GN4_DB_MODE=dev` - "start without a database" mode. -* The auth stub on `:8081` is *not* a leftover any more — without gn-auth - it is the auth server — but it must stay reachable from genenetwork4 - only: **never** route it publicly. Its signing key - (`test/data/test_rsa_private.pem`) and JWKS are in this repository, so - anyone who can fetch its JWKS can mint tokens that genenetwork4 accepts - (see "Token issuance" below). - -**Token issuance (no gn-auth: the stub, one token per user)** - -Without gn-auth the deployment's auth server is -`scripts/gn-auth-stub.py`, run once and left running (supervise it like -genenetwork4 — see "Tokens are stateless" in the README for why an hour -of downtime is survivable and longer is not): - -```sh -AUTH_SERVER_URL=http://127.0.0.1:8081/ \ - scripts/gn-auth-stub.py --token-file /run/gn4/stub.token -``` - -`AUTH_SERVER_URL` is where genenetwork4 fetches the key set -(`/auth/public-jwks`), so it stays a loopback URL here: the public -`/gn4` prefix has nothing to do with it (docs/nginx.md). - -Then one token per user, minted on the box, without touching the running -stub: - -```sh -AUTH_SERVER_URL=http://127.0.0.1:8081/ \ - scripts/gn-auth-stub.py --mint-only --subject alice \ - --expires-in 30d --token-file /run/gn4/alice.token -``` - -`--mint-only` mints, prints, writes the file and exits; `--issuer` -defaults to `AUTH_SERVER_URL` when that is set, which is what makes the -token verify against the running stub. Hand the file's contents over out -of band. - -What that token is, and is not: - -* **public data only.** tokens minted without `--grant-resource-id` carry - no privileges at all: genenetwork4 decides what may go out from each - dataset's `public`/`confidentiality`/`create_time` columns, and asks - the authorisation lookup about the rest (`src/auth.rs`'s `check`). The - lookup says "no privileges", so public datasets read and every - non-public one is a 403. Dataset *metadata* stays visible either way, - by design. **Never** mint with `--grant-resource-id` for a shared or - long-lived token: that hands the holder that non-public resource, and - there is no way to take it back before `exp`. -* **expiry is the only revocation.** There is no denylist, so a token is - good until its `exp` and no longer; `--expires-in` is therefore the - withdrawal mechanism (30d for a known user is fine, shorter for - anything you might need to pull). Revoking *everything* means changing - `AUTH_SERVER_URL` (or the key) — `iss` is compared verbatim, so a new - host/scheme/trailing slash invalidates every outstanding token. -* **`sub` is attribution, not authentication.** It names the user in the - token and is used as an authorisation-cache key; it grants nothing. - Anyone holding a token is that `sub` as far as the API can tell, and - the `scope` claim is not consulted for data access. -* **different users, different tokens** — `--subject alice`, - `--subject bob`: same key and issuer, independent `exp`, so one - person's token can be rotated or left to lapse without touching anyone - else's. -* verify before handing over: `curl -s -H "Authorization: Bearer $TOKEN" - "$API/groups" | head -c 80` should return an envelope, and - `401 invalid_token` with `InvalidIssuer` means `AUTH_SERVER_URL` and - the minted `iss` disagree. - -When gn-auth lands, nothing above changes for the AI reader except where -the token comes from: `POST /get-test-token` (gn-auth's dev password -grant) replaces `--mint-only`, and the tokens gain gn-auth's own -lifetime (7–14 days, bounded by JWKS rotation) and its per-user grants. -That route must never be proxied — see "Must be off" above. - -**Data exposure** - -* Public rule: `public > 0 && confidentiality < 1`, or older than five - years (`PUBLIC_AFTER_YEARS`, `src/auth.rs`). Values endpoints enforce it - per dataset; `/groups/{group}/traits` checks the group's publish freeze. -* Metadata endpoints are `token_only` by design and will reveal the names - and descriptions of *non-public* datasets (e.g. `GET /datasets/{name}` - for a `public=0` ProbeSetFreeze returns the record). Decide whether that - is acceptable; if not, gate `datasets.rs::dataset_by_name`, - `groups.rs::{groups, group_by_name, group_datasets, group_strains, - group_case_attributes}`, `species.rs` and `traits.rs` metadata on the - same public rule. -* Rebuild, do not inherit, the search index: GN4's indexer applies - `public > 0 AND confidentiality < 1` at crawl time (`src/indexer/rows.rs`), - whereas gn3's `index-genenetwork` has no `WHERE` clause. Verified on the - current full index: `dataset:hcppublish` (a `confidentiality=1` freeze) - and `dataset:br_u_0303_m` (a `public=0` freeze) both return 0 hits. -* Error bodies can carry internal detail: a `db_unavailable` 503 includes - the driver error with the database address. Strip or generalise `detail` - at the proxy for `/` outside the deployment, or accept disclosing - `127.0.0.1:3306`. -* Keep `/openapi.json`, `/version` and `/` open (they are useful and - harmless), and serve everything over TLS. - -**Optional but worth it** - -* Rate limit per token at the proxy (`limit_req`), with `Retry-After`. -* Log `request_id`, token `sub`, path and status; alert on 4xx spikes from - a single `sub` (a leaking client) and on `/compute` or `/get-test-token` - being reached at all. -* A `/search` `total`/facet would end the paging-to-count pattern that - costs ~23 s per 15 000 hits; see `docs/search-engines.md`. +Operator-side notes (deployment, token issuance, what must not be +routed) are in [`operator-checklist.md`](operator-checklist.md). You do +not need them to use the API. diff --git a/docs/www/operator-checklist.md b/docs/www/operator-checklist.md new file mode 100644 index 0000000..583983e --- /dev/null +++ b/docs/www/operator-checklist.md @@ -0,0 +1,175 @@ +# Operator checklist: deploying genenetwork4 for public, read-only use + +Companion to [`ai-instruction.md`](ai-instruction.md), which is what the +end user (typically an AI assistant) is given. This file is for whoever +runs the deployment: what has to stay off the public surface, how tokens +are issued without gn-auth, and what is knowingly exposed. + +The reference deployment is `https://vesuvio.pu.ac.ke/gn4/` proxied by +nginx to `127.0.0.1:3000` — see [`../nginx.md`](../nginx.md) and +[`../../etc/nginx-vesuvio.pu.ac.ke.conf`](../../etc/nginx-vesuvio.pu.ac.ke.conf). + +``` +nginx 443 / 80 public; TLS; /gn4/ -> 127.0.0.1:3000 +genenetwork4 127.0.0.1:3000 loopback only, SEARCH_ENGINE=native +gn-auth stub 127.0.0.1:8081 loopback only, serves the JWKS +MariaDB 127.0.0.1:3306 +xapian index XAPIAN_DB_PATH +``` + +## Must be off / not routed + +* `POST /get-test-token` — irrelevant while there is no gn-auth (tokens + come from `scripts/gn-auth-stub.py --mint-only`, see below), but the + moment gn-auth sits behind it this is a minting route, and the only + thing standing between it and the internet is a loopback peer check + plus a refusal of requests carrying proxy headers + (`src/routes/dev.rs`). Behind nginx the peer *is* loopback, so a proxy + that sets none of those headers makes it open to the world. Belt and + braces, all three: `GENENETWORK4_TEST_TOKEN=false` (route not + registered), bind to `127.0.0.1`, and the `location = + /gn4/get-test-token { return 404; }` refusal in the nginx config. +* `POST /compute/*` (`/compute/gemma`, `/compute/gemma/ravanan`, + `/compute/rqtl`, `/compute/correlation`) — these take **server-side + file paths** (`genotype_file`, `snps_file`) and run gemma/ravanan/R on + them; with a token they are an arbitrary-read (and -execution) + surface. They are `token_only`, not resource-checked. The nginx config + answers `403` for `/gn4/compute` and `/gn4/compute/…`; keep that + unless every token holder may read the box's filesystem. Same for + `/compute/*/status/{id}`, which tails run logs by name. +* The auth stub on `:8081` is not a dev leftover — without gn-auth it + *is* the auth server — but it must stay reachable from genenetwork4 + only: **never** route it publicly. Its signing key + (`test/data/test_rsa_private.pem`) and JWKS are in this repository, so + anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. +* Dev leftovers: `target/debug` binaries, `GN4_DB_MODE=dev` ("start + without a database"). + +## Token issuance (no gn-auth: the stub, one token per user) + +Run the stub once and leave it running, supervised like genenetwork4 — +see "Tokens are stateless" in the README for why an hour of downtime is +survivable and longer is not: + +```sh +AUTH_SERVER_URL=http://127.0.0.1:8081/ \ + scripts/gn-auth-stub.py --token-file /run/gn4/stub.token +``` + +`AUTH_SERVER_URL` is where genenetwork4 fetches the key set +(`/auth/public-jwks`); it stays a loopback URL, and the public `/gn4` +prefix has nothing to do with it. + +Then one token per user, on the box, without touching the running stub: + +```sh +AUTH_SERVER_URL=http://127.0.0.1:8081/ \ + scripts/gn-auth-stub.py --mint-only --subject alice \ + --expires-in 30d --token-file /run/gn4/alice.token +``` + +`--mint-only` mints, prints, writes the file and exits; `--issuer` +defaults to `AUTH_SERVER_URL`, which is what makes the token verify +against the running stub. Hand the file's contents over out of band. + +What that token is, and is not: + +* **public data only.** A token minted without `--grant-resource-id` + carries no privileges: genenetwork4 decides what may go out from each + dataset's `public`/`confidentiality`/`create_time` columns and asks + the authorisation lookup about the rest (`src/auth.rs`'s `check`). The + lookup answers "no privileges", so public data reads and every + non-public dataset is a `403`. **Never** mint with + `--grant-resource-id` for a shared or long-lived token: it hands the + holder that non-public resource and cannot be taken back before `exp`. +* **expiry is the only revocation.** No denylist exists, so a token is + good until its `exp`; `--expires-in` is the withdrawal mechanism. + Revoking *everything* means changing `AUTH_SERVER_URL` or the signing + key — `iss` is compared verbatim, so a new host, scheme or trailing + slash invalidates every outstanding token with `InvalidIssuer`. +* **`sub` is attribution, not authentication.** It names the user, is + used as an authorisation-cache key, and grants nothing. Anyone holding + a token is that `sub` as far as the API can tell, and the `scope` + claim is not consulted for data access. +* **per user, per expiry** — `--subject alice`, `--subject bob`: same + key and issuer, independent `exp`, so one token can lapse without + touching the others. +* verify before handing over: `curl -s -H "Authorization: Bearer $TOKEN" + https://vesuvio.pu.ac.ke/gn4/groups | head -c 80` should return an + envelope. + +When gn-auth lands, only the source of tokens changes: +`POST /get-test-token` (gn-auth's dev password grant) replaces +`--mint-only`, and tokens gain gn-auth's own lifetime (7–14 days, +bounded by JWKS rotation) and its per-user grants. + +## Data exposure + +* Public rule: `public > 0 && confidentiality < 1`, or older than five + years (`PUBLIC_AFTER_YEARS`, `src/auth.rs`). Values endpoints enforce + it per dataset; `/groups/{group}/traits` checks the group's publish + freeze. +* Metadata endpoints are `token_only` by design and reveal the names and + descriptions of *non-public* datasets (e.g. `GET /datasets/{name}` for + a `public=0` ProbeSetFreeze returns the record, with its `public` and + `confidentiality` flags). This was a deliberate decision: a caller + needs to know a dataset exists to know it is being refused. If that is + not acceptable, gate `datasets.rs::dataset_by_name`, + `groups.rs::{groups, group_by_name, group_datasets, group_strains, + group_case_attributes}`, `species.rs` and `traits.rs` metadata on the + same rule. +* Rebuild, do not inherit, the search index: GN4's indexer applies + `public > 0 AND confidentiality < 1` at crawl time + (`src/indexer/rows.rs`), whereas gn3's `index-genenetwork` has no + `WHERE` clause. Verified on the current full index: + `dataset:hcppublish` (a `confidentiality=1` freeze) and + `dataset:br_u_0303_m` (a `public=0` freeze) both return 0 hits. +* Error bodies can carry internal detail: a `db_unavailable` `503` + includes the driver error with the database address. Strip or + generalise `detail` at the proxy, or accept disclosing + `127.0.0.1:3306`. +* Serve everything over TLS (port 80 redirects), keep `/openapi.json`, + `/version` and the version envelope's git revision open — they are + useful and harmless. + +## Verified on this deployment + +``` +GET / 302 -> /gn4/ (HTTP/2) +GET /gn4 301 -> /gn4/ +GET /gn4/version 200 envelope, git revision +GET /gn4/openapi.json 200 +GET /gn4/groups/BXD no token 401 / token 200 +GET /gn4/groups/BXD/traits?limit=2 200 +GET /gn4/search?q=alcohol&limit=3 200 state: complete +GET /gn4/datasets/{non-public}/… 403 +GET /gn4/species/nope 404 links rewritten to /gn4/… +POST /gn4/get-test-token 404 +POST /gn4/compute/correlation 403 +http://…/ 301 -> https://… +``` + +`next` links come back already prefixed (`/gn4/groups/BXD/strains`) and +resolving one through the proxy returns 200, so the `sub_filter` +rewriting in nginx is doing its job for success and error bodies alike. +Check it after any nginx change: without it every link points outside +the mount point, and the API's own tests cannot see that (they run +in-process, where there is no prefix). + +## Optional but worth it + +* Rate limit per address at the proxy: `limit_req_zone` + + `limit_req`/`limit_req_status 429` are commented in the config. 429 + rather than 503, because `Retry-After` on 429 is what the API's + clients are told to expect. +* Put genenetwork4 and the stub under the same supervision (systemd + units, not a shell): the stub's availability bounds every + authenticated request, and `JWKS_TTL` (an hour) is the grace period. +* Logs: the nginx `gn4` log format records status, `rt`, the + `x-request-id` and the user agent. genenetwork4 does **not** log the + token's `sub`, so attribution stops at the token you handed out — if + that matters, mint one token per consumer and correlate by the token + file, or wait for gn-auth. +* A `/search` `total`/facet would end the paging-to-count pattern that + costs ~23 s per 15 000 hits; see + [`../search-engines.md`](../search-engines.md). |
