# Publishing genenetwork4 under a path prefix with nginx How to put genenetwork4 on `https://vesuvio.pu.ac.ke/gn4/` while the server itself listens on a loopback port, with no gn-auth: a bearer token minted from `scripts/gn-auth-stub.py` (see "Authentication" below). Three things separate this from "just proxy it": - genenetwork4 assumes it is served at `/`. Every link it emits is **root-relative** (`"href": "/groups"`, `spec_url: "/openapi.json"`, `type: "/errors/..."`), so under `/gn4` they point outside the mount point unless nginx (or the client) puts the prefix back. See "Links the API emits". - `/compute/*` runs for minutes in a single request; nginx's default one-minute read timeout cuts it off mid-computation. - `POST /get-test-token` is deliberately unusable through a proxy (see README, "Behind nginx or in a container"). With no gn-auth, tokens come from the stub instead. ## The server behind nginx ``` GENENETWORK4_BIND=127.0.0.1 \ GENENETWORK4_PORT=3000 \ GENENETWORK4_TEST_TOKEN=false \ AUTH_SERVER_URL=http://127.0.0.1:8081/ \ SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \ XAPIAN_DB_PATH=/path/to/xapian-index \ SEARCH_ENGINE=native \ genenetwork4 ``` `SEARCH_ENGINE=native` is what makes `/search` use `XAPIAN_DB_PATH`. The default is `forward`, which proxies to GeneNetwork3 at `GN3_URL` (`127.0.0.1:5003`) -- correct when gn3 is running alongside, and a `502 backend_unreachable` when it is not, which is easy to mistake for the proxy being misconfigured. See docs/search-engines.md. - `GENENETWORK4_BIND=127.0.0.1` -- nginx is on the same host, so nothing else needs the port. The default is `0.0.0.0` (all interfaces). - `GENENETWORK4_TEST_TOKEN=false` -- do not register `POST /get-test-token` at all. Belt and braces with nginx's own refusal below; the setting fails closed (only `true`/`1`/`yes`/`on` keep it). - `AUTH_SERVER_URL` stays **local**: it is where *this process* fetches the signing keys from, not a URL that clients need. Nothing about the public `/gn4` prefix enters it. - Any of `SQL_URI`, `XAPIAN_DB_PATH`, `GN3_URL`, `GEMMA_*`, ... as that deployment needs; see README. ## nginx ```nginx server { listen 443 ssl; http2 on; server_name vesuvio.pu.ac.ke; ssl_certificate /etc/nginx/ssl/vesuvio.pu.ac.ke.pem; ssl_certificate_key /etc/nginx/ssl/vesuvio.pu.ac.ke.key; # genenetwork4 is mounted at /gn4/. The trailing slash on proxy_pass # is what translates the prefix away: /gn4/species -> /species. # Without it nginx forwards "/gn4/species" and every request 404s. location /gn4/ { proxy_pass http://127.0.0.1:3000/; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; # The API says it is forwarding, so genenetwork4 refuses # POST /get-test-token for these requests even if it is # registered (GENENETWORK4_TEST_TOKEN=false above is the # stronger measure; this is the one that survives a config # mistake). # # Belt and braces, independent of the application: location = /gn4/get-test-token { return 404; } # /compute/gemma, /compute/gemma/ravanan and /compute/rqtl are # synchronous: one request, one long computation (16-20s on the # tiny BXD fixtures, minutes on real datasets, and # /compute/gemma/ravanan on Slurm measured 158s with Slurm's # default scheduler interval). The default 60s read timeout # returns a 504 while the server keeps working. proxy_read_timeout 3600; proxy_send_timeout 3600; # nginx's 1m default is far below what the API accepts: a # /compute/* body carries one value per individual plus covariates, # and /plot/* is posted a whole mapping result. Keep this at least # as large as the application's own 64 MiB limit # (BODY_LIMIT_BYTES in src/lib.rs), or nginx refuses first. client_max_body_size 64m; # No streaming responses: the /compute/*/status/{run_id} # endpoints are polled and return accumulated lines as JSON, so # buffering is fine and left at its default. # Root-relative links in JSON bodies, rewritten to stay under # /gn4. See "Links the API emits" below before enabling this. sub_filter_types application/json; sub_filter_once off; sub_filter '"href":"/' '"href":"/gn4/'; sub_filter '"spec_url":"/' '"spec_url":"/gn4/'; sub_filter '"type":"/errors' '"type":"/gn4/errors'; sub_filter '"next":"/' '"next":"/gn4/'; sub_filter '"href": "/' '"href": "/gn4/'; sub_filter '"spec_url": "/' '"spec_url": "/gn4/'; sub_filter '"type": "/errors' '"type": "/gn4/errors'; sub_filter '"next": "/' '"next": "/gn4/'; } # /gn4 without the trailing slash: redirect so relative and absolute # links agree about where the mount point is. location = /gn4 { return 301 /gn4/; } } ``` ### The access log, and who is using the API The format in `etc/nginx-vesuvio.pu.ac.ke.conf` records the caller's identity, which the API sends back as a response header: ```nginx log_format gn4 '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent rt=$request_time ' 'rid=$upstream_http_x_request_id ' 'user=$upstream_http_x_gn4_user ' 'ua="$http_user_agent"'; ``` `user=` is the token's subject (`pjotrp`, `robw`, ...) -- the API sets `x-gn4-user` on every authenticated response for exactly this. Without it a log answers "which address" and never "which user", and per-user success becomes guesswork from user agents and behaviour; `user=-` marks a request that carried no valid token (scanners, anonymous reads). With it, per-user traffic and failure rates are one `awk` away: ```sh awk '{for(i=1;i<=NF;i++){if($i ~ /^user=/)u=$i; if($i ~ /^[0-9][0-9][0-9]$/)s=$i} print u, s}' \ /var/log/nginx/gn4.access.log | sort | uniq -c | sort -rn | head ``` Restart or reload nginx after changing the format; the old and new lines differ in field count, so cut over between rotations. **Install it *instead of* the distribution's default site.** A `sites-enabled/default` that certbot has edited to serve this hostname will win over this block, silently: `sites-enabled/*` is globbed in alphabetical order and nginx uses the **first** server block whose `server_name` matches, so `default` takes 443 and the gn4 block is the one nginx discards. The only sign is a warning at startup or reload -- ``` nginx: [warn] conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored ``` -- and the symptoms are a stock "Welcome to nginx!" page at `/` and nginx's own `404` (not genenetwork4's JSON) for `/gn4/...`, at HTTP/1.1 even though this block sets `http2 on`. Remove or disable the default site (`rm /etc/nginx/sites-enabled/default`, or delete its certbot-managed `server` block for this hostname) and reload; grep the error log for `conflicting server name` after any change to the vhost set. For the `vesuvio.pu.ac.ke/gn4` deployment as configured, [`etc/nginx-vesuvio.pu.ac.ke.conf`](../etc/nginx-vesuvio.pu.ac.ke.conf) is the same thing as a complete, drop-in file -- TLS, log format, the `/gn4` redirect, and the `/get-test-token` refusal. `/compute/*` (R/qtl, GEMMA) is proxied and gated on a valid token, at the cost of letting any token holder name server-side paths for R and gemma to read — the operator checklist in [`docs/www/operator-checklist.md`](www/operator-checklist.md) covers that trade, what the box needs installed, and how to re-block it. `http2 on;` needs nginx >= 1.25.1; on older versions use `listen 443 ssl http2;`. The block above (minus the `ssl_*` lines and the TLS listener) is tested against nginx 1.28: `nginx -t` accepts it, and with a real genenetwork4 behind it `/gn4/version` answers, the `/gn4` redirect and the `/gn4/get-test-token` refusal fire, envelopes come back with `/gn4/...` hrefs, and an error body comes back with `"type":"/gn4/errors/..."`, `"spec_url":"/gn4/openapi.json"` and `"next":"/gn4/species"`. ## Links the API emits Every success envelope carries `next: [{"href": ...}]`, and every error carries `spec_url`, `type` and sometimes `next`. All of them are root-relative by design -- genenetwork4 has no idea it is behind a prefix -- so a client at `https://vesuvio.pu.ac.ke/gn4/species` is told to call `https://vesuvio.pu.ac.ke/groups`. Three ways to deal with that, in order of preference: 1. **Give the service its own host** (`https://gn4.example.org/`) and drop the prefix. Nothing needs rewriting, and this is what the rest of the README assumes. 2. **Rewrite the three keys in nginx**, as in the config above. It is narrow on purpose: only the values of `"href"`, `"next"`, `"spec_url"` and `"type"` are touched, so the OpenAPI document's `"paths"` (which are legitimately root-relative, and which clients resolve against the document's own URL) are left alone. Four keys, each needing both a compact and a spaced spelling: `"href"` and `"next"` are objects in an envelope's `next` array and bare strings in an error body, `"spec_url"` is always a bare string, `"type"` appears only on errors. `"next":[` (an array) deliberately does not match the `"next":"/` filter, so an envelope keeps its `href`-rewritten objects and nothing else. `sub_filter_types application/json;` is required -- nginx rewrites only `text/html` by default -- and `sub_filter_once off;` because an envelope carries several links. Three consequences worth knowing: the server's `serde` output has no space after the colon, so the compact spellings are the ones that actually fire; nginx drops `Content-Length` and sends these responses chunked; and the rewrite is textual, so a `detail` string that happened to contain `"href":"/` would be rewritten too (nothing does: `notes`/`detail` come from this server, never from database text or request input). 3. **Let clients re-root**: prepend `/gn4` to any `href`, `spec_url` or `type` that starts with `/`. Correct without touching the server, but every client has to know to do it. `/openapi.json` itself is served fine through the proxy. Its `paths` are root-relative, which is correct OpenAPI: a tool that reads the document from `https://vesuvio.pu.ac.ke/gn4/openapi.json` resolves them under `/gn4` on its own. A tool that hard-codes `paths` against the host root will call the wrong place -- that is the same prefix problem, one level further out. ## Authentication without gn-auth The stub (`scripts/gn-auth-stub.py`, see README "Tests") is enough for a deployment that only needs bearer tokens checked: ``` scripts/gn-auth-stub.py --token-file /run/gn4/public.token ``` and set `AUTH_SERVER_URL=http://127.0.0.1:8081/` (the stub's default) on genenetwork4. Notes: - Keep `AUTH_SERVER_URL` **internal**. Do not route `/gn4/` or anything else on the public site to the stub: its signing key (`test/data/test_rsa_private.pem`) is committed to this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. The JWKS only needs to be reachable from the genenetwork4 process. - Tokens minted this way carry `iss` = `http://127.0.0.1:8081/`, which is what genenetwork4 compares against `AUTH_SERVER_URL` verbatim (trailing slash included). They are valid only while that `AUTH_SERVER_URL` is unchanged, and only against this server. - A token minted without `--grant-resource-id` is a **public-data-only** token: public datasets read normally, every non-public one is a 403, dataset metadata stays visible. `--expires-in 30d` gives it a month; nothing invalidates it but its own `exp`, and it keeps working as long as the stub is reachable when genenetwork4 refreshes its copy of the keys (cached an hour). - Hand the token out of band. `POST /gn4/get-test-token` is not usable through the proxy: nginx connects from `127.0.0.1`, which would make every client look local, so genenetwork4 refuses any request carrying `X-Forwarded-For`/`Forwarded`/`X-Real-IP` -- and the config above sets them. That refusal is the point, not an obstacle; keep it. Swapping in a real gn-auth later changes only `AUTH_SERVER_URL` (and whatever `iss` gn-auth mints -- it must be the same string). See docs/gn-auth.md. ## Checking it From a client, plainly: ``` curl -s https://vesuvio.pu.ac.ke/gn4/version curl -s https://vesuvio.pu.ac.ke/gn4/openapi.json | head -c 200 TOKEN=$(cat /run/gn4/public.token) curl -s -H "Authorization: Bearer $TOKEN" \ "https://vesuvio.pu.ac.ke/gn4/species" | head -c 200 curl -s -H "Authorization: Bearer $TOKEN" \ "https://vesuvio.pu.ac.ke/gn4/groups/BXD/traits?limit=3" ``` On the box itself, two things trip up the obvious `curl -H "Host: ..." https://localhost`: * TLS validates the **hostname in the URL**, not the `Host:` header, which only arrives after the handshake -- so `https://localhost` fails against a certificate for `vesuvio.pu.ac.ke` no matter what `Host:` says. Use `--resolve` to connect to loopback while still verifying the real name. * use the name that is actually on the certificate. Check it rather than assuming (`openssl s_client -connect 127.0.0.1:443 -servername vesuvio.pu.ac.ke /gn4/ curl -sSI $R https://vesuvio.pu.ac.ke/gn4 | head -3 # 301 -> /gn4/ curl -s $R https://vesuvio.pu.ac.ke/gn4/version # the version envelope curl -s $R -o /dev/null -w '%{http_code}\n' -X POST \ https://vesuvio.pu.ac.ke/gn4/get-test-token # 404: not registered curl -s $R -o /dev/null -w '%{http_code}\n' -X POST \ https://vesuvio.pu.ac.ke/gn4/compute/correlation # 401: proxied, token needed ``` A 502 instead means nginx is fine and genenetwork4 is not listening on `GENENETWORK4_PORT`; check `curl -s http://127.0.0.1:3000/version` directly. The `/version` reply also names the git revision and build date, which is the quickest way to confirm the proxy is talking to the process you think it is. To check the link rewriting, follow a `next` from a body: ``` curl -s -H "Authorization: Bearer $TOKEN" \ https://vesuvio.pu.ac.ke/gn4/groups/BXD | grep -o '"href": "[^"]*"' | head -3 ``` Each one should start with `/gn4/`. Without the `sub_filter` lines they start with `/`, and resolve outside the mount point. The same check on an error (`/gn4/species/nonexistent`) shows `/gn4/errors/...`, `/gn4/openapi.json` and `/gn4/species`. ## When a request looks like it hangs `/compute/gemma` and `/compute/rqtl` really do run for minutes: they are documented as synchronous, single-request computations. Pass `?id=` and poll `/gn4/compute/gemma/status/{run_id}` (or `/compute/rqtl/status/`) on a second connection to see progress; both are ordinary JSON responses, so nothing about the proxy needs to change for them. `/compute/gemma/ravanan` additionally waits on ravanan (and on Slurm, when `RAVANAN_BATCH_SYSTEM=slurm-api` is set), which is where the two-minute orders of magnitude come from -- see docs/ravanan-slurm.md.