Publishing genenetwork4 under a path prefix with nginx
How to put genenetwork4 on https://vesuvio.pu.ac.ke/gn4/ while the
server itself listens on a loopback port, with no gn-auth: a bearer
token minted from scripts/gn-auth-stub.py (see "Authentication"
below).
Three things separate this from "just proxy it":
- genenetwork4 assumes it is served at
/. Every link it emits is root-relative ("href": "/groups",spec_url: "/openapi.json",type: "/errors/..."), so under/gn4they point outside the mount point unless nginx (or the client) puts the prefix back. See "Links the API emits". /compute/*runs for minutes in a single request; nginx's default one-minute read timeout cuts it off mid-computation.POST /get-test-tokenis deliberately unusable through a proxy (see README, "Behind nginx or in a container"). With no gn-auth, tokens come from the stub instead.
The server behind nginx
GENENETWORK4_BIND=127.0.0.1 \
GENENETWORK4_PORT=3000 \
GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/path/to/xapian-index \
SEARCH_ENGINE=native \
genenetwork4
SEARCH_ENGINE=native is what makes /search use XAPIAN_DB_PATH. The
default is forward, which proxies to GeneNetwork3 at GN3_URL
(127.0.0.1:5003) -- correct when gn3 is running alongside, and a 502
backend_unreachable when it is not, which is easy to mistake for the
proxy being misconfigured. See docs/search-engines.md.
GENENETWORK4_BIND=127.0.0.1-- nginx is on the same host, so nothing else needs the port. The default is0.0.0.0(all interfaces).GENENETWORK4_TEST_TOKEN=false-- do not registerPOST /get-test-tokenat all. Belt and braces with nginx's own refusal below; the setting fails closed (onlytrue/1/yes/onkeep it).AUTH_SERVER_URLstays local: it is where this process fetches the signing keys from, not a URL that clients need. Nothing about the public/gn4prefix enters it.- Any of
SQL_URI,XAPIAN_DB_PATH,GN3_URL,GEMMA_*, ... as that deployment needs; see README.
nginx
server {
listen 443 ssl;
http2 on;
server_name vesuvio.pu.ac.ke;
ssl_certificate /etc/nginx/ssl/vesuvio.pu.ac.ke.pem;
ssl_certificate_key /etc/nginx/ssl/vesuvio.pu.ac.ke.key;
# genenetwork4 is mounted at /gn4/. The trailing slash on proxy_pass
# is what translates the prefix away: /gn4/species -> /species.
# Without it nginx forwards "/gn4/species" and every request 404s.
location /gn4/ {
proxy_pass http://127.0.0.1:3000/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
# The API says it is forwarding, so genenetwork4 refuses
# POST /get-test-token for these requests even if it is
# registered (GENENETWORK4_TEST_TOKEN=false above is the
# stronger measure; this is the one that survives a config
# mistake).
#
# Belt and braces, independent of the application:
location = /gn4/get-test-token { return 404; }
# /compute/gemma, /compute/gemma/ravanan and /compute/rqtl are
# synchronous: one request, one long computation (16-20s on the
# tiny BXD fixtures, minutes on real datasets, and
# /compute/gemma/ravanan on Slurm measured 158s with Slurm's
# default scheduler interval). The default 60s read timeout
# returns a 504 while the server keeps working.
proxy_read_timeout 3600;
proxy_send_timeout 3600;
# nginx's 1m default is far below what the API accepts: a
# /compute/* body carries one value per individual plus covariates,
# and /plot/* is posted a whole mapping result. Keep this at least
# as large as the application's own 64 MiB limit
# (BODY_LIMIT_BYTES in src/lib.rs), or nginx refuses first.
client_max_body_size 64m;
# No streaming responses: the /compute/*/status/{run_id}
# endpoints are polled and return accumulated lines as JSON, so
# buffering is fine and left at its default.
# Root-relative links in JSON bodies, rewritten to stay under
# /gn4. See "Links the API emits" below before enabling this.
sub_filter_types application/json;
sub_filter_once off;
sub_filter '"href":"/' '"href":"/gn4/';
sub_filter '"spec_url":"/' '"spec_url":"/gn4/';
sub_filter '"type":"/errors' '"type":"/gn4/errors';
sub_filter '"next":"/' '"next":"/gn4/';
sub_filter '"href": "/' '"href": "/gn4/';
sub_filter '"spec_url": "/' '"spec_url": "/gn4/';
sub_filter '"type": "/errors' '"type": "/gn4/errors';
sub_filter '"next": "/' '"next": "/gn4/';
}
# /gn4 without the trailing slash: redirect so relative and absolute
# links agree about where the mount point is.
location = /gn4 { return 301 /gn4/; }
}
The access log, and who is using the API
The format in etc/nginx-vesuvio.pu.ac.ke.conf records the caller's
identity, which the API sends back as a response header:
log_format gn4 '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent rt=$request_time '
'rid=$upstream_http_x_request_id '
'user=$upstream_http_x_gn4_user '
'ua="$http_user_agent"';
user= is the token's subject (pjotrp, robw, ...) -- the API sets
x-gn4-user on every authenticated response for exactly this. Without it
a log answers "which address" and never "which user", and per-user
success becomes guesswork from user agents and behaviour; user=- marks a
request that carried no valid token (scanners, anonymous reads). With it,
per-user traffic and failure rates are one awk away:
awk '{for(i=1;i<=NF;i++){if($i ~ /^user=/)u=$i; if($i ~ /^[0-9][0-9][0-9]$/)s=$i} print u, s}' \
/var/log/nginx/gn4.access.log | sort | uniq -c | sort -rn | head
Restart or reload nginx after changing the format; the old and new lines differ in field count, so cut over between rotations.
Install it instead of the distribution's default site. A
sites-enabled/default that certbot has edited to serve this hostname
will win over this block, silently: sites-enabled/* is globbed in
alphabetical order and nginx uses the first server block whose
server_name matches, so default takes 443 and the gn4 block is the
one nginx discards. The only sign is a warning at startup or reload --
nginx: [warn] conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored
-- and the symptoms are a stock "Welcome to nginx!" page at / and
nginx's own 404 (not genenetwork4's JSON) for /gn4/..., at HTTP/1.1
even though this block sets http2 on. Remove or disable the default
site (rm /etc/nginx/sites-enabled/default, or delete its
certbot-managed server block for this hostname) and reload; grep the
error log for conflicting server name after any change to the vhost
set.
For the vesuvio.pu.ac.ke/gn4 deployment as configured,
etc/nginx-vesuvio.pu.ac.ke.conf is
the same thing as a complete, drop-in file -- TLS, log format, the /gn4
redirect, and the /get-test-token refusal. /compute/* (R/qtl, GEMMA)
is proxied and gated on a valid token, at the cost of letting any token
holder name server-side paths for R and gemma to read — the operator
checklist in docs/www/operator-checklist.md
covers that trade, what the box needs installed, and how to re-block it.
http2 on; needs nginx >= 1.25.1; on older versions use listen 443
ssl http2;. The block above (minus the ssl_* lines and the TLS
listener) is tested against nginx 1.28: nginx -t accepts it, and with
a real genenetwork4 behind it /gn4/version answers, the /gn4
redirect and the /gn4/get-test-token refusal fire, envelopes come back
with /gn4/... hrefs, and an error body comes back with
"type":"/gn4/errors/...", "spec_url":"/gn4/openapi.json" and
"next":"/gn4/species".
Links the API emits
Every success envelope carries next: [{"href": ...}], and every error
carries spec_url, type and sometimes next. All of them are
root-relative by design -- genenetwork4 has no idea it is behind a
prefix -- so a client at https://vesuvio.pu.ac.ke/gn4/species is told
to call https://vesuvio.pu.ac.ke/groups.
Three ways to deal with that, in order of preference:
- Give the service its own host (
https://gn4.example.org/) and drop the prefix. Nothing needs rewriting, and this is what the rest of the README assumes. - Rewrite the three keys in nginx, as in the config above. It is
narrow on purpose: only the values of
"href","next","spec_url"and"type"are touched, so the OpenAPI document's"paths"(which are legitimately root-relative, and which clients resolve against the document's own URL) are left alone. Four keys, each needing both a compact and a spaced spelling:"href"and"next"are objects in an envelope'snextarray and bare strings in an error body,"spec_url"is always a bare string,"type"appears only on errors."next":[(an array) deliberately does not match the"next":"/filter, so an envelope keeps itshref-rewritten objects and nothing else.sub_filter_types application/json;is required -- nginx rewrites onlytext/htmlby default -- andsub_filter_once off;because an envelope carries several links. Three consequences worth knowing: the server'sserdeoutput has no space after the colon, so the compact spellings are the ones that actually fire; nginx dropsContent-Lengthand sends these responses chunked; and the rewrite is textual, so adetailstring that happened to contain"href":"/would be rewritten too (nothing does:notes/detailcome from this server, never from database text or request input). - Let clients re-root: prepend
/gn4to anyhref,spec_urlortypethat starts with/. Correct without touching the server, but every client has to know to do it.
/openapi.json itself is served fine through the proxy. Its paths are
root-relative, which is correct OpenAPI: a tool that reads the document
from https://vesuvio.pu.ac.ke/gn4/openapi.json resolves them under
/gn4 on its own. A tool that hard-codes paths against the host root
will call the wrong place -- that is the same prefix problem, one level
further out.
Authentication without gn-auth
The stub (scripts/gn-auth-stub.py, see README "Tests") is enough for a
deployment that only needs bearer tokens checked:
scripts/gn-auth-stub.py --token-file /run/gn4/public.token
and set AUTH_SERVER_URL=http://127.0.0.1:8081/ (the stub's default) on
genenetwork4. Notes:
- Keep
AUTH_SERVER_URLinternal. Do not route/gn4/or anything else on the public site to the stub: its signing key (test/data/test_rsa_private.pem) is committed to this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. The JWKS only needs to be reachable from the genenetwork4 process. - Tokens minted this way carry
iss=http://127.0.0.1:8081/, which is what genenetwork4 compares againstAUTH_SERVER_URLverbatim (trailing slash included). They are valid only while thatAUTH_SERVER_URLis unchanged, and only against this server. - A token minted without
--grant-resource-idis a public-data-only token: public datasets read normally, every non-public one is a 403, dataset metadata stays visible.--expires-in 30dgives it a month; nothing invalidates it but its ownexp, and it keeps working as long as the stub is reachable when genenetwork4 refreshes its copy of the keys (cached an hour). - Hand the token out of band.
POST /gn4/get-test-tokenis not usable through the proxy: nginx connects from127.0.0.1, which would make every client look local, so genenetwork4 refuses any request carryingX-Forwarded-For/Forwarded/X-Real-IP-- and the config above sets them. That refusal is the point, not an obstacle; keep it.
Swapping in a real gn-auth later changes only AUTH_SERVER_URL (and
whatever iss gn-auth mints -- it must be the same string). See
docs/gn-auth.md.
Checking it
From a client, plainly:
curl -s https://vesuvio.pu.ac.ke/gn4/version
curl -s https://vesuvio.pu.ac.ke/gn4/openapi.json | head -c 200
TOKEN=$(cat /run/gn4/public.token)
curl -s -H "Authorization: Bearer $TOKEN" \
"https://vesuvio.pu.ac.ke/gn4/species" | head -c 200
curl -s -H "Authorization: Bearer $TOKEN" \
"https://vesuvio.pu.ac.ke/gn4/groups/BXD/traits?limit=3"
On the box itself, two things trip up the obvious curl -H "Host: ..."
https://localhost:
- TLS validates the hostname in the URL, not the
Host:header, which only arrives after the handshake -- sohttps://localhostfails against a certificate forvesuvio.pu.ac.keno matter whatHost:says. Use--resolveto connect to loopback while still verifying the real name. - use the name that is actually on the certificate. Check it rather than assuming (`openssl s_client -connect 127.0.0.1:443 -servername vesuvio.pu.ac.ke
