about summary refs log tree commit diff

Publishing genenetwork4 under a path prefix with nginx

How to put genenetwork4 on https://vesuvio.pu.ac.ke/gn4/ while the server itself listens on a loopback port, with no gn-auth: a bearer token minted from scripts/gn-auth-stub.py (see "Authentication" below).

Three things separate this from "just proxy it":

  • genenetwork4 assumes it is served at /. Every link it emits is root-relative ("href": "/groups", spec_url: "/openapi.json", type: "/errors/..."), so under /gn4 they point outside the mount point unless nginx (or the client) puts the prefix back. See "Links the API emits".
  • /compute/* runs for minutes in a single request; nginx's default one-minute read timeout cuts it off mid-computation.
  • POST /get-test-token is deliberately unusable through a proxy (see README, "Behind nginx or in a container"). With no gn-auth, tokens come from the stub instead.

The server behind nginx

GENENETWORK4_BIND=127.0.0.1 \
GENENETWORK4_PORT=3000 \
GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/path/to/xapian-index \
SEARCH_ENGINE=native \
genenetwork4

SEARCH_ENGINE=native is what makes /search use XAPIAN_DB_PATH. The default is forward, which proxies to GeneNetwork3 at GN3_URL (127.0.0.1:5003) -- correct when gn3 is running alongside, and a 502 backend_unreachable when it is not, which is easy to mistake for the proxy being misconfigured. See docs/search-engines.md.

  • GENENETWORK4_BIND=127.0.0.1 -- nginx is on the same host, so nothing else needs the port. The default is 0.0.0.0 (all interfaces).
  • GENENETWORK4_TEST_TOKEN=false -- do not register POST /get-test-token at all. Belt and braces with nginx's own refusal below; the setting fails closed (only true/1/yes/on keep it).
  • AUTH_SERVER_URL stays local: it is where this process fetches the signing keys from, not a URL that clients need. Nothing about the public /gn4 prefix enters it.
  • Any of SQL_URI, XAPIAN_DB_PATH, GN3_URL, GEMMA_*, ... as that deployment needs; see README.

nginx

server {
    listen 443 ssl;
    http2 on;
    server_name vesuvio.pu.ac.ke;

    ssl_certificate     /etc/nginx/ssl/vesuvio.pu.ac.ke.pem;
    ssl_certificate_key /etc/nginx/ssl/vesuvio.pu.ac.ke.key;

    # genenetwork4 is mounted at /gn4/. The trailing slash on proxy_pass
    # is what translates the prefix away: /gn4/species -> /species.
    # Without it nginx forwards "/gn4/species" and every request 404s.
    location /gn4/ {
        proxy_pass http://127.0.0.1:3000/;

        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP         $remote_addr;

        # The API says it is forwarding, so genenetwork4 refuses
        # POST /get-test-token for these requests even if it is
        # registered (GENENETWORK4_TEST_TOKEN=false above is the
        # stronger measure; this is the one that survives a config
        # mistake).
        #
        # Belt and braces, independent of the application:
        location = /gn4/get-test-token { return 404; }

        # /compute/gemma, /compute/gemma/ravanan and /compute/rqtl are
        # synchronous: one request, one long computation (16-20s on the
        # tiny BXD fixtures, minutes on real datasets, and
        # /compute/gemma/ravanan on Slurm measured 158s with Slurm's
        # default scheduler interval). The default 60s read timeout
        # returns a 504 while the server keeps working.
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;

        # nginx's 1m default is far below what the API accepts: a
        # /compute/* body carries one value per individual plus covariates,
        # and /plot/* is posted a whole mapping result. Keep this at least
        # as large as the application's own 64 MiB limit
        # (BODY_LIMIT_BYTES in src/lib.rs), or nginx refuses first.
        client_max_body_size 64m;

        # No streaming responses: the /compute/*/status/{run_id}
        # endpoints are polled and return accumulated lines as JSON, so
        # buffering is fine and left at its default.

        # Root-relative links in JSON bodies, rewritten to stay under
        # /gn4. See "Links the API emits" below before enabling this.
        sub_filter_types application/json;
        sub_filter_once off;
        sub_filter '"href":"/'        '"href":"/gn4/';
        sub_filter '"spec_url":"/'    '"spec_url":"/gn4/';
        sub_filter '"type":"/errors'  '"type":"/gn4/errors';
        sub_filter '"next":"/'        '"next":"/gn4/';
        sub_filter '"href": "/'       '"href": "/gn4/';
        sub_filter '"spec_url": "/'   '"spec_url": "/gn4/';
        sub_filter '"type": "/errors' '"type": "/gn4/errors';
        sub_filter '"next": "/'       '"next": "/gn4/';
    }

    # /gn4 without the trailing slash: redirect so relative and absolute
    # links agree about where the mount point is.
    location = /gn4 { return 301 /gn4/; }
}

The access log, and who is using the API

The format in etc/nginx-vesuvio.pu.ac.ke.conf records the caller's identity, which the API sends back as a response header:

log_format gn4 '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent rt=$request_time '
               'rid=$upstream_http_x_request_id '
               'user=$upstream_http_x_gn4_user '
               'ua="$http_user_agent"';

user= is the token's subject (pjotrp, robw, ...) -- the API sets x-gn4-user on every authenticated response for exactly this. Without it a log answers "which address" and never "which user", and per-user success becomes guesswork from user agents and behaviour; user=- marks a request that carried no valid token (scanners, anonymous reads). With it, per-user traffic and failure rates are one awk away:

awk '{for(i=1;i<=NF;i++){if($i ~ /^user=/)u=$i; if($i ~ /^[0-9][0-9][0-9]$/)s=$i} print u, s}' \
  /var/log/nginx/gn4.access.log | sort | uniq -c | sort -rn | head

Restart or reload nginx after changing the format; the old and new lines differ in field count, so cut over between rotations.

Install it instead of the distribution's default site. A sites-enabled/default that certbot has edited to serve this hostname will win over this block, silently: sites-enabled/* is globbed in alphabetical order and nginx uses the first server block whose server_name matches, so default takes 443 and the gn4 block is the one nginx discards. The only sign is a warning at startup or reload --

nginx: [warn] conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored

-- and the symptoms are a stock "Welcome to nginx!" page at / and nginx's own 404 (not genenetwork4's JSON) for /gn4/..., at HTTP/1.1 even though this block sets http2 on. Remove or disable the default site (rm /etc/nginx/sites-enabled/default, or delete its certbot-managed server block for this hostname) and reload; grep the error log for conflicting server name after any change to the vhost set.

For the vesuvio.pu.ac.ke/gn4 deployment as configured, etc/nginx-vesuvio.pu.ac.ke.conf is the same thing as a complete, drop-in file -- TLS, log format, the /gn4 redirect, and the /get-test-token refusal. /compute/* (R/qtl, GEMMA) is proxied and gated on a valid token, at the cost of letting any token holder name server-side paths for R and gemma to read — the operator checklist in docs/www/operator-checklist.md covers that trade, what the box needs installed, and how to re-block it.

http2 on; needs nginx >= 1.25.1; on older versions use listen 443 ssl http2;. The block above (minus the ssl_* lines and the TLS listener) is tested against nginx 1.28: nginx -t accepts it, and with a real genenetwork4 behind it /gn4/version answers, the /gn4 redirect and the /gn4/get-test-token refusal fire, envelopes come back with /gn4/... hrefs, and an error body comes back with "type":"/gn4/errors/...", "spec_url":"/gn4/openapi.json" and "next":"/gn4/species".

Every success envelope carries next: [{"href": ...}], and every error carries spec_url, type and sometimes next. All of them are root-relative by design -- genenetwork4 has no idea it is behind a prefix -- so a client at https://vesuvio.pu.ac.ke/gn4/species is told to call https://vesuvio.pu.ac.ke/groups.

Three ways to deal with that, in order of preference:

  1. Give the service its own host (https://gn4.example.org/) and drop the prefix. Nothing needs rewriting, and this is what the rest of the README assumes.
  2. Rewrite the three keys in nginx, as in the config above. It is narrow on purpose: only the values of "href", "next", "spec_url" and "type" are touched, so the OpenAPI document's "paths" (which are legitimately root-relative, and which clients resolve against the document's own URL) are left alone. Four keys, each needing both a compact and a spaced spelling: "href" and "next" are objects in an envelope's next array and bare strings in an error body, "spec_url" is always a bare string, "type" appears only on errors. "next":[ (an array) deliberately does not match the "next":"/ filter, so an envelope keeps its href-rewritten objects and nothing else. sub_filter_types application/json; is required -- nginx rewrites only text/html by default -- and sub_filter_once off; because an envelope carries several links. Three consequences worth knowing: the server's serde output has no space after the colon, so the compact spellings are the ones that actually fire; nginx drops Content-Length and sends these responses chunked; and the rewrite is textual, so a detail string that happened to contain "href":"/ would be rewritten too (nothing does: notes/detail come from this server, never from database text or request input).
  3. Let clients re-root: prepend /gn4 to any href, spec_url or type that starts with /. Correct without touching the server, but every client has to know to do it.

/openapi.json itself is served fine through the proxy. Its paths are root-relative, which is correct OpenAPI: a tool that reads the document from https://vesuvio.pu.ac.ke/gn4/openapi.json resolves them under /gn4 on its own. A tool that hard-codes paths against the host root will call the wrong place -- that is the same prefix problem, one level further out.

Authentication without gn-auth

The stub (scripts/gn-auth-stub.py, see README "Tests") is enough for a deployment that only needs bearer tokens checked:

scripts/gn-auth-stub.py --token-file /run/gn4/public.token

and set AUTH_SERVER_URL=http://127.0.0.1:8081/ (the stub's default) on genenetwork4. Notes:

  • Keep AUTH_SERVER_URL internal. Do not route /gn4/ or anything else on the public site to the stub: its signing key (test/data/test_rsa_private.pem) is committed to this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. The JWKS only needs to be reachable from the genenetwork4 process.
  • Tokens minted this way carry iss = http://127.0.0.1:8081/, which is what genenetwork4 compares against AUTH_SERVER_URL verbatim (trailing slash included). They are valid only while that AUTH_SERVER_URL is unchanged, and only against this server.
  • A token minted without --grant-resource-id is a public-data-only token: public datasets read normally, every non-public one is a 403, dataset metadata stays visible. --expires-in 30d gives it a month; nothing invalidates it but its own exp, and it keeps working as long as the stub is reachable when genenetwork4 refreshes its copy of the keys (cached an hour).
  • Hand the token out of band. POST /gn4/get-test-token is not usable through the proxy: nginx connects from 127.0.0.1, which would make every client look local, so genenetwork4 refuses any request carrying X-Forwarded-For/Forwarded/X-Real-IP -- and the config above sets them. That refusal is the point, not an obstacle; keep it.

Swapping in a real gn-auth later changes only AUTH_SERVER_URL (and whatever iss gn-auth mints -- it must be the same string). See docs/gn-auth.md.

Checking it

From a client, plainly:

curl -s https://vesuvio.pu.ac.ke/gn4/version
curl -s https://vesuvio.pu.ac.ke/gn4/openapi.json | head -c 200
TOKEN=$(cat /run/gn4/public.token)
curl -s -H "Authorization: Bearer $TOKEN" \
     "https://vesuvio.pu.ac.ke/gn4/species" | head -c 200
curl -s -H "Authorization: Bearer $TOKEN" \
     "https://vesuvio.pu.ac.ke/gn4/groups/BXD/traits?limit=3"

On the box itself, two things trip up the obvious curl -H "Host: ..." https://localhost:

  • TLS validates the hostname in the URL, not the Host: header, which only arrives after the handshake -- so https://localhost fails against a certificate for vesuvio.pu.ac.ke no matter what Host: says. Use --resolve to connect to loopback while still verifying the real name.
  • use the name that is actually on the certificate. Check it rather than assuming (`openssl s_client -connect 127.0.0.1:443 -servername vesuvio.pu.ac.ke