about summary refs log tree commit diff

Operator checklist: deploying genenetwork4 for public, read-only use

Companion to ai-instruction.md, which is what the end user (typically an AI assistant) is given. This file is for whoever runs the deployment: what has to stay off the public surface, how tokens are issued without gn-auth, and what is knowingly exposed.

The reference deployment is https://vesuvio.pu.ac.ke/gn4/ proxied by nginx to 127.0.0.1:3000 — see ../nginx.md and ../../etc/nginx-vesuvio.pu.ac.ke.conf.

nginx            443 / 80      public; TLS; /gn4/ -> 127.0.0.1:3000
genenetwork4     127.0.0.1:3000 loopback only, SEARCH_ENGINE=native
gn-auth stub     127.0.0.1:8081 loopback only, serves the JWKS
MariaDB          127.0.0.1:3306
xapian index     XAPIAN_DB_PATH

Must be off / not routed

  • POST /get-test-token — irrelevant while there is no gn-auth (tokens come from scripts/gn-auth-stub.py --mint-only, see below), but the moment gn-auth sits behind it this is a minting route, and the only thing standing between it and the internet is a loopback peer check plus a refusal of requests carrying proxy headers (src/routes/dev.rs). Behind nginx the peer is loopback, so a proxy that sets none of those headers makes it open to the world. Belt and braces, all three: GENENETWORK4_TEST_TOKEN=false (route not registered), bind to 127.0.0.1, and the location = /gn4/get-test-token { return 404; } refusal in the nginx config.
  • POST /compute/* (/compute/gemma, /compute/gemma/ravanan, /compute/rqtl, /compute/correlation) is now proxied — R/qtl and GEMMA are part of the service, gated on a valid token (see "Compute" below). Understand what that grants: those handlers take server-side file paths (genotype_file, snps_file) and run R/gemma on them, so every token holder can ask the box to read its own filesystem. It is token_only, not resource-checked: the public-data-only restriction that applies to traits and datasets does not apply here. The nginx config comments show how to re-block it (location /gn4/compute/ { return 403; }) if that is no longer wanted. /compute/*/status/{id} tails run logs by name and shares the same gate.
  • The auth stub on :8081 is not a dev leftover — without gn-auth it is the auth server — but it must stay reachable from genenetwork4 only: never route it publicly. Its signing key (test/data/test_rsa_private.pem) and JWKS are in this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts.
  • Dev leftovers: target/debug binaries, GN4_DB_MODE=dev ("start without a database").

Token issuance (no gn-auth: the stub, one token per user)

The request path is: the AI assistant has no token, so it asks the user (see §1 of ai-instruction.md); the user asks the GeneNetwork team; someone here mints a token for that user and hands it back out of band; the user passes it to the assistant. Expect the request to say who it is for, what data, and for how long — that is what --subject and --expires-in need.

Run the stub once and leave it running, supervised like genenetwork4 — see "Tokens are stateless" in the README for why an hour of downtime is survivable and longer is not:

AUTH_SERVER_URL=http://127.0.0.1:8081/ \
  scripts/gn-auth-stub.py --token-file /run/gn4/stub.token

AUTH_SERVER_URL is where genenetwork4 fetches the key set (/auth/public-jwks); it stays a loopback URL, and the public /gn4 prefix has nothing to do with it.

Then one token per user, on the box, without touching the running stub:

AUTH_SERVER_URL=http://127.0.0.1:8081/ \
  scripts/gn-auth-stub.py --mint-only --subject alice \
      --expires-in 30d --token-file /run/gn4/alice.token

--mint-only mints, prints, writes the file and exits; --issuer defaults to AUTH_SERVER_URL, which is what makes the token verify against the running stub. Hand the file's contents over out of band.

What that token is, and is not:

  • public data only. A token minted without --grant-resource-id carries no privileges: genenetwork4 decides what may go out from each dataset's public/confidentiality/create_time columns and asks the authorisation lookup about the rest (src/auth.rs's check). The lookup answers "no privileges", so public data reads and every non-public dataset is a 403. Never mint with --grant-resource-id for a shared or long-lived token: it hands the holder that non-public resource and cannot be taken back before exp.
  • expiry is the only revocation. No denylist exists, so a token is good until its exp; --expires-in is the withdrawal mechanism. Revoking everything means changing AUTH_SERVER_URL or the signing key — iss is compared verbatim, so a new host, scheme or trailing slash invalidates every outstanding token with InvalidIssuer.
  • sub is attribution, not authentication. It names the user, is used as an authorisation-cache key, and grants nothing. Anyone holding a token is that sub as far as the API can tell, and the scope claim is not consulted for data access.
  • per user, per expiry — --subject alice, --subject bob: same key and issuer, independent exp, so one token can lapse without touching the others.
  • verify before handing over: curl -s -H "Authorization: Bearer $TOKEN" https://vesuvio.pu.ac.ke/gn4/groups | head -c 80 should return an envelope.

When gn-auth lands, only the source of tokens changes: POST /get-test-token (gn-auth's dev password grant) replaces --mint-only, and tokens gain gn-auth's own lifetime (7–14 days, bounded by JWKS rotation) and its per-user grants.

Compute: what the box needs

R/qtl and GEMMA run as child processes, so the genenetwork4 process's environment has to be able to find them. From a plain shell it is not enough to have the binary in a store path — the R packages matter too. The deployment shell gives both:

P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT')
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ genenetwork4 …other env…
  • $P/bin carries Rscript, gemma-wrapper and gemma. If the profile's gemma-wrapper is older than the pin in guix.scm, point GEMMA_WRAPPER_CMD at the store path directly rather than editing PATH: -pve reaches GEMMA either way, but the per-run .log.txt files (where pve lives) are only kept beside the archive from 5eb53fa on.
  • R_LIBS_SITE=$P/site-library/ is what lets scripts/rqtl_wrapper.R load optparse, qtl, stringi and stringr. Without it the request fails with rqtl_failed and there is no package called 'optparse' — an environment problem, not a token or proxy problem.

Real panels are served from GENOTYPE_FILES (default /home/wrk/tmp/ai-root/factory/gn4-data/genotype-files), in GeneNetwork2's layout, and GET /groups/{group}/genotypes exposes them by version only — no paths, no file names — which is what POST /compute/{rqtl,gemma} resolve group + genotype_version against. The layout: genotype/BXD.8.geno (WGS Mar2022, 235 individuals × 21 057 markers) with GEMMA's genotype/bimbam/BXD.8_geno.txt + _snps.txt, and genotype/HET3-ITP.12.geno (6 447 individuals × 894 markers) for UM-HET3. docs/genotypes.md covers how the current version is chosen, which engine each group offers, and the measured runs.

Both engines match values to individuals by position; the API hides that by aligning named samples onto the panel itself, and by dropping parents/F1s the only way that works — a panel-subset genotype file under the engine's cache directory (/tmp/gn4-rqtl/prepared, /tmp/gn4-gemma/prepared), reused by content and panel. If those caches are wiped, the next mapping recreates them (a few seconds on BXD).

Parents and F1s are excluded by default, as in GN2/GN3 (GN2's incparentsf1, GN3's get_sample_list(inc_par=False, inc_f1=False)). BXD.8's panel is 198 RI strains + 37 F1s and no parent columns, so that default means "the 198 strains"; HET3-ITP.12 is individual animals only. null-filling a non-strain individual keeps its phenotype out of the fit but not its genotype out of GEMMA's kinship matrix or R/qtl's cross, so excluding it means subsetting the genotype file — which is what the API should do when it takes over the alignment.

Which engines a group offers is InbredSet.MappingMethodId (1 → GEMMA, QTLReaper, R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA, PLINK): BXD = 1, HET3-ITP = 3. So UM-HET3 is R/qtl only, correct for a 4-way cross even though bimbam/HET3-ITP.* exists.

Measured on this box with the newest genotypes: BXD (BXD.8, 235 individuals, trait 10001 with 34 values) 17.1 s / 21 055 marker rows on GEMMA and 7.5 s / 21 056 markers on R/qtl; UM-HET3 lifespan over HET3-ITP.12 (6 447 individuals, 6 438 with values) 83.1 s / 893 markers, peak LOD 7.95 at chr 1, 24.04 Mb. Details and the misgivings about HET3-ITP's versioning are in docs/genotypes.md.

/compute/gemma/ravanan additionally needs ravanan and ccwl on PATH, and with RAVANAN_BATCH_SYSTEM=slurm-api a reachable slurmrestd (see docs/ravanan-slurm.md).

Repeat calls are cached by input (gemma-wrapper's content-hash cache, and src/rqtl.rs's own hash of genotype+phenotype), so the second identical request is cheap: the same GEMMA run measured ~2 s cached against ~20 s cold. Worth telling users, because the alternative they reach for — downloading the values and fitting a scan themselves — is both slower and wrong (their genotypes are not the panel's).

Verified through the proxy with a token: POST /gn4/compute/rqtl returned 200 in 0.4 s with the fixture's marker rows, POST /gn4/compute/gemma 200 in 2.1 s with 7 320 marker rows, and GET /gn4/compute/gemma/status/<id> ended with GN4: computation complete. Without a token both are 401.

Two operational consequences: the nginx proxy_read_timeout (3600 s) must stay generous, since a real mapping takes minutes; and each request is CPU-bound on the box, so rate-limit per token or keep the audience small — limit_req in the config is one way, but it keys on address, not token.

Database connections

SQL_CHECK_HEALTH (default true) makes the pool ping a connection at checkout and replace it when the ping fails. Keep it on: without it, a connection MariaDB closed while idle becomes a 500 database_error on whichever request draws it, one per dead socket, until the process restarts — that is what happened on 2026-11-07. GET /version now carries "database": "reachable" | "unreachable" (cached ten seconds), so an up-but-useless service is visible without calling a data endpoint.

Compute concurrency

The gates exist to stop large computations piling up, so they only apply above a threshold that differs by endpoint: compute_gate::GEMMA_LARGE_PANEL is 50 individuals (all GEMMA runs share GEMMA_CACHE_DIR, where gemma-wrapper extracts per-chromosome kinship archives by content hash -- two runs at once are two writers over the same hundreds of megabytes), compute_gate::RQTL_LARGE_PANEL is 1 000 (a scanone over a few hundred individuals is seconds of one core). A panel below its endpoint's threshold takes no slot. Above it, POST /compute/gemma takes one job at a time, POST /compute/rqtl RQTL_MAX_CONCURRENT (default 8) and /compute/gemma/ravanan one at a time, each one large job per caller. All three are in-process gates (src/compute_gate.rs), so they are per server process and reset on restart -- a job's child process, however, outlives the server. Waits are bounded only by nginx's proxy_read_timeout.

The slot count is fixed when a gate is first built (its first large request), so raising RQTL_MAX_CONCURRENT takes a restart, not a reload. Changing a threshold is the same: both are constants, and the reasons for the numbers are in their doc comments.

Plotting

POST /plot/rqtl and /plot/gemma take a posted mapping response -- no mapping is re-run -- and hand it to the deployment's plotter: scripts/plot-rqtl.py / scripts/plot-gemma.py, executed by PLOT_PYTHON (default python3) -- so python3 must be on the server process's PATH, alongside Rscript/gemma-wrapper for the mapping itself. They return image/svg+xml rather than an envelope, and the plotter's peak summary in x-plot-summary. Point RQTL_PLOTTER / GEMMA_PLOTTER elsewhere if the scripts move; a missing script is a 502 plot_failed naming the interpreter and script it tried.

PheWAS

GET /groups/{group}/phewas reads PublishXRef only -- the peak marker, LRS and the join to Phenotype/Publication -- plus the group's genotype marker table for positions. It is token-only like the other metadata routes, and it inherits the exposure question below: PublishXRef covers published traits, whose datasets are the *Publish freezes.

Data exposure

  • Public rule: public > 0 && confidentiality < 1, or older than five years (PUBLIC_AFTER_YEARS, src/auth.rs). Values endpoints enforce it per dataset; /groups/{group}/traits checks the group's publish freeze.
  • Metadata endpoints are token_only by design and reveal the names and descriptions of non-public datasets (e.g. GET /datasets/{name} for a public=0 ProbeSetFreeze returns the record, with its public and confidentiality flags). This was a deliberate decision: a caller needs to know a dataset exists to know it is being refused. If that is not acceptable, gate datasets.rs::dataset_by_name, groups.rs::{groups, group_by_name, group_datasets, group_strains, group_case_attributes}, species.rs and traits.rs metadata on the same rule.
  • Rebuild, do not inherit, the search index: GN4's indexer applies public > 0 AND confidentiality < 1 at crawl time (src/indexer/rows.rs), whereas gn3's index-genenetwork has no WHERE clause. Verified on the current full index: dataset:hcppublish (a confidentiality=1 freeze) and dataset:br_u_0303_m (a public=0 freeze) both return 0 hits.
  • Error bodies can carry internal detail: a db_unavailable 503 includes the driver error with the database address. Strip or generalise detail at the proxy, or accept disclosing 127.0.0.1:3306.
  • Serve everything over TLS (port 80 redirects), keep /openapi.json, /version and the version envelope's git revision open — they are useful and harmless.

Verified on this deployment

GET  /                                302 -> /gn4/              (HTTP/2)
GET  /gn4                             301 -> /gn4/
GET  /gn4/version                     200  envelope, git revision
GET  /gn4/openapi.json                200
GET  /gn4/groups/BXD                  no token 401 / token 200
GET  /gn4/groups/BXD/traits?limit=2   200
GET  /gn4/search?q=alcohol&limit=3     200  state: complete
GET  /gn4/datasets/{non-public}/…     403
GET  /gn4/species/nope                404  links rewritten to /gn4/…
POST /gn4/get-test-token              404
POST /gn4/compute/correlation         200 with a token / 401 without
http://…/                             301 -> https://…

next links come back already prefixed (/gn4/groups/BXD/strains) and resolving one through the proxy returns 200, so the sub_filter rewriting in nginx is doing its job for success and error bodies alike. Check it after any nginx change: without it every link points outside the mount point, and the API's own tests cannot see that (they run in-process, where there is no prefix).

When you update the running build

This deployment is updated in place, so the build revision users see at GET /gn4/version moves under them. Two things follow:

  • their assistants are told (in ai-instruction.md) to pull the repository and re-read it whenever git is a revision they have not seen before, because the page they cached no longer describes the API they are now calling. Nothing enforces that from this side, so if a release adds endpoints or options, say so where you hand out tokens;
  • source_digest changes with every build, dirty or not -- it identifies the binary, so quote it in bug reports rather than the commit alone.

Restarting it: trust /version, not the pid file

nohup env … ./target/release/genenetwork4 & does not reliably leave the server's own pid in $! — on this box it named an intermediate process, so kill $(cat /tmp/gn4.pid) killed that and left the server answering. The replacement then died with AddrInUse (in its log, not on the console) while the old build kept serving: everything looked successful and nothing had changed.

Two checks make that impossible to miss:

# who is actually serving (no ps here; read /proc, and match argv[0])
for d in /proc/[0-9]*; do a0=$(tr '\0' '\n' < $d/cmdline 2>/dev/null | head -1); \
  case "$a0" in ./target/release/genenetwork4) echo "${d#/proc/}";; esac; done

# and what is actually answering: git must be the commit you just built
curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r '.data | "\(.git) \(.dirty)"'

Match argv[0] against the binary path, not the whole command line: nohup env … leaves a wrapper process whose command line also contains that path, so a substring match finds the wrapper -- which is how the pid file ended up naming a process that exits while the server keeps running. A git value older than the build you just made is this mistake, not a build problem: write the resolved pid into the pid file (and start the replacement only after :3000 has stopped answering). source_digest changes on every build, so it distinguishes two builds of the same commit.

Optional but worth it

  • Rate limit per address at the proxy: limit_req_zone + limit_req/limit_req_status 429 are commented in the config. 429 rather than 503, because Retry-After on 429 is what the API's clients are told to expect.
  • Put genenetwork4 and the stub under the same supervision (systemd units, not a shell): the stub's availability bounds every authenticated request, and JWKS_TTL (an hour) is the grace period.
  • Logs: the nginx gn4 log format records status, rt, the x-request-id, user= (the API's x-gn4-user response header, i.e. the token's subject, so per-user success is awk '{print $9, $10}' away instead of inferred from addresses) and the user agent. user=- means the request carried no valid token: scanners and anonymous reads, never a caller whose token failed after authenticating.
  • genenetwork4 writes its own access line too, one JSON object per request on stderr, naming the caller:

json {"event":"request","method":"GET","path":"/groups/BXD","status":200,"took_ms":10,"request_id":"…","user":"zachs"}

So tail -f on the process log answers "who ran this job" without going through the proxy's log — which matters while watching a run, and it is the same log the event: error lines go to, correlated by request_id. user is absent (not -) when no token was verified. The path is logged without its query string, and the token itself is never logged — only its subject. Turn it off with GN4_ACCESS_LOG=false where the proxy's access log is the record of truth. * A /search total/facet would end the paging-to-count pattern that costs ~23 s per 15 000 hits; see ../search-engines.md.

Production layout on this box (2026-11-15, no container)

The API runs the store binary directly, as wrk, with no guix shell container around it. Two instances (3000, 3001) are started by factory/gn4-prod/run.sh, which sources factory/gn4-prod/env.sh:

runuser -u wrk -- /home/wrk/tmp/ai-root/factory/gn4-prod/run.sh restart

run.sh also takes start, stop and status; the instances log to factory/gn4-prod/logs/<port>.log (appending across restarts, so the access lines survive a deploy). Deploying is therefore: guix build -f guix.scm, edit env.sh's GEN4=, run.sh restart.

Then run the monitor. Every time, without being asked:

bin/monitor-gn4 --token-file /tmp/prodtok.file --tier all

And run it cold as well as warm after anything that touches the environment, and weekly:

bin/monitor-gn4 --token-file /tmp/prodtok.file --tier compute --cold

A warm run answers from caches that were filled by the previous binary, so it can pass while the cold path -- the one a new user's first request takes -- is broken. --cold sends each cached engine a payload with one value nudged, which every engine here keys on (gemma-wrapper's content hash, ravanan's interned inputs, rqtl/qtl2/bulklmm's own hashes), then repeats the original payload for the warm hit. Both runs must pass, and the line reports cold X / warm Y. Measured on this box with a 30-sample fixture: ravanan 102.0 s cold / 3.7 s warm; gemma 3.1 / 2.0; rqtl, qtl2, fitqtl and BulkLMM are compute-bound at this size and show cold ~= warm, which is a fact about the fixture, not a fault. Two costs to know: ravanan's cold run takes minutes, and every cold run leaves another entry in the caches (GEMMA_CACHE_DIR, RAVANAN_CACHE_DIR, …), so run it deliberately rather than in a loop.

/version only says which commit is running; it cannot see that an engine lost its tool path, that a config key moved, or that an endpoint now answers a shape nobody expected. On 2026-11-15 the service was "deployed and up" with database: reachable while every shell-out engine answered 502 and /search answered 502 in 1.4 ms, and the only thing that caught it was this monitor. A deploy that has not been through it is not finished. Expect 39+ ok, 0 failed and read any skip: the listener check skips whenever the service runs outside the monitor's PID namespace, which is normal on this box and is the only skip that should ever appear.

Everything mutable is under factory/gn4-prod/ — tokens.sqlite (the only copy: losing it loses every token), gemma-cache, rqtl-cache, qtl2-cache, bulklmm-cache, bnw-cache. The ravanan cache is the exception: it must be under /home/wrk/tmp/ai-root/slurm-share/ because the Slurm container bind-mounts nothing else (see ../ravanan-slurm.md).

Two things that are easy to get wrong, both measured on this box:

  • The workflow and manifest come from the checkout, not from the store build: GEMMA_RAVANAN_WORKFLOW and GEMMA_RAVANAN_MANIFEST point into factory/code/genenetwork/genenetwork4/cwl/gemma/, so the checkout has to be at the commit the binary was built from — otherwise the CWL that runs is not the one that was tested.
  • The ravanan in env.sh is a locally patched build. /gnu/store/amn5dfh4…-ravanan-0.2.0 comes from factory/code/ravanan (commit 6f9031b) and carries three patches the upstream 0.2.0 lacks: --slurm-api-endpoint parsed into a URI, user-error printed unconditionally (job scripts have no log level, so their failures were silent), and {"job": …} instead of {"jobs": […]} on submit. That last one is worth remembering: the plural form is slurmrestd's heterogeneous job submission, so every step arrived as a one-component het job holding a whole node, and a 20-chromosome scatter ran strictly 1-wide (cold 24.7 s). With it fixed: 20-wide, cold 9.4 s. Rebuild with guix build --with-source=ravanan=<checkout> --without-tests=ravanan ravanan (its own test suite wants node and fails in the sandbox).
  • ravanan 0.2.0 exists on this box in two variants, and only one works with --slurm-api-endpoint. In /gnu/store/1ydn2bdw…-ravanan-0.2.0 (and 20aqba00…) the option handler converts its argument, (acons 'slurm-api-endpoint (string->uri arg) result); in rim2v0pi… and vyqs04g9… it does not, so slurm-api.scm calls uri-scheme on a string and the run dies with In procedure struct-vtable: Wrong type argument …: "http://127.0.0.1:6820" after staging every input, i.e. looking like a Slurm problem. env.sh pins the working one.

GENENETWORK4_TEST_TOKEN=false in production: /get-test-token mints a token for any local caller, which is a development convenience only. Mint with POST /tokens (localhost-only) instead.

The tool block env.sh must keep

Every engine that shells out needs absolute paths in env.sh; without them the read surface and search stay healthy while all of compute answers 502 (measured 2026-11-15: 1 ok / 7 failed / 1 busy before, 8 ok / 1 failed after). gn4-container/gn4.env is the reference for the keys:

PATH=$R_BIN:$WRAPPER:/usr/bin:/bin     # Rscript first: no R in any system profile
R_LIBS_SITE=<profile>/site-library      # r-qtl and r-qtl2 live here
GEMMA_WRAPPER_CMD, GEMMA_DIR, GEMMA_WRAPPER_DIR   # + gemma/time/tar/xz/parallel/pfff
RQTL_WRAPPER_PATH, FITQTL_SCRIPT_PATH, QTL2_SCRIPT_PATH,
GEMMA_PLOTTER, BULKLMM_PLOTTER, BULKLMM_SCRIPT, GENOTYPE_PLOTTER, BNW_PLOTTER
BNW_CMD, BNW_BIN_DIR, BNW_EXAMPLES_DIR
BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT
BASKET_DIR                              # per-user SQLite, not /tmp

Two of these are easy to get wrong:

  • The script paths are absolute because the service does not run from the checkout — config.rs's defaults (scripts/rqtl_wrapper.R) resolve against the working directory, and run.sh cd's to the checkout only as a belt-and-braces measure.
  • R_LIBS_SITE must match the R that reads it: the profile below is built for R 4.6.1, so R_BIN has to be an r-minimal-4.6.1, not whichever R the store happens to offer first.

BulkLMM needs a populated Julia depot, once. Its project (gn4-julia-env) is registry-based and Julia had none of its packages, so the endpoint failed with Package CSV ... is required but does not seem to be installed: Run Pkg.instantiate(). The fix is one command, run as wrk:

JULIA_DEPOT_PATH= HOME=/home/wrk \
  /gnu/store/…-julia-1.8.5/bin/julia --project=/home/wrk/tmp/ai-root/factory/gn4-julia-env \
  -e 'using Pkg; Pkg.instantiate(); using BulkLMM, CSV, DataFrames'

It downloads from the Julia registry (142 MB here) and takes a few minutes; afterwards /compute/bulklmm answers 21 056 rows in ~19 s.

BULKLMM_DEPOT cannot choose where those packages live: Guix's julia is a wrapper that prepends a colon to JULIA_DEPOT_PATH, so the default depot ($HOME/.julia, the service's HOME=/home/wrk) is always first and the variable only appends. Point it at /home/wrk/.julia and leave it alone.