Operator checklist: deploying genenetwork4 for public, read-only use
Companion to ai-instruction.md, which is what the
end user (typically an AI assistant) is given. This file is for whoever
runs the deployment: what has to stay off the public surface, how tokens
are issued without gn-auth, and what is knowingly exposed.
The reference deployment is https://vesuvio.pu.ac.ke/gn4/ proxied by
nginx to 127.0.0.1:3000 — see ../nginx.md and
../../etc/nginx-vesuvio.pu.ac.ke.conf.
nginx 443 / 80 public; TLS; /gn4/ -> 127.0.0.1:3000
genenetwork4 127.0.0.1:3000 loopback only, SEARCH_ENGINE=native
gn-auth stub 127.0.0.1:8081 loopback only, serves the JWKS
MariaDB 127.0.0.1:3306
xapian index XAPIAN_DB_PATH
Must be off / not routed
POST /get-test-token— irrelevant while there is no gn-auth (tokens come fromscripts/gn-auth-stub.py --mint-only, see below), but the moment gn-auth sits behind it this is a minting route, and the only thing standing between it and the internet is a loopback peer check plus a refusal of requests carrying proxy headers (src/routes/dev.rs). Behind nginx the peer is loopback, so a proxy that sets none of those headers makes it open to the world. Belt and braces, all three:GENENETWORK4_TEST_TOKEN=false(route not registered), bind to127.0.0.1, and thelocation = /gn4/get-test-token { return 404; }refusal in the nginx config.POST /compute/*(/compute/gemma,/compute/gemma/ravanan,/compute/rqtl,/compute/correlation) is now proxied — R/qtl and GEMMA are part of the service, gated on a valid token (see "Compute" below). Understand what that grants: those handlers take server-side file paths (genotype_file,snps_file) and run R/gemma on them, so every token holder can ask the box to read its own filesystem. It istoken_only, not resource-checked: the public-data-only restriction that applies to traits and datasets does not apply here. The nginx config comments show how to re-block it (location /gn4/compute/ { return 403; }) if that is no longer wanted./compute/*/status/{id}tails run logs by name and shares the same gate.- The auth stub on
:8081is not a dev leftover — without gn-auth it is the auth server — but it must stay reachable from genenetwork4 only: never route it publicly. Its signing key (test/data/test_rsa_private.pem) and JWKS are in this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. - Dev leftovers:
target/debugbinaries,GN4_DB_MODE=dev("start without a database").
Token issuance (no gn-auth: the stub, one token per user)
The request path is: the AI assistant has no token, so it asks the user
(see §1 of ai-instruction.md); the user asks the
GeneNetwork team; someone here mints a token for that user and hands it
back out of band; the user passes it to the assistant. Expect the
request to say who it is for, what data, and for how long — that is what
--subject and --expires-in need.
Run the stub once and leave it running, supervised like genenetwork4 — see "Tokens are stateless" in the README for why an hour of downtime is survivable and longer is not:
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
scripts/gn-auth-stub.py --token-file /run/gn4/stub.token
AUTH_SERVER_URL is where genenetwork4 fetches the key set
(/auth/public-jwks); it stays a loopback URL, and the public /gn4
prefix has nothing to do with it.
Then one token per user, on the box, without touching the running stub:
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
scripts/gn-auth-stub.py --mint-only --subject alice \
--expires-in 30d --token-file /run/gn4/alice.token
--mint-only mints, prints, writes the file and exits; --issuer
defaults to AUTH_SERVER_URL, which is what makes the token verify
against the running stub. Hand the file's contents over out of band.
What that token is, and is not:
- public data only. A token minted without
--grant-resource-idcarries no privileges: genenetwork4 decides what may go out from each dataset'spublic/confidentiality/create_timecolumns and asks the authorisation lookup about the rest (src/auth.rs'scheck). The lookup answers "no privileges", so public data reads and every non-public dataset is a403. Never mint with--grant-resource-idfor a shared or long-lived token: it hands the holder that non-public resource and cannot be taken back beforeexp. - expiry is the only revocation. No denylist exists, so a token is
good until its
exp;--expires-inis the withdrawal mechanism. Revoking everything means changingAUTH_SERVER_URLor the signing key —issis compared verbatim, so a new host, scheme or trailing slash invalidates every outstanding token withInvalidIssuer. subis attribution, not authentication. It names the user, is used as an authorisation-cache key, and grants nothing. Anyone holding a token is thatsubas far as the API can tell, and thescopeclaim is not consulted for data access.- per user, per expiry —
--subject alice,--subject bob: same key and issuer, independentexp, so one token can lapse without touching the others. - verify before handing over:
curl -s -H "Authorization: Bearer $TOKEN" https://vesuvio.pu.ac.ke/gn4/groups | head -c 80should return an envelope.
When gn-auth lands, only the source of tokens changes:
POST /get-test-token (gn-auth's dev password grant) replaces
--mint-only, and tokens gain gn-auth's own lifetime (7–14 days,
bounded by JWKS rotation) and its per-user grants.
Compute: what the box needs
R/qtl and GEMMA run as child processes, so the genenetwork4 process's environment has to be able to find them. From a plain shell it is not enough to have the binary in a store path — the R packages matter too. The deployment shell gives both:
P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT')
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ genenetwork4 …other env…
$P/bincarriesRscript,gemma-wrapperandgemma. If the profile'sgemma-wrapperis older than the pin inguix.scm, pointGEMMA_WRAPPER_CMDat the store path directly rather than editingPATH:-pvereaches GEMMA either way, but the per-run.log.txtfiles (wherepvelives) are only kept beside the archive from5eb53faon.R_LIBS_SITE=$P/site-library/is what letsscripts/rqtl_wrapper.Rloadoptparse,qtl,stringiandstringr. Without it the request fails withrqtl_failedandthere is no package called 'optparse'— an environment problem, not a token or proxy problem.
Real panels are served from GENOTYPE_FILES (default
/home/wrk/tmp/ai-root/factory/gn4-data/genotype-files), in
GeneNetwork2's layout, and GET /groups/{group}/genotypes exposes them
by version only — no paths, no file names — which is what
POST /compute/{rqtl,gemma} resolve group + genotype_version
against. The layout: genotype/BXD.8.geno (WGS
Mar2022, 235 individuals × 21 057 markers) with GEMMA's
genotype/bimbam/BXD.8_geno.txt + _snps.txt, and
genotype/HET3-ITP.12.geno (6 447 individuals × 894 markers) for
UM-HET3. docs/genotypes.md covers how the current version is chosen,
which engine each group offers, and the measured runs.
Both engines match values to individuals by position; the API hides
that by aligning named samples onto the panel itself, and by dropping
parents/F1s the only way that works — a panel-subset genotype file under
the engine's cache directory (/tmp/gn4-rqtl/prepared,
/tmp/gn4-gemma/prepared), reused by content and panel. If those caches
are wiped, the next mapping recreates them (a few seconds on BXD).
Parents and F1s are excluded by default, as in GN2/GN3 (GN2's
incparentsf1, GN3's get_sample_list(inc_par=False, inc_f1=False)).
BXD.8's panel is 198 RI strains + 37 F1s and no parent columns, so that
default means "the 198 strains"; HET3-ITP.12 is individual animals
only. null-filling a non-strain individual keeps its phenotype out of
the fit but not its genotype out of GEMMA's kinship matrix or R/qtl's
cross, so excluding it means subsetting the genotype file — which is what
the API should do when it takes over the alignment.
Which engines a group offers is InbredSet.MappingMethodId (1 → GEMMA,
QTLReaper, R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA, PLINK): BXD = 1,
HET3-ITP = 3. So UM-HET3 is R/qtl only, correct for a 4-way cross even
though bimbam/HET3-ITP.* exists.
Measured on this box with the newest genotypes: BXD (BXD.8, 235
individuals, trait 10001 with 34 values) 17.1 s / 21 055 marker rows on
GEMMA and 7.5 s / 21 056 markers on R/qtl; UM-HET3 lifespan over
HET3-ITP.12 (6 447 individuals, 6 438 with values) 83.1 s / 893
markers, peak LOD 7.95 at chr 1, 24.04 Mb. Details and the misgivings
about HET3-ITP's versioning are in docs/genotypes.md.
/compute/gemma/ravanan additionally needs ravanan and ccwl on
PATH, and with RAVANAN_BATCH_SYSTEM=slurm-api a reachable slurmrestd
(see docs/ravanan-slurm.md).
Repeat calls are cached by input (gemma-wrapper's content-hash cache, and src/rqtl.rs's own hash of genotype+phenotype), so the second identical request is cheap: the same GEMMA run measured ~2 s cached against ~20 s cold. Worth telling users, because the alternative they reach for — downloading the values and fitting a scan themselves — is both slower and wrong (their genotypes are not the panel's).
Verified through the proxy with a token: POST /gn4/compute/rqtl
returned 200 in 0.4 s with the fixture's marker rows, POST
/gn4/compute/gemma 200 in 2.1 s with 7 320 marker rows, and
GET /gn4/compute/gemma/status/<id> ended with GN4: computation
complete. Without a token both are 401.
Two operational consequences: the nginx proxy_read_timeout (3600 s)
must stay generous, since a real mapping takes minutes; and each request
is CPU-bound on the box, so rate-limit per token or keep the audience
small — limit_req in the config is one way, but it keys on address,
not token.
Database connections
SQL_CHECK_HEALTH (default true) makes the pool ping a connection at
checkout and replace it when the ping fails. Keep it on: without it, a
connection MariaDB closed while idle becomes a 500 database_error on
whichever request draws it, one per dead socket, until the process
restarts — that is what happened on 2026-11-07. GET /version now carries
"database": "reachable" | "unreachable" (cached ten seconds), so an
up-but-useless service is visible without calling a data endpoint.
Compute concurrency
The gates exist to stop large computations piling up, so they only
apply above a threshold that differs by endpoint:
compute_gate::GEMMA_LARGE_PANEL is 50 individuals (all GEMMA runs
share GEMMA_CACHE_DIR, where gemma-wrapper extracts per-chromosome
kinship archives by content hash -- two runs at once are two writers over
the same hundreds of megabytes), compute_gate::RQTL_LARGE_PANEL is
1 000 (a scanone over a few hundred individuals is seconds of one
core). A panel below its endpoint's threshold takes no slot. Above it, POST /compute/gemma takes one job at a time,
POST /compute/rqtl RQTL_MAX_CONCURRENT (default 8) and
/compute/gemma/ravanan one at a time, each one large job per caller. All
three are in-process gates (src/compute_gate.rs), so they are per server
process and reset on restart -- a job's child process, however, outlives
the server. Waits are bounded only by nginx's proxy_read_timeout.
The slot count is fixed when a gate is first built (its first large
request), so raising RQTL_MAX_CONCURRENT takes a restart, not a reload.
Changing a threshold is the same: both are constants, and the reasons for
the numbers are in their doc comments.
Plotting
POST /plot/rqtl and /plot/gemma take a posted mapping response --
no mapping is re-run -- and hand it to the deployment's plotter: scripts/plot-rqtl.py / scripts/plot-gemma.py,
executed by PLOT_PYTHON (default python3) -- so python3 must be on
the server process's PATH, alongside Rscript/gemma-wrapper for the
mapping itself. They return image/svg+xml rather than an envelope, and
the plotter's peak summary in x-plot-summary. Point RQTL_PLOTTER /
GEMMA_PLOTTER elsewhere if the scripts move; a missing script is a
502 plot_failed naming the interpreter and script it tried.
PheWAS
GET /groups/{group}/phewas reads PublishXRef only -- the peak marker,
LRS and the join to Phenotype/Publication -- plus the group's
genotype marker table for positions. It is token-only like the other
metadata routes, and it inherits the exposure question below:
PublishXRef covers published traits, whose datasets are the *Publish
freezes.
Data exposure
- Public rule:
public > 0 && confidentiality < 1, or older than five years (PUBLIC_AFTER_YEARS,src/auth.rs). Values endpoints enforce it per dataset;/groups/{group}/traitschecks the group's publish freeze. - Metadata endpoints are
token_onlyby design and reveal the names and descriptions of non-public datasets (e.g.GET /datasets/{name}for apublic=0ProbeSetFreeze returns the record, with itspublicandconfidentialityflags). This was a deliberate decision: a caller needs to know a dataset exists to know it is being refused. If that is not acceptable, gatedatasets.rs::dataset_by_name,groups.rs::{groups, group_by_name, group_datasets, group_strains, group_case_attributes},species.rsandtraits.rsmetadata on the same rule. - Rebuild, do not inherit, the search index: GN4's indexer applies
public > 0 AND confidentiality < 1at crawl time (src/indexer/rows.rs), whereas gn3'sindex-genenetworkhas noWHEREclause. Verified on the current full index:dataset:hcppublish(aconfidentiality=1freeze) anddataset:br_u_0303_m(apublic=0freeze) both return 0 hits. - Error bodies can carry internal detail: a
db_unavailable503includes the driver error with the database address. Strip or generalisedetailat the proxy, or accept disclosing127.0.0.1:3306. - Serve everything over TLS (port 80 redirects), keep
/openapi.json,/versionand the version envelope's git revision open — they are useful and harmless.
Verified on this deployment
GET / 302 -> /gn4/ (HTTP/2)
GET /gn4 301 -> /gn4/
GET /gn4/version 200 envelope, git revision
GET /gn4/openapi.json 200
GET /gn4/groups/BXD no token 401 / token 200
GET /gn4/groups/BXD/traits?limit=2 200
GET /gn4/search?q=alcohol&limit=3 200 state: complete
GET /gn4/datasets/{non-public}/… 403
GET /gn4/species/nope 404 links rewritten to /gn4/…
POST /gn4/get-test-token 404
POST /gn4/compute/correlation 200 with a token / 401 without
http://…/ 301 -> https://…
next links come back already prefixed (/gn4/groups/BXD/strains) and
resolving one through the proxy returns 200, so the sub_filter
rewriting in nginx is doing its job for success and error bodies alike.
Check it after any nginx change: without it every link points outside
the mount point, and the API's own tests cannot see that (they run
in-process, where there is no prefix).
When you update the running build
This deployment is updated in place, so the build revision users see at
GET /gn4/version moves under them. Two things follow:
- their assistants are told (in
ai-instruction.md) to pull the repository and re-read it whenevergitis a revision they have not seen before, because the page they cached no longer describes the API they are now calling. Nothing enforces that from this side, so if a release adds endpoints or options, say so where you hand out tokens; source_digestchanges with every build, dirty or not -- it identifies the binary, so quote it in bug reports rather than the commit alone.
Restarting it: trust /version, not the pid file
nohup env … ./target/release/genenetwork4 & does not reliably leave the
server's own pid in $! — on this box it named an intermediate process, so
kill $(cat /tmp/gn4.pid) killed that and left the server answering.
The replacement then died with AddrInUse (in its log, not on the
console) while the old build kept serving: everything looked successful and
nothing had changed.
Two checks make that impossible to miss:
# who is actually serving (no ps here; read /proc, and match argv[0])
for d in /proc/[0-9]*; do a0=$(tr '\0' '\n' < $d/cmdline 2>/dev/null | head -1); \
case "$a0" in ./target/release/genenetwork4) echo "${d#/proc/}";; esac; done
# and what is actually answering: git must be the commit you just built
curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r '.data | "\(.git) \(.dirty)"'
Match argv[0] against the binary path, not the whole command line:
nohup env … leaves a wrapper process whose command line also contains
that path, so a substring match finds the wrapper -- which is how the pid
file ended up naming a process that exits while the server keeps running.
A git value older than the build you just made is this mistake, not a
build problem: write the resolved pid into the pid file (and start the
replacement only after :3000 has stopped answering). source_digest
changes on every build, so it distinguishes two builds of the same commit.
Optional but worth it
- Rate limit per address at the proxy:
limit_req_zone+limit_req/limit_req_status 429are commented in the config. 429 rather than 503, becauseRetry-Afteron 429 is what the API's clients are told to expect. - Put genenetwork4 and the stub under the same supervision (systemd
units, not a shell): the stub's availability bounds every
authenticated request, and
JWKS_TTL(an hour) is the grace period. - Logs: the nginx
gn4log format records status,rt, thex-request-id,user=(the API'sx-gn4-userresponse header, i.e. the token's subject, so per-user success isawk '{print $9, $10}'away instead of inferred from addresses) and the user agent.user=-means the request carried no valid token: scanners and anonymous reads, never a caller whose token failed after authenticating. - genenetwork4 writes its own access line too, one JSON object per request on stderr, naming the caller:
json
{"event":"request","method":"GET","path":"/groups/BXD","status":200,"took_ms":10,"request_id":"…","user":"zachs"}
So tail -f on the process log answers "who ran this job" without going
through the proxy's log — which matters while watching a run, and it is the
same log the event: error lines go to, correlated by request_id.
user is absent (not -) when no token was verified. The path is
logged without its query string, and the token itself is never logged —
only its subject. Turn it off with GN4_ACCESS_LOG=false where the proxy's
access log is the record of truth.
* A /search total/facet would end the paging-to-count pattern that
costs ~23 s per 15 000 hits; see
../search-engines.md.
Production layout on this box (2026-11-15, no container)
The API runs the store binary directly, as wrk, with no guix shell
container around it. Two instances (3000, 3001) are started by
factory/gn4-prod/run.sh, which sources factory/gn4-prod/env.sh:
runuser -u wrk -- /home/wrk/tmp/ai-root/factory/gn4-prod/run.sh restart
run.sh also takes start, stop and status; the instances log to
factory/gn4-prod/logs/<port>.log (appending across restarts, so the access
lines survive a deploy). Deploying is therefore: guix build -f guix.scm,
edit env.sh's GEN4=, run.sh restart.
Then run the monitor. Every time, without being asked:
bin/monitor-gn4 --token-file /tmp/prodtok.file --tier all
And run it cold as well as warm after anything that touches the environment, and weekly:
bin/monitor-gn4 --token-file /tmp/prodtok.file --tier compute --cold
A warm run answers from caches that were filled by the previous binary, so
it can pass while the cold path -- the one a new user's first request takes --
is broken. --cold sends each cached engine a payload with one value nudged,
which every engine here keys on (gemma-wrapper's content hash, ravanan's
interned inputs, rqtl/qtl2/bulklmm's own hashes), then repeats the original
payload for the warm hit. Both runs must pass, and the line reports
cold X / warm Y. Measured on this box with a 30-sample fixture: ravanan
102.0 s cold / 3.7 s warm; gemma 3.1 / 2.0; rqtl, qtl2, fitqtl and BulkLMM are
compute-bound at this size and show cold ~= warm, which is a fact about the
fixture, not a fault. Two costs to know: ravanan's cold run takes minutes, and
every cold run leaves another entry in the caches (GEMMA_CACHE_DIR,
RAVANAN_CACHE_DIR, …), so run it deliberately rather than in a loop.
/version only says which commit is running; it cannot see that an engine
lost its tool path, that a config key moved, or that an endpoint now answers
a shape nobody expected. On 2026-11-15 the service was "deployed and up" with
database: reachable while every shell-out engine answered 502 and
/search answered 502 in 1.4 ms, and the only thing that caught it was this
monitor. A deploy that has not been through it is not finished. Expect
39+ ok, 0 failed and read any skip: the listener check skips whenever the
service runs outside the monitor's PID namespace, which is normal on this box
and is the only skip that should ever appear.
Everything mutable is under factory/gn4-prod/ — tokens.sqlite (the only
copy: losing it loses every token), gemma-cache, rqtl-cache,
qtl2-cache, bulklmm-cache, bnw-cache. The ravanan cache is the
exception: it must be under /home/wrk/tmp/ai-root/slurm-share/ because the
Slurm container bind-mounts nothing else (see
../ravanan-slurm.md).
Two things that are easy to get wrong, both measured on this box:
- The workflow and manifest come from the checkout, not from the store
build:
GEMMA_RAVANAN_WORKFLOWandGEMMA_RAVANAN_MANIFESTpoint intofactory/code/genenetwork/genenetwork4/cwl/gemma/, so the checkout has to be at the commit the binary was built from — otherwise the CWL that runs is not the one that was tested. - The ravanan in
env.shis a locally patched build./gnu/store/amn5dfh4…-ravanan-0.2.0comes fromfactory/code/ravanan(commit6f9031b) and carries three patches the upstream 0.2.0 lacks:--slurm-api-endpointparsed into a URI,user-errorprinted unconditionally (job scripts have no log level, so their failures were silent), and{"job": …}instead of{"jobs": […]}on submit. That last one is worth remembering: the plural form is slurmrestd's heterogeneous job submission, so every step arrived as a one-component het job holding a whole node, and a 20-chromosome scatter ran strictly 1-wide (cold 24.7 s). With it fixed: 20-wide, cold 9.4 s. Rebuild withguix build --with-source=ravanan=<checkout> --without-tests=ravanan ravanan(its own test suite wants node and fails in the sandbox). - ravanan 0.2.0 exists on this box in two variants, and only one works
with
--slurm-api-endpoint. In/gnu/store/1ydn2bdw…-ravanan-0.2.0(and20aqba00…) the option handler converts its argument,(acons 'slurm-api-endpoint (string->uri arg) result); inrim2v0pi…andvyqs04g9…it does not, soslurm-api.scmcallsuri-schemeon a string and the run dies withIn procedure struct-vtable: Wrong type argument …: "http://127.0.0.1:6820"after staging every input, i.e. looking like a Slurm problem.env.shpins the working one.
GENENETWORK4_TEST_TOKEN=false in production: /get-test-token mints a
token for any local caller, which is a development convenience only. Mint
with POST /tokens (localhost-only) instead.
The tool block env.sh must keep
Every engine that shells out needs absolute paths in env.sh; without them
the read surface and search stay healthy while all of compute answers 502
(measured 2026-11-15: 1 ok / 7 failed / 1 busy before, 8 ok / 1 failed after).
gn4-container/gn4.env is the reference for the keys:
PATH=$R_BIN:$WRAPPER:/usr/bin:/bin # Rscript first: no R in any system profile
R_LIBS_SITE=<profile>/site-library # r-qtl and r-qtl2 live here
GEMMA_WRAPPER_CMD, GEMMA_DIR, GEMMA_WRAPPER_DIR # + gemma/time/tar/xz/parallel/pfff
RQTL_WRAPPER_PATH, FITQTL_SCRIPT_PATH, QTL2_SCRIPT_PATH,
GEMMA_PLOTTER, BULKLMM_PLOTTER, BULKLMM_SCRIPT, GENOTYPE_PLOTTER, BNW_PLOTTER
BNW_CMD, BNW_BIN_DIR, BNW_EXAMPLES_DIR
BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT
BASKET_DIR # per-user SQLite, not /tmp
Two of these are easy to get wrong:
- The script paths are absolute because the service does not run from the
checkout —
config.rs's defaults (scripts/rqtl_wrapper.R) resolve against the working directory, andrun.shcd's to the checkout only as a belt-and-braces measure. R_LIBS_SITEmust match the R that reads it: the profile below is built for R 4.6.1, soR_BINhas to be anr-minimal-4.6.1, not whichever R the store happens to offer first.
BulkLMM needs a populated Julia depot, once. Its project
(gn4-julia-env) is registry-based and Julia had none of its packages, so the
endpoint failed with Package CSV ... is required but does not seem to be
installed: Run Pkg.instantiate(). The fix is one command, run as wrk:
JULIA_DEPOT_PATH= HOME=/home/wrk \
/gnu/store/…-julia-1.8.5/bin/julia --project=/home/wrk/tmp/ai-root/factory/gn4-julia-env \
-e 'using Pkg; Pkg.instantiate(); using BulkLMM, CSV, DataFrames'
It downloads from the Julia registry (142 MB here) and takes a few minutes;
afterwards /compute/bulklmm answers 21 056 rows in ~19 s.
BULKLMM_DEPOT cannot choose where those packages live: Guix's julia
is a wrapper that prepends a colon to JULIA_DEPOT_PATH, so the default
depot ($HOME/.julia, the service's HOME=/home/wrk) is always first and the
variable only appends. Point it at /home/wrk/.julia and leave it alone.
