# Operator checklist: deploying genenetwork4 for public, read-only use Companion to [`ai-instruction.md`](ai-instruction.md), which is what the end user (typically an AI assistant) is given. This file is for whoever runs the deployment: what has to stay off the public surface, how tokens are issued without gn-auth, and what is knowingly exposed. The reference deployment is `https://vesuvio.pu.ac.ke/gn4/` proxied by nginx to `127.0.0.1:3000` — see [`../nginx.md`](../nginx.md) and [`../../etc/nginx-vesuvio.pu.ac.ke.conf`](../../etc/nginx-vesuvio.pu.ac.ke.conf). ``` nginx 443 / 80 public; TLS; /gn4/ -> 127.0.0.1:3000 genenetwork4 127.0.0.1:3000 loopback only, SEARCH_ENGINE=native gn-auth stub 127.0.0.1:8081 loopback only, serves the JWKS MariaDB 127.0.0.1:3306 xapian index XAPIAN_DB_PATH ``` ## Must be off / not routed * `POST /get-test-token` — irrelevant while there is no gn-auth (tokens come from `scripts/gn-auth-stub.py --mint-only`, see below), but the moment gn-auth sits behind it this is a minting route, and the only thing standing between it and the internet is a loopback peer check plus a refusal of requests carrying proxy headers (`src/routes/dev.rs`). Behind nginx the peer *is* loopback, so a proxy that sets none of those headers makes it open to the world. Belt and braces, all three: `GENENETWORK4_TEST_TOKEN=false` (route not registered), bind to `127.0.0.1`, and the `location = /gn4/get-test-token { return 404; }` refusal in the nginx config. * `POST /compute/*` (`/compute/gemma`, `/compute/gemma/ravanan`, `/compute/rqtl`, `/compute/correlation`) is **now proxied** — R/qtl and GEMMA are part of the service, gated on a valid token (see "Compute" below). Understand what that grants: those handlers take **server-side file paths** (`genotype_file`, `snps_file`) and run R/gemma on them, so every token holder can ask the box to read its own filesystem. It is `token_only`, not resource-checked: the public-data-only restriction that applies to traits and datasets does **not** apply here. The nginx config comments show how to re-block it (`location /gn4/compute/ { return 403; }`) if that is no longer wanted. `/compute/*/status/{id}` tails run logs by name and shares the same gate. * The auth stub on `:8081` is not a dev leftover — without gn-auth it *is* the auth server — but it must stay reachable from genenetwork4 only: **never** route it publicly. Its signing key (`test/data/test_rsa_private.pem`) and JWKS are in this repository, so anyone who can fetch its JWKS can mint tokens genenetwork4 accepts. * Dev leftovers: `target/debug` binaries, `GN4_DB_MODE=dev` ("start without a database"). ## Token issuance (no gn-auth: the stub, one token per user) The request path is: the AI assistant has no token, so it asks the user (see §1 of [`ai-instruction.md`](ai-instruction.md)); the user asks the GeneNetwork team; someone here mints a token for that user and hands it back out of band; the user passes it to the assistant. Expect the request to say who it is for, what data, and for how long — that is what `--subject` and `--expires-in` need. Run the stub once and leave it running, supervised like genenetwork4 — see "Tokens are stateless" in the README for why an hour of downtime is survivable and longer is not: ```sh AUTH_SERVER_URL=http://127.0.0.1:8081/ \ scripts/gn-auth-stub.py --token-file /run/gn4/stub.token ``` `AUTH_SERVER_URL` is where genenetwork4 fetches the key set (`/auth/public-jwks`); it stays a loopback URL, and the public `/gn4` prefix has nothing to do with it. Then one token per user, on the box, without touching the running stub: ```sh AUTH_SERVER_URL=http://127.0.0.1:8081/ \ scripts/gn-auth-stub.py --mint-only --subject alice \ --expires-in 30d --token-file /run/gn4/alice.token ``` `--mint-only` mints, prints, writes the file and exits; `--issuer` defaults to `AUTH_SERVER_URL`, which is what makes the token verify against the running stub. Hand the file's contents over out of band. What that token is, and is not: * **public data only.** A token minted without `--grant-resource-id` carries no privileges: genenetwork4 decides what may go out from each dataset's `public`/`confidentiality`/`create_time` columns and asks the authorisation lookup about the rest (`src/auth.rs`'s `check`). The lookup answers "no privileges", so public data reads and every non-public dataset is a `403`. **Never** mint with `--grant-resource-id` for a shared or long-lived token: it hands the holder that non-public resource and cannot be taken back before `exp`. * **expiry is the only revocation.** No denylist exists, so a token is good until its `exp`; `--expires-in` is the withdrawal mechanism. Revoking *everything* means changing `AUTH_SERVER_URL` or the signing key — `iss` is compared verbatim, so a new host, scheme or trailing slash invalidates every outstanding token with `InvalidIssuer`. * **`sub` is attribution, not authentication.** It names the user, is used as an authorisation-cache key, and grants nothing. Anyone holding a token is that `sub` as far as the API can tell, and the `scope` claim is not consulted for data access. * **per user, per expiry** — `--subject alice`, `--subject bob`: same key and issuer, independent `exp`, so one token can lapse without touching the others. * verify before handing over: `curl -s -H "Authorization: Bearer $TOKEN" https://vesuvio.pu.ac.ke/gn4/groups | head -c 80` should return an envelope. When gn-auth lands, only the source of tokens changes: `POST /get-test-token` (gn-auth's dev password grant) replaces `--mint-only`, and tokens gain gn-auth's own lifetime (7–14 days, bounded by JWKS rotation) and its per-user grants. ## Compute: what the box needs R/qtl and GEMMA run as child processes, so the genenetwork4 process's environment has to be able to find them. From a plain shell it is not enough to have the binary in a store path — the R packages matter too. The deployment shell gives both: ```sh P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT') PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ genenetwork4 …other env… ``` - `$P/bin` carries `Rscript`, `gemma-wrapper` and `gemma`. If the profile's `gemma-wrapper` is older than the pin in `guix.scm`, point `GEMMA_WRAPPER_CMD` at the store path directly rather than editing `PATH`: `-pve` reaches GEMMA either way, but the per-run `.log.txt` files (where `pve` lives) are only kept beside the archive from `5eb53fa` on. - `R_LIBS_SITE=$P/site-library/` is what lets `scripts/rqtl_wrapper.R` load `optparse`, `qtl`, `stringi` and `stringr`. Without it the request fails with `rqtl_failed` and `there is no package called 'optparse'` — an environment problem, not a token or proxy problem. Real panels are served from `GENOTYPE_FILES` (default `/home/wrk/tmp/ai-root/factory/gn4-data/genotype-files`), in GeneNetwork2's layout, and `GET /groups/{group}/genotypes` exposes them by **version only** — no paths, no file names — which is what `POST /compute/{rqtl,gemma}` resolve `group` + `genotype_version` against. The layout: `genotype/BXD.8.geno` (WGS Mar2022, 235 individuals × 21 057 markers) with GEMMA's `genotype/bimbam/BXD.8_geno.txt` + `_snps.txt`, and `genotype/HET3-ITP.12.geno` (6 447 individuals × 894 markers) for UM-HET3. `docs/genotypes.md` covers how the current version is chosen, which engine each group offers, and the measured runs. Both engines match values to individuals **by position**; the API hides that by aligning named `samples` onto the panel itself, and by dropping parents/F1s the only way that works — a panel-subset genotype file under the engine's cache directory (`/tmp/gn4-rqtl/prepared`, `/tmp/gn4-gemma/prepared`), reused by content and panel. If those caches are wiped, the next mapping recreates them (a few seconds on BXD). Parents and F1s are excluded by default, as in GN2/GN3 (GN2's `incparentsf1`, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). BXD.8's panel is 198 RI strains + 37 F1s and no parent columns, so that default means "the 198 strains"; `HET3-ITP.12` is individual animals only. `null`-filling a non-strain individual keeps its phenotype out of the fit but *not* its genotype out of GEMMA's kinship matrix or R/qtl's cross, so excluding it means subsetting the genotype file — which is what the API should do when it takes over the alignment. Which engines a group offers is `InbredSet.MappingMethodId` (1 → GEMMA, QTLReaper, R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA, PLINK): BXD = 1, HET3-ITP = 3. So UM-HET3 is R/qtl only, correct for a 4-way cross even though `bimbam/HET3-ITP.*` exists. Measured on this box with the newest genotypes: BXD (`BXD.8`, 235 individuals, trait 10001 with 34 values) 17.1 s / 21 055 marker rows on GEMMA and 7.5 s / 21 056 markers on R/qtl; UM-HET3 lifespan over `HET3-ITP.12` (6 447 individuals, 6 438 with values) 83.1 s / 893 markers, peak LOD 7.95 at chr 1, 24.04 Mb. Details and the misgivings about HET3-ITP's versioning are in `docs/genotypes.md`. `/compute/gemma/ravanan` additionally needs `ravanan` and `ccwl` on `PATH`, and with `RAVANAN_BATCH_SYSTEM=slurm-api` a reachable slurmrestd (see `docs/ravanan-slurm.md`). Repeat calls are cached by input (gemma-wrapper's content-hash cache, and src/rqtl.rs's own hash of genotype+phenotype), so the second identical request is cheap: the same GEMMA run measured ~2 s cached against ~20 s cold. Worth telling users, because the alternative they reach for — downloading the values and fitting a scan themselves — is both slower and wrong (their genotypes are not the panel's). Verified through the proxy with a token: `POST /gn4/compute/rqtl` returned 200 in 0.4 s with the fixture's marker rows, `POST /gn4/compute/gemma` 200 in 2.1 s with 7 320 marker rows, and `GET /gn4/compute/gemma/status/` ended with `GN4: computation complete`. Without a token both are `401`. Two operational consequences: the nginx `proxy_read_timeout` (3600 s) must stay generous, since a real mapping takes minutes; and each request is CPU-bound on the box, so rate-limit per token or keep the audience small — `limit_req` in the config is one way, but it keys on address, not token. ## Database connections `SQL_CHECK_HEALTH` (default `true`) makes the pool ping a connection at checkout and replace it when the ping fails. Keep it on: without it, a connection MariaDB closed while idle becomes a `500 database_error` on whichever request draws it, one per dead socket, until the process restarts — that is what happened on 2026-11-07. `GET /version` now carries `"database": "reachable" | "unreachable"` (cached ten seconds), so an up-but-useless service is visible without calling a data endpoint. ## Compute concurrency The gates exist to stop **large** computations piling up, so they only apply above a threshold that differs by endpoint: `compute_gate::GEMMA_LARGE_PANEL` is **50** individuals (all GEMMA runs share `GEMMA_CACHE_DIR`, where `gemma-wrapper` extracts per-chromosome kinship archives by content hash -- two runs at once are two writers over the same hundreds of megabytes), `compute_gate::RQTL_LARGE_PANEL` is **1 000** (a `scanone` over a few hundred individuals is seconds of one core). A panel below its endpoint's threshold takes no slot. Above it, `POST /compute/gemma` takes one job at a time, `POST /compute/rqtl` `RQTL_MAX_CONCURRENT` (default 8) and `/compute/gemma/ravanan` one at a time, each one large job per caller. All three are in-process gates (`src/compute_gate.rs`), so they are per server process and reset on restart -- a job's child process, however, outlives the server. Waits are bounded only by nginx's `proxy_read_timeout`. The slot count is fixed when a gate is first built (its first large request), so raising `RQTL_MAX_CONCURRENT` takes a restart, not a reload. Changing a threshold is the same: both are constants, and the reasons for the numbers are in their doc comments. ## Plotting `POST /plot/rqtl` and `/plot/gemma` take a **posted mapping response** -- no mapping is re-run -- and hand it to the deployment's plotter: `scripts/plot-rqtl.py` / `scripts/plot-gemma.py`, executed by `PLOT_PYTHON` (default `python3`) -- so **python3 must be on the server process's `PATH`**, alongside `Rscript`/`gemma-wrapper` for the mapping itself. They return `image/svg+xml` rather than an envelope, and the plotter's peak summary in `x-plot-summary`. Point `RQTL_PLOTTER` / `GEMMA_PLOTTER` elsewhere if the scripts move; a missing script is a `502 plot_failed` naming the interpreter and script it tried. ## PheWAS `GET /groups/{group}/phewas` reads `PublishXRef` only -- the peak marker, LRS and the join to `Phenotype`/`Publication` -- plus the group's genotype marker table for positions. It is token-only like the other metadata routes, and it inherits the exposure question below: `PublishXRef` covers published traits, whose datasets are the `*Publish` freezes. ## Data exposure * Public rule: `public > 0 && confidentiality < 1`, or older than five years (`PUBLIC_AFTER_YEARS`, `src/auth.rs`). Values endpoints enforce it per dataset; `/groups/{group}/traits` checks the group's publish freeze. * Metadata endpoints are `token_only` by design and reveal the names and descriptions of *non-public* datasets (e.g. `GET /datasets/{name}` for a `public=0` ProbeSetFreeze returns the record, with its `public` and `confidentiality` flags). This was a deliberate decision: a caller needs to know a dataset exists to know it is being refused. If that is not acceptable, gate `datasets.rs::dataset_by_name`, `groups.rs::{groups, group_by_name, group_datasets, group_strains, group_case_attributes}`, `species.rs` and `traits.rs` metadata on the same rule. * Rebuild, do not inherit, the search index: GN4's indexer applies `public > 0 AND confidentiality < 1` at crawl time (`src/indexer/rows.rs`), whereas gn3's `index-genenetwork` has no `WHERE` clause. Verified on the current full index: `dataset:hcppublish` (a `confidentiality=1` freeze) and `dataset:br_u_0303_m` (a `public=0` freeze) both return 0 hits. * Error bodies can carry internal detail: a `db_unavailable` `503` includes the driver error with the database address. Strip or generalise `detail` at the proxy, or accept disclosing `127.0.0.1:3306`. * Serve everything over TLS (port 80 redirects), keep `/openapi.json`, `/version` and the version envelope's git revision open — they are useful and harmless. ## Verified on this deployment ``` GET / 302 -> /gn4/ (HTTP/2) GET /gn4 301 -> /gn4/ GET /gn4/version 200 envelope, git revision GET /gn4/openapi.json 200 GET /gn4/groups/BXD no token 401 / token 200 GET /gn4/groups/BXD/traits?limit=2 200 GET /gn4/search?q=alcohol&limit=3 200 state: complete GET /gn4/datasets/{non-public}/… 403 GET /gn4/species/nope 404 links rewritten to /gn4/… POST /gn4/get-test-token 404 POST /gn4/compute/correlation 200 with a token / 401 without http://…/ 301 -> https://… ``` `next` links come back already prefixed (`/gn4/groups/BXD/strains`) and resolving one through the proxy returns 200, so the `sub_filter` rewriting in nginx is doing its job for success and error bodies alike. Check it after any nginx change: without it every link points outside the mount point, and the API's own tests cannot see that (they run in-process, where there is no prefix). ## When you update the running build This deployment is updated in place, so the build revision users see at `GET /gn4/version` moves under them. Two things follow: * their assistants are told (in `ai-instruction.md`) to pull the repository and re-read it whenever `git` is a revision they have not seen before, because the page they cached no longer describes the API they are now calling. Nothing enforces that from this side, so if a release adds endpoints or options, say so where you hand out tokens; * `source_digest` changes with every build, dirty or not -- it identifies the binary, so quote it in bug reports rather than the commit alone. ## Restarting it: trust `/version`, not the pid file `nohup env … ./target/release/genenetwork4 &` does not reliably leave the server's own pid in `$!` — on this box it named an intermediate process, so `kill $(cat /tmp/gn4.pid)` killed that and left the *server* answering. The replacement then died with `AddrInUse` (in its log, not on the console) while the old build kept serving: everything looked successful and nothing had changed. Two checks make that impossible to miss: ```sh # who is actually serving (no ps here; read /proc, and match argv[0]) for d in /proc/[0-9]*; do a0=$(tr '\0' '\n' < $d/cmdline 2>/dev/null | head -1); \ case "$a0" in ./target/release/genenetwork4) echo "${d#/proc/}";; esac; done # and what is actually answering: git must be the commit you just built curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r '.data | "\(.git) \(.dirty)"' ``` Match **argv[0] against the binary path**, not the whole command line: `nohup env …` leaves a wrapper process whose command line also contains that path, so a substring match finds the wrapper -- which is how the pid file ended up naming a process that exits while the server keeps running. A `git` value older than the build you just made is this mistake, not a build problem: write the resolved pid into the pid file (and start the replacement only after `:3000` has stopped answering). `source_digest` changes on every build, so it distinguishes two builds of the same commit. ## Optional but worth it * Rate limit per address at the proxy: `limit_req_zone` + `limit_req`/`limit_req_status 429` are commented in the config. 429 rather than 503, because `Retry-After` on 429 is what the API's clients are told to expect. * Put genenetwork4 and the stub under the same supervision (systemd units, not a shell): the stub's availability bounds every authenticated request, and `JWKS_TTL` (an hour) is the grace period. * Logs: the nginx `gn4` log format records status, `rt`, the `x-request-id`, **`user=`** (the API's `x-gn4-user` response header, i.e. the token's subject, so per-user success is `awk '{print $9, $10}'` away instead of inferred from addresses) and the user agent. `user=-` means the request carried no valid token: scanners and anonymous reads, never a caller whose token failed after authenticating. * **genenetwork4 writes its own access line too**, one JSON object per request on stderr, naming the caller: ```json {"event":"request","method":"GET","path":"/groups/BXD","status":200,"took_ms":10,"request_id":"…","user":"zachs"} ``` So `tail -f` on the process log answers "who ran this job" without going through the proxy's log — which matters while watching a run, and it is the same log the `event: error` lines go to, correlated by `request_id`. `user` is **absent** (not `-`) when no token was verified. The path is logged without its query string, and the token itself is never logged — only its subject. Turn it off with `GN4_ACCESS_LOG=false` where the proxy's access log is the record of truth. * A `/search` `total`/facet would end the paging-to-count pattern that costs ~23 s per 15 000 hits; see [`../search-engines.md`](../search-engines.md). ## Production layout on this box (2026-11-15, no container) The API runs the **store binary directly**, as `wrk`, with no `guix shell` container around it. Two instances (3000, 3001) are started by `factory/gn4-prod/run.sh`, which sources `factory/gn4-prod/env.sh`: ```sh runuser -u wrk -- /home/wrk/tmp/ai-root/factory/gn4-prod/run.sh restart ``` `run.sh` also takes `start`, `stop` and `status`; the instances log to `factory/gn4-prod/logs/.log` (appending across restarts, so the access lines survive a deploy). Deploying is therefore: `guix build -f guix.scm`, edit `env.sh`'s `GEN4=`, `run.sh restart`. **Then run the monitor. Every time, without being asked:** ```sh bin/monitor-gn4 --token-file /tmp/prodtok.file --tier all ``` And **run it cold as well as warm** after anything that touches the environment, and weekly: ```sh bin/monitor-gn4 --token-file /tmp/prodtok.file --tier compute --cold ``` A warm run answers from caches that were filled by the *previous* binary, so it can pass while the cold path -- the one a new user's first request takes -- is broken. `--cold` sends each cached engine a payload with one value nudged, which every engine here keys on (gemma-wrapper's content hash, ravanan's interned inputs, rqtl/qtl2/bulklmm's own hashes), then repeats the original payload for the warm hit. Both runs must pass, and the line reports `cold X / warm Y`. Measured on this box with a 30-sample fixture: ravanan 102.0 s cold / 3.7 s warm; gemma 3.1 / 2.0; rqtl, qtl2, fitqtl and BulkLMM are compute-bound at this size and show cold ~= warm, which is a fact about the fixture, not a fault. Two costs to know: ravanan's cold run takes minutes, and every cold run leaves another entry in the caches (`GEMMA_CACHE_DIR`, `RAVANAN_CACHE_DIR`, …), so run it deliberately rather than in a loop. `/version` only says which commit is running; it cannot see that an engine lost its tool path, that a config key moved, or that an endpoint now answers a shape nobody expected. On 2026-11-15 the service was "deployed and up" with `database: reachable` while **every** shell-out engine answered 502 and `/search` answered 502 in 1.4 ms, and the only thing that caught it was this monitor. A deploy that has not been through it is not finished. Expect `39+ ok, 0 failed` and read any skip: the `listener` check skips whenever the service runs outside the monitor's PID namespace, which is normal on this box and is the only skip that should ever appear. Everything mutable is under `factory/gn4-prod/` — `tokens.sqlite` (the only copy: losing it loses every token), `gemma-cache`, `rqtl-cache`, `qtl2-cache`, `bulklmm-cache`, `bnw-cache`. The ravanan cache is the exception: it must be under `/home/wrk/tmp/ai-root/slurm-share/` because the Slurm container bind-mounts nothing else (see [`../ravanan-slurm.md`](../ravanan-slurm.md)). Two things that are easy to get wrong, both measured on this box: * **The workflow and manifest come from the checkout**, not from the store build: `GEMMA_RAVANAN_WORKFLOW` and `GEMMA_RAVANAN_MANIFEST` point into `factory/code/genenetwork/genenetwork4/cwl/gemma/`, so the checkout has to be at the commit the binary was built from — otherwise the CWL that runs is not the one that was tested. * **The ravanan in `env.sh` is a locally patched build.** `/gnu/store/amn5dfh4…-ravanan-0.2.0` comes from `factory/code/ravanan` (commit `6f9031b`) and carries three patches the upstream 0.2.0 lacks: `--slurm-api-endpoint` parsed into a URI, `user-error` printed unconditionally (job scripts have no log level, so their failures were silent), and `{"job": …}` instead of `{"jobs": […]}` on submit. That last one is worth remembering: the plural form is slurmrestd's *heterogeneous* job submission, so every step arrived as a one-component het job holding a whole node, and a 20-chromosome scatter ran strictly 1-wide (cold 24.7 s). With it fixed: 20-wide, cold 9.4 s. Rebuild with `guix build --with-source=ravanan= --without-tests=ravanan ravanan` (its own test suite wants node and fails in the sandbox). * **ravanan 0.2.0 exists on this box in two variants**, and only one works with `--slurm-api-endpoint`. In `/gnu/store/1ydn2bdw…-ravanan-0.2.0` (and `20aqba00…`) the option handler converts its argument, `(acons 'slurm-api-endpoint (string->uri arg) result)`; in `rim2v0pi…` and `vyqs04g9…` it does not, so `slurm-api.scm` calls `uri-scheme` on a string and the run dies with `In procedure struct-vtable: Wrong type argument …: "http://127.0.0.1:6820"` *after* staging every input, i.e. looking like a Slurm problem. `env.sh` pins the working one. `GENENETWORK4_TEST_TOKEN=false` in production: `/get-test-token` mints a token for any local caller, which is a development convenience only. Mint with `POST /tokens` (localhost-only) instead. ### The tool block `env.sh` must keep Every engine that shells out needs absolute paths in `env.sh`; without them the read surface and search stay healthy while all of compute answers 502 (measured 2026-11-15: 1 ok / 7 failed / 1 busy before, 8 ok / 1 failed after). `gn4-container/gn4.env` is the reference for the keys: ```sh PATH=$R_BIN:$WRAPPER:/usr/bin:/bin # Rscript first: no R in any system profile R_LIBS_SITE=/site-library # r-qtl and r-qtl2 live here GEMMA_WRAPPER_CMD, GEMMA_DIR, GEMMA_WRAPPER_DIR # + gemma/time/tar/xz/parallel/pfff RQTL_WRAPPER_PATH, FITQTL_SCRIPT_PATH, QTL2_SCRIPT_PATH, GEMMA_PLOTTER, BULKLMM_PLOTTER, BULKLMM_SCRIPT, GENOTYPE_PLOTTER, BNW_PLOTTER BNW_CMD, BNW_BIN_DIR, BNW_EXAMPLES_DIR BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT BASKET_DIR # per-user SQLite, not /tmp ``` Two of these are easy to get wrong: * The script paths are absolute because the service does not run from the checkout — `config.rs`'s defaults (`scripts/rqtl_wrapper.R`) resolve against the working directory, and `run.sh` cd's to the checkout only as a belt-and-braces measure. * `R_LIBS_SITE` must match the R that reads it: the profile below is built for **R 4.6.1**, so `R_BIN` has to be an `r-minimal-4.6.1`, not whichever R the store happens to offer first. **BulkLMM needs a populated Julia depot, once.** Its project (`gn4-julia-env`) is registry-based and Julia had none of its packages, so the endpoint failed with `Package CSV ... is required but does not seem to be installed: Run Pkg.instantiate()`. The fix is one command, run as `wrk`: ```sh JULIA_DEPOT_PATH= HOME=/home/wrk \ /gnu/store/…-julia-1.8.5/bin/julia --project=/home/wrk/tmp/ai-root/factory/gn4-julia-env \ -e 'using Pkg; Pkg.instantiate(); using BulkLMM, CSV, DataFrames' ``` It downloads from the Julia registry (142 MB here) and takes a few minutes; afterwards `/compute/bulklmm` answers 21 056 rows in ~19 s. `BULKLMM_DEPOT` **cannot** choose where those packages live: Guix's `julia` is a wrapper that prepends a colon to `JULIA_DEPOT_PATH`, so the default depot (`$HOME/.julia`, the service's `HOME=/home/wrk`) is always first and the variable only appends. Point it at `/home/wrk/.julia` and leave it alone.