about summary refs log tree commit diff
path: root/docs/www/operator-checklist.md
blob: a6267de4861b9c6add1b12158ef8613c8ae34b94 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
# Operator checklist: deploying genenetwork4 for public, read-only use

Companion to [`ai-instruction.md`](ai-instruction.md), which is what the
end user (typically an AI assistant) is given. This file is for whoever
runs the deployment: what has to stay off the public surface, how tokens
are issued without gn-auth, and what is knowingly exposed.

The reference deployment is `https://vesuvio.pu.ac.ke/gn4/` proxied by
nginx to `127.0.0.1:3000` — see [`../nginx.md`](../nginx.md) and
[`../../etc/nginx-vesuvio.pu.ac.ke.conf`](../../etc/nginx-vesuvio.pu.ac.ke.conf).

```
nginx            443 / 80      public; TLS; /gn4/ -> 127.0.0.1:3000
genenetwork4     127.0.0.1:3000 loopback only, SEARCH_ENGINE=native
gn-auth stub     127.0.0.1:8081 loopback only, serves the JWKS
MariaDB          127.0.0.1:3306
xapian index     XAPIAN_DB_PATH
```

## Must be off / not routed

* `POST /get-test-token` — irrelevant while there is no gn-auth (tokens
  come from `scripts/gn-auth-stub.py --mint-only`, see below), but the
  moment gn-auth sits behind it this is a minting route, and the only
  thing standing between it and the internet is a loopback peer check
  plus a refusal of requests carrying proxy headers
  (`src/routes/dev.rs`). Behind nginx the peer *is* loopback, so a proxy
  that sets none of those headers makes it open to the world. Belt and
  braces, all three: `GENENETWORK4_TEST_TOKEN=false` (route not
  registered), bind to `127.0.0.1`, and the `location =
  /gn4/get-test-token { return 404; }` refusal in the nginx config.
* `POST /compute/*` (`/compute/gemma`, `/compute/gemma/ravanan`,
  `/compute/rqtl`, `/compute/correlation`) is **now proxied** — R/qtl and
  GEMMA are part of the service, gated on a valid token (see "Compute"
  below). Understand what that grants: those handlers take **server-side
  file paths** (`genotype_file`, `snps_file`) and run R/gemma on them, so
  every token holder can ask the box to read its own filesystem. It is
  `token_only`, not resource-checked: the public-data-only restriction
  that applies to traits and datasets does **not** apply here. The nginx
  config comments show how to re-block it
  (`location /gn4/compute/ { return 403; }`) if that is no longer
  wanted. `/compute/*/status/{id}` tails run logs by name and shares the
  same gate.
* The auth stub on `:8081` is not a dev leftover — without gn-auth it
  *is* the auth server — but it must stay reachable from genenetwork4
  only: **never** route it publicly. Its signing key
  (`test/data/test_rsa_private.pem`) and JWKS are in this repository, so
  anyone who can fetch its JWKS can mint tokens genenetwork4 accepts.
* Dev leftovers: `target/debug` binaries, `GN4_DB_MODE=dev` ("start
  without a database").

## Token issuance (no gn-auth: the stub, one token per user)

The request path is: the AI assistant has no token, so it asks the user
(see §1 of [`ai-instruction.md`](ai-instruction.md)); the user asks the
GeneNetwork team; someone here mints a token for that user and hands it
back out of band; the user passes it to the assistant. Expect the
request to say who it is for, what data, and for how long — that is what
`--subject` and `--expires-in` need.

Run the stub once and leave it running, supervised like genenetwork4 —
see "Tokens are stateless" in the README for why an hour of downtime is
survivable and longer is not:

```sh
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
  scripts/gn-auth-stub.py --token-file /run/gn4/stub.token
```

`AUTH_SERVER_URL` is where genenetwork4 fetches the key set
(`/auth/public-jwks`); it stays a loopback URL, and the public `/gn4`
prefix has nothing to do with it.

Then one token per user, on the box, without touching the running stub:

```sh
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
  scripts/gn-auth-stub.py --mint-only --subject alice \
      --expires-in 30d --token-file /run/gn4/alice.token
```

`--mint-only` mints, prints, writes the file and exits; `--issuer`
defaults to `AUTH_SERVER_URL`, which is what makes the token verify
against the running stub. Hand the file's contents over out of band.

What that token is, and is not:

* **public data only.** A token minted without `--grant-resource-id`
  carries no privileges: genenetwork4 decides what may go out from each
  dataset's `public`/`confidentiality`/`create_time` columns and asks
  the authorisation lookup about the rest (`src/auth.rs`'s `check`). The
  lookup answers "no privileges", so public data reads and every
  non-public dataset is a `403`. **Never** mint with
  `--grant-resource-id` for a shared or long-lived token: it hands the
  holder that non-public resource and cannot be taken back before `exp`.
* **expiry is the only revocation.** No denylist exists, so a token is
  good until its `exp`; `--expires-in` is the withdrawal mechanism.
  Revoking *everything* means changing `AUTH_SERVER_URL` or the signing
  key — `iss` is compared verbatim, so a new host, scheme or trailing
  slash invalidates every outstanding token with `InvalidIssuer`.
* **`sub` is attribution, not authentication.** It names the user, is
  used as an authorisation-cache key, and grants nothing. Anyone holding
  a token is that `sub` as far as the API can tell, and the `scope`
  claim is not consulted for data access.
* **per user, per expiry** — `--subject alice`, `--subject bob`: same
  key and issuer, independent `exp`, so one token can lapse without
  touching the others.
* verify before handing over: `curl -s -H "Authorization: Bearer $TOKEN"
  https://vesuvio.pu.ac.ke/gn4/groups | head -c 80` should return an
  envelope.

When gn-auth lands, only the source of tokens changes:
`POST /get-test-token` (gn-auth's dev password grant) replaces
`--mint-only`, and tokens gain gn-auth's own lifetime (7–14 days,
bounded by JWKS rotation) and its per-user grants.

## Compute: what the box needs

R/qtl and GEMMA run as child processes, so the genenetwork4 process's
environment has to be able to find them. From a plain shell it is not
enough to have the binary in a store path — the R packages matter too.
The deployment shell gives both:

```sh
P=$(guix shell -D -f guix.scm -- bash -c 'echo $GUIX_ENVIRONMENT')
PATH=$P/bin:$PATH R_LIBS_SITE=$P/site-library/ genenetwork4 …other env…
```

- `$P/bin` carries `Rscript`, `gemma-wrapper` and `gemma`. If the profile's
  `gemma-wrapper` is older than the pin in `guix.scm`, point
  `GEMMA_WRAPPER_CMD` at the store path directly rather than editing `PATH`:
  `-pve` reaches GEMMA either way, but the per-run `.log.txt` files (where
  `pve` lives) are only kept beside the archive from `5eb53fa` on.
- `R_LIBS_SITE=$P/site-library/` is what lets `scripts/rqtl_wrapper.R`
  load `optparse`, `qtl`, `stringi` and `stringr`. Without it the request
  fails with `rqtl_failed` and `there is no package called 'optparse'`
  — an environment problem, not a token or proxy problem.

Real panels are served from `GENOTYPE_FILES` (default
`/home/wrk/tmp/ai-root/factory/gn4-data/genotype-files`), in
GeneNetwork2's layout, and `GET /groups/{group}/genotypes` exposes them
by **version only** — no paths, no file names — which is what
`POST /compute/{rqtl,gemma}` resolve `group` + `genotype_version`
against. The layout: `genotype/BXD.8.geno` (WGS
Mar2022, 235 individuals × 21 057 markers) with GEMMA's
`genotype/bimbam/BXD.8_geno.txt` + `_snps.txt`, and
`genotype/HET3-ITP.12.geno` (6 447 individuals × 894 markers) for
UM-HET3. `docs/genotypes.md` covers how the current version is chosen,
which engine each group offers, and the measured runs.

Both engines match values to individuals **by position**; the API hides
that by aligning named `samples` onto the panel itself, and by dropping
parents/F1s the only way that works — a panel-subset genotype file under
the engine's cache directory (`/tmp/gn4-rqtl/prepared`,
`/tmp/gn4-gemma/prepared`), reused by content and panel. If those caches
are wiped, the next mapping recreates them (a few seconds on BXD).

Parents and F1s are excluded by default, as in GN2/GN3 (GN2's
`incparentsf1`, GN3's `get_sample_list(inc_par=False, inc_f1=False)`).
BXD.8's panel is 198 RI strains + 37 F1s and no parent columns, so that
default means "the 198 strains"; `HET3-ITP.12` is individual animals
only. `null`-filling a non-strain individual keeps its phenotype out of
the fit but *not* its genotype out of GEMMA's kinship matrix or R/qtl's
cross, so excluding it means subsetting the genotype file — which is what
the API should do when it takes over the alignment.

Which engines a group offers is `InbredSet.MappingMethodId` (1 → GEMMA,
QTLReaper, R/qtl; 2 → GEMMA; 3 → R/qtl; 4 → GEMMA, PLINK): BXD = 1,
HET3-ITP = 3. So UM-HET3 is R/qtl only, correct for a 4-way cross even
though `bimbam/HET3-ITP.*` exists.

Measured on this box with the newest genotypes: BXD (`BXD.8`, 235
individuals, trait 10001 with 34 values) 17.1 s / 21 055 marker rows on
GEMMA and 7.5 s / 21 056 markers on R/qtl; UM-HET3 lifespan over
`HET3-ITP.12` (6 447 individuals, 6 438 with values) 83.1 s / 893
markers, peak LOD 7.95 at chr 1, 24.04 Mb. Details and the misgivings
about HET3-ITP's versioning are in `docs/genotypes.md`.

`/compute/gemma/ravanan` additionally needs `ravanan` and `ccwl` on
`PATH`, and with `RAVANAN_BATCH_SYSTEM=slurm-api` a reachable slurmrestd
(see `docs/ravanan-slurm.md`).

Repeat calls are cached by input (gemma-wrapper's content-hash cache,
and src/rqtl.rs's own hash of genotype+phenotype), so the second
identical request is cheap: the same GEMMA run measured ~2 s cached
against ~20 s cold. Worth telling users, because the alternative they
reach for — downloading the values and fitting a scan themselves — is
both slower and wrong (their genotypes are not the panel's).

Verified through the proxy with a token: `POST /gn4/compute/rqtl`
returned 200 in 0.4 s with the fixture's marker rows, `POST
/gn4/compute/gemma` 200 in 2.1 s with 7 320 marker rows, and
`GET /gn4/compute/gemma/status/<id>` ended with `GN4: computation
complete`. Without a token both are `401`.

Two operational consequences: the nginx `proxy_read_timeout` (3600 s)
must stay generous, since a real mapping takes minutes; and each request
is CPU-bound on the box, so rate-limit per token or keep the audience
small — `limit_req` in the config is one way, but it keys on address,
not token.

## Database connections

`SQL_CHECK_HEALTH` (default `true`) makes the pool ping a connection at
checkout and replace it when the ping fails. Keep it on: without it, a
connection MariaDB closed while idle becomes a `500 database_error` on
whichever request draws it, one per dead socket, until the process
restarts — that is what happened on 2026-11-07. `GET /version` now carries
`"database": "reachable" | "unreachable"` (cached ten seconds), so an
up-but-useless service is visible without calling a data endpoint.

## Compute concurrency

The gates exist to stop **large** computations piling up, so they only
apply above a threshold that differs by endpoint:
`compute_gate::GEMMA_LARGE_PANEL` is **50** individuals (all GEMMA runs
share `GEMMA_CACHE_DIR`, where `gemma-wrapper` extracts per-chromosome
kinship archives by content hash -- two runs at once are two writers over
the same hundreds of megabytes), `compute_gate::RQTL_LARGE_PANEL` is
**1 000** (a `scanone` over a few hundred individuals is seconds of one
core). A panel below its endpoint's threshold takes no slot. Above it, `POST /compute/gemma` takes one job at a time,
`POST /compute/rqtl` `RQTL_MAX_CONCURRENT` (default 8) and
`/compute/gemma/ravanan` one at a time, each one large job per caller. All
three are in-process gates (`src/compute_gate.rs`), so they are per server
process and reset on restart -- a job's child process, however, outlives
the server. Waits are bounded only by nginx's `proxy_read_timeout`.

The slot count is fixed when a gate is first built (its first large
request), so raising `RQTL_MAX_CONCURRENT` takes a restart, not a reload.
Changing a threshold is the same: both are constants, and the reasons for
the numbers are in their doc comments.

## Plotting

`POST /plot/rqtl` and `/plot/gemma` take a **posted mapping response** --
no mapping is re-run -- and hand it to the deployment's plotter: `scripts/plot-rqtl.py` / `scripts/plot-gemma.py`,
executed by `PLOT_PYTHON` (default `python3`) -- so **python3 must be on
the server process's `PATH`**, alongside `Rscript`/`gemma-wrapper` for the
mapping itself. They return `image/svg+xml` rather than an envelope, and
the plotter's peak summary in `x-plot-summary`. Point `RQTL_PLOTTER` /
`GEMMA_PLOTTER` elsewhere if the scripts move; a missing script is a
`502 plot_failed` naming the interpreter and script it tried.

## PheWAS

`GET /groups/{group}/phewas` reads `PublishXRef` only -- the peak marker,
LRS and the join to `Phenotype`/`Publication` -- plus the group's
genotype marker table for positions. It is token-only like the other
metadata routes, and it inherits the exposure question below:
`PublishXRef` covers published traits, whose datasets are the `*Publish`
freezes.

## Data exposure

* Public rule: `public > 0 && confidentiality < 1`, or older than five
  years (`PUBLIC_AFTER_YEARS`, `src/auth.rs`). Values endpoints enforce
  it per dataset; `/groups/{group}/traits` checks the group's publish
  freeze.
* Metadata endpoints are `token_only` by design and reveal the names and
  descriptions of *non-public* datasets (e.g. `GET /datasets/{name}` for
  a `public=0` ProbeSetFreeze returns the record, with its `public` and
  `confidentiality` flags). This was a deliberate decision: a caller
  needs to know a dataset exists to know it is being refused. If that is
  not acceptable, gate `datasets.rs::dataset_by_name`,
  `groups.rs::{groups, group_by_name, group_datasets, group_strains,
  group_case_attributes}`, `species.rs` and `traits.rs` metadata on the
  same rule.
* Rebuild, do not inherit, the search index: GN4's indexer applies
  `public > 0 AND confidentiality < 1` at crawl time
  (`src/indexer/rows.rs`), whereas gn3's `index-genenetwork` has no
  `WHERE` clause. Verified on the current full index:
  `dataset:hcppublish` (a `confidentiality=1` freeze) and
  `dataset:br_u_0303_m` (a `public=0` freeze) both return 0 hits.
* Error bodies can carry internal detail: a `db_unavailable` `503`
  includes the driver error with the database address. Strip or
  generalise `detail` at the proxy, or accept disclosing
  `127.0.0.1:3306`.
* Serve everything over TLS (port 80 redirects), keep `/openapi.json`,
  `/version` and the version envelope's git revision open — they are
  useful and harmless.

## Verified on this deployment

```
GET  /                                302 -> /gn4/              (HTTP/2)
GET  /gn4                             301 -> /gn4/
GET  /gn4/version                     200  envelope, git revision
GET  /gn4/openapi.json                200
GET  /gn4/groups/BXD                  no token 401 / token 200
GET  /gn4/groups/BXD/traits?limit=2   200
GET  /gn4/search?q=alcohol&limit=3     200  state: complete
GET  /gn4/datasets/{non-public}/…     403
GET  /gn4/species/nope                404  links rewritten to /gn4/…
POST /gn4/get-test-token              404
POST /gn4/compute/correlation         200 with a token / 401 without
http://…/                             301 -> https://…
```

`next` links come back already prefixed (`/gn4/groups/BXD/strains`) and
resolving one through the proxy returns 200, so the `sub_filter`
rewriting in nginx is doing its job for success and error bodies alike.
Check it after any nginx change: without it every link points outside
the mount point, and the API's own tests cannot see that (they run
in-process, where there is no prefix).

## When you update the running build

This deployment is updated in place, so the build revision users see at
`GET /gn4/version` moves under them. Two things follow:

* their assistants are told (in `ai-instruction.md`) to pull the
  repository and re-read it whenever `git` is a revision they have not
  seen before, because the page they cached no longer describes the API
  they are now calling. Nothing enforces that from this side, so if a
  release adds endpoints or options, say so where you hand out tokens;
* `source_digest` changes with every build, dirty or not -- it identifies
  the binary, so quote it in bug reports rather than the commit alone.

## Restarting it: trust `/version`, not the pid file

`nohup env … ./target/release/genenetwork4 &` does not reliably leave the
server's own pid in `$!` — on this box it named an intermediate process, so
`kill $(cat /tmp/gn4.pid)` killed that and left the *server* answering.
The replacement then died with `AddrInUse` (in its log, not on the
console) while the old build kept serving: everything looked successful and
nothing had changed.

Two checks make that impossible to miss:

```sh
# who is actually serving (no ps here; read /proc, and match argv[0])
for d in /proc/[0-9]*; do a0=$(tr '\0' '\n' < $d/cmdline 2>/dev/null | head -1); \
  case "$a0" in ./target/release/genenetwork4) echo "${d#/proc/}";; esac; done

# and what is actually answering: git must be the commit you just built
curl -s https://vesuvio.pu.ac.ke/gn4/version | jq -r '.data | "\(.git) \(.dirty)"'
```

Match **argv[0] against the binary path**, not the whole command line:
`nohup env …` leaves a wrapper process whose command line also contains
that path, so a substring match finds the wrapper -- which is how the pid
file ended up naming a process that exits while the server keeps running.
A `git` value older than the build you just made is this mistake, not a
build problem: write the resolved pid into the pid file (and start the
replacement only after `:3000` has stopped answering). `source_digest`
changes on every build, so it distinguishes two builds of the same commit.

## Optional but worth it

* Rate limit per address at the proxy: `limit_req_zone` +
  `limit_req`/`limit_req_status 429` are commented in the config. 429
  rather than 503, because `Retry-After` on 429 is what the API's
  clients are told to expect.
* Put genenetwork4 and the stub under the same supervision (systemd
  units, not a shell): the stub's availability bounds every
  authenticated request, and `JWKS_TTL` (an hour) is the grace period.
* Logs: the nginx `gn4` log format records status, `rt`, the
  `x-request-id`, **`user=`** (the API's `x-gn4-user` response header, i.e.
  the token's subject, so per-user success is `awk '{print $9, $10}'` away
  instead of inferred from addresses) and the user agent. `user=-` means
  the request carried no valid token: scanners and anonymous reads, never
  a caller whose token failed after authenticating.
* **genenetwork4 writes its own access line too**, one JSON object per request
  on stderr, naming the caller:

  ```json
  {"event":"request","method":"GET","path":"/groups/BXD","status":200,"took_ms":10,"request_id":"…","user":"zachs"}
  ```

  So `tail -f` on the process log answers "who ran this job" without going
  through the proxy's log — which matters while watching a run, and it is the
  same log the `event: error` lines go to, correlated by `request_id`.
  `user` is **absent** (not `-`) when no token was verified. The path is
  logged without its query string, and the token itself is never logged —
  only its subject. Turn it off with `GN4_ACCESS_LOG=false` where the proxy's
  access log is the record of truth.
* A `/search` `total`/facet would end the paging-to-count pattern that
  costs ~23 s per 15 000 hits; see
  [`../search-engines.md`](../search-engines.md).

## Production layout on this box (2026-11-15, no container)

The API runs the **store binary directly**, as `wrk`, with no `guix shell`
container around it. Two instances (3000, 3001) are started by
`factory/gn4-prod/run.sh`, which sources `factory/gn4-prod/env.sh`:

```sh
runuser -u wrk -- /home/wrk/tmp/ai-root/factory/gn4-prod/run.sh restart
```

`run.sh` also takes `start`, `stop` and `status`; the instances log to
`factory/gn4-prod/logs/<port>.log` (appending across restarts, so the access
lines survive a deploy). Deploying is therefore: `guix build -f guix.scm`,
edit `env.sh`'s `GEN4=`, `run.sh restart`.

**Then run the monitor. Every time, without being asked:**

```sh
bin/monitor-gn4 --token-file /tmp/prodtok.file --tier all
```

And **run it cold as well as warm** after anything that touches the
environment, and weekly:

```sh
bin/monitor-gn4 --token-file /tmp/prodtok.file --tier compute --cold
```

A warm run answers from caches that were filled by the *previous* binary, so
it can pass while the cold path -- the one a new user's first request takes --
is broken. `--cold` sends each cached engine a payload with one value nudged,
which every engine here keys on (gemma-wrapper's content hash, ravanan's
interned inputs, rqtl/qtl2/bulklmm's own hashes), then repeats the original
payload for the warm hit. Both runs must pass, and the line reports
`cold X / warm Y`. Measured on this box with a 30-sample fixture: ravanan
102.0 s cold / 3.7 s warm; gemma 3.1 / 2.0; rqtl, qtl2, fitqtl and BulkLMM are
compute-bound at this size and show cold ~= warm, which is a fact about the
fixture, not a fault. Two costs to know: ravanan's cold run takes minutes, and
every cold run leaves another entry in the caches (`GEMMA_CACHE_DIR`,
`RAVANAN_CACHE_DIR`, …), so run it deliberately rather than in a loop.

`/version` only says which commit is running; it cannot see that an engine
lost its tool path, that a config key moved, or that an endpoint now answers
a shape nobody expected. On 2026-11-15 the service was "deployed and up" with
`database: reachable` while **every** shell-out engine answered 502 and
`/search` answered 502 in 1.4 ms, and the only thing that caught it was this
monitor. A deploy that has not been through it is not finished. Expect
`39+ ok, 0 failed` and read any skip: the `listener` check skips whenever the
service runs outside the monitor's PID namespace, which is normal on this box
and is the only skip that should ever appear.

Everything mutable is under `factory/gn4-prod/` — `tokens.sqlite` (the only
copy: losing it loses every token), `gemma-cache`, `rqtl-cache`,
`qtl2-cache`, `bulklmm-cache`, `bnw-cache`. The ravanan cache is the
exception: it must be under `/home/wrk/tmp/ai-root/slurm-share/` because the
Slurm container bind-mounts nothing else (see
[`../ravanan-slurm.md`](../ravanan-slurm.md)).

Two things that are easy to get wrong, both measured on this box:

* **The workflow and manifest come from the checkout**, not from the store
  build: `GEMMA_RAVANAN_WORKFLOW` and `GEMMA_RAVANAN_MANIFEST` point into
  `factory/code/genenetwork/genenetwork4/cwl/gemma/`, so the checkout has to
  be at the commit the binary was built from — otherwise the CWL that runs
  is not the one that was tested.
* **The ravanan in `env.sh` is a locally patched build.**
  `/gnu/store/amn5dfh4…-ravanan-0.2.0` comes from
  `factory/code/ravanan` (commit `6f9031b`) and carries three patches the
  upstream 0.2.0 lacks: `--slurm-api-endpoint` parsed into a URI, `user-error`
  printed unconditionally (job scripts have no log level, so their failures
  were silent), and `{"job": …}` instead of `{"jobs": […]}` on submit. That
  last one is worth remembering: the plural form is slurmrestd's
  *heterogeneous* job submission, so every step arrived as a one-component het
  job holding a whole node, and a 20-chromosome scatter ran strictly 1-wide
  (cold 24.7 s). With it fixed: 20-wide, cold 9.4 s. Rebuild with
  `guix build --with-source=ravanan=<checkout> --without-tests=ravanan
  ravanan` (its own test suite wants node and fails in the sandbox).
* **ravanan 0.2.0 exists on this box in two variants**, and only one works
  with `--slurm-api-endpoint`. In `/gnu/store/1ydn2bdw…-ravanan-0.2.0`
  (and `20aqba00…`) the option handler converts its argument,
  `(acons 'slurm-api-endpoint (string->uri arg) result)`; in
  `rim2v0pi…` and `vyqs04g9…` it does not, so `slurm-api.scm` calls
  `uri-scheme` on a string and the run dies with
  `In procedure struct-vtable: Wrong type argument …: "http://127.0.0.1:6820"`
  *after* staging every input, i.e. looking like a Slurm problem. `env.sh`
  pins the working one.

`GENENETWORK4_TEST_TOKEN=false` in production: `/get-test-token` mints a
token for any local caller, which is a development convenience only. Mint
with `POST /tokens` (localhost-only) instead.

### The tool block `env.sh` must keep

Every engine that shells out needs absolute paths in `env.sh`; without them
the read surface and search stay healthy while all of compute answers 502
(measured 2026-11-15: 1 ok / 7 failed / 1 busy before, 8 ok / 1 failed after).
`gn4-container/gn4.env` is the reference for the keys:

```sh
PATH=$R_BIN:$WRAPPER:/usr/bin:/bin     # Rscript first: no R in any system profile
R_LIBS_SITE=<profile>/site-library      # r-qtl and r-qtl2 live here
GEMMA_WRAPPER_CMD, GEMMA_DIR, GEMMA_WRAPPER_DIR   # + gemma/time/tar/xz/parallel/pfff
RQTL_WRAPPER_PATH, FITQTL_SCRIPT_PATH, QTL2_SCRIPT_PATH,
GEMMA_PLOTTER, BULKLMM_PLOTTER, BULKLMM_SCRIPT, GENOTYPE_PLOTTER, BNW_PLOTTER
BNW_CMD, BNW_BIN_DIR, BNW_EXAMPLES_DIR
BULKLMM_CMD, BULKLMM_ENV, BULKLMM_DEPOT
BASKET_DIR                              # per-user SQLite, not /tmp
```

Two of these are easy to get wrong:

* The script paths are absolute because the service does not run from the
  checkout — `config.rs`'s defaults (`scripts/rqtl_wrapper.R`) resolve against
  the working directory, and `run.sh` cd's to the checkout only as a
  belt-and-braces measure.
* `R_LIBS_SITE` must match the R that reads it: the profile below is built for
  **R 4.6.1**, so `R_BIN` has to be an `r-minimal-4.6.1`, not whichever R the
  store happens to offer first.

**BulkLMM needs a populated Julia depot, once.** Its project
(`gn4-julia-env`) is registry-based and Julia had none of its packages, so the
endpoint failed with `Package CSV ... is required but does not seem to be
installed: Run Pkg.instantiate()`. The fix is one command, run as `wrk`:

```sh
JULIA_DEPOT_PATH= HOME=/home/wrk \
  /gnu/store/…-julia-1.8.5/bin/julia --project=/home/wrk/tmp/ai-root/factory/gn4-julia-env \
  -e 'using Pkg; Pkg.instantiate(); using BulkLMM, CSV, DataFrames'
```

It downloads from the Julia registry (142 MB here) and takes a few minutes;
afterwards `/compute/bulklmm` answers 21 056 rows in ~19 s.

`BULKLMM_DEPOT` **cannot** choose where those packages live: Guix's `julia`
is a wrapper that prepends a colon to `JULIA_DEPOT_PATH`, so the default
depot (`$HOME/.julia`, the service's `HOME=/home/wrk`) is always first and the
variable only appends. Point it at `/home/wrk/.julia` and leave it alone.