about summary refs log tree commit diff
path: root/docs/www/operator-checklist.md
AgeCommit message (Collapse)Author
31 hoursdocs: the auth stub is a second process, and its absence looks like a ↵ HEAD mainPjotr Prins
working API run.sh restart replaces the two instances only. On 2026-11-19 the stub on :8081 was down, so every authenticated endpoint answered 401 invalid_token: fetching http://127.0.0.1:8081/auth/public-jwks: io: Connection refused (os error 111) while /version and /mapping-methods still answered -- the deployment looked alive and the read surface was entirely dark. A monitor run is what names it, so the checklist now says what to look for (18 failed, all 401), how to start the stub, and that its key is fixed, which is why tokens minted before the outage were still valid afterwards. The stub's token file belongs under the deployment's state directory: /run does not survive a reboot, and this is the failure that follows.
4 daysoperator checklist: the ravanan build is a patched one, and whyPjotr Prins
Three local patches, the third of which is the reason a 20-chromosome scatter now runs 20-wide instead of 1-wide: {"job": ...} versus {"jobs": [...]}, which is slurmrestd's heterogeneous-job form. With the node declaring 30 CPUs (cold 24.7 s) and the submit fixed (cold 9.4 s), ravanan is within 3x of gemma-wrapper's 3.1 s for the same rows instead of 33x.
5 daysmonitor: run the cached engines cold, not just warmPjotr Prins
A warm run answers from caches the previous binary filled, so it can pass while the cold path -- what a new caller's first request actually does -- is broken. `--cold` sends each cached engine a payload with one sample value nudged by a thousandth, then repeats the original: every engine here keys its cache on content (gemma-wrapper's hash, ravanan's interned inputs, rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both runs are judged; the line reports "cold X / warm Y". Measured on production with the 30-sample fixture: compute-ravanan 102.0 s cold / 3.7 s warm compute-gemma 3.1 s cold / 2.0 s warm compute-rqtl 7.5 s cold / 7.3 s warm compute-qtl2 3.3 s cold / 3.3 s warm compute-fitqtl 6.7 s cold / 6.9 s warm compute-bulklmm 19.6 s cold / 19.0 s warm ravanan is the one where the cache is the difference between minutes and seconds; the rest are compute-bound at this size, and the honest thing is to print the pair rather than assert a speedup. A check whose payload cannot be perturbed skips with that reason instead of reporting two warm runs as cold and warm. The operator checklist gains the cold run as part of deploying, with both costs named: ravanan's cold run takes minutes, and every cold run adds cache entries.
5 dayscorrelation: the matrix endpoint, and the rule that a deploy ends with a ↵Pjotr Prins
monitor run `POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}` references -- a collection, as GN2's matrix form posts it, which spans datasets -- and answers an N x N matrix of r, p and n. The five decisions that are not obvious, each recorded where a reader will look for them: * **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to Spearman mid-row once a Pearson r passes 0.999, which the self-pair always does, so its matrix is asymmetric by construction. Here the method is chosen once and r[i][j] == r[j][i] exactly. * **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine calls, not N^2. * **The diagonal is by construction**: r = 1, p = null, n = the trait's own count, so half the traits cannot come back as 1.0000000000000002 with a NaN p. * **Per-cell n is its own matrix**, and a cell under `min_overlap` (default 6) is null in r and p while its n still says how many points there were. GN2 accepts 2, where |r| = 1 by definition. * **The sample axis is the first trait's group** (#106), resolved once and stated in the reply with the number of individuals it kept. Fetches are one query per dataset (`get_dataset_values_for_traits`, `get_publish_values_for_traits`), not one per trait: a matrix chooses a handful out of twenty thousand. Capped at 1 000 traits, because the cost is quadratic in the correlations and in the reply. No PCA -- the r matrix has to be trusted on its own first. Verified twice: cells agree with scipy to the last bit (BXDPublish 10001/10002/10005: 0.6756270426459695, 0.6950450737755965, 0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964, 0.8819879006700073), and a cell equals what the row endpoint says for the same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix, `min_overlap` above every cell's n, an unknown trait, `bicor`, and no traits at all. The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier all`, and says why: `/version` can report a commit and a reachable database while every engine that shells out is dead. The monitor gained a matrix check whose judge enforces square, symmetric and a unit diagonal.
5 daysmonitor: a --filter for "just retest this", and track the toolPjotr Prins
`--only` needs exact check names, which is no help when what you know is the dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring against the check's name, the sentence describing what it needs, and its hint -- so `--filter julia` finds compute-bulklmm even though its name says nothing about Julia. It selects tiers like `--only` does, and says so when nothing matches *because* discovery has not run (the compute checks do not exist until a group with samples is known). Also un-ignores bin/: the monitor is source, and an ops tool that is not versioned is the problem ai-memory #38 already named. Bytecode stays out. Co-report: the whole compute tier is green now (9 ok, 0 failed), including BulkLMM -- the recipe and the depot quirk are in the operator checklist.
5 daysoperator checklist: the tool block env.sh must keep, and why BulkLMM cannot ↵Pjotr Prins
work yet Writing down what the monitor found and what fixed it, because it is config that lives outside the repository and is therefore easy to lose at the next deploy: with only the cache dirs set, every engine that shells out answers 502 while the whole read surface and search look healthy. Measured on production after the fix (bin/monitor-gn4 --tier compute): 8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453), fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed, 1 busy. The remaining failure is BulkLMM and it is not a config line: the Julia project is registry-based and no depot on this box has its packages, so it stops at "Run Pkg.instantiate()". Populating a depot downloads from the Julia registry, which should be a decision rather than a side effect.
5 daysoperator checklist: how production actually runs now (store binary, no ↵Pjotr Prins
container) The layout, the restart command, where mutable state lives, and the two mistakes that cost time today: the CWL comes from the checkout rather than the store, and only two of the four ravanan 0.2.0 builds on this box convert --slurm-api-endpoint to a URI -- the others die inside slurm-api.scm with struct-vtable on a string, after staging every input, which reads like a Slurm fault rather than an argument one.
6 dayslogs: one access line per request, naming the callerPjotr
The proxy logs the caller through x-gn4-user, but the process log -- the one an operator tails while a job runs -- named nobody, so "who started this GEMMA run" meant going to the nginx log or guessing. genenetwork4 now writes one compact JSON line per request on stderr: {"event":"request","method":"GET","path":"/groups/BXD","status":200, "took_ms":10,"request_id":"…","user":"zachs"} Decisions in it: `user` is absent, not "-", when no token was verified (a dash is a value a reader can mistake for a user name); the path is logged without its query string, because a query can carry a credential and has no business in a log; the token value itself is never logged, only its subject; and the line carries the `request_id`, so it sits beside the existing `event: error` lines for 5xx and one grep joins a failure to the request that caused it. `GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of truth. Verified on a probe instance: with the test token the line says user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated 401 has no user field at all. docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's sub" -- true until now, which is exactly the kind of sentence that rots; it is replaced by the line above, what it contains and what it deliberately omits. README's x-gn4-user paragraph gains the same, next to the identity it already described.
9 daysdocs: how pve gets deployed, and what the deployed box runsPjotr
GEMMA_WRAPPER_CMD is the lever: the profile's gemma-wrapper may be older than the pin, and pointing the variable at the store path is enough -- -pve reaches GEMMA either way, but the per-run .log.txt files where pve lives are only kept beside the archive from 5eb53fa on. Recorded the verified production state too: both processes restarted with gemma-wrapper-1.00-pre1-4.5eb53fa, and POST /compute/gemma for BXD::10001 on the running instance returns pve 0.3125 to 0.9558 over 20 chromosomes -- measured there, not on a fixture.
11 daysapi: name the caller on the response, so the log can say whoPjotr
Answering "is robw using this successfully, and the others?" from the access logs took an hour of inference -- user agents, search terms, which traits were mapped -- because nginx records the address and the subject is only in the token. Two callers can share an address (an agent and a person behind one NAT), one caller can move between addresses (an LLM's egress is a /24), and the answer for a given person was "probably 80.115.1.132". The API now sets `x-gn4-user: <token sub>` on every **authenticated** response, and the `gn4` log format records it as `user=`. The subject is only known inside the handler (the `AuthToken` extractor verifies the token there), so a middleware opens a slot in the request extensions, the extractor fills it, and the header is set on the way out; a request with no valid token gets no header, which is the honest answer and reads as `user=-`. The caller only ever sees an identity it already presented. * src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording its subject. * src/lib.rs: the `tag_caller` middleware, outermost so the slot exists before any extractor runs. * tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header equals the token's `sub`, and `/version` (no identity) carries none. * etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=` in the log format. * README.md, docs/nginx.md (the format, an `awk` one-liner for per-user counts, and the caveat that old and new lines differ in field count), docs/www/operator-checklist.md (`user=-` means no token, not a token that failed late). 281 tests pass; the same four as before do not.
12 dayscompute: gate GEMMA from 50 individuals, R/qtl from 1 000Pjotr
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
12 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
12 daysdocs: match the server by argv[0], not by a substring of its command linePjotr
The restart section added a moment ago pointed at the trap and then walked into its other half: `nohup env … ./target/release/genenetwork4 &` leaves an `env` wrapper whose command line also contains the binary path, so matching `*release/genenetwork4*` finds the wrapper -- the pid file named a process that exits while the server keeps running, and the next `kill $(cat /tmp/gn4.pid)` did nothing at all. Read `argv[0]` (the first NUL-terminated word) and compare it to the binary path instead; that finds the serving process and only it. Applied here, it is what finally moved production off the previous build.
12 daysdocs: a restart that reports success and changes nothingPjotr
Deploying today, `nohup env … ./target/release/genenetwork4 &` left an intermediate process's pid in `$!`, so `kill $(cat /tmp/gn4.pid)` stopped that and the server kept answering. The replacement died with `AddrInUse` -- in its own log, where nobody was looking -- and `/version` still reported the previous commit. Everything looked fine: the port answered, the smoke tests passed, the notes were the old ones. The checklist now says what actually decides it: read `/proc` for the process running the release binary (there is no `ps` on this box), and believe `/version`'s `git` over a successful-looking start -- a `git` older than the build just made is this mistake, not a build problem. Start the replacement only once `:3000` has stopped answering, and write the resolved pid into the pid file.
12 daysdocs: tell users (and operators) to refetch the instructions when the build ↵Pjotr
moves `/version` reports the revision the *server* is running, and this deployment is updated in place, so an assistant that cached `ai-instruction.md` can end up calling an API that has grown options its copy never mentions -- by default, silently. The instructions now say what to do about it, in the two places a reader sees a revision: pull the repository and re-read the page when `git` is not the revision the session started with, and the same in the `/version` section where that field is explained. It matters for the last few commits in particular: covariates in `/compute/rqtl` and `/compute/gemma` were added, and a stale copy would never know to ask for them. `docs/www/operator-checklist.md` gets the operator's half: nothing here enforces it, so a release that adds endpoints or options should say so where tokens are handed out, and `source_digest` -- not the commit alone -- is what identifies a build in a bug report.
12 dayscompute: gate ravanan at one job, and name the binary that failedPjotr
`POST /compute/gemma/ravanan` is now gated like the other two: one workflow at a time, one per caller, `409 compute_busy` naming the run and linking `/compute/gemma/status/{id}` (ravanan has no status route of its own). One slot because ravanan already fans its per-chromosome steps out -- over Slurm, when that backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from `/compute/gemma`'s: different runners, different cache directories, so one of each at once is the overlap `/compute/rqtl` is already allowed. Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the slot for the full 5 s, the same caller's second request was refused at once, and another caller's waited for the slot and then ran. While testing that, `GemmaError::Io` turned out to print "failed to run gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced "failed to run gemma-wrapper: No such file or directory", which sent me looking at the wrong binary. Spawn failures are now their own variant naming the command, and plain file I/O says "I/O error: …". * src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()` exposes the slot count, and a test pins all three endpoints' counts (gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT). * src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with `status = "/compute/gemma/status"`. * src/gemma.rs (Spawn): names the command; the four run_streaming call sites use it; unit test for both messages. * README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
12 dayscompute: gate R/qtl at eight concurrent scans, one per callerPjotr
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default 8, read when the gate is built) with the same one-job-per-caller rule as GEMMA: a second submission from the same token while its scan is running or queued is refused `409 compute_busy`; a submission that finds every slot taken waits for one to come free. Eight because an R/qtl scan is a single-threaded Rscript on a 96-core box; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. Unlike GEMMA's single slot, concurrent scans share no cache (R/qtl results are uncached), so the only cost is the CPU they take. Two flaws only appeared once there was more than one slot: - the "what is running" record was a single `Option<Active>`, so a refusal quoted whichever job had most recently taken a slot -- robw's 409 named JoM's run id. It is now a map keyed by subject, and `a_refusal_quotes_the_callers_own_run_id` pins it; - `Busy::describe` took the endpoint identifier, so the message read "your rqtl job"; the decorator takes a separate `label` ("R/qtl") from the status path. The test suite hit the rule itself: six compute tests in tests/rqtl.rs share one token, and cargo runs them on several threads, so three were refused instead of run. They now serialise on a module mutex, with a comment saying why -- the same lesson clients need, that parallelism means more tokens, not more requests from one. * src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`; `Gate::new(permits)` was already parameterised. `running` becomes a map keyed by subject, since one record cannot name N jobs. Module doc talks about slots rather than a single slot. Tests: eight subjects run and the ninth waits, a caller still may not take two even with slots free, and a refusal quotes the caller's own run id. * src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use. * gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are independent. * src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which it needs. * tests/rqtl.rs: compute tests serialised. * README.md (/compute/rqtl), docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the numbers per endpoint, and that more parallelism means more tokens.
12 daysdb: recover from a connection the server closed, and report the databasePjotr
Every database-backed endpoint 500'd with `CodecError { IO error: Broken pipe }` while the process answered /version quite happily. The cause was ours: `pool()` turns off the driver's `check_health` (a ping at checkout, dead connections replaced) to save a round trip -- sound against a database 130 ms away, worth nothing against loopback, and it costs the pool its self-healing. One server-side close of an idle session then becomes one failed request per dead socket: ten in a row, until a restart. * src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping; `is_connection_loss` recognises the errors that mean the peer went away (broken pipe, reset, went away, lost connection, closed); `forget_pool` drops the cached pool so the next call rebuilds it; `health` probes reachability, cached ten seconds. One pool static shared by pool()/forget_pool (they had drifted into two). * src/error.rs (ApiError::from_db): on a connection loss, forget the pool -- the one place every database error passes through, so recovery does not depend on each caller. * src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a service that answers while nothing behind it works is now visible. * src/config.rs: the switch, with the trade written down. * tests/db_recovery.rs: kills every pooled session from a second connection, then asserts the next twelve requests all succeed -- the scenario that produced the outage, now a regression test. * tests/...: unit test for the classifier (its own IO error text and the "this is a bad query, do not retry" cases). * README.md ("Database connections"), docs/www/operator-checklist.md: the knob, why it defaults on, and what /version reports.
13 daysplot: take a posted result, and never re-run the mappingPjotr
The plot endpoints ran the mapping themselves, which meant the picture could come from a different run than the one the caller was looking at, and cost a scan to draw. They now take the mapping's own response -- the JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight to the plotter, which is what the scripts were written for (`python3 plot-rqtl.py result.json -o out.svg --trait …`). Validation replaces the run: `/plot/rqtl` needs `data.markers` with positional 4+ column rows, `/plot/gemma` needs `data` rows *and* `meta.columns` (its plotter reads the column names to find `p_lrt`), and anything else is a 422 saying "post that response back". `/plot/rqtl` also accepts just the `data` member; GEMMA cannot, and says why. The mapping's `actions` suggestion now describes posting its own response and carries no body -- duplicating a 21 000-row response into a suggestion would double the envelope for nothing, and the caller already holds it. * src/routes/plot.rs: posted result, shape validation, `?trait`/`?title` and `?bonferroni=false` passthrough to the scripts; no compute imports. * src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body. * tests/plot.rs: canned results instead of real mappings -- the tests now need only the plotter, no Rscript or gemma; covers both engines, the bare-`data` form, and the 422 for a body that is a request. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: "post the response back", and why GEMMA needs `meta.columns`.
13 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).
13 daysphewas: every trait peaking within a window of one positionPjotr
PublishXRef records the marker each published trait's QTL peaks at (Locus) with its LRS, which is a PheWAS waiting to happen: fix a position, find the markers within a window of it, and ask which traits peak at those markers. GET /groups/{group}/phewas?locus=rs30524230&window=10000 GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000 The window is in base pairs, centred, default 10 kb either side -- the "10K up and downstream" case. Positions come from the group's genotype marker table (the endpoint added last round), so `genotype_version` selects whose markers define the window and the answer reports them alongside the traits: an empty `traits` list is then readable as "nothing peaks that close" rather than a failure, and a ten-times wider window is offered as `next`. Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 -- ANPEP_band, then microbiome/haematology/infectious-disease traits. That cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say. * src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak markers, joining Phenotype/Publication through the existing trait_info_sql, strongest first; marker names are re-validated before they reach the IN list. * src/routes/phewas.rs: the endpoint, token-gated; window/centre validation (422 for none or negative), 404 for an unknown locus, notes for the empty cases, and a unit test for the version/group helper. * tests/phewas.rs: markers in the window, traits sorted by LRS with lod = lrs/4.604, every hit peaking at a listed marker, zero-width window, the two failure modes, and that the suggested wider window resolves. * README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS), docs/www/operator-checklist.md: the endpoint and its reading.
13 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.
13 daysdocs: parents and F1s are excluded by default, measured both waysPjotr
Asked whether the BXD mappings ran GN2/GN3's default (no parents, no F1s): they did not. BXD.8's panel is 198 RI strains + 37 F1s and no parent columns at all, and the runs passed all 235 with `null` for the F1s -- phenotypes right, but GEMMA's kinship and R/qtl's cross spanning 235 individuals instead of 198. Measured on trait 10001, both engines, both panels: identical answers (GEMMA p 5.9e-07 at rsm10000005700 chr 8, R/qtl LOD 2.9320 at rs48756159 chr 8), so nothing published from those runs is wrong. The lesson is that null-filling excludes an individual from the fit but not from the model; excluding it means subsetting the genotype file. * docs/genotypes.md: "Parents and F1s are excluded by default" -- the GN2 (`incparentsf1`) and GN3 (`get_sample_list(inc_par=False, inc_f1=False)`) mechanisms, what each panel actually holds, the both-ways measurement, and the rule for the alignment work. * docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the same, in a sentence and a paragraph respectively.
13 daysdocs: name the real genotype files, panels and timings everywherePjotr
docs/genotypes.md explained the convention; the endpoint docs still left a caller guessing which file to pass and how big the panel is. So: * README.md (/compute/gemma): GEMMA wants the BIMBAM pair `genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno` file; `trait_values` is one entry per individual in that file's column order; BXD's current panel stated (235 x 21056), with the 17.1 s measurement against the 19 s the 7 320-marker fixtures take. * README.md (/compute/rqtl): the file is `genotype/<file>.geno` (BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId` decides the engine, and the two measured runs with the UM-HET3 peak. * README.md (/compute/gemma/ravanan): same input convention as gemma. * docs/genotypes.md: how each measurement was made, step by step, so they can be repeated; and that step 3 (expanding values onto the panel) is the API's job in the next iteration. * docs/www/ai-instruction.md (4a): a table of the BXD files per engine with panel sizes, the UM-HET3 file, "ask the operator for any other panel" with where GN2 records which one is current, and the measured times. Values must be one per individual, in file order -- position, not name -- and the API does not re-index yet, so a mapping is prepared for the caller rather than guessed. * docs/www/operator-checklist.md: MappingMethodId per group, and the three measurements.
13 daysdocs: which genotype file is current, and three real mappings measuredPjotr
The compute endpoints take server-side paths, so callers need to know the deployment's genotype-file layout and which version is current -- neither of which is discoverable from this repository, and both of which are spread across GN2 and GN3 (GN2's per-group genotype/<Group>.json whose `genofile` list is newest-first and is the actual authority -- the GenoFile table still lists only the classic files; GEMMA fed from genotype/bimbam/<file>_geno.txt, R/qtl from genotype/<file>.geno; GN3 reading the sample order out of the .geno header; InbredSet. MappingMethodId deciding which engine a group offers). Measured on this deployment, with the newest genotypes and values re-indexed onto each panel's sample order: BXD BXD.8.geno (WGS Mar2022) GEMMA 17.1s 21055 rows BXD BXD.8.geno R/qtl 7.5s 21056 markers HET3-ITP.12 (Apr2023, 6447 individuals) R/qtl 83.1s 893 markers, lifespan peak LOD 7.95 at 1_24042124 (chr1, 24.04 Mb) That last run is the shape of what Rob Williams wants: UM-HET3 is a 4-way cross (MappingMethodId=3, R/qtl), and its genotype files are versioned per cohort with no JSON recording which is current, and no per-cohort trait split -- 6438 of lifespan's 23879 animal values fall in the .12 panel, the rest are silently dropped. Recorded as the known messiness rather than hidden. * docs/genotypes.md: new. Layout, how GN2/GN3 choose, the alignment rule, per-group engines, the measurements, what is still messy. * README.md: pointer. * docs/www/operator-checklist.md: replace "only the test fixtures exist" with the real panels, and name the missing alignment step.
13 daysrqtl: refuse a phenotype whose sample count is not the cross'sPjotr
Asked to map a 9-strain trait, the wrapper scanned it against the 6-individual fixture cross and returned six markers with LOD scores and `state: complete` -- a plausible answer belonging to nobody. R recycled and truncated the values silently, and nothing downstream could tell. GeneNetwork3's copy of this script has the same hole. The cross-file format carries no per-sample names, so the count is the only check available; order remains the caller's responsibility (and the reason values and genotype file must come from the same panel). With the check, the same request is a 502 rqtl_failed quoting both counts, and the correctly-shaped request still returns the documented LODs. * scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of the cross, and stop. * README.md (/compute/rqtl): the second fix relative to gn3, and what stays the caller's job. * docs/www/ai-instruction.md (4a): values and genotypes must describe the same individuals, one for one, in the same order. * docs/www/operator-checklist.md (Compute): what genotype files actually exist on this box (test fixtures, 198 BXD individuals) and why a database trait, with its own strain subset, cannot be mapped against them.
13 daysvalues: suggest R/qtl and GEMMA in the envelope's `actions`Pjotr
Fetching one trait's value per strain is usually the step before mapping that trait, and the API can do the mapping. Those calls are POSTs with a body, so they cannot go in `next` -- a `next` link is a GET that returns 200, which is what makes `next` safe to follow blindly and what tests/envelope.rs relies on. Hence a fifth envelope member, `actions`, holding method + href + description + body, and omitted entirely when there is nothing to suggest (most responses keep the four-member shape). The three per-trait values endpoints now carry a `POST /compute/rqtl` and a `POST /compute/gemma` suggestion with the values already in the body, in the response's own sample order. Genotype/SNP paths stay bracketed placeholders: they must exist on the server, which the server cannot know on the caller's behalf. The instruction page says, in the same breath, never to run R/qtl or GEMMA locally -- a local scan would use genotypes that are not the panel's and would disagree with GeneNetwork's own results -- and that results are cached, so repeat calls are cheap (measured: ~2 s cached against ~20 s cold on the BXD fixtures). Also fixes a flake the new test exposed rather than caused: config::tests read and write process-global env vars and Config::load() reads all of them, so the RAVANAN_* tests could fail search_engine_keys_follow_env_over_default_layering on an unrelated assertion. One module-level lock serialises them. * src/response.rs (SuggestedCall, Envelope::actions/with_action/ with_actions): the new member, omitted when empty; unit tests for both. * src/suggest.rs: new. Builds the two suggestions for either value type (db::TraitValue, db::ProbeSetValue), with its own unit tests. * src/routes/{traits,groups,datasets}.rs: the three values handlers. * tests/compute_suggestions.rs: new. Suggestions present with values in order, on all three routes; absent elsewhere; `next` still GET-only and still returns 200. * src/config.rs (tests): env_guard, taken by the 11 config tests. * README.md (Success responses): the `actions` member, with an example. * docs/www/ai-instruction.md (4a): never compute locally, why, caching, and how to use the suggested bodies. * docs/www/operator-checklist.md: the caching numbers.
13 dayscompute: proxy /compute/* for any valid tokenPjotr
R/qtl and GEMMA are part of the service now, so the blanket 403 goes: the handlers already require an AuthToken, which is the whole gate ("token_only", not resource-checked). What that buys and costs is now written down rather than implied -- any token holder can name server-side paths for R and gemma to read, and the public-data-only restriction on traits/datasets does not apply here at all. The box also has to be able to run them: PATH must carry Rscript, gemma-wrapper and gemma, *and* R_LIBS_SITE must point at the profile's site-library or scripts/rqtl_wrapper.R dies on `there is no package called 'optparse'` -- which reads like a token or proxy fault and is neither. Verified through nginx with a token: POST /gn4/compute/rqtl 200 in 0.4s with the fixture's marker rows, POST /gn4/compute/gemma 200 in 2.1s with 7 320 marker rows (identical to the documented run's first row), GET /gn4/compute/gemma/status/<id> ending in "GN4: computation complete", /compute/correlation 200; all four 401 without a token. * etc/nginx-vesuvio.pu.ac.ke.conf: drop the /compute 403 locations, keep them commented as the way back; timeouts/body-size comments now apply. * docs/www/ai-instruction.md: new §4a, "Running QTL mapping (compute)" -- shapes, server-side paths, ?id= + status polling, what to do with a 504. * docs/www/operator-checklist.md: compute moves from "must be off" to "proxied, know what it grants", plus what the box needs (PATH, R_LIBS_SITE, ravanan) and the measured results. * docs/nginx.md: same, in the deployment and checking sections.
13 daysdocs/www: say who issues tokens, and what the assistant should ask forPjotr
The page said a human issues tokens and to "stop and ask", without saying whose door to knock on -- and the operator commands belong in the operator file anyway. State it as the GeneNetwork team, with wording the assistant can relay to its user: what a token is for, how long the default lasts, and to pass it out of band rather than through a public issue or a shared chat. Same wording for renewal, since a 401 is the one failure the assistant cannot fix itself. * docs/www/ai-instruction.md (1. Get a token): "If you do not have one" and "If it stops working", the latter quoting request_id and retrying exactly once; the summary line and the 401 row of the error table say GeneNetwork team rather than "the operator". * docs/www/operator-checklist.md (Token issuance): the request path, so the team knows what the ask they receive looks like. * .gitignore: *.swp, after one got swept into this commit.
13 daysdocs/www: rewrite the AI instructions for the live deploymentPjotr
ai-instruction.md was written before anything was deployed: it took a placeholder base URL, did not say where the API is mounted, and buried the operator's deployment notes in an appendix labelled "not for the AI reader" -- awkward to hand to an assistant, and confusing when it is. So: the AI-facing page now names the deployment (https://vesuvio.pu.ac.ke/gn4) and is self-contained, and the operator material moves to operator-checklist.md. Everything in the page was exercised against the running deployment. Content changes, beyond the move: * `next` hrefs arrive already prefixed (`/gn4/groups/BXD/strains`), so they resolve against the origin -- joining them to the current URL's directory double-prefixes them. Said plainly, since it is the one thing the prefix breaks. * says what cannot work and should not be retried: get-test-token is a 404, /compute/* is a 403 from the front end, and neither is a permissions problem to work around. * `/publications/{pubmed_id}/traits` wants the PubMed id, not the internal `publication_id` in the same trait record, which 404s. * states the current rate-limit position (none configured) instead of implying a 429 policy that does not exist yet. * docs/www/ai-instruction.md: rewritten, paste-ready. * docs/www/operator-checklist.md: new file, the former appendix, updated with what is now enforced at the proxy and a record of the verified request matrix. * README.md, docs/nginx.md, etc/nginx-vesuvio.pu.ac.ke.conf: point at the operator checklist by its new name.