| Age | Commit message (Collapse) | Author |
|
working API
run.sh restart replaces the two instances only. On 2026-11-19 the stub on
:8081 was down, so every authenticated endpoint answered
401 invalid_token: fetching http://127.0.0.1:8081/auth/public-jwks:
io: Connection refused (os error 111)
while /version and /mapping-methods still answered -- the deployment looked
alive and the read surface was entirely dark. A monitor run is what names
it, so the checklist now says what to look for (18 failed, all 401), how to
start the stub, and that its key is fixed, which is why tokens minted before
the outage were still valid afterwards.
The stub's token file belongs under the deployment's state directory: /run
does not survive a reboot, and this is the failure that follows.
|
|
Three local patches, the third of which is the reason a 20-chromosome scatter
now runs 20-wide instead of 1-wide: {"job": ...} versus {"jobs": [...]},
which is slurmrestd's heterogeneous-job form. With the node declaring 30 CPUs
(cold 24.7 s) and the submit fixed (cold 9.4 s), ravanan is within 3x of
gemma-wrapper's 3.1 s for the same rows instead of 33x.
|
|
A warm run answers from caches the previous binary filled, so it can pass
while the cold path -- what a new caller's first request actually does -- is
broken. `--cold` sends each cached engine a payload with one sample value
nudged by a thousandth, then repeats the original: every engine here keys its
cache on content (gemma-wrapper's hash, ravanan's interned inputs,
rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both
runs are judged; the line reports "cold X / warm Y".
Measured on production with the 30-sample fixture:
compute-ravanan 102.0 s cold / 3.7 s warm
compute-gemma 3.1 s cold / 2.0 s warm
compute-rqtl 7.5 s cold / 7.3 s warm
compute-qtl2 3.3 s cold / 3.3 s warm
compute-fitqtl 6.7 s cold / 6.9 s warm
compute-bulklmm 19.6 s cold / 19.0 s warm
ravanan is the one where the cache is the difference between minutes and
seconds; the rest are compute-bound at this size, and the honest thing is to
print the pair rather than assert a speedup. A check whose payload cannot be
perturbed skips with that reason instead of reporting two warm runs as cold
and warm.
The operator checklist gains the cold run as part of deploying, with both
costs named: ravanan's cold run takes minutes, and every cold run adds cache
entries.
|
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
`--only` needs exact check names, which is no help when what you know is the
dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring
against the check's name, the sentence describing what it needs, and its
hint -- so `--filter julia` finds compute-bulklmm even though its name says
nothing about Julia. It selects tiers like `--only` does, and says so when
nothing matches *because* discovery has not run (the compute checks do not
exist until a group with samples is known).
Also un-ignores bin/: the monitor is source, and an ops tool that is not
versioned is the problem ai-memory #38 already named. Bytecode stays out.
Co-report: the whole compute tier is green now (9 ok, 0 failed), including
BulkLMM -- the recipe and the depot quirk are in the operator checklist.
|
|
work yet
Writing down what the monitor found and what fixed it, because it is config
that lives outside the repository and is therefore easy to lose at the next
deploy: with only the cache dirs set, every engine that shells out answers
502 while the whole read surface and search look healthy.
Measured on production after the fix (bin/monitor-gn4 --tier compute):
8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453),
fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed,
1 busy.
The remaining failure is BulkLMM and it is not a config line: the Julia
project is registry-based and no depot on this box has its packages, so
it stops at "Run Pkg.instantiate()". Populating a depot downloads from the
Julia registry, which should be a decision rather than a side effect.
|
|
container)
The layout, the restart command, where mutable state lives, and the two
mistakes that cost time today: the CWL comes from the checkout rather than
the store, and only two of the four ravanan 0.2.0 builds on this box
convert --slurm-api-endpoint to a URI -- the others die inside
slurm-api.scm with struct-vtable on a string, after staging every input,
which reads like a Slurm fault rather than an argument one.
|
|
The proxy logs the caller through x-gn4-user, but the process log -- the one an
operator tails while a job runs -- named nobody, so "who started this GEMMA
run" meant going to the nginx log or guessing. genenetwork4 now writes one
compact JSON line per request on stderr:
{"event":"request","method":"GET","path":"/groups/BXD","status":200,
"took_ms":10,"request_id":"…","user":"zachs"}
Decisions in it: `user` is absent, not "-", when no token was verified (a dash
is a value a reader can mistake for a user name); the path is logged without
its query string, because a query can carry a credential and has no business
in a log; the token value itself is never logged, only its subject; and the
line carries the `request_id`, so it sits beside the existing `event: error`
lines for 5xx and one grep joins a failure to the request that caused it.
`GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of
truth.
Verified on a probe instance: with the test token the line says
user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated
401 has no user field at all.
docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's
sub" -- true until now, which is exactly the kind of sentence that rots; it is
replaced by the line above, what it contains and what it deliberately omits.
README's x-gn4-user paragraph gains the same, next to the identity it already
described.
|
|
GEMMA_WRAPPER_CMD is the lever: the profile's gemma-wrapper may be older
than the pin, and pointing the variable at the store path is enough --
-pve reaches GEMMA either way, but the per-run .log.txt files where pve
lives are only kept beside the archive from 5eb53fa on.
Recorded the verified production state too: both processes restarted with
gemma-wrapper-1.00-pre1-4.5eb53fa, and POST /compute/gemma for BXD::10001
on the running instance returns pve 0.3125 to 0.9558 over 20 chromosomes
-- measured there, not on a fixture.
|
|
Answering "is robw using this successfully, and the others?" from the
access logs took an hour of inference -- user agents, search terms, which
traits were mapped -- because nginx records the address and the subject is
only in the token. Two callers can share an address (an agent and a person
behind one NAT), one caller can move between addresses (an LLM's egress is
a /24), and the answer for a given person was "probably 80.115.1.132".
The API now sets `x-gn4-user: <token sub>` on every **authenticated**
response, and the `gn4` log format records it as `user=`. The subject is
only known inside the handler (the `AuthToken` extractor verifies the
token there), so a middleware opens a slot in the request extensions, the
extractor fills it, and the header is set on the way out; a request with no
valid token gets no header, which is the honest answer and reads as
`user=-`. The caller only ever sees an identity it already presented.
* src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording
its subject.
* src/lib.rs: the `tag_caller` middleware, outermost so the slot exists
before any extractor runs.
* tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header
equals the token's `sub`, and `/version` (no identity) carries none.
* etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=`
in the log format.
* README.md, docs/nginx.md (the format, an `awk` one-liner for per-user
counts, and the caveat that old and new lines differ in field count),
docs/www/operator-checklist.md (`user=-` means no token, not a token
that failed late).
281 tests pass; the same four as before do not.
|
|
One threshold for both engines was the wrong shape. For R/qtl the
interesting line is where a `scanone` stops being seconds (a few hundred
individuals is seconds of one core), so 1 000 stays. For GEMMA it is much
earlier, and not because of the runtime:
- every GEMMA run goes through **one shared cache directory**
(`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome
kinship archives of hundreds of megabytes by content hash. Two runs at
once are two writers over the same archives, and a half-extracted kinship
another run reads surfaces as a GEMMA failure;
- a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382
individuals write 207 MB of kinship per chromosome.
So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every
mapping request is gated in practice, and only a handful of individuals (a
four-way cross's parents, a probe of the pipeline) runs free. `ravanan`
uses the same number, since it writes into the same cache. The threshold
moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint
carries its own; the policy, the slots and the 409 are unchanged.
Tests, asked and answered: at 50 the three gemma tests that do real work
(fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated
and share one token, so on a cold cache two of them at once would collide
into `409 compute_busy` -- a gate doing its job, failing a test that only
wanted to map something. They now take a lock, and only they do; the
validation tests stay parallel, and the rqtl suite is untouched (its gate
still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an
empty directory, `cargo test --test gemma` is green twice in a row, 18 s
per run, i.e. really recomputing.
* src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL`
(50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`;
the test that pinned one number now pins both and that each endpoint is
wired to its own.
* tests/gemma.rs: the three real-job tests take the lock, and say why.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
the two thresholds and the cache-directory reason for the lower one.
277 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
The restart section added a moment ago pointed at the trap and then walked
into its other half: `nohup env … ./target/release/genenetwork4 &` leaves
an `env` wrapper whose command line also contains the binary path, so
matching `*release/genenetwork4*` finds the wrapper -- the pid file named
a process that exits while the server keeps running, and the next
`kill $(cat /tmp/gn4.pid)` did nothing at all.
Read `argv[0]` (the first NUL-terminated word) and compare it to the
binary path instead; that finds the serving process and only it. Applied
here, it is what finally moved production off the previous build.
|
|
Deploying today, `nohup env … ./target/release/genenetwork4 &` left an
intermediate process's pid in `$!`, so `kill $(cat /tmp/gn4.pid)` stopped
that and the server kept answering. The replacement died with `AddrInUse`
-- in its own log, where nobody was looking -- and `/version` still
reported the previous commit. Everything looked fine: the port answered,
the smoke tests passed, the notes were the old ones.
The checklist now says what actually decides it: read `/proc` for the
process running the release binary (there is no `ps` on this box), and
believe `/version`'s `git` over a successful-looking start -- a `git` older
than the build just made is this mistake, not a build problem. Start the
replacement only once `:3000` has stopped answering, and write the
resolved pid into the pid file.
|
|
moves
`/version` reports the revision the *server* is running, and this
deployment is updated in place, so an assistant that cached
`ai-instruction.md` can end up calling an API that has grown options its
copy never mentions -- by default, silently.
The instructions now say what to do about it, in the two places a reader
sees a revision: pull the repository and re-read the page when `git` is
not the revision the session started with, and the same in the `/version`
section where that field is explained. It matters for the last few
commits in particular: covariates in `/compute/rqtl` and `/compute/gemma`
were added, and a stale copy would never know to ask for them.
`docs/www/operator-checklist.md` gets the operator's half: nothing here
enforces it, so a release that adds endpoints or options should say so
where tokens are handed out, and `source_digest` -- not the commit alone
-- is what identifies a build in a bug report.
|
|
`POST /compute/gemma/ravanan` is now gated like the other two: one
workflow at a time, one per caller, `409 compute_busy` naming the run and
linking `/compute/gemma/status/{id}` (ravanan has no status route of its
own). One slot because ravanan already fans its per-chromosome steps out
-- over Slurm, when that backend is selected -- so a second request would
only queue behind the same scratch directory. Its gate is separate from
`/compute/gemma`'s: different runners, different cache directories, so one
of each at once is the overlap `/compute/rqtl` is already allowed.
Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the
slot for the full 5 s, the same caller's second request was refused at
once, and another caller's waited for the slot and then ran.
While testing that, `GemmaError::Io` turned out to print "failed to run
gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced
"failed to run gemma-wrapper: No such file or directory", which sent me
looking at the wrong binary. Spawn failures are now their own variant
naming the command, and plain file I/O says "I/O error: …".
* src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()`
exposes the slot count, and a test pins all three endpoints' counts
(gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT).
* src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with
`status = "/compute/gemma/status"`.
* src/gemma.rs (Spawn): names the command; the four run_streaming call
sites use it; unit test for both messages.
* README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the
slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
|
|
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default
8, read when the gate is built) with the same one-job-per-caller rule as
GEMMA: a second submission from the same token while its scan is running
or queued is refused `409 compute_busy`; a submission that finds every
slot taken waits for one to come free. Eight because an R/qtl scan is a
single-threaded Rscript on a 96-core box; not more, because a scan on a
3 000-individual cross is minutes of CPU and the same machine serves the
database. Unlike GEMMA's single slot, concurrent scans share no cache
(R/qtl results are uncached), so the only cost is the CPU they take.
Two flaws only appeared once there was more than one slot:
- the "what is running" record was a single `Option<Active>`, so a
refusal quoted whichever job had most recently taken a slot -- robw's
409 named JoM's run id. It is now a map keyed by subject, and
`a_refusal_quotes_the_callers_own_run_id` pins it;
- `Busy::describe` took the endpoint identifier, so the message read
"your rqtl job"; the decorator takes a separate `label` ("R/qtl") from
the status path.
The test suite hit the rule itself: six compute tests in tests/rqtl.rs
share one token, and cargo runs them on several threads, so three were
refused instead of run. They now serialise on a module mutex, with a
comment saying why -- the same lesson clients need, that parallelism
means more tokens, not more requests from one.
* src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`;
`Gate::new(permits)` was already parameterised. `running` becomes a map
keyed by subject, since one record cannot name N jobs. Module doc talks
about slots rather than a single slot. Tests: eight subjects run and the
ninth waits, a caller still may not take two even with slots free, and a
refusal quotes the caller's own run id.
* src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use.
* gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are
independent.
* src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which
it needs.
* tests/rqtl.rs: compute tests serialised.
* README.md (/compute/rqtl), docs/www/ai-instruction.md (4a),
docs/www/operator-checklist.md: the numbers per endpoint, and that more
parallelism means more tokens.
|
|
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
|
|
The plot endpoints ran the mapping themselves, which meant the picture
could come from a different run than the one the caller was looking at,
and cost a scan to draw. They now take the mapping's own response -- the
JSON /compute/{rqtl,gemma} returned, posted back -- and hand it straight
to the plotter, which is what the scripts were written for
(`python3 plot-rqtl.py result.json -o out.svg --trait …`).
Validation replaces the run: `/plot/rqtl` needs `data.markers` with
positional 4+ column rows, `/plot/gemma` needs `data` rows *and*
`meta.columns` (its plotter reads the column names to find `p_lrt`), and
anything else is a 422 saying "post that response back". `/plot/rqtl`
also accepts just the `data` member; GEMMA cannot, and says why.
The mapping's `actions` suggestion now describes posting its own response
and carries no body -- duplicating a 21 000-row response into a
suggestion would double the envelope for nothing, and the caller already
holds it.
* src/routes/plot.rs: posted result, shape validation, `?trait`/`?title`
and `?bonferroni=false` passthrough to the scripts; no compute imports.
* src/routes/{rqtl,gemma}.rs: the plot suggestion loses its body.
* tests/plot.rs: canned results instead of real mappings -- the tests now
need only the plotter, no Rscript or gemma; covers both engines, the
bare-`data` form, and the 422 for a body that is a request.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md:
"post the response back", and why GEMMA needs `meta.columns`.
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|
|
PublishXRef records the marker each published trait's QTL peaks at
(Locus) with its LRS, which is a PheWAS waiting to happen: fix a
position, find the markers within a window of it, and ask which traits
peak at those markers.
GET /groups/{group}/phewas?locus=rs30524230&window=10000
GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000
The window is in base pairs, centred, default 10 kb either side -- the
"10K up and downstream" case. Positions come from the group's genotype
marker table (the endpoint added last round), so `genotype_version`
selects whose markers define the window and the answer reports them
alongside the traits: an empty `traits` list is then readable as
"nothing peaks that close" rather than a failure, and a ten-times wider
window is offered as `next`.
Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD
QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 --
ANPEP_band, then microbiome/haematology/infectious-disease traits. That
cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say.
* src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak
markers, joining Phenotype/Publication through the existing
trait_info_sql, strongest first; marker names are re-validated before
they reach the IN list.
* src/routes/phewas.rs: the endpoint, token-gated; window/centre
validation (422 for none or negative), 404 for an unknown locus, notes
for the empty cases, and a unit test for the version/group helper.
* tests/phewas.rs: markers in the window, traits sorted by LRS with
lod = lrs/4.604, every hit peaking at a listed marker, zero-width window,
the two failure modes, and that the suggested wider window resolves.
* README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS),
docs/www/operator-checklist.md: the endpoint and its reading.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Asked whether the BXD mappings ran GN2/GN3's default (no parents, no
F1s): they did not. BXD.8's panel is 198 RI strains + 37 F1s and no
parent columns at all, and the runs passed all 235 with `null` for the
F1s -- phenotypes right, but GEMMA's kinship and R/qtl's cross spanning
235 individuals instead of 198.
Measured on trait 10001, both engines, both panels: identical answers
(GEMMA p 5.9e-07 at rsm10000005700 chr 8, R/qtl LOD 2.9320 at
rs48756159 chr 8), so nothing published from those runs is wrong. The
lesson is that null-filling excludes an individual from the fit but not
from the model; excluding it means subsetting the genotype file.
* docs/genotypes.md: "Parents and F1s are excluded by default" -- the
GN2 (`incparentsf1`) and GN3 (`get_sample_list(inc_par=False,
inc_f1=False)`) mechanisms, what each panel actually holds, the
both-ways measurement, and the rule for the alignment work.
* docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the
same, in a sentence and a paragraph respectively.
|
|
docs/genotypes.md explained the convention; the endpoint docs still left
a caller guessing which file to pass and how big the panel is. So:
* README.md (/compute/gemma): GEMMA wants the BIMBAM pair
`genotype/bimbam/<file>_geno.txt` + `<file>_snps.txt`, not the `.geno`
file; `trait_values` is one entry per individual in that file's column
order; BXD's current panel stated (235 x 21056), with the 17.1 s
measurement against the 19 s the 7 320-marker fixtures take.
* README.md (/compute/rqtl): the file is `genotype/<file>.geno`
(BXD.8: 235 x 21057; HET3-ITP.12: 6447 x 894), `MappingMethodId`
decides the engine, and the two measured runs with the UM-HET3 peak.
* README.md (/compute/gemma/ravanan): same input convention as gemma.
* docs/genotypes.md: how each measurement was made, step by step, so
they can be repeated; and that step 3 (expanding values onto the panel)
is the API's job in the next iteration.
* docs/www/ai-instruction.md (4a): a table of the BXD files per engine
with panel sizes, the UM-HET3 file, "ask the operator for any other
panel" with where GN2 records which one is current, and the measured
times. Values must be one per individual, in file order -- position, not
name -- and the API does not re-index yet, so a mapping is prepared for
the caller rather than guessed.
* docs/www/operator-checklist.md: MappingMethodId per group, and the
three measurements.
|
|
The compute endpoints take server-side paths, so callers need to know
the deployment's genotype-file layout and which version is current --
neither of which is discoverable from this repository, and both of which
are spread across GN2 and GN3 (GN2's per-group genotype/<Group>.json
whose `genofile` list is newest-first and is the actual authority -- the
GenoFile table still lists only the classic files; GEMMA fed from
genotype/bimbam/<file>_geno.txt, R/qtl from genotype/<file>.geno; GN3
reading the sample order out of the .geno header; InbredSet.
MappingMethodId deciding which engine a group offers).
Measured on this deployment, with the newest genotypes and values
re-indexed onto each panel's sample order:
BXD BXD.8.geno (WGS Mar2022) GEMMA 17.1s 21055 rows
BXD BXD.8.geno R/qtl 7.5s 21056 markers
HET3-ITP.12 (Apr2023, 6447 individuals) R/qtl 83.1s 893 markers,
lifespan peak LOD 7.95 at 1_24042124 (chr1, 24.04 Mb)
That last run is the shape of what Rob Williams wants: UM-HET3 is a
4-way cross (MappingMethodId=3, R/qtl), and its genotype files are
versioned per cohort with no JSON recording which is current, and no
per-cohort trait split -- 6438 of lifespan's 23879 animal values fall in
the .12 panel, the rest are silently dropped. Recorded as the known
messiness rather than hidden.
* docs/genotypes.md: new. Layout, how GN2/GN3 choose, the alignment
rule, per-group engines, the measurements, what is still messy.
* README.md: pointer.
* docs/www/operator-checklist.md: replace "only the test fixtures exist"
with the real panels, and name the missing alignment step.
|
|
Asked to map a 9-strain trait, the wrapper scanned it against the
6-individual fixture cross and returned six markers with LOD scores and
`state: complete` -- a plausible answer belonging to nobody. R recycled
and truncated the values silently, and nothing downstream could tell.
GeneNetwork3's copy of this script has the same hole.
The cross-file format carries no per-sample names, so the count is the
only check available; order remains the caller's responsibility (and the
reason values and genotype file must come from the same panel). With the
check, the same request is a 502 rqtl_failed quoting both counts, and the
correctly-shaped request still returns the documented LODs.
* scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of
the cross, and stop.
* README.md (/compute/rqtl): the second fix relative to gn3, and what
stays the caller's job.
* docs/www/ai-instruction.md (4a): values and genotypes must describe the
same individuals, one for one, in the same order.
* docs/www/operator-checklist.md (Compute): what genotype files actually
exist on this box (test fixtures, 198 BXD individuals) and why a database
trait, with its own strain subset, cannot be mapped against them.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
R/qtl and GEMMA are part of the service now, so the blanket 403 goes:
the handlers already require an AuthToken, which is the whole gate
("token_only", not resource-checked). What that buys and costs is now
written down rather than implied -- any token holder can name
server-side paths for R and gemma to read, and the public-data-only
restriction on traits/datasets does not apply here at all.
The box also has to be able to run them: PATH must carry Rscript,
gemma-wrapper and gemma, *and* R_LIBS_SITE must point at the profile's
site-library or scripts/rqtl_wrapper.R dies on `there is no package
called 'optparse'` -- which reads like a token or proxy fault and is
neither.
Verified through nginx with a token: POST /gn4/compute/rqtl 200 in 0.4s
with the fixture's marker rows, POST /gn4/compute/gemma 200 in 2.1s with
7 320 marker rows (identical to the documented run's first row),
GET /gn4/compute/gemma/status/<id> ending in "GN4: computation
complete", /compute/correlation 200; all four 401 without a token.
* etc/nginx-vesuvio.pu.ac.ke.conf: drop the /compute 403 locations, keep
them commented as the way back; timeouts/body-size comments now apply.
* docs/www/ai-instruction.md: new §4a, "Running QTL mapping (compute)" --
shapes, server-side paths, ?id= + status polling, what to do with a 504.
* docs/www/operator-checklist.md: compute moves from "must be off" to
"proxied, know what it grants", plus what the box needs (PATH,
R_LIBS_SITE, ravanan) and the measured results.
* docs/nginx.md: same, in the deployment and checking sections.
|
|
The page said a human issues tokens and to "stop and ask", without
saying whose door to knock on -- and the operator commands belong in the
operator file anyway. State it as the GeneNetwork team, with wording the
assistant can relay to its user: what a token is for, how long the
default lasts, and to pass it out of band rather than through a public
issue or a shared chat. Same wording for renewal, since a 401 is the one
failure the assistant cannot fix itself.
* docs/www/ai-instruction.md (1. Get a token): "If you do not have one"
and "If it stops working", the latter quoting request_id and retrying
exactly once; the summary line and the 401 row of the error table say
GeneNetwork team rather than "the operator".
* docs/www/operator-checklist.md (Token issuance): the request path, so
the team knows what the ask they receive looks like.
* .gitignore: *.swp, after one got swept into this commit.
|
|
ai-instruction.md was written before anything was deployed: it took a
placeholder base URL, did not say where the API is mounted, and buried
the operator's deployment notes in an appendix labelled "not for the AI
reader" -- awkward to hand to an assistant, and confusing when it is.
So: the AI-facing page now names the deployment
(https://vesuvio.pu.ac.ke/gn4) and is self-contained, and the operator
material moves to operator-checklist.md. Everything in the page was
exercised against the running deployment.
Content changes, beyond the move:
* `next` hrefs arrive already prefixed (`/gn4/groups/BXD/strains`), so
they resolve against the origin -- joining them to the current URL's
directory double-prefixes them. Said plainly, since it is the one
thing the prefix breaks.
* says what cannot work and should not be retried: get-test-token is a
404, /compute/* is a 403 from the front end, and neither is a
permissions problem to work around.
* `/publications/{pubmed_id}/traits` wants the PubMed id, not the
internal `publication_id` in the same trait record, which 404s.
* states the current rate-limit position (none configured) instead of
implying a 429 policy that does not exist yet.
* docs/www/ai-instruction.md: rewritten, paste-ready.
* docs/www/operator-checklist.md: new file, the former appendix, updated
with what is now enforced at the proxy and a record of the verified
request matrix.
* README.md, docs/nginx.md, etc/nginx-vesuvio.pu.ac.ke.conf: point at
the operator checklist by its new name.
|