| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
`"pca": true` on the matrix request now folds the correlation matrix into
principal components: eigenvalues descending, explained variance ratios,
loadings per kept trait, and the eigenvalues' sum reported so a caller can
check the decomposition is sane instead of trusting it.
The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no
new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and
this is for a collection, not a dataset.
Two decisions from ai-memory #103, both visible in the reply:
* **Traits with any null cell are dropped and named**, with the reason. A
component from a matrix with holes is not defined; GN2 silently mutates its
shared-sample list instead, and nobody can reproduce that component.
* **The sign is fixed**: within a component the largest absolute loading is
positive. Eigenvectors are defined up to sign, so without a rule two correct
implementations disagree and no parity test could be exact.
Verified against numpy.linalg.eigh on the same r the reply carries
(scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007):
eigenvalues: worst relative difference 3.2e-14
loadings: worst absolute difference 9.99e-16
trace: 5.000000000000 for 5 traits
Unit tests cover the identity (unit eigenvalues), the closed form for 2x2
(1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The
monitor gained `compute-correlation-pca`, whose judge checks that the
eigenvalues sum to the trait count and the ratios to 1.
|
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
work yet
Writing down what the monitor found and what fixed it, because it is config
that lives outside the repository and is therefore easy to lose at the next
deploy: with only the cache dirs set, every engine that shells out answers
502 while the whole read surface and search look healthy.
Measured on production after the fix (bin/monitor-gn4 --tier compute):
8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453),
fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed,
1 busy.
The remaining failure is BulkLMM and it is not a config line: the Julia
project is registry-based and no depot on this box has its packages, so
it stops at "Run Pkg.instantiate()". Populating a depot downloads from the
Julia registry, which should be a decision rather than a side effect.
|
|
src/correlation.rs now holds the part the engine does not offer in memory,
and nothing of the arithmetic: `Pearson`/`Spearman` and their p-values are
correlation_rust's (and GSL's). What is ours is the alignment into the
engine's parser, the minimum-overlap gate, the ordering and top-N:
- alignment is by individual, never by position: an individual the target
trait lacks becomes "NA", which the engine's parser drops together with
its reference value, so a missing observation costs one pair, not the
row;
- minimum overlap 6, so a two- or four-point +-1 cannot appear;
- ordering reuses correlation_rust's own `custom_float_sorter` -- |r|
descending, NaNs last -- so GN2, GN3 and GN4 sort identically;
- top-N after the sort, clamped to GN2's 20000 ceiling;
- bicor is refused by name with the methods that exist, rather than
answered with something else.
Tests cover both methods, the six-pair minimum, name alignment across a
reordered and sparse candidate, the ordering with top-N, and the method
parsing. The route is still the stub; wiring it to the bulk fetch is next.
|
|
The access log said who (`user`) but not with what: with short tokens a
request can be made by a shared credential, so "which person" and "which
credential" are now separate fields -- `scheme` ("jwt" | "opaque") and,
for an opaque token, `token` (its registry handle, e.g.
`agent-ravanan-3`). Never the token or its hash, and the path still
carries no query string.
The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is
self-describing rather than relying on journald's prefix. Hand-rolled
civil-from-days rather than a new crate: one format, and no Cargo.lock
drift.
scripts/gn4-log.py reads such a log and shows it as a table with the
handle joined against tokens.sqlite: the subject as the registry has it
now (flagging a handle whose subject has been repointed), and the token's
status today -- [revoked], [expired], [unknown] -- which is how "an agent
was working at 14:02" is told apart from "an agent was working at 14:02
with a credential revoked since". It reads the registry read-only, passes
non-JSON lines through in table mode (the process log interleaves
backtraces), and offers --json and --summary for jq and for counting.
CallerSlot now carries the whole Caller (scheme, id, subject) rather than
just the subject string.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
The proxy logs the caller through x-gn4-user, but the process log -- the one an
operator tails while a job runs -- named nobody, so "who started this GEMMA
run" meant going to the nginx log or guessing. genenetwork4 now writes one
compact JSON line per request on stderr:
{"event":"request","method":"GET","path":"/groups/BXD","status":200,
"took_ms":10,"request_id":"…","user":"zachs"}
Decisions in it: `user` is absent, not "-", when no token was verified (a dash
is a value a reader can mistake for a user name); the path is logged without
its query string, because a query can carry a credential and has no business
in a log; the token value itself is never logged, only its subject; and the
line carries the `request_id`, so it sits beside the existing `event: error`
lines for 5xx and one grep joins a failure to the request that caused it.
`GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of
truth.
Verified on a probe instance: with the test token the line says
user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated
401 has no user field at all.
docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's
sub" -- true until now, which is exactly the kind of sentence that rots; it is
replaced by the line above, what it contains and what it deliberately omits.
README's x-gn4-user paragraph gains the same, next to the identity it already
described.
|
|
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only
implied. It reports the F1 strains, the maternal and paternal strain (from the
deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups),
the @type of the group's genotype file, the individuals each role names where
the metadata names them (BXD-Longevity does, most do not), and -- the part a
caller needs -- which members of the current panel the default rule drops and
which the `F1` name heuristic catches.
Two things that came out of building it and comparison with GN2:
* the JSON's paternal/maternal labels were INVERTED against GN2's reading.
GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal
strain is C57BL/6J; our file said paternal. No mapping was affected
(non_strains takes both regardless of role) but the metadata was wrong, and
anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat
note -- would have been backwards. The file is rewritten from ParInfo itself
(ast-parsed, not retyped), 23 groups' roles filled in and the 24th
(BXD-Longevity, which GN2 does not have) left as it was.
* an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's
own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports
excluded_from_panel and excluded_by_name separately and says which is doing
the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1
names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming
exclusion entries that match nothing.
genotypes::non_strains is rebuilt on the same typed parser (DesignRole /
ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit
test for the roles, empty-name handling and the unlisted-group source, and an
integration test asserting BXD's maternal/paternal orientation and the note.
|
|
Two additions that make the endpoint usable rather than merely correct.
POST /compute/bnw/features -- from a network to a mappable trait. Every node
with at least one parent at the threshold is a module (the node plus its
parents), scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module -- module, members, size,
variance_explained -- where variance_explained says whether the score is a fair
summary at all. With ?module=NAME it is that module's per-sample scores in the
{name, value} shape every mapping endpoint already takes, so the chain runs
network -> module -> /compute/rqtl or /compute/gemma with no translation layer.
That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes
name the construction (a citable module score must say what it is) and repeat
the caveat: a module is a correlation structure, not a QTL.
GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked
examples, for an agent to explore instead of inventing data. The five datasets
its tutorial and examples pages present are listed with their sizes, and each is
served as a matrix in the values shape /compute/bnw takes, with a ready-to-post
action carrying the body. ?columns= restricts the matrix (the spleen example
has 14 variables and a caller may want fewer). genotype_column warns that a
column is a genotype code: this driver is continuous-only, so it enters the
network as a number, the way the web application's GeneNetwork path treats it.
Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/
example_datasets); a deployment without it answers 404 no_examples.
Measured on the dev instance: features over five BXD traits gives two modules
(BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with
BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl
and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs
covers the module list, one module's scores in mapping shape, the unknown-module
422, the example list, one example's matrix and its ready-to-post action, and a
404 for a name that does not exist.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
POST /compute/bnw learns a network over a set of variables and returns one row
per candidate edge: from, to, posterior, in_structure. Inputs are published
traits of a group (group + traits) or an explicit matrix (values: one entry per
variable with per-sample values), which covers the case the database cannot --
a basket's columns, a handful of probe sets.
BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x
samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and
parses the reply: nothing about the three C programs, the private work
directory or the type line in the input file lives here, so BNW's pipeline can
change without a GN4 release. The driver's failures map onto the API's own:
bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed
computation 502 bnw_failed in the driver's words.
Decisions worth recording:
* parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold
0.5, read from its create_tiers_gom_part1.php rather than chosen here --
and the reply names the values used, because a network at 0.5 and one at
0.9 are different answers to the same question;
* at most 40 variables and at least 3 samples complete across every variable.
The first is the structure search's exponential in-degree; the second is
BNW modelling a complete multivariate normal, so a sample missing any value
is dropped -- and the note says how many survived rather than silently
shrinking the panel;
* its own gate, always claimed (threshold 0): there is no such thing as a
small BNW run, the cost is the search, not the number of individuals;
* the notes say plainly that a posterior is not a p-value and that a network
is not a QTL, because that is the easiest thing to over-read.
tests/bnw.rs compares the endpoint against the answer the BNW *web
application* gives for its own six-trait example (the same fixture as the
driver's parity check, gn-bnw/test/golden/): the five edges must match, and the
notes must name the parameters and carry the caveat. It skips when the driver
or its binaries are absent.
Not deployed to production.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.
An upload updates a population, it does not invent one, and the checks
say so:
* the base version must exist (resolved through the same
genotypes::resolve_named the rest of the API uses);
* at most 50 individuals the base does not have, and at least one that
it does -- no overlap is a different population, not an update;
* at most 200 000 markers, and the refusal says the sysadmin has to
organize a file that size, because this is a request path, not an
ingest;
* the values must be the same kind as the base's: letters, whole
numbers or probabilities. A letters panel holding 0/1 would be read
as the wrong thing silently, so it is refused naming both kinds.
GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.
Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
|
|
The other half of changing data: bringing new columns in. A matrix is a
delimited table -- a header row, then one row per individual, the first
column naming them -- stored in the uploader's own SQLite file and checked
against an existing genotype version's panel before it is accepted.
* POST /matrices {name, genotype_version, contents, kinds?}: parse the
table (tab, comma or semicolon, guessed from the header), then check
it against that version's individuals. The count must match and every
name must be in the panel; 92 of 93, or a name the panel does not
have, is a 422 that says which. A mapping that silently drops
individuals is worse than an error.
* kinds marks columns that are covariates; everything else is a trait.
* Each column becomes a basket named <matrix>.<column> -- a dot, not a
slash, because a basket name has to survive a URL path segment -- so
a column maps on any engine with {"basket": "my-traits.bw"} and
inherits the revision history and digest without a line of new
mapping code.
* GET /matrices lists them, GET /matrices/{name} their columns and
basket names, GET /matrices/{name}/values the table back (parsed
values, null where missing), DELETE removes the matrix and its
columns.
Two bugs this found in the basket store, both real:
* SQLite has foreign keys OFF by default, so the ON DELETE CASCADE
never fired: deleting a basket left its rows behind, rowids were
reused, and the next create hit "UNIQUE constraint failed:
basket_version.basket_id, revision". The pragma is now set, and
deletion removes the dependants explicitly, because files written
before it have orphans.
* The first attempt named a column's basket "<matrix>/<column>", which
the basket-name rule (rightly) refuses. The dot keeps both rules.
scripts/example-matrix.py writes a table from a genotype file's own header
with deterministic values, so the fixture passes the panel check by
construction -- tests/matrices.rs builds its own for the same reason and
walks the loop: upload, read back, map a column with R/qtl2, delete, plus
the three refusals (too few individuals, a name not in the panel, an
unknown version).
|
|
Three things the first cut was missing, and both were asked for.
* An automatic version history. Revision 1 is the column as first
stored and each PATCH adds one -- with its own digest and Unix
timestamp -- and earlier revisions are kept. GET
/baskets/{name}/revisions is the history, ?revision=N reads an old
column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is
the point of a digest: a result quoted yesterday can still be
reproduced after the column has moved on.
* Files named for the user, not a hash of the subject:
BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not,
so a new token finds the same baskets -- and an operator can see
whose baskets a file holds. A subject that is not filename-safe is
sanitised and disambiguated with a short digest, and a pre-naming
hash file is renamed on first use rather than orphaned.
* /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket`
too, so all four engines share one meaning: a basket resolves to the
same (name, value) pairs a client would have posted, and every reply
records the revision and digest it ran.
The schema upgrade is a real one, not a reset: an older file's single
column becomes revision 1 of the history, cells copied across with the
digest and timestamp they already had. Tested by building the old schema
by hand and opening it (an_older_file_is_upgraded_with_its_column_intact),
which is also how I found that the migration was recreating a table the
base schema had already made.
Verified on all four engines, head and pinned: rqtl 3811 rows, gemma
3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...'
revision N (digest ...)".
|
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|
|
The question that follows "why do GEMMA and R/qtl2 peak at different
markers": what does the trait look like within each genotype class at one
marker. The frequencies endpoints report the distribution of calls, not
the values behind them, and the answer must not be "serve the matrix" --
this API never does.
GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id>
-> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}}
The calls are joined to the trait's per-strain values inside the server
(the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values)
and reduced to those five numbers per class. Nothing in the payload names
an animal. A class with fewer than ten animals is called out in notes,
because a peak resting on two is a different claim from one resting on a
hundred.
It settles the DOL trait 10003 question. At GEMMA's top marker
JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean
2.53, B n=171 mean 3.95 -- the A class is two animals and they are the
trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's
convert2riself), so it keeps 173 animals there but only 2 of one class:
nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps
the 20 hets at dosage 0.5, and that is where its sharper p-value comes
from. GEMMA's next two hits are the same pattern in opposite directions
(one A among 171, then two B among 171), so the chr16 block is a haplotype
carried by a handful of extreme animals, not a population effect.
Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the
BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for
JAX00418132 is half the B-allele frequency 0.935, not the A-allele
frequency 0.065. The bimbam file and DOL.geno agree at that marker.
tests/frequencies.rs gains a route test: the class fields, A < B, the
small-class note, the "not served" note, no individual names anywhere in
the payload, and a 404 for a trait the group does not have.
|
|
A GeneNetwork trait page shows the gene's aliases next to the symbol,
and GN4 had no way to ask for them: gn-guile has a Wikidata lookup
(gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that
lookup with the caching Wikidata deserves -- it is a shared public
service and an alias list is not news.
* GET /genes/{symbol}/aliases: two SPARQL queries against
WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human,
mouse or rat, labelled with the symbol; then its skos:altLabel s in
English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB
file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion
rather than a directory of two).
* TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value
is refetched; ?refresh=true forces that.
* The label is matched case-insensitively, which the gn-guile query
did not: human genes are labelled SHH and mouse ones Shh, so an
exact match finds only one species. ?species=human now returns the
ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a
mouse trait page shows. With no species named the pick is
deterministic (mouse, rat, human), so a repeated request cannot
change entity.
* A failed refresh keeps the last value: the answer says stale: true,
the envelope is partial, and a note explains it. Nothing cached at
all is a 502 that says so. An empty alias list and an unreachable
upstream are different facts, and only one of them is about the
gene.
Seven unit tests (key scoping, TTL including a clock that went backwards,
the query shape and quote-escaping, SPARQL flattening, an LMDB
round-trip that asserts the file is gene-aliases.lmdb, and
stale-on-failure with and without a cache) and two route tests (the
second call is served from the cache -- same fetched_at -- and the
species switch), which skip with a 502 assertion when there is no
network.
|
|
A probeset id is not a Xapian field, so /search cannot find 1427571_at
exactly and the only route to it was guessing a dataset and calling
/datasets/{ds}/probesets/{name} until one answered. A gene symbol could
be searched, but the reply was capped at 100 hits, interleaved probesets
and platforms, and carried no total -- so "did I see all of Shh?" had no
answer. Both questions are one indexed query each: ProbeSet.Name_IDX and
symbol_IDX.
* GET /probesets/{name}: one row per dataset that carries the probeset,
with the panel (group, species, platform) and the scan's statistics
(mean, LRS, LOD, locus, additive). Suffixed chip variants match too
-- 1427571_at also finds 1427571_at_A -- and each row's `name` says
which matched, so the answer never hides a substitution. LIKE's
wildcards are escaped, or every Affymetrix `_at` would match any
character. 155 rows over 155 datasets in 32 ms.
* GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a
real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and
paging -- "BXD datasets where Shh is a strong eQTL" is one call.
1225 rows over 17 datasets and 13 probesets in 38 ms.
* Both serve metadata, not values. A token is enough for the index;
trait values stay per-dataset and authorized per dataset, and `next`
goes to /datasets/{ds}/probesets/{name}/values, which checks. There
is no cross-dataset values endpoint, deliberately: that is bulk work
and /groups/{g}/traits is the bulk call.
The one-statement join took 5.2 s for Shh -- MySQL rescans
InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102
panels are fetched separately and joined in memory: 70 ms each. nsamples
is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset
record already reports strains.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Two questions the genotype surface could not answer: what a panel is
actually called as, and which markers it holds. The listing gives counts
and the compute endpoints give scans; a panel whose calls are not the
codes its header declares, or a marker that is 99% missing, stayed
invisible until a scan came back flat -- and then the search for the
cause started at the wrong end.
* GET /groups/{group}/genotypes/{version}/frequencies counts the calls
in the version's .geno and returns them as call -> frequency
({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over
the calls that are present, so they sum to 1; missing codes (U, N,
NA, an empty cell) are counted separately and reported in notes,
because the missing rate is the diagnosis.
* A probability-valued file is binned 0.05 wide (twenty bins over
[0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same
statement about a marker; ?bin= sets another width. A file with a few
distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported
exactly -- binning three values hides what is being looked at.
* .../markers/{marker}/frequencies is one locus, and ?marker= on the
panel route is the same call. Counting goes through an interner, so
cost is one hash lookup per cell and one allocation per distinct
call; a .geno over 256 MB is refused (413 genotype_too_large) rather
than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x
6147 individuals) demanded.
* The version's /markers now lists marker NAMES only, one column, and
.../markers/{marker}/location returns {"name","chr","cM","Mb"} as
named fields: a page of names stays small, and a position is a
lookup rather than a table the client must hold. The same size guard
applies, so an oversized panel's listing is a 413 in 3 ms.
* Files no group lists -- a panel in preparation, a fixture, the DO's
do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the
same four calls. `unlisted` is a namespace, not a group: only its
genotype subtree answers. One prefix for the whole genotype surface
beats a second spelling for the same tally, so the earlier
/genotypes/{name} spelling is gone.
* genotypes::resolve_named (moved out of the frequencies module, since
it is genotype metadata) resolves a version, a .geno name, or a path
relative to the genotype files directory; MAX_SCAN_BYTES is shared by
every whole-file read.
Tests limited to the feature: genotypes 3 (names-only columns, paging,
location fields, marker_not_found, unlisted parity, 413), frequencies 5,
frequencies unit 7.
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
Answering "is robw using this successfully, and the others?" from the
access logs took an hour of inference -- user agents, search terms, which
traits were mapped -- because nginx records the address and the subject is
only in the token. Two callers can share an address (an agent and a person
behind one NAT), one caller can move between addresses (an LLM's egress is
a /24), and the answer for a given person was "probably 80.115.1.132".
The API now sets `x-gn4-user: <token sub>` on every **authenticated**
response, and the `gn4` log format records it as `user=`. The subject is
only known inside the handler (the `AuthToken` extractor verifies the
token there), so a middleware opens a slot in the request extensions, the
extractor fills it, and the header is set on the way out; a request with no
valid token gets no header, which is the honest answer and reads as
`user=-`. The caller only ever sees an identity it already presented.
* src/auth.rs: `USER_HEADER`, `CallerSlot`, and the extractor recording
its subject.
* src/lib.rs: the `tag_caller` middleware, outermost so the slot exists
before any extractor runs.
* tests/auth.rs: `an_authenticated_response_names_its_caller` -- the header
equals the token's `sub`, and `/version` (no identity) carries none.
* etc/nginx-vesuvio.pu.ac.ke.conf (and the deployed copy of it): `user=`
in the log format.
* README.md, docs/nginx.md (the format, an `awk` one-liner for per-user
counts, and the caveat that old and new lines differ in field count),
docs/www/operator-checklist.md (`user=-` means no token, not a token
that failed late).
281 tests pass; the same four as before do not.
|
|
A user reported that the token was fine and the service was up, but
"everything backed by the relational database is failing" -- while
`/search` (Xapian) and `/version` answered. By the time I looked, every
data endpoint answered 200 and `/version` said `database: reachable`, and
there was **no record anywhere** of what they had hit: the service logged
nothing at all, so a caller's report could not be matched to a cause, and
`/version`'s `database` is a *now* probe that says nothing about the
minute before.
Two things now make that diagnosable:
* every 5xx is written to stderr as one JSON line -- status, code,
operation_id, **request_id**, took_ms and a truncated detail -- so
"500 at 12:31, request id ..." can be looked up. 4xx stay unlogged: they
are the caller's business, and a 401/422 stream would drown this;
* the last database failure is remembered (`db::record_failure`, called
from `ApiError::from_db`, the one place every database error passes
through) and `/version` reports it in `notes` for ten minutes, with its
kind (`unavailable`, `connection_lost`, `query`) and the driver's
message. That is the gap that misled the user: a healthy probe beside
failing endpoints looks like an auth problem from outside.
`is_connection_loss` gained a `_detail` form so the caller that already
has the rendered message does not render it twice.
* src/error.rs: the log line, and recording failures by kind;
* src/db.rs: `FailureKind`, `Failure`, `record_failure`, `recent_failure`,
and two unit tests (a failure is remembered and ages out; a driver
message is classified correctly);
* src/lib.rs: the `/version` note, absent unless something failed recently.
Checked against a deliberately broken `SQL_URI` on a scratch instance: the
data endpoint answers `503 db_unavailable`, the log carries the
request id, and `/version` shows the note while still answering 200.
|
|
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl`
does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample
numbers -- and fits it with `-c` on the association step only. That needed
a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample
names, positional over the genotype panel, numbers only. GEMMA has no
factors, so a categorical covariate is dummy-expanded here into k-1
columns named after their levels (`Status.Live`, the first level being the
reference), where R/qtl's wrapper does that expansion for itself.
The spec shape, the alignment onto the panel and the text coding are now
one module, `src/covariates.rs`, called by both routes; `/compute/rqtl`
behaves as before except that its note also reports the complete cases
(`... 34 with both the trait and the covariate`), which README and the
instructions had been promising all along.
A missing covariate value is written `NA`, which GEMMA treats exactly as
it treats a missing phenotype -- that individual leaves the fit (measured:
one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's
`-9`, which GEMMA fits as the value minus nine.
Found on live data, not in the fixtures: a level that exists in the panel
but that none of the *fitted* individuals carries becomes an all-zero dummy
column, and GEMMA stops with `LUDecomp ... input domain error` rather than
dropping the term the way R would. Columns that do not vary among the
individuals with both a trait and a covariate are now left out, and the
response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a
covariate left with no columns runs without one, with a note saying so.
Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes
its file arguments, so a different covariate set is a different cache entry
while the kinship already computed is still reused. Covariates need a
`group`: individuals are aligned by name, which an explicit
`genotype_file`/`snps_file` request does not carry (`422
covariate_unavailable`), and an attribute the group does not define lists
the ones it has.
* src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now
`CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`,
`Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file`
(moved out of src/rqtl.rs) and `write_gemma_covariates_file`.
* src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate
file, `-c` after `-a` and before `-lmm`, and the notes about dropped
individuals and dropped columns.
* src/routes/gemma.rs: the `covariates` field, resolved after the trait
values so the note can count complete cases; `RqtlCovariate` replaced by
the shared spec.
* tests: six new -- the GEMMA file (numerical with `NA`; two and three
levels; inline codes; a dropped degenerate column) and an end-to-end
`compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with
`Status`, which also pins that `-c` is in the GWA command and not in the
kinship one, and that the covariate changes the fit rather than reusing
the plain run's cache. The gemma tests now take a lock: the gate answers
a second job from the same caller with 409 instead of queueing it.
* README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl
points at the shared shape and explains `x` vs `NA`), docs/www/
ai-instruction.md (both endpoints; what differs in `notes`).
Also in the instructions: `/version` reporting a `git` value you have not
seen before now says explicitly to pull the repository and re-read
`docs/www/ai-instruction.md` before the next call. This deployment is
updated in place and a stale page does not announce itself -- the API
accepts the request without the new option, or answers with fields the old
text never mentions.
272 tests pass; the four that do not (benchmark, no_db, and two search
parity/proxy replays) fail identically at the parent commit.
|
|
`POST /compute/gemma` now takes a single slot: one job runs at a time
across all callers, a second submission from the same token is refused
immediately with `409 compute_busy` (whatever it would have waited for,
its own job is already in the way), and a submission from another token
waits for the running job and then runs. Nothing else is gated -- ravanan
and /compute/rqtl are untouched.
The gate is one `tokio::sync::Semaphore(1)` (fair: waiters take the
permit in arrival order) plus a `Mutex<HashSet<String>>` of subjects
holding the slot or queued for it, in a per-process static. The claim is
moved into the spawned job task, so a client disconnect -- which does not
kill the child process -- cannot hand the slot to somebody else while the
CPU is still busy.
The endpoint carries none of that: `#[compute_gate(gemma, run_id =
params.id.as_deref())]` expands to a claim and a run, and requires the
handler to take `Ctx` and `AuthToken` (every compute endpoint is
token-only, and the token's sub is the key), so an endpoint that is not
token-gated cannot be decorated. The policy -- the 409's status, code,
wording and next link -- stays in plain Rust (`Gate::claim_or_busy`), so
the gate's own tests cover it; the attribute is decoration over two calls.
* src/compute_gate.rs: new. Gate, Claim (released on drop, including a
panic), Busy, claim/claim_or_busy/run, and four tests, two of them on a
four-thread runtime (the earlier ones ran single-threaded and so never
exercised the cross-thread hand-off; one now asserts at most one task
inside the slot across eight tasks).
* gnauth-macros/src/lib.rs: the `compute_gate` attribute, alongside the
`gnauth` markers it shares `syn`/`quote` with; arguments documented
(gate, endpoint, status, run_id) for the endpoints still to be gated.
* src/routes/gemma.rs: the decorator replaces 57 lines of claim and
spawn plumbing in compute_gemma.
* docs/www/ai-instruction.md (4a, 5): one job at a time per caller, and
that a `compute_busy` 409 means poll, not resubmit.
|
|
A caller posting a mapping result back to /plot/rqtl -- or a compute
request carrying a value per individual plus covariates -- was refused
with `413 Failed to buffer the request body: length limit exceeded`. That
is axum's DefaultBodyLimit, 2 MiB, which nothing here had raised: a real
GEMMA run on the BXD WGS panel is ~1.9 MB, so the plot endpoints sat just
under the cap and anything bigger (a genome-sized run, or every animal's
covariate value) fell over it. nginx's 10m was never the binding limit.
* src/lib.rs (BODY_LIMIT_BYTES): 64 MiB, with the reasoning and the note
that a proxy has to allow as much or it refuses first.
* etc/nginx-vesuvio.pu.ac.ke.conf, docs/nginx.md: client_max_body_size
64m, so the two limits agree instead of the proxy cutting in at 10m.
* README.md ("Request body size"): both caps, and that a request does not
need to be large -- `{"case_attribute": "Sex"}` has the server read the
attribute and align it, where posting a value per animal is pure payload.
Verified: the 11.9 MB body that was refused now returns an SVG; 67 MB
still gets axum's 413.
|
|
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
|
|
The trait-values endpoint stays about trait values; a covariate needs the
study's metadata for the same samples, aligned the same way. So:
GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute}
returns one attribute's values for exactly that trait's samples, **in the
trait's own order**, which is what makes it usable without the caller
re-sorting anything:
/groups/BXD-Longevity/traits/10001/case-attributes/Sex
-> 3281 values in AgeAtDeath's order, F x 3101, M x 180
-> order identical to /traits/10001/values (asserted in the tests)
Alongside the values it reports `distinct` (each value with its count,
commonest first) and `num_values`. That is the check that decides whether
an attribute can serve as a covariate at all -- a constant one cannot --
and it saves a second request just to find out.
Attributes are per group, so an unknown one is a 404 whose `next` is the
group's attribute list: BXD has no Sex, BXD-Longevity does.
* src/db.rs (get_trait_case_attribute, get_group_trait_samples): the
group's attribute values aligned onto the trait's sample list, joining
CaseAttribute/CaseAttributeXRefNew on (InbredSetId, CaseAttributeId) as
the group endpoint does; empty text is reported as no value.
* src/routes/groups.rs: the handler, token-gated, with the response
shapes and the two `next` links (the trait's values, the group's
attributes).
* tests/case_attributes.rs: alignment against an independent /values
response, `distinct` accounting, ordering, and the 404s for BXD (no Sex)
and for a non-existent attribute.
* README.md, docs/www/ai-instruction.md (3. Case attributes): the
endpoint, and "ask per trait rather than sorting the group's attributes
yourself".
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|
|
PublishXRef records the marker each published trait's QTL peaks at
(Locus) with its LRS, which is a PheWAS waiting to happen: fix a
position, find the markers within a window of it, and ask which traits
peak at those markers.
GET /groups/{group}/phewas?locus=rs30524230&window=10000
GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000
The window is in base pairs, centred, default 10 kb either side -- the
"10K up and downstream" case. Positions come from the group's genotype
marker table (the endpoint added last round), so `genotype_version`
selects whose markers define the window and the answer reports them
alongside the traits: an empty `traits` list is then readable as
"nothing peaks that close" rather than a failure, and a ten-times wider
window is offered as `next`.
Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD
QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 --
ANPEP_band, then microbiome/haematology/infectious-disease traits. That
cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say.
* src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak
markers, joining Phenotype/Publication through the existing
trait_info_sql, strongest first; marker names are re-validated before
they reach the IN list.
* src/routes/phewas.rs: the endpoint, token-gated; window/centre
validation (422 for none or negative), 404 for an unknown locus, notes
for the empty cases, and a unit test for the version/group helper.
* tests/phewas.rs: markers in the window, traits sorted by LRS with
lod = lrs/4.604, every hit peaking at a listed marker, zero-width window,
the two failure modes, and that the suggested wider window resolves.
* README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS),
docs/www/operator-checklist.md: the endpoint and its reading.
|
|
*tests/envelope.rs* uses /version as its example of a response whose
`notes` is an empty array; adding a note there broke that check for no
gain -- the fields document themselves, and tests/version.rs asserts
their shape.
* src/lib.rs (version): drop the with_note, with a comment saying why.
|
|
/version reported 4831e5a while HEAD had moved on to 18d18b1: the stamp
is written at build time and build.rs only declared .git/HEAD, which
keeps saying "ref: refs/heads/main" across commits, so a rebuild after a
commit reused the old hash. It now declares the ref file and the index
as well, which is what actually makes a rebuild re-stamp.
Three additions make the stamp trustworthy rather than merely newer:
- `-dirty` when the working tree had uncommitted changes at build time,
so a binary built pre-commit can never claim to be that commit;
- `source_digest`, a hash over the compiled source files, which tells
apart two builds of the same commit -- or of the same dirty tree --
where a commit hash cannot;
- `branch`, so a detached HEAD is visible.
* build.rs: watch .git/HEAD, the ref it points at, and .git/index;
stamp dirty, digest and branch as well as the hash and commit date.
* src/lib.rs (VersionInfo): the three new fields, plus a note saying a
stamp is never advanced by committing without rebuilding.
|
|
The listing named versions but not files; a caller could pick one and
still not see what was in it. Three token-gated routes now describe a
version completely, short of the matrix:
GET /groups/{group}/genotypes versions, files, sizes, current
GET /groups/{group}/genotypes/{version} file name, header labels,
individuals, strains, chromosomes
GET /groups/{group}/genotypes/{version}/markers markers (name, chr, cM, Mb), paged
The detail reads the file's own `@`-labels (type, mat, pat, het, unk,
scale) and its panel in column order, and reports both the whole panel
and the strains a mapping uses after the parents/F1s rule -- so a caller
can check a request before sending it and interpret a result after.
Markers come back as positional rows with `meta.columns`, paged with
`?page`/`?limit` and filterable with `?chr`, following /compute/gemma's
shape rather than inventing one.
The genotype calls stay on the server: file_meta() reads the matrix to
skip it, and nothing returns a cell.
* src/genotypes.rs (Marker, FileMeta, file_meta; GenotypeFile::file):
header labels, individuals and marker positions; unit test.
* src/routes/genotypes.rs: the two new handlers, token-gated like the
listing, plus `file` in the listing and counts in the detail.
* tests/genotypes.rs: the listing carries file names but no paths.
* README.md, docs/genotypes.md, docs/www/ai-instruction.md: what the
three routes give, and that the matrix is not among them.
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
/compute/gemma can genuinely take minutes (see the README's ~100K
marker caveat), and until now gave no visibility into a run already
in flight. Same shape GeneNetwork2/3 use for R/qtl
(gn3/computations/streaming.py, gn3/api/streaming.py): the caller
picks a run id, passes it as POST /compute/gemma?id=<id>, and polls
GET /compute/gemma/status/{id}?peak=<offset> on a second connection
while the first is still blocked.
What's actually tailed is gemma-wrapper's stderr, not stdout: under
--json, all of its human-readable progress (banner, chromosome list,
"Executing: ...", per-step info) goes to stderr, and stdout carries
only the one final JSON record, printed once at the very end -- so
there is nothing to stream on stdout. run_streaming() replaces the
old buffered Command::output() call with one that reads stderr line
by line as the process runs, appending each line to a log file if one
was given, across both the kinship and GWA steps into a single
continuous log; stdout is still read in full (unchanged, one JSON
blob). A completion marker ("GN4: computation complete") is appended
once the whole request finishes, the same convention GeneNetwork3
uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this
project's own name.
run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever
joined into a path, on both the POST's ?id= and the status endpoint's
path segment -- GeneNetwork3's own /api/stream/{run_id} does no such
check on its equivalent id, which is not knowingly replicated here.
Verified end to end, not just unit tested: the integration test spawns
the POST on its own task and polls the status endpoint from this one
while the ~19s real computation is still running, asserting it
actually observed non-empty progress lines mid-run (not merely after
the fact), then checks the finished log for both real gemma-wrapper
progress text and the completion marker. cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|