| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
The proxy logs the caller through x-gn4-user, but the process log -- the one an
operator tails while a job runs -- named nobody, so "who started this GEMMA
run" meant going to the nginx log or guessing. genenetwork4 now writes one
compact JSON line per request on stderr:
{"event":"request","method":"GET","path":"/groups/BXD","status":200,
"took_ms":10,"request_id":"…","user":"zachs"}
Decisions in it: `user` is absent, not "-", when no token was verified (a dash
is a value a reader can mistake for a user name); the path is logged without
its query string, because a query can carry a credential and has no business
in a log; the token value itself is never logged, only its subject; and the
line carries the `request_id`, so it sits beside the existing `event: error`
lines for 5xx and one grep joins a failure to the request that caused it.
`GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of
truth.
Verified on a probe instance: with the test token the line says
user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated
401 has no user field at all.
docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's
sub" -- true until now, which is exactly the kind of sentence that rots; it is
replaced by the line above, what it contains and what it deliberately omits.
README's x-gn4-user paragraph gains the same, next to the identity it already
described.
|
|
Two additions that make the endpoint usable rather than merely correct.
POST /compute/bnw/features -- from a network to a mappable trait. Every node
with at least one parent at the threshold is a module (the node plus its
parents), scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module -- module, members, size,
variance_explained -- where variance_explained says whether the score is a fair
summary at all. With ?module=NAME it is that module's per-sample scores in the
{name, value} shape every mapping endpoint already takes, so the chain runs
network -> module -> /compute/rqtl or /compute/gemma with no translation layer.
That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes
name the construction (a citable module score must say what it is) and repeat
the caveat: a module is a correlation structure, not a QTL.
GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked
examples, for an agent to explore instead of inventing data. The five datasets
its tutorial and examples pages present are listed with their sizes, and each is
served as a matrix in the values shape /compute/bnw takes, with a ready-to-post
action carrying the body. ?columns= restricts the matrix (the spleen example
has 14 variables and a caller may want fewer). genotype_column warns that a
column is a genotype code: this driver is continuous-only, so it enters the
network as a number, the way the web application's GeneNetwork path treats it.
Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/
example_datasets); a deployment without it answers 404 no_examples.
Measured on the dev instance: features over five BXD traits gives two modules
(BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with
BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl
and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs
covers the module list, one module's scores in mapping shape, the unknown-module
422, the example list, one example's matrix and its ready-to-post action, and a
404 for a name that does not exist.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
POST /compute/bnw learns a network over a set of variables and returns one row
per candidate edge: from, to, posterior, in_structure. Inputs are published
traits of a group (group + traits) or an explicit matrix (values: one entry per
variable with per-sample values), which covers the case the database cannot --
a basket's columns, a handful of probe sets.
BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x
samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and
parses the reply: nothing about the three C programs, the private work
directory or the type line in the input file lives here, so BNW's pipeline can
change without a GN4 release. The driver's failures map onto the API's own:
bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed
computation 502 bnw_failed in the driver's words.
Decisions worth recording:
* parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold
0.5, read from its create_tiers_gom_part1.php rather than chosen here --
and the reply names the values used, because a network at 0.5 and one at
0.9 are different answers to the same question;
* at most 40 variables and at least 3 samples complete across every variable.
The first is the structure search's exponential in-degree; the second is
BNW modelling a complete multivariate normal, so a sample missing any value
is dropped -- and the note says how many survived rather than silently
shrinking the panel;
* its own gate, always claimed (threshold 0): there is no such thing as a
small BNW run, the cost is the search, not the number of individuals;
* the notes say plainly that a posterior is not a p-value and that a network
is not a QTL, because that is the easiest thing to over-read.
tests/bnw.rs compares the endpoint against the answer the BNW *web
application* gives for its own six-trait example (the same fixture as the
driver's parity check, gn-bnw/test/golden/): the five edges must match, and the
notes must name the parameters and carry the caveat. It skips when the driver
or its binaries are absent.
Not deployed to production.
|
|
The marker blocks a single-trait scan is split into were the driver's
default (Threads.nthreads()) and changed with --threads. They are a
separate knob now:
BULKLMM_THREADS Julia threads (--threads) default 4
BULKLMM_BLOCKS independent marker blocks, scanned default 4
in parallel (1 = serial scan)
Blocks are exact either way -- LOD 9.589938 and the same permutation
thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says
what ran ("split into 4 independent blocks and scanned on 4 Julia
thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to
"is it multicore?" should be in the response and not in the log.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|
|
A GeneNetwork trait page shows the gene's aliases next to the symbol,
and GN4 had no way to ask for them: gn-guile has a Wikidata lookup
(gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that
lookup with the caching Wikidata deserves -- it is a shared public
service and an alias list is not news.
* GET /genes/{symbol}/aliases: two SPARQL queries against
WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human,
mouse or rat, labelled with the symbol; then its skos:altLabel s in
English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB
file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion
rather than a directory of two).
* TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value
is refetched; ?refresh=true forces that.
* The label is matched case-insensitively, which the gn-guile query
did not: human genes are labelled SHH and mouse ones Shh, so an
exact match finds only one species. ?species=human now returns the
ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a
mouse trait page shows. With no species named the pick is
deterministic (mouse, rat, human), so a repeated request cannot
change entity.
* A failed refresh keeps the last value: the answer says stale: true,
the envelope is partial, and a note explains it. Nothing cached at
all is a 502 that says so. An empty alias list and an unreachable
upstream are different facts, and only one of them is about the
gene.
Seven unit tests (key scoping, TTL including a clock that went backwards,
the query shape and quote-escaping, SPARQL flattening, an LMDB
round-trip that asserts the file is gene-aliases.lmdb, and
stale-on-failure with and without a cache) and two route tests (the
second call is served from the cache -- same fetched_at -- and the
species switch), which skip with a 502 assertion when there is no
network.
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|
|
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default
8, read when the gate is built) with the same one-job-per-caller rule as
GEMMA: a second submission from the same token while its scan is running
or queued is refused `409 compute_busy`; a submission that finds every
slot taken waits for one to come free. Eight because an R/qtl scan is a
single-threaded Rscript on a 96-core box; not more, because a scan on a
3 000-individual cross is minutes of CPU and the same machine serves the
database. Unlike GEMMA's single slot, concurrent scans share no cache
(R/qtl results are uncached), so the only cost is the CPU they take.
Two flaws only appeared once there was more than one slot:
- the "what is running" record was a single `Option<Active>`, so a
refusal quoted whichever job had most recently taken a slot -- robw's
409 named JoM's run id. It is now a map keyed by subject, and
`a_refusal_quotes_the_callers_own_run_id` pins it;
- `Busy::describe` took the endpoint identifier, so the message read
"your rqtl job"; the decorator takes a separate `label` ("R/qtl") from
the status path.
The test suite hit the rule itself: six compute tests in tests/rqtl.rs
share one token, and cargo runs them on several threads, so three were
refused instead of run. They now serialise on a module mutex, with a
comment saying why -- the same lesson clients need, that parallelism
means more tokens, not more requests from one.
* src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`;
`Gate::new(permits)` was already parameterised. `running` becomes a map
keyed by subject, since one record cannot name N jobs. Module doc talks
about slots rather than a single slot. Tests: eight subjects run and the
ninth waits, a caller still may not take two even with slots free, and a
refusal quotes the caller's own run id.
* src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use.
* gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are
independent.
* src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which
it needs.
* tests/rqtl.rs: compute tests serialised.
* README.md (/compute/rqtl), docs/www/ai-instruction.md (4a),
docs/www/operator-checklist.md: the numbers per endpoint, and that more
parallelism means more tokens.
|
|
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
POST /compute/gemma/ravanan used to run every CWL step locally, because
compute_ravanan never passed ravanan a batch system. Add a
RavananBatch setting:
- RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing
closed on anything else, exactly like GN4_DB_MODE;
- slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint,
--slurm-jwt (a *file*, which is what ravanan reads) and, when set,
--slurm-partition/--slurm-nice; a slurm selection without an endpoint
or a JWT file refuses to start rather than 502-ing every request;
- --scratch defaults to ravanan_cache_dir for slurm, so the store and
the job scratch space live on the same shared filesystem (with slurm
that directory has to be visible to the compute nodes);
- the response's ravanan note records which backend ran.
The switch building is RavananBatch::ravanan_args so it is unit
testable; compute_ravanan takes the backend and the scratch directory,
and the route passes them from config. docs/ravanan-slurm.md documents
the keys, the shared-directory requirement, and the slurm-side auth
wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in
slurmrestd's environment) that makes REST-submitted jobs actually run.
|
|
# Conflicts:
# src/gemma.rs
# src/rqtl.rs
# src/streaming.rs
|
|
gemma-loco.cwl was a generated artifact (ccwl compile's output), but
was being committed like hand-written source. Remove it, and have
compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL
file under its own cache dir on every call instead -- cheap (pure
macro expansion, no Guix build involved), so there's no real cost to
regenerating it fresh each time, and no risk of a stale compiled file
drifting from its ccwl source.
Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH,
alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now
names the .scm source, not a .cwl file. Verified end to end by running
the exact compile-then-run sequence compute_ravanan now performs
against the real ccwl/ravanan/gemma-wrapper binaries and the BXD
fixtures: same correctness and timings as before.
|
|
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper
invocation covering all chromosomes, relying on gemma-wrapper's own
internal GNU-parallel fan-out) with one gemma-wrapper invocation PER
CHROMOSOME, restricted via its existing --chromosomes/--no-parallel
flags and scattered by ravanan itself (ScatterFeatureRequirement,
scatterMethod: dotproduct). gemma-wrapper still does the correct
per-chromosome LOCO math (exclude this chromosome from kinship, test
only its markers) against the full genotype/SNP files -- only which
chromosome(s) it loops over changes. compute_ravanan now computes the
chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own
get_chromosomes) and passes it as the workflow's scatter input, and
collates the resulting per-chromosome gwa_json/archive array outputs.
Measured 16.7s wall-clock end-to-end against the BXD fixtures (20
chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for
the previous sequential ravanan workflow -- confirmed byte-for-byte
identical output (rs31443144's exact values, 7320 total rows).
Also fixes three real, previously-unexercised bugs in compute_ravanan
found getting this to run at all (POST /compute/gemma/ravanan had no
automated test until this commit):
- missing --guix-manifest (ravanan requires it; added
gemma_ravanan_manifest config key and cwl/gemma/manifest.scm)
- missing --store files/logs/data subdirectories (ravanan does not
create them itself)
- --output-directory dropped entirely: every chromosome's gwa archive
shares gemma-wrapper's own content-hash basename (the hash doesn't
factor in --chromosomes), so ravanan's optional flat-symlink step
collides across them; unneeded anyway since results are read
straight from ravanan's --store via its stdout JSON record.
Also found and worked around a ravanan scheduler bug: a step scattered
over exactly one input needs an explicit scatterMethod even though CWL
only requires one for multiple scattered inputs -- ravanan has no
fallback case for a missing one and silently misbehaves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
tied to dev mode.
The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind
address. It decides per request whether to answer.
* src/routes/dev.rs (get_test_token): answers a caller only when the TCP
peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped
127.x) and the request carries no X-Forwarded-For, Forwarded or
X-Real-IP header (a reverse proxy on the same host would otherwise make
every client look local). Anything else, including an unknown peer, is a
403. Local is never established from a header.
* src/main.rs: serve with into_make_service_with_connect_info, which
provides the peer address.
* src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and
dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it
is not a clearly affirmative value (true/1/yes/on) the route is not
registered: a 404, absent from /openapi.json. It fails closed, so a
typo ("flase") turns the endpoint off.
* src/lib.rs: register the route unless that switch is off.
* tests: tests/get_test_token.rs rewritten (remote, unknown and proxied
peers refused; 127.x and ::1 answered; always documented by default);
tests/get_test_token_off.rs for the switch (own process: it sets an
environment variable); common::post_from supplies a peer address to the
in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV
is now empty (no setting needed).
* README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document
the check, the switch, and "Behind nginx or in a container" (a bridge
container sees the gateway as the peer; a host-network container behind
nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For,
or switch the endpoint off).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
New configuration for the native search engine (PORT-DESIGN C1.2 /
AMENDMENT D pre-decided mechanical naming):
- xapian_db_path (env XAPIAN_DB_PATH): directory of the xapian index the
native engine queries. Mirrors gn3's own key at gn3/api/search.py:295
so the two services share one convention. Empty default: forwarding
needs no index; the native engine reports the missing key as a local
misconfiguration rather than guessing a path.
- search_engine (env SEARCH_ENGINE): "forward" (default, the existing
gn3 proxy) or "native" (query the xapian index directly). Opt-in, so
the default behavior of every deployment is unchanged and flipping is
a one-line config change.
Both follow the existing layering (defaults -> GN4_CONF/GN4_SECRETS ->
environment) exactly like gn3_url / gn3_timeout_secs; unit tests cover
the defaults and the env-over-default layering.
Evidence: ~/spikes/port-execution/logs/c1-2.log (config unit tests 12
passed; full suite EXIT=0, parity floor still 54/54 through the
forwarding engine).
|
|
- Cargo.toml: drop duplicate serde_json (keep 1.0.151).
- Cargo.lock: keep origin/main's rustc-1.85-compatible pins; minimally add reqwest/url.
- README: drop stale /search bullet.
- test/python: drop dead 'import socket' (shared database_reachable now).
- src/config.rs: dev_fields test literal gains gn3_url/gn3_timeout_secs.
NOTE: pre-commit fmt --check bypassed; local rustfmt 1.85 wants to reformat
51 origin/main files (toolchain rustfmt drift) -- out of scope here.
|
|
total timeout for gn3 calls
reqwest::get built a fresh client per query (no keepalive) and set no
timeout, while gn3 sets none of its own -- a stalled upstream would
hang the handler past any front-proxy timeout. One shared client in a
OnceLock now, mirroring the shared-DB-pool pattern.
* src/config.rs (Config): add GN3_TIMEOUT_SECS (default 30);
normalize GN3_URL to end with a slash.
* tests/search.rs test/python/test_search.py (search): one scenario
mapping a gn3 5xx to the 502 backend_error shape.
|
|
Base URL of the GeneNetwork3 API for proxy endpoints (phase A
search), following the existing layering: built-in default
(http://127.0.0.1:5003/api/), overridable via GN4_CONF/GN4_SECRETS
files and the GN3_URL environment.
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's
well-known development user and returns the result, so local testing
no longer needs a manual curl/gn-auth round trip of its own.
Only ever registered -- not just rejected at request time, genuinely
absent from the router and from /openapi.json -- when two new Config
fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback
bind (GENENETWORK4_BIND, also new). Both are config, not something
genenetwork4 infers from its actual network exposure, by design: dev
mode alone, or a loopback bind alone, each leave it disabled, and the
check (Config::dev_token_enabled) is a plain function so the condition
itself has direct unit test coverage rather than only being exercised
indirectly through the route.
src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0",
via a proper SocketAddr rather than string formatting -- the latter
would mishandle an IPv6 bind address like "::1" (needs brackets).
GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to
gn-auth's own published dev fixtures (flask init-dev-clients,
gn_auth/wsgi.py), so this works unconfigured against an unmodified dev
gn-auth instance -- confirmed live, including using the returned token
against a real gated endpoint (/traits/BXD/10001), not just checking
the token response's shape.
tests/dev_token.rs covers all four combinations of the two settings in
one sequential test function (mutating the same two env vars from more
than one #[tokio::test] in a file would race, since they run
concurrently by default) plus the /openapi.json visibility check.
test/python/test_dev_token.py mirrors this using separate test classes
instead, which is safe here since each spawns its own server
subprocess with its own environment (ServerTestCase gains EXTRA_ENV for
this).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* Cargo.toml (jsonwebtoken, ureq): New dependencies, RS256 JWT
verification and gn-auth's HTTP API respectively. ureq pulls in ring
for RSA verification, needing a C compiler at build time -- already
provided by cargo-build-system's standard inputs, nothing added to
guix.scm for it. Promote serde_json from dev-dependencies to
dependencies; src/auth.rs needs it outside tests now.
(mysql-common-derive, simple_asn1, time): Pin to the newest versions
still compatible with the Guix-packaged rustc 1.85.1 (same trap as
before: ureq's dependency chain pulls darling 0.23/time 0.3.46+, both
needing rustc 1.88).
* rust-package.scm: Regenerate with `guix import crate -f Cargo.lock
genenetwork4`.
* guix.scm (genenetwork4-git)[native-inputs]: Update crate list to
match.
* src/auth.rs (AuthToken): New extractor. Verifies a bearer token's
RS256 signature and standard time claims (exp/nbf, checked only when
present -- gn-auth's own validator treats them as optional too)
against gn-auth's JWKS, cached and refreshed periodically
(OnceLock<Mutex<HashMap<...>>>, same shape as db::pool()) rather than
fetched per request.
(is_publicly_readable): New function. Matches GeneNetwork2's own
threshold (public > 0 AND confidentiality < 1, used throughout
wqflask/api/router.py) for whether a dataset needs an authorization
check at all; InbredSet/group rows carry a public column too, but GN2
never filters on it, so groups/species endpoints need no gating.
(authorise): New function. Wraps gn-auth's POST
/auth/resource/authorisation, caching the per-(user, resource) answer
briefly (AUTHORISATION_TTL) so repeated requests for the same resource
by the same user don't each cost a round trip. Not yet called from any
route -- see RELEASE_NOTES.md.
* src/config.rs (Config)[auth_server_url]: New field, AUTH_SERVER_URL
config key, matching GeneNetwork3's own key of the same name.
* src/error.rs (ApiError): Derive Debug; needed by tests/auth.rs's
.unwrap() on a Result<_, ApiError>, and every error type should
implement it regardless.
* src/lib.rs: Register the new auth module.
* tests/auth.rs: New file. Runs AuthToken verification and authorise()
(including its cache) against a real gn-auth instance; set
GN4_TEST_AUTH_TOKEN/GN4_TEST_RESOURCE_ID (see docs/gn-auth.md), skips
otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Line-wrapping and canonical import ordering only (verified with
git diff -w). Establishes rustfmt's output as the canonical style
before the search/correlation work lands.
|
|
|
|
* src/config.rs: New module. Layered configuration,
defaults -> GN4_CONF -> GN4_SECRETS -> environment, mirroring
GeneNetwork3's SQL_URI convention so the two services can share
configuration. Missing config files are ignored.
* src/response.rs: New module. The descriptive success envelope
(data + next actions) from the foundation design note
(topics/gn4/gn4-api-foundation.gmi). Not yet adopted by existing
endpoints; that is a deliberate wire-format change to agree on
before it ships.
* src/routes/mod.rs, species.rs, groups.rs: Move the species and
groups handlers out of lib.rs into one module per domain, per the
foundation structure. Handlers are pub; lib.rs just registers them.
* src/lib.rs (app): Register the moved routes. hello_world and
version predate the domain split and stay as top-level utility
routes.
* src/db.rs (sql_uri): Resolve through config::Config instead of
reading the environment directly.
* src/main.rs: Bind the port from config::Config.
|