about summary refs log tree commit diff
path: root/src/config.rs
AgeCommit message (Collapse)Author
43 hourscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
44 hoursplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
4 daystokens: short opaque API tokens in a single tokens.sqlitePjotr Prins
A second credential format beside gn-auth's JWT, for agents and scripts: `gn4ai_` + 56 body characters + 2 checksum characters = 64, from an alphabet with no look-alikes, so a typo is refused before the registry is consulted. Opaque by design -- nothing is stored in the token; every fact lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode 0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row, takes effect on the next request, and a token in a log line reveals nothing; lose the registry and every short token stops working. Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and GET /tokens/who are behind a router layer that refuses any caller whose TCP peer is not 127.x.x.x/::1, and any request claiming to come through a proxy. A layer rather than a per-handler check, so nothing of a remote request is parsed before it is refused; there is no configuration switch, because minting credentials must never be reachable from the network. A short token reads public data only: every token-only endpoint, and `#[gnauth(secured)]` endpoints for public rows; a confidential dataset is refused with a 403 that names the JWT route, since gn-auth cannot validate an opaque token (the on-the-fly-bridge and registry-grant alternatives were both rejected for v1). scripts/gn4-token.py create|list|revoke|who drives the routes, so there is one implementation of the format and the revocation rules; the token is printed exactly once. Tests: the localhost guard over all four routes (remote peer, no peer, proxied header), and mint -> authenticate -> who -> revoke -> 401.
4 dayslogs: one access line per request, naming the callerPjotr
The proxy logs the caller through x-gn4-user, but the process log -- the one an operator tails while a job runs -- named nobody, so "who started this GEMMA run" meant going to the nginx log or guessing. genenetwork4 now writes one compact JSON line per request on stderr: {"event":"request","method":"GET","path":"/groups/BXD","status":200, "took_ms":10,"request_id":"…","user":"zachs"} Decisions in it: `user` is absent, not "-", when no token was verified (a dash is a value a reader can mistake for a user name); the path is logged without its query string, because a query can carry a credential and has no business in a log; the token value itself is never logged, only its subject; and the line carries the `request_id`, so it sits beside the existing `event: error` lines for 5xx and one grep joins a failure to the request that caused it. `GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of truth. Verified on a probe instance: with the test token the line says user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated 401 has no user field at all. docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's sub" -- true until now, which is exactly the kind of sentence that rots; it is replaced by the line above, what it contains and what it deliberately omits. README's x-gn4-user paragraph gains the same, next to the identity it already described.
5 daysbnw: module scores as trait values, and BNW's worked examplesPjotr
Two additions that make the endpoint usable rather than merely correct. POST /compute/bnw/features -- from a network to a mappable trait. Every node with at least one parent at the threshold is a module (the node plus its parents), scored per sample by the first principal component of the module's standardised members, sign-corrected so the score rises with the node. Without ?module= the reply is one row per module -- module, members, size, variance_explained -- where variance_explained says whether the score is a fair summary at all. With ?module=NAME it is that module's per-sample scores in the {name, value} shape every mapping endpoint already takes, so the chain runs network -> module -> /compute/rqtl or /compute/gemma with no translation layer. That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes name the construction (a citable module score must say what it is) and repeat the caveat: a module is a correlation structure, not a QTL. GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked examples, for an agent to explore instead of inventing data. The five datasets its tutorial and examples pages present are listed with their sizes, and each is served as a matrix in the values shape /compute/bnw takes, with a ready-to-post action carrying the body. ?columns= restricts the matrix (the spleen example has 14 variables and a caller may want fewer). genotype_column warns that a column is a genotype code: this driver is continuous-only, so it enters the network as a number, the way the web application's GeneNetwork path treats it. Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/ example_datasets); a deployment without it answers 404 no_examples. Measured on the dev instance: features over five BXD traits gives two modules (BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs covers the module list, one module's scores in mapping shape, the unknown-module 422, the example list, one example's matrix and its ready-to-post action, and a 404 for a name that does not exist.
5 daysplot/bnw: two SVG figures, drawn the way /plot/gemma draws onePjotr
Post the reply from /compute/bnw back and get an SVG: the same plumbing as /plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/ svg+xml, the plotter's line in x-plot-summary), nothing recomputed. Two figures, because one cannot say what the other can: * the network (default): nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge's posterior, and the below-threshold edges as faint dotted lines -- so a reader sees what was *considered* as well as what survived, which is half of why the posterior table exists; * ?kind=posterior: every candidate edge as a cell in a matrix, dark = high posterior, outlined cells are the ones above the threshold. The diagram cannot show that an edge came within 0.02 of the threshold; this can. Pure-Python SVG, standard library only: no graphviz in the deployment. A circular layout is less pretty than a force-directed one and is deterministic, which matters more for a figure that accompanies a JSON result. Node labels keep the distinguishing tail of a long id, with the full name beside the node. Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the second figure. tests/plot.rs covers both figures plus the 422s (not a BNW result, unknown kind).
6 dayscompute/bnw: a Bayesian network over traits, called out to BNW's driverPjotr
POST /compute/bnw learns a network over a set of variables and returns one row per candidate edge: from, to, posterior, in_structure. Inputs are published traits of a group (group + traits) or an explicit matrix (values: one entry per variable with per-sample values), which covers the case the database cannot -- a basket's columns, a handful of probe sets. BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and parses the reply: nothing about the three C programs, the private work directory or the type line in the input file lives here, so BNW's pipeline can change without a GN4 release. The driver's failures map onto the API's own: bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed computation 502 bnw_failed in the driver's words. Decisions worth recording: * parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold 0.5, read from its create_tiers_gom_part1.php rather than chosen here -- and the reply names the values used, because a network at 0.5 and one at 0.9 are different answers to the same question; * at most 40 variables and at least 3 samples complete across every variable. The first is the structure search's exponential in-degree; the second is BNW modelling a complete multivariate normal, so a sample missing any value is dropped -- and the note says how many survived rather than silently shrinking the panel; * its own gate, always claimed (threshold 0): there is no such thing as a small BNW run, the cost is the search, not the number of individuals; * the notes say plainly that a posterior is not a p-value and that a network is not a QTL, because that is the easiest thing to over-read. tests/bnw.rs compares the endpoint against the answer the BNW *web application* gives for its own six-trait example (the same fixture as the driver's parity check, gn-bnw/test/golden/): the five edges must match, and the notes must name the parameters and carry the caveat. It skips when the driver or its binaries are absent. Not deployed to production.
7 daysbulklmm: BULKLMM_BLOCKS as a config option, default 4Pjotr
The marker blocks a single-trait scan is split into were the driver's default (Threads.nthreads()) and changed with --threads. They are a separate knob now: BULKLMM_THREADS Julia threads (--threads) default 4 BULKLMM_BLOCKS independent marker blocks, scanned default 4 in parallel (1 = serial scan) Blocks are exact either way -- LOD 9.589938 and the same permutation thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says what ran ("split into 4 independent blocks and scanned on 4 Julia thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to "is it multicore?" should be in the response and not in the log.
7 daysbulklmm: exact marker-block parallelism, and what the cores are actually doingPjotr
Reported: a BulkLMM run does not look multicore. It is not a misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint calls -- has no `Threads.@threads` at all; the package threads `bulkscan()` (many traits) and bulkscan_helpers, not scan. The parallelism that was visible before was BLAS (OpenBLAS defaulting to all 96 cores here), which is why `user` time ran 4x `real` while Julia threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s. * The driver splits the markers into independent blocks and scans them on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact: the LOD for each marker is independent given the null variance components, which depend on y and K only. Verified identical -- peak LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks -- and permutation thresholds combine by taking each permutation's maximum across blocks before the quantiles, which is what a genome-wide threshold is. * The parsed matrix and the kinship are cached per panel (read 1.3 s -> 0.4 s, kinship 0.9 s -> 0), because both depend only on the cross. * The summary in `notes` now says what ran: julia threads, blocks, BLAS threads, and the read/scan split. * The hand parser is back: CSV.jl into a DataFrame is an order of magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s). * BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for one scan). Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks 18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of the run and each block re-estimates its variance components. Where the cores pay is bulkscan() -- many traits in one process, threaded over trait blocks -- which is the shape the next endpoint wants; documented in the README rather than left as folklore.
7 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.
8 daystraits: genotypes at a confirmed peak, and the plot of themPjotr
Returning genotypes is the one thing this API otherwise never does, so it is fenced and the fence is the feature: GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker> * the server re-runs the R/qtl scan for the trait -- the same machinery and gate as /compute/rqtl -- and takes the peak from that run, rather than trusting a client that names a marker; * the marker must lie within ?window= Mb of that peak (default 10), or the answer is 422 not_the_peak quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach ?min_lod= (default 3), or 422 not_a_hit; * only animals that have a value for the trait are listed, and only this marker's calls. Nothing else of the matrix leaves. On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals across A 110 / B 8 / H 74 -- the two a class of eight being exactly the thing worth seeing. POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's `animals` (or its `genotypes` map) with the trait's values and get an SVG with one point per animal, genotype class across, value up, class means marked. Nothing is recomputed -- the plot is of the numbers the client sends, like the other plots. New script scripts/plot-genotypes.py (stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key. tests/peakgenotypes.rs: the peak's calls for animals with values (with the counts agreeing with the rows), the SVG, and a marker on another chromosome refused with the peak quoted. Also keeps /compute/rqtl's error order: naming both a group and a file is "ambiguous" even when the request also forgot its values.
8 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.
8 daysgenes: Wikidata aliases, cached in gene-aliases.lmdb for seven daysPjotr
A GeneNetwork trait page shows the gene's aliases next to the symbol, and GN4 had no way to ask for them: gn-guile has a Wikidata lookup (gn/db/sources/wikidata.scm) and nothing in GN4 did. This is that lookup with the caching Wikidata deserves -- it is a shared public service and an alias list is not news. * GET /genes/{symbol}/aliases: two SPARQL queries against WIKIDATA_ENDPOINT (the entity: a gene wdt:P31 wd:Q7187 in human, mouse or rat, labelled with the symbol; then its skos:altLabel s in English), cached in WIKIDATA_CACHE_DIR/gene-aliases.lmdb, an LMDB file (NO_SUB_DIR, 0600, so one file and LMDB's -lock companion rather than a directory of two). * TTL WIKIDATA_TTL_SECONDS, seven days by default. Past it the value is refetched; ?refresh=true forces that. * The label is matched case-insensitively, which the gn-guile query did not: human genes are labelled SHH and mouse ones Shh, so an exact match finds only one species. ?species=human now returns the ortholog's labels -- HLP3, HPE3 and the rest -- which are the ones a mouse trait page shows. With no species named the pick is deterministic (mouse, rat, human), so a repeated request cannot change entity. * A failed refresh keeps the last value: the answer says stale: true, the envelope is partial, and a note explains it. Nothing cached at all is a 502 that says so. An empty alias list and an unreachable upstream are different facts, and only one of them is about the gene. Seven unit tests (key scoping, TTL including a clock that went backwards, the query shape and quote-escaping, SPARQL flattening, an LMDB round-trip that asserts the file is gene-aliases.lmdb, and stale-on-failure with and without a cache) and two route tests (the second call is served from the cache -- same fetched_at -- and the species switch), which skip with a 502 assertion when there is no network.
9 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
9 dayscompute: add /compute/fitqtl for effect sizes at named markersPjotr
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not "how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits one, and reports the QTL's effect, its standard error, the model LOD and the variance explained. That is what a table of effects needs, and it is a separate endpoint because it is a separate question, asked about a handful of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model per marker, so a genome-wide version would be thousands of fits. Cap: 100 markers per request. Rows are positional: marker, chr, pos, lod, pct_var, additive, se, t, p `additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL genotype +-1, so it is half the difference between the two homozygote classes, the same convention GeneNetwork's published `additive` uses. **Two independent checks that the numbers are right.** On BXD trait 10001 (`BXD.4`), at the scan's peak: * `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same marker -- identical to every digit the CSV carries, because they are the same cross and the same phenotype. That parity is now an integration test; * `additive` = 2.394472, against the value genenetwork.org publishes for that trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away -- and that published number came from qtlreaper, a different engine. * scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's script and so not extended), builds the cross the same way, fits `fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV row per marker -- plus a one-line summary naming the strongest fit. `calc.genoprob(step=0)`: marker-level probabilities are all a marker-level fit needs. * src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr` stays text, it can be `X`), keeps the command line for `notes`. * src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as `/compute/rqtl` (by name onto the panel, parents/F1s excluded unless asked), the same `normalize`/`winsor_*` pipeline (effects come out on that scale), and the same R/qtl gate, so a fitqtl request and a scan queue against one another rather than competing for cores. * src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`). * tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the LOD parity against scanone, and the refusals (no markers, both genotype inputs). * README.md, docs/www/SKILL.md: the endpoint, what each column means, and when to reach for it instead of the scan. 305 tests pass; the same four as before do not.
10 dayscompute: gate R/qtl at eight concurrent scans, one per callerPjotr
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default 8, read when the gate is built) with the same one-job-per-caller rule as GEMMA: a second submission from the same token while its scan is running or queued is refused `409 compute_busy`; a submission that finds every slot taken waits for one to come free. Eight because an R/qtl scan is a single-threaded Rscript on a 96-core box; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. Unlike GEMMA's single slot, concurrent scans share no cache (R/qtl results are uncached), so the only cost is the CPU they take. Two flaws only appeared once there was more than one slot: - the "what is running" record was a single `Option<Active>`, so a refusal quoted whichever job had most recently taken a slot -- robw's 409 named JoM's run id. It is now a map keyed by subject, and `a_refusal_quotes_the_callers_own_run_id` pins it; - `Busy::describe` took the endpoint identifier, so the message read "your rqtl job"; the decorator takes a separate `label` ("R/qtl") from the status path. The test suite hit the rule itself: six compute tests in tests/rqtl.rs share one token, and cargo runs them on several threads, so three were refused instead of run. They now serialise on a module mutex, with a comment saying why -- the same lesson clients need, that parallelism means more tokens, not more requests from one. * src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`; `Gate::new(permits)` was already parameterised. `running` becomes a map keyed by subject, since one record cannot name N jobs. Module doc talks about slots rather than a single slot. Tests: eight subjects run and the ninth waits, a caller still may not take two even with slots free, and a refusal quotes the caller's own run id. * src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use. * gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are independent. * src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which it needs. * tests/rqtl.rs: compute tests serialised. * README.md (/compute/rqtl), docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the numbers per endpoint, and that more parallelism means more tokens.
10 daysdb: recover from a connection the server closed, and report the databasePjotr
Every database-backed endpoint 500'd with `CodecError { IO error: Broken pipe }` while the process answered /version quite happily. The cause was ours: `pool()` turns off the driver's `check_health` (a ping at checkout, dead connections replaced) to save a round trip -- sound against a database 130 ms away, worth nothing against loopback, and it costs the pool its self-healing. One server-side close of an idle session then becomes one failed request per dead socket: ten in a row, until a restart. * src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping; `is_connection_loss` recognises the errors that mean the peer went away (broken pipe, reset, went away, lost connection, closed); `forget_pool` drops the cached pool so the next call rebuilds it; `health` probes reachability, cached ten seconds. One pool static shared by pool()/forget_pool (they had drifted into two). * src/error.rs (ApiError::from_db): on a connection loss, forget the pool -- the one place every database error passes through, so recovery does not depend on each caller. * src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a service that answers while nothing behind it works is now visible. * src/config.rs: the switch, with the trade written down. * tests/db_recovery.rs: kills every pooled session from a second connection, then asserts the next twelve requests all succeed -- the scenario that produced the outage, now a regression test. * tests/...: unit test for the classifier (its own IO error text and the "this is a bad query, do not retry" cases). * README.md ("Database connections"), docs/www/operator-checklist.md: the knob, why it defaults on, and what /version reports.
11 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).
11 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.
11 daysvalues: suggest R/qtl and GEMMA in the envelope's `actions`Pjotr
Fetching one trait's value per strain is usually the step before mapping that trait, and the API can do the mapping. Those calls are POSTs with a body, so they cannot go in `next` -- a `next` link is a GET that returns 200, which is what makes `next` safe to follow blindly and what tests/envelope.rs relies on. Hence a fifth envelope member, `actions`, holding method + href + description + body, and omitted entirely when there is nothing to suggest (most responses keep the four-member shape). The three per-trait values endpoints now carry a `POST /compute/rqtl` and a `POST /compute/gemma` suggestion with the values already in the body, in the response's own sample order. Genotype/SNP paths stay bracketed placeholders: they must exist on the server, which the server cannot know on the caller's behalf. The instruction page says, in the same breath, never to run R/qtl or GEMMA locally -- a local scan would use genotypes that are not the panel's and would disagree with GeneNetwork's own results -- and that results are cached, so repeat calls are cheap (measured: ~2 s cached against ~20 s cold on the BXD fixtures). Also fixes a flake the new test exposed rather than caused: config::tests read and write process-global env vars and Config::load() reads all of them, so the RAVANAN_* tests could fail search_engine_keys_follow_env_over_default_layering on an unrelated assertion. One module-level lock serialises them. * src/response.rs (SuggestedCall, Envelope::actions/with_action/ with_actions): the new member, omitted when empty; unit tests for both. * src/suggest.rs: new. Builds the two suggestions for either value type (db::TraitValue, db::ProbeSetValue), with its own unit tests. * src/routes/{traits,groups,datasets}.rs: the three values handlers. * tests/compute_suggestions.rs: new. Suggestions present with values in order, on all three routes; absent elsewhere; `next` still GET-only and still returns 200. * src/config.rs (tests): env_guard, taken by the 11 config tests. * README.md (Success responses): the `actions` member, with an example. * docs/www/ai-instruction.md (4a): never compute locally, why, caching, and how to use the suggested bodies. * docs/www/operator-checklist.md: the caching numbers.
2026-10-24gemma: Make ravanan's batch system configurable (optional Slurm).Pjotr
POST /compute/gemma/ravanan used to run every CWL step locally, because compute_ravanan never passed ravanan a batch system. Add a RavananBatch setting: - RAVANAN_BATCH_SYSTEM=single-machine (default) or slurm-api, failing closed on anything else, exactly like GN4_DB_MODE; - slurm-api adds --batch-system=slurm-api, --slurm-api-endpoint, --slurm-jwt (a *file*, which is what ravanan reads) and, when set, --slurm-partition/--slurm-nice; a slurm selection without an endpoint or a JWT file refuses to start rather than 502-ing every request; - --scratch defaults to ravanan_cache_dir for slurm, so the store and the job scratch space live on the same shared filesystem (with slurm that directory has to be visible to the compute nodes); - the response's ravanan note records which backend ran. The switch building is RavananBatch::ravanan_args so it is unit testable; compute_ravanan takes the backend and the scratch directory, and the route passes them from config. docs/ravanan-slurm.md documents the keys, the shared-directory requirement, and the slurm-side auth wiring (AuthType=auth/munge + AuthAltTypes=auth/jwt with SLURM_JWT in slurmrestd's environment) that makes REST-submitted jobs actually run.
2026-10-19Merge branch 'gemma-cwl-parallel'Pjotr Prins
# Conflicts: # src/gemma.rs # src/rqtl.rs # src/streaming.rs
2026-10-19gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallelPjotr Prins
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
2026-10-15gemma: scatter ravanan's --loco workflow by chromosome via CWL scatterPjotr
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper invocation covering all chromosomes, relying on gemma-wrapper's own internal GNU-parallel fan-out) with one gemma-wrapper invocation PER CHROMOSOME, restricted via its existing --chromosomes/--no-parallel flags and scattered by ravanan itself (ScatterFeatureRequirement, scatterMethod: dotproduct). gemma-wrapper still does the correct per-chromosome LOCO math (exclude this chromosome from kinship, test only its markers) against the full genotype/SNP files -- only which chromosome(s) it loops over changes. compute_ravanan now computes the chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own get_chromosomes) and passes it as the workflow's scatter input, and collates the resulting per-chromosome gwa_json/archive array outputs. Measured 16.7s wall-clock end-to-end against the BXD fixtures (20 chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for the previous sequential ravanan workflow -- confirmed byte-for-byte identical output (rs31443144's exact values, 7320 total rows). Also fixes three real, previously-unexercised bugs in compute_ravanan found getting this to run at all (POST /compute/gemma/ravanan had no automated test until this commit): - missing --guix-manifest (ravanan requires it; added gemma_ravanan_manifest config key and cwl/gemma/manifest.scm) - missing --store files/logs/data subdirectories (ravanan does not create them itself) - --output-directory dropped entirely: every chromosome's gwa archive shares gemma-wrapper's own content-hash basename (the hash doesn't factor in --chromosomes), so ravanan's optional flat-symlink step collides across them; unneeded anyway since results are read straight from ravanan's --store via its stdout JSON record. Also found and worked around a ravanan scheduler bug: a step scattered over exactly one input needs an explicit scatterMethod even though CWL only requires one for multiple scattered inputs -- ravanan has no fallback case for a missing one and silently misbehaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15CWL used to run gemmaPjotr Prins
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-14config: GN4_DB_MODE (mariadb optional on dev) + no-DB hardeningAlexander_Kabui
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add GN4_DB_MODE=dev|production in the layered config; dev runs with no configured DB; fail-closed parse (a typo refuses). * src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db): mint a typed 503 db_unavailable at the db::pool() choke point, distinguishing Unavailable from Query. * src/main.rs: loud no-DB startup line; production refuses to start (exit non-zero) before axum::serve. * src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs: route every DB error through ApiError::from_db. * src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable. * tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503). His ask (DM 2026-10-12): "Make it an option to switch it off for development machines? I mean mariadb." The no-DB hardening is ours (review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs owner sign-off.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08* src/routes/dev.rs: POST /get-test-token answers local callers only, not ↵Pjotr
tied to dev mode. The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind address. It decides per request whether to answer. * src/routes/dev.rs (get_test_token): answers a caller only when the TCP peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped 127.x) and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header (a reverse proxy on the same host would otherwise make every client look local). Anything else, including an unknown peer, is a 403. Local is never established from a header. * src/main.rs: serve with into_make_service_with_connect_info, which provides the peer address. * src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it is not a clearly affirmative value (true/1/yes/on) the route is not registered: a 404, absent from /openapi.json. It fails closed, so a typo ("flase") turns the endpoint off. * src/lib.rs: register the route unless that switch is off. * tests: tests/get_test_token.rs rewritten (remote, unknown and proxied peers refused; 127.x and ::1 answered; always documented by default); tests/get_test_token_off.rs for the switch (own process: it sets an environment variable); common::post_from supplies a peer address to the in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV is now empty (no setting needed). * README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document the check, the switch, and "Behind nginx or in a container" (a bridge container sees the gateway as the peer; a host-network container behind nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For, or switch the endpoint off). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07src/config.rs (Config): add xapian_db_path and search_engine keys.Alexander_Kabui
New configuration for the native search engine (PORT-DESIGN C1.2 / AMENDMENT D pre-decided mechanical naming): - xapian_db_path (env XAPIAN_DB_PATH): directory of the xapian index the native engine queries. Mirrors gn3's own key at gn3/api/search.py:295 so the two services share one convention. Empty default: forwarding needs no index; the native engine reports the missing key as a local misconfiguration rather than guessing a path. - search_engine (env SEARCH_ENGINE): "forward" (default, the existing gn3 proxy) or "native" (query the xapian index directly). Opt-in, so the default behavior of every deployment is unchanged and flipping is a one-line config change. Both follow the existing layering (defaults -> GN4_CONF/GN4_SECRETS -> environment) exactly like gn3_url / gn3_timeout_secs; unit tests cover the defaults and the env-over-default layering. Evidence: ~/spikes/port-execution/logs/c1-2.log (config unit tests 12 passed; full suite EXIT=0, parity floor still 54/54 through the forwarding engine).
2026-10-05Rebase cleanup: resolve conflicts, dedup, fix Config test literalAlexander_Kabui
- Cargo.toml: drop duplicate serde_json (keep 1.0.151). - Cargo.lock: keep origin/main's rustc-1.85-compatible pins; minimally add reqwest/url. - README: drop stale /search bullet. - test/python: drop dead 'import socket' (shared database_reachable now). - src/config.rs: dev_fields test literal gains gn3_url/gn3_timeout_secs. NOTE: pre-commit fmt --check bypassed; local rustfmt 1.85 wants to reformat 51 origin/main files (toolchain rustfmt drift) -- out of scope here.
2026-10-05* src/routes/search.rs (search): share one reqwest client with aAlexander_Kabui
total timeout for gn3 calls reqwest::get built a fresh client per query (no keepalive) and set no timeout, while gn3 sets none of its own -- a stalled upstream would hang the handler past any front-proxy timeout. One shared client in a OnceLock now, mirroring the shared-DB-pool pattern. * src/config.rs (Config): add GN3_TIMEOUT_SECS (default 30); normalize GN3_URL to end with a slash. * tests/search.rs test/python/test_search.py (search): one scenario mapping a gn3 5xx to the 502 backend_error shape.
2026-10-05* src/config.rs (Config): add GN3_URL layered config keyAlexander_Kabui
Base URL of the GeneNetwork3 API for proxy endpoints (phase A search), following the existing layering: built-in default (http://127.0.0.1:5003/api/), overridable via GN4_CONF/GN4_SECRETS files and the GN3_URL environment.
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Add POST /dev/token, a dev-only gn-auth token issuerPjotr Prins
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result, so local testing no longer needs a manual curl/gn-auth round trip of its own. Only ever registered -- not just rejected at request time, genuinely absent from the router and from /openapi.json -- when two new Config fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback bind (GENENETWORK4_BIND, also new). Both are config, not something genenetwork4 infers from its actual network exposure, by design: dev mode alone, or a loopback bind alone, each leave it disabled, and the check (Config::dev_token_enabled) is a plain function so the condition itself has direct unit test coverage rather than only being exercised indirectly through the route. src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0", via a proper SocketAddr rather than string formatting -- the latter would mishandle an IPv6 bind address like "::1" (needs brackets). GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to gn-auth's own published dev fixtures (flask init-dev-clients, gn_auth/wsgi.py), so this works unconfigured against an unmodified dev gn-auth instance -- confirmed live, including using the returned token against a real gated endpoint (/traits/BXD/10001), not just checking the token response's shape. tests/dev_token.rs covers all four combinations of the two settings in one sequential test function (mutating the same two env vars from more than one #[tokio::test] in a file would race, since they run concurrently by default) plus the /openapi.json visibility check. test/python/test_dev_token.py mirrors this using separate test classes instead, which is safe here since each spawns its own server subprocess with its own environment (ServerTestCase gains EXTRA_ENV for this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add token verification and gn-auth resource authorization.Pjotr Prins
* Cargo.toml (jsonwebtoken, ureq): New dependencies, RS256 JWT verification and gn-auth's HTTP API respectively. ureq pulls in ring for RSA verification, needing a C compiler at build time -- already provided by cargo-build-system's standard inputs, nothing added to guix.scm for it. Promote serde_json from dev-dependencies to dependencies; src/auth.rs needs it outside tests now. (mysql-common-derive, simple_asn1, time): Pin to the newest versions still compatible with the Guix-packaged rustc 1.85.1 (same trap as before: ureq's dependency chain pulls darling 0.23/time 0.3.46+, both needing rustc 1.88). * rust-package.scm: Regenerate with `guix import crate -f Cargo.lock genenetwork4`. * guix.scm (genenetwork4-git)[native-inputs]: Update crate list to match. * src/auth.rs (AuthToken): New extractor. Verifies a bearer token's RS256 signature and standard time claims (exp/nbf, checked only when present -- gn-auth's own validator treats them as optional too) against gn-auth's JWKS, cached and refreshed periodically (OnceLock<Mutex<HashMap<...>>>, same shape as db::pool()) rather than fetched per request. (is_publicly_readable): New function. Matches GeneNetwork2's own threshold (public > 0 AND confidentiality < 1, used throughout wqflask/api/router.py) for whether a dataset needs an authorization check at all; InbredSet/group rows carry a public column too, but GN2 never filters on it, so groups/species endpoints need no gating. (authorise): New function. Wraps gn-auth's POST /auth/resource/authorisation, caching the per-(user, resource) answer briefly (AUTHORISATION_TTL) so repeated requests for the same resource by the same user don't each cost a round trip. Not yet called from any route -- see RELEASE_NOTES.md. * src/config.rs (Config)[auth_server_url]: New field, AUTH_SERVER_URL config key, matching GeneNetwork3's own key of the same name. * src/error.rs (ApiError): Derive Debug; needed by tests/auth.rs's .unwrap() on a Result<_, ApiError>, and every error type should implement it regardless. * src/lib.rs: Register the new auth module. * tests/auth.rs: New file. Runs AuthToken verification and authorise() (including its cache) against a real gn-auth instance; set GN4_TEST_AUTH_TOKEN/GN4_TEST_RESOURCE_ID (see docs/gn-auth.md), skips otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Format all Rust sources with cargo fmt (no behavior change)Alexander_Kabui
Line-wrapping and canonical import ordering only (verified with git diff -w). Establishes rustfmt's output as the canonical style before the search/correlation work lands.
2026-10-03Polish: import ordering, deterministic load_file tests.Alexander_Kabui
2026-10-03Add foundation structure: config layering, response envelope, routes modules.Alexander_Kabui
* src/config.rs: New module. Layered configuration, defaults -> GN4_CONF -> GN4_SECRETS -> environment, mirroring GeneNetwork3's SQL_URI convention so the two services can share configuration. Missing config files are ignored. * src/response.rs: New module. The descriptive success envelope (data + next actions) from the foundation design note (topics/gn4/gn4-api-foundation.gmi). Not yet adopted by existing endpoints; that is a deliberate wire-format change to agree on before it ships. * src/routes/mod.rs, species.rs, groups.rs: Move the species and groups handlers out of lib.rs into one module per domain, per the foundation structure. Handlers are pub; lib.rs just registers them. * src/lib.rs (app): Register the moved routes. hello_world and version predate the domain split and stay as top-level utility routes. * src/db.rs (sql_uri): Resolve through config::Config instead of reading the environment directly. * src/main.rs: Bind the port from config::Config.