about summary refs log tree commit diff
path: root/src/routes/mod.rs
AgeCommit message (Collapse)Author
4 dayscorrelation: the matrix endpoint, and the rule that a deploy ends with a ↵Pjotr Prins
monitor run `POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}` references -- a collection, as GN2's matrix form posts it, which spans datasets -- and answers an N x N matrix of r, p and n. The five decisions that are not obvious, each recorded where a reader will look for them: * **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to Spearman mid-row once a Pearson r passes 0.999, which the self-pair always does, so its matrix is asymmetric by construction. Here the method is chosen once and r[i][j] == r[j][i] exactly. * **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine calls, not N^2. * **The diagonal is by construction**: r = 1, p = null, n = the trait's own count, so half the traits cannot come back as 1.0000000000000002 with a NaN p. * **Per-cell n is its own matrix**, and a cell under `min_overlap` (default 6) is null in r and p while its n still says how many points there were. GN2 accepts 2, where |r| = 1 by definition. * **The sample axis is the first trait's group** (#106), resolved once and stated in the reply with the number of individuals it kept. Fetches are one query per dataset (`get_dataset_values_for_traits`, `get_publish_values_for_traits`), not one per trait: a matrix chooses a handful out of twenty thousand. Capped at 1 000 traits, because the cost is quadratic in the correlations and in the reply. No PCA -- the r matrix has to be trusted on its own first. Verified twice: cells agree with scipy to the last bit (BXDPublish 10001/10002/10005: 0.6756270426459695, 0.6950450737755965, 0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964, 0.8819879006700073), and a cell equals what the row endpoint says for the same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix, `min_overlap` above every cell's n, an unknown trait, `bicor`, and no traits at all. The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier all`, and says why: `/version` can report a commit and a reachable database while every engine that shells out is dead. The monitor gained a matrix check whose judge enforces square, symmetric and a unit diagonal.
5 daystokens: short opaque API tokens in a single tokens.sqlitePjotr Prins
A second credential format beside gn-auth's JWT, for agents and scripts: `gn4ai_` + 56 body characters + 2 checksum characters = 64, from an alphabet with no look-alikes, so a typo is refused before the registry is consulted. Opaque by design -- nothing is stored in the token; every fact lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode 0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row, takes effect on the next request, and a token in a log line reveals nothing; lose the registry and every short token stops working. Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and GET /tokens/who are behind a router layer that refuses any caller whose TCP peer is not 127.x.x.x/::1, and any request claiming to come through a proxy. A layer rather than a per-handler check, so nothing of a remote request is parsed before it is refused; there is no configuration switch, because minting credentials must never be reachable from the network. A short token reads public data only: every token-only endpoint, and `#[gnauth(secured)]` endpoints for public rows; a confidential dataset is refused with a 403 that names the JWT route, since gn-auth cannot validate an opaque token (the on-the-fly-bridge and registry-grant alternatives were both rejected for v1). scripts/gn4-token.py create|list|revoke|who drives the routes, so there is one implementation of the format and the revocation rules; the token is printed exactly once. Tests: the localhost guard over all four routes (remote peer, no peer, proxied header), and mint -> authenticate -> who -> revoke -> 401.
7 dayscompute/bnw: a Bayesian network over traits, called out to BNW's driverPjotr
POST /compute/bnw learns a network over a set of variables and returns one row per candidate edge: from, to, posterior, in_structure. Inputs are published traits of a group (group + traits) or an explicit matrix (values: one entry per variable with per-sample values), which covers the case the database cannot -- a basket's columns, a handful of probe sets. BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and parses the reply: nothing about the three C programs, the private work directory or the type line in the input file lives here, so BNW's pipeline can change without a GN4 release. The driver's failures map onto the API's own: bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed computation 502 bnw_failed in the driver's words. Decisions worth recording: * parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold 0.5, read from its create_tiers_gom_part1.php rather than chosen here -- and the reply names the values used, because a network at 0.5 and one at 0.9 are different answers to the same question; * at most 40 variables and at least 3 samples complete across every variable. The first is the structure search's exponential in-degree; the second is BNW modelling a complete multivariate normal, so a sample missing any value is dropped -- and the note says how many survived rather than silently shrinking the panel; * its own gate, always claimed (threshold 0): there is no such thing as a small BNW run, the cost is the search, not the number of individuals; * the notes say plainly that a posterior is not a p-value and that a network is not a QTL, because that is the easiest thing to over-read. tests/bnw.rs compares the endpoint against the answer the BNW *web application* gives for its own six-trait example (the same fixture as the driver's parity check, gn-bnw/test/golden/): the five edges must match, and the notes must name the parameters and carry the caveat. It skips when the driver or its binaries are absent. Not deployed to production.
9 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.
9 daystraits: genotypes at a confirmed peak, and the plot of themPjotr
Returning genotypes is the one thing this API otherwise never does, so it is fenced and the fence is the feature: GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker> * the server re-runs the R/qtl scan for the trait -- the same machinery and gate as /compute/rqtl -- and takes the peak from that run, rather than trusting a client that names a marker; * the marker must lie within ?window= Mb of that peak (default 10), or the answer is 422 not_the_peak quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach ?min_lod= (default 3), or 422 not_a_hit; * only animals that have a value for the trait are listed, and only this marker's calls. Nothing else of the matrix leaves. On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals across A 110 / B 8 / H 74 -- the two a class of eight being exactly the thing worth seeing. POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's `animals` (or its `genotypes` map) with the trait's values and get an SVG with one point per animal, genotype class across, value up, class means marked. Nothing is recomputed -- the plot is of the numbers the client sends, like the other plots. New script scripts/plot-genotypes.py (stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key. tests/peakgenotypes.rs: the peak's calls for animals with values (with the counts agreeing with the rows), the SVG, and a marker on another chromosome refused with the peak quoted. Also keeps /compute/rqtl's error order: naming both a group and a file is "ambiguous" even when the request also forgot its values.
9 daysuploads: a genotype matrix that updates an existing populationPjotr
Tab-delimited, header names the individuals from column 4, then marker, chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own SQLite file -- nothing goes near MariaDB -- and written out on demand as a GN .geno carrying the base's @-metadata, so the R scripts read exactly what was uploaded. An upload updates a population, it does not invent one, and the checks say so: * the base version must exist (resolved through the same genotypes::resolve_named the rest of the API uses); * at most 50 individuals the base does not have, and at least one that it does -- no overlap is a different population, not an update; * at most 200 000 markers, and the refusal says the sysadmin has to organize a file that size, because this is a request path, not an ingest; * the values must be the same kind as the base's: letters, whole numbers or probabilities. A letters panel holding 0/1 would be read as the wrong thing silently, so it is refused naming both kinds. GET /uploads/genotypes lists them, /{name} its metadata (overlap, new individuals, kind, digest), /{name}/markers pages the marker table, and /{name}/geno writes the file and returns the path for `genotype_file` on the compute endpoints. DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update -- a subset of markers, a few individuals dropped and added -- and tests/genoupload.rs builds its own from the deployment's BXD.4 for the same reason: upload, page the markers, write the .geno and check the layout, delete, then break each rule (no overlap, numbers in a letters panel, commas) and check the refusal says which. Unit tests cover the three value kinds, missing cells, ragged rows, a non-numeric position, the 50-new-individual limit, and the .geno layout (Chr first, then the marker -- GN's order, not the upload's).
9 daysmatrices: upload new traits and covariates as one named tablePjotr
The other half of changing data: bringing new columns in. A matrix is a delimited table -- a header row, then one row per individual, the first column naming them -- stored in the uploader's own SQLite file and checked against an existing genotype version's panel before it is accepted. * POST /matrices {name, genotype_version, contents, kinds?}: parse the table (tab, comma or semicolon, guessed from the header), then check it against that version's individuals. The count must match and every name must be in the panel; 92 of 93, or a name the panel does not have, is a 422 that says which. A mapping that silently drops individuals is worse than an error. * kinds marks columns that are covariates; everything else is a trait. * Each column becomes a basket named <matrix>.<column> -- a dot, not a slash, because a basket name has to survive a URL path segment -- so a column maps on any engine with {"basket": "my-traits.bw"} and inherits the revision history and digest without a line of new mapping code. * GET /matrices lists them, GET /matrices/{name} their columns and basket names, GET /matrices/{name}/values the table back (parsed values, null where missing), DELETE removes the matrix and its columns. Two bugs this found in the basket store, both real: * SQLite has foreign keys OFF by default, so the ON DELETE CASCADE never fired: deleting a basket left its rows behind, rowids were reused, and the next create hit "UNIQUE constraint failed: basket_version.basket_id, revision". The pragma is now set, and deletion removes the dependants explicitly, because files written before it have orphans. * The first attempt named a column's basket "<matrix>/<column>", which the basket-name rule (rightly) refuses. The dot keeps both rules. scripts/example-matrix.py writes a table from a genotype file's own header with deterministic values, so the fixture passes the panel check by construction -- tests/matrices.rs builds its own for the same reason and walks the loop: upload, read back, map a column with R/qtl2, delete, plus the three refusals (too few individuals, a name not in the panel, an unknown version).
9 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.
9 daysprobesets: cross-dataset lookups by probeset id and by gene symbolPjotr
A probeset id is not a Xapian field, so /search cannot find 1427571_at exactly and the only route to it was guessing a dataset and calling /datasets/{ds}/probesets/{name} until one answered. A gene symbol could be searched, but the reply was capped at 100 hits, interleaved probesets and platforms, and carried no total -- so "did I see all of Shh?" had no answer. Both questions are one indexed query each: ProbeSet.Name_IDX and symbol_IDX. * GET /probesets/{name}: one row per dataset that carries the probeset, with the panel (group, species, platform) and the scan's statistics (mean, LRS, LOD, locus, additive). Suffixed chip variants match too -- 1427571_at also finds 1427571_at_A -- and each row's `name` says which matched, so the answer never hides a substitution. LIKE's wildcards are escaped, or every Affymetrix `_at` would match any character. 155 rows over 155 datasets in 32 ms. * GET /genes/{symbol}/traits: the same rows filtered on Symbol, with a real total, ?sort=lod, ?min_lod=, ?group=, ?species=, ?platform= and paging -- "BXD datasets where Shh is a strong eQTL" is one call. 1225 rows over 17 datasets and 13 probesets in 38 ms. * Both serve metadata, not values. A token is enough for the index; trait values stay per-dataset and authorized per dataset, and `next` goes to /datasets/{ds}/probesets/{name}/values, which checks. There is no cross-dataset values endpoint, deliberately: that is bulk work and /groups/{g}/traits is the bulk call. The one-statement join took 5.2 s for Shh -- MySQL rescans InbredSet/Species per row -- so the 933 ProbeSetFreeze rows and the 102 panels are fetched separately and joined in memory: 70 ms each. nsamples is left out: it is a ProbeSetData count per row (+0.5 s) and the dataset record already reports strains.
9 daysgenotypes: call frequencies, marker names, and a location per markerPjotr
Two questions the genotype surface could not answer: what a panel is actually called as, and which markers it holds. The listing gives counts and the compute endpoints give scans; a panel whose calls are not the codes its header declares, or a marker that is 99% missing, stayed invisible until a scan came back flat -- and then the search for the cause started at the wrong end. * GET /groups/{group}/genotypes/{version}/frequencies counts the calls in the version's .geno and returns them as call -> frequency ({"B": 0.4889, "D": 0.492109, "H": 0.018991}). Frequencies are over the calls that are present, so they sum to 1; missing codes (U, N, NA, an empty cell) are counted separately and reported in notes, because the missing rate is the diagnosis. * A probability-valued file is binned 0.05 wide (twenty bins over [0, 1], keys are the bin edges), since 0.4187 and 0.4191 are the same statement about a marker; ?bin= sets another width. A file with a few distinct numbers (0, 0.5, 1 for a biallelic dosage) is reported exactly -- binning three values hides what is being looked at. * .../markers/{marker}/frequencies is one locus, and ?marker= on the panel route is the same call. Counting goes through an interner, so cost is one hash lookup per cell and one allocation per distinct call; a .geno over 256 MB is refused (413 genotype_too_large) rather than read, which is what HSNIH-Palmer_true (3.7 GB, 134927 markers x 6147 individuals) demanded. * The version's /markers now lists marker NAMES only, one column, and .../markers/{marker}/location returns {"name","chr","cM","Mb"} as named fields: a page of names stays small, and a position is a lookup rather than a table the client must hold. The same size guard applies, so an oversized panel's listing is a 413 in 3 ms. * Files no group lists -- a panel in preparation, a fixture, the DO's do/ cross -- live at /groups/unlisted/genotypes/{name}/... with the same four calls. `unlisted` is a namespace, not a group: only its genotype subtree answers. One prefix for the whole genotype surface beats a second spelling for the same tally, so the earlier /genotypes/{name} spelling is gone. * genotypes::resolve_named (moved out of the frequencies module, since it is genotype metadata) resolves a version, a .geno name, or a path relative to the genotype files directory; MAX_SCAN_BYTES is shared by every whole-file read. Tests limited to the feature: genotypes 3 (names-only columns, paging, location fields, marker_not_found, unlisted parity, 413), frequencies 5, frequencies unit 7.
10 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
10 dayscompute: add /compute/fitqtl for effect sizes at named markersPjotr
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not "how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits one, and reports the QTL's effect, its standard error, the model LOD and the variance explained. That is what a table of effects needs, and it is a separate endpoint because it is a separate question, asked about a handful of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model per marker, so a genome-wide version would be thousands of fits. Cap: 100 markers per request. Rows are positional: marker, chr, pos, lod, pct_var, additive, se, t, p `additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL genotype +-1, so it is half the difference between the two homozygote classes, the same convention GeneNetwork's published `additive` uses. **Two independent checks that the numbers are right.** On BXD trait 10001 (`BXD.4`), at the scan's peak: * `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same marker -- identical to every digit the CSV carries, because they are the same cross and the same phenotype. That parity is now an integration test; * `additive` = 2.394472, against the value genenetwork.org publishes for that trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away -- and that published number came from qtlreaper, a different engine. * scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's script and so not extended), builds the cross the same way, fits `fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV row per marker -- plus a one-line summary naming the strongest fit. `calc.genoprob(step=0)`: marker-level probabilities are all a marker-level fit needs. * src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr` stays text, it can be `X`), keeps the command line for `notes`. * src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as `/compute/rqtl` (by name onto the panel, parents/F1s excluded unless asked), the same `normalize`/`winsor_*` pipeline (effects come out on that scale), and the same R/qtl gate, so a fitqtl request and a scan queue against one another rather than competing for cores. * src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`). * tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the LOD parity against scanone, and the refusals (no markers, both genotype inputs). * README.md, docs/www/SKILL.md: the endpoint, what each column means, and when to reach for it instead of the scan. 305 tests pass; the same four as before do not.
12 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).
12 daysphewas: every trait peaking within a window of one positionPjotr
PublishXRef records the marker each published trait's QTL peaks at (Locus) with its LRS, which is a PheWAS waiting to happen: fix a position, find the markers within a window of it, and ask which traits peak at those markers. GET /groups/{group}/phewas?locus=rs30524230&window=10000 GET /groups/{group}/phewas?chr=9&mb=29.940961&window=10000 The window is in base pairs, centred, default 10 kb either side -- the "10K up and downstream" case. Positions come from the group's genotype marker table (the endpoint added last round), so `genotype_version` selects whose markers define the window and the answer reports them alongside the traits: an empty `traits` list is then readable as "nothing peaks that close" rather than a failure, and a ten-times wider window is offered as `next`. Measured on real data, ±10 kb around rs30524230 (chr 9, 29.94 Mb, a BXD QTL peak at LRS 460): 4 markers, 10 traits, from LOD 99.9 down to 2.68 -- ANPEP_band, then microbiome/haematology/infectious-disease traits. That cluster of 99.9s is the 460 LRS cap, not a spectrum, which the notes say. * src/db.rs (get_phewas_traits): PublishXRef rows for a set of peak markers, joining Phenotype/Publication through the existing trait_info_sql, strongest first; marker names are re-validated before they reach the IN list. * src/routes/phewas.rs: the endpoint, token-gated; window/centre validation (422 for none or negative), 404 for an unknown locus, notes for the empty cases, and a unit test for the version/group helper. * tests/phewas.rs: markers in the window, traits sorted by LRS with lod = lrs/4.604, every hit peaking at a listed marker, zero-width window, the two failure modes, and that the suggested wider window resolves. * README.md, docs/www/ai-instruction.md (new 4b: how to read a PheWAS), docs/www/operator-checklist.md: the endpoint and its reading.
12 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08* src/db.rs, src/routes: add mapping_method, tissue, /mapping-methods, ↵Pjotr
/accessions. Answers four more questions of aisearch's full_benchmark.csv from tables the API did not expose. * src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain `mapping_method`, the name of the MappingMethod row their mapping_method_id points to (qtlreaper, Rqtl, ...), or null. * src/db.rs (Dataset): datasets gain `tissue`, the Tissue their ProbeFreeze points to (e.g. "Brain mRNA"). * src/routes/mapping_methods.rs: GET /mapping-methods lists the MappingMethod table, including tools no group uses (PLINK, FastMap). * src/routes/accessions.rs: GET /accessions/{accession} resolves a GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the dataset's name, title, category (publish/probeset) and group. * docs/genenetwork_datatypes.md: MappingMethodId is a key into the MappingMethod table (the authoritative reading); GN2's code-based reading is noted as superseded. * tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new full-benchmark rows that pass (full rows 0..114; search rows run against recorded production gn3, the rest need the database), and full rows 33 (mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only GTEx v8 has TPM datasets). The untested rows are listed in the file's header comment with the reason for each. * tests, test/python: tests for the new fields and endpoints; both endpoints added to the OpenAPI and next-link crawl tests. * README.md, RELEASE_NOTES.md: document the new fields and endpoints. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05* src/routes/search.rs (search): add GET /search, the phase-A gn3Alexander_Kabui
search proxy. Validates q/type/limit at the boundary (400 with suggestions; limit capped at 100), forwards to {GN3_URL}/search/ with proper percent-encoding, and re-wraps hits in the descriptive envelope. Hits are typed as SearchHit (all 14 gn3 fields optional, unknown fields preserved via serde flatten) so /openapi.json documents a real schema while staying transparent to gn3 shape changes. gn3's 400+parsererror becomes 422 invalid_query carrying gn3's own diagnostic; every other backend failure is one 502 shape distinguished by code. * src/lib.rs, src/routes/mod.rs (routes): register the search route. * tests/search.rs, test/data/search_alcohol_phenotype.json (tests): in-process gn3 stub replays the recorded production fixture; four scenarios run offline -- passthrough+envelope, validation, 422 forwarding, limit cap.
2026-10-05Add POST /dev/token, a dev-only gn-auth token issuerPjotr Prins
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result, so local testing no longer needs a manual curl/gn-auth round trip of its own. Only ever registered -- not just rejected at request time, genuinely absent from the router and from /openapi.json -- when two new Config fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback bind (GENENETWORK4_BIND, also new). Both are config, not something genenetwork4 infers from its actual network exposure, by design: dev mode alone, or a loopback bind alone, each leave it disabled, and the check (Config::dev_token_enabled) is a plain function so the condition itself has direct unit test coverage rather than only being exercised indirectly through the route. src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0", via a proper SocketAddr rather than string formatting -- the latter would mishandle an IPv6 bind address like "::1" (needs brackets). GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to gn-auth's own published dev fixtures (flask init-dev-clients, gn_auth/wsgi.py), so this works unconfigured against an unmodified dev gn-auth instance -- confirmed live, including using the returned token against a real gated endpoint (/traits/BXD/10001), not just checking the token response's shape. tests/dev_token.rs covers all four combinations of the two settings in one sequential test function (mutating the same two env vars from more than one #[tokio::test] in a file would race, since they run concurrently by default) plus the /openapi.json visibility check. test/python/test_dev_token.py mirrors this using separate test classes instead, which is safe here since each spawns its own server subprocess with its own environment (ServerTestCase gains EXTRA_ENV for this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /traits/{dataset}/... spanning ProbeSet and Publish dataPjotr Prins
A new, additional endpoint family that addresses both trait categories the same way -- (dataset, trait) -- matching GeneNetwork2's own generic wqflask.api.router.trait_sample_data/get_trait_info, rather than the two category-specific shapes genenetwork4 already has (/datasets/{dataset}/probesets... and /groups/{group}/traits...), which are unchanged and keep working exactly as before. - db::resolve_trait_dataset(name) resolves a dataset name against ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze row, via the new get_publish_freeze_by_name) -- whichever matches decides the category. This means only dataset names are ever "guessed" across categories, not trait names/ids, which is the much smaller and safer surface: GN2's own equivalent tries ProbeSet.Name and PublishXRef.Id against the same bare trait string and silently falls back, which can return the wrong trait on a same-named collision. get_publish_freeze_by_name accepts a PublishFreeze's own Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/ Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or "BXD"). - GET /traits/{dataset}: list every trait in a dataset, tagged by category. - GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric trait against a Publish dataset is a 404 (can't match a PublishXRef.Id), not a 500. - GET /traits/{dataset}/{trait}/values: that trait's per-strain values. TraitValue and ProbeSetValue are already identical in shape, so this unifies with no information loss, unlike the metadata endpoint above. Metadata responses are a tagged union ({"category": "probeset"| "publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than a lossy common-fields schema -- the two share little beyond mean/locus/lrs/additive. utoipa generates this as a oneOf with a category discriminator, verified against /openapi.json. Gated the same way as /datasets/{dataset}/probesets..., extended to Publish data for the first time: /groups/{group}/traits itself is still not gated, but reaching the same data through /traits/{group} now requires a token and checks PublishFreeze.public/confidentiality. No gn-auth-side change was needed -- POST /auth/data/authorisation already dispatches *Publish-suffixed dataset names to its phenotype resource resolver (gn_auth/auth/authorisation/data/phenotypes.py), the exact mechanism this integration already relies on for mRNA data. auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to routes::auth (pub) so routes::traits can reuse it instead of duplicating the string. tests/traits.rs and its Python mirror cover both categories (including both ways of addressing a Publish dataset), the 401/403/404 cases, and the non-numeric-trait-against-Publish-dataset edge case, run against a real gn-auth instance. The Publish branch's deny path itself is only exercised via gn-auth's "unlinked dataset" fallback (already proven for ProbeSet) -- no non-public PublishFreeze exists in the database this was tested against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add ProbeSetXRef endpoints: datasets, probesets, probeset values.Pjotr Prins
The expression (ProbeSet) counterpart to the PublishXRef endpoints: ProbeSetXRef has no single Id column of its own (its primary key is DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)), so a trait here is addressed by dataset name plus ProbeSet.Name instead of a single numeric trait id. * src/db.rs (Dataset): New struct, one ProbeSetFreeze row. (get_group_datasets): New function. Joins ProbeSetFreeze and ProbeFreeze by group Id, matching GeneNetwork2's wqflask.api.router.get_datasets_for_group. (get_dataset_by_name): New function. Matches Name, Name2, FullName, ShortName or numeric Id, matching wqflask.api.router.get_dataset_info's ProbeSet-branch matching logic. A dataset name is globally unique on its own, unlike a group's traits, so this isn't scoped by group. (ProbeSetInfo): New struct, a probe's ProbeSetXRef mapping stats plus its ProbeSet gene annotation. (get_dataset_probeset_info): New function, matching wqflask.api.router.get_trait_info's ProbeSet branch. (ProbeSetValue): New struct. (get_dataset_probeset_values): New function, matching wqflask.api.router.trait_sample_data's ProbeSet branch. * src/lib.rs (DatasetProbesetLookup): New enum, the expression-dataset equivalent of GroupTraitLookup -- distinguishes a missing dataset from a missing probeset in an existing dataset. (group_datasets): New handler for GET /groups/{group}/datasets. (dataset_by_name): New handler for GET /datasets/{dataset}. (get_dataset_probeset_info_result, dataset_probeset_info): New function and handler for GET /datasets/{dataset}/probesets/{probeset}. (get_dataset_probeset_values_result, dataset_probeset_values): New function and handler for GET /datasets/{dataset}/probesets/{probeset}/values. (app): Register all four. * tests/group_datasets.rs, tests/dataset_by_name.rs, tests/dataset_probeset_info.rs, tests/dataset_probeset_values.rs, test/python/test_group_datasets.py, test/python/test_dataset_by_name.py, test/python/test_dataset_probeset_info.py, test/python/test_dataset_probeset_values.py: New files. Br_U_0803_M (dataset) / 100001_at (probeset, gene symbol Cd3g) in BXD are real rows used as fixtures. * tests/openapi.rs, test/python/test_openapi.py: Check the four new paths are in the generated spec. * README.md: Document the four new endpoints and list the new test files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Document all endpoints with utoipa summaries; add POST /api/v2/correlation stubAlexander_Kabui
Summaries appear in the generated /openapi.json and make the spec readable on its own. The correlation endpoint is a stub: it validates the request shape and returns an envelope with canned data and a null next pointer, establishing the contract ahead of the real engine.
2026-10-03Add foundation structure: config layering, response envelope, routes modules.Alexander_Kabui
* src/config.rs: New module. Layered configuration, defaults -> GN4_CONF -> GN4_SECRETS -> environment, mirroring GeneNetwork3's SQL_URI convention so the two services can share configuration. Missing config files are ignored. * src/response.rs: New module. The descriptive success envelope (data + next actions) from the foundation design note (topics/gn4/gn4-api-foundation.gmi). Not yet adopted by existing endpoints; that is a deliberate wire-format change to agree on before it ships. * src/routes/mod.rs, species.rs, groups.rs: Move the species and groups handlers out of lib.rs into one module per domain, per the foundation structure. Handlers are pub; lib.rs just registers them. * src/lib.rs (app): Register the moved routes. hello_world and version predate the domain split and stay as top-level utility routes. * src/db.rs (sql_uri): Resolve through config::Config instead of reading the environment directly. * src/main.rs: Bind the port from config::Config.