about summary refs log tree commit diff
path: root/scripts
AgeCommit message (Collapse)Author
2 dayscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
2 daysplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
2 dayscorrelation matrix: PCA, behind its parity testPjotr Prins
`"pca": true` on the matrix request now folds the correlation matrix into principal components: eigenvalues descending, explained variance ratios, loadings per kept trait, and the eigenvalues' sum reported so a caller can check the decomposition is sane instead of trusting it. The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and this is for a collection, not a dataset. Two decisions from ai-memory #103, both visible in the reply: * **Traits with any null cell are dropped and named**, with the reason. A component from a matrix with holes is not defined; GN2 silently mutates its shared-sample list instead, and nobody can reproduce that component. * **The sign is fixed**: within a component the largest absolute loading is positive. Eigenvectors are defined up to sign, so without a rule two correct implementations disagree and no parity test could be exact. Verified against numpy.linalg.eigh on the same r the reply carries (scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007): eigenvalues: worst relative difference 3.2e-14 loadings: worst absolute difference 9.99e-16 trace: 5.000000000000 for 5 traits Unit tests cover the identity (unit eigenvalues), the closed form for 2x2 (1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The monitor gained `compute-correlation-pca`, whose judge checks that the eigenvalues sum to the trait count and the ratios to 1.
2 daysgemma-compare: make the parity gate run, and say what it is measuringPjotr Prins
The script could not have caught anything: it posted without an Authorization header (401 against a token-protected server), asked for a trait's values as `{"name": n, "values": [<trait name>]}` (422: samples need numbers), read `columns`/`rows` from the envelope's top level (they are under `meta` and `data`), and compared relatively -- which calls beta 0.0131 against 0.0157 a 16% disagreement on a marker where both runs agree to three decimals. Now: `--token`/`$GN4_TOKEN`, trait values fetched from the API, the envelope shape handled, and an ABSOLUTE `--tolerance` (default 0 = exact). It fetches, compares and reports through the server as deployed. Verdict on the current deployment, BXDPublish 10001 over 34 individuals: rows: gemma-wrapper 21055, ravanan+gemma 21055, shared 21055 worst absolute difference: rs230640887 beta -1.015599 vs -1.018739 (0.00314) --tolerance 0.005 -> agree; --tolerance 0 -> 63 163 values beyond it So the row difference this started from is gone: same markers, same count. The residual is that the two paths run *different GEMMA builds* -- the wrapper's log names gemma-gn2-0.98.5-8cd4cdb, the ravanan manifest gemma 0.98.6 -- which moves beta/se/p_lrt in the third decimal while leaving af, n_miss, l_mle and the marker set identical (and the top hits in the same order).
3 daysqtl2: permutations for the LOD thresholdsPjotr
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply gains meta.permutation {n, seed, significant, suggestive, values} -- the genome-wide maximum of every permutation, so a histogram needs no second call. `data` stays the positional rows array on purpose: /plot/rqtl posts it straight back and the monitor counts its rows, so the null went beside the rows rather than around them (a generic Envelope::with_meta/Meta::extra, skip_serializing_if, so no other endpoint's response changes). Thresholds come from R/qtl2's scan1perm over the same genotype probabilities, kinship and covariates as the scan, with chr_lengths from the map -- without those it derives chromosome lengths from marker positions and the numbers stop being comparable with R/qtl's. The permutation CSV is written in R/qtl's two-column shape so one parser and one percentile routine serve both engines: PermutationResult::from_values is now shared with /compute/rqtl instead of duplicated. Semantics, each with the note that says it: * scan1perm shuffles phenotypes across individuals -- right for an outbred cross, only approximate for a selfing RIL panel, where qtl2perm is the correct package (not installed here, and not packaged); * perm_strata takes a case attribute name (sex, Epoch, ...) and permutes within it, because shuffling across sexes inflates the null; * fewer than 100 permutations gets a note that the 0.05 quantile is a quantile of too few values; more than 10 000 is refused, since each permutation is a whole genome scan; * a seed is reported always -- without one, two identical requests would give different thresholds. Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s, 1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at 1000 on the raw probe) and are reproducible: the same seed gives the same numbers on 1 and 4 cores. Four things that only surfaced by running it, all commented where they bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so opt$perm_out was always NULL and the file landed somewhere the caller did not look); perm_strata must be a named character vector, not a factor; the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the permutation index first, and a first-field parser reads that as the LOD); and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not 0.67. bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging meta.permutation) so a deploy that loses the R path shows up as a red check rather than a surprise.
4 dayslog: name the person and the credential, and resolve handles from the registryPjotr Prins
The access log said who (`user`) but not with what: with short tokens a request can be made by a shared credential, so "which person" and "which credential" are now separate fields -- `scheme` ("jwt" | "opaque") and, for an opaque token, `token` (its registry handle, e.g. `agent-ravanan-3`). Never the token or its hash, and the path still carries no query string. The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is self-describing rather than relying on journald's prefix. Hand-rolled civil-from-days rather than a new crate: one format, and no Cargo.lock drift. scripts/gn4-log.py reads such a log and shows it as a table with the handle joined against tokens.sqlite: the subject as the registry has it now (flagging a handle whose subject has been repointed), and the token's status today -- [revoked], [expired], [unknown] -- which is how "an agent was working at 14:02" is told apart from "an agent was working at 14:02 with a credential revoked since". It reads the registry read-only, passes non-JSON lines through in table mode (the process log interleaves backtraces), and offers --json and --summary for jq and for counting. CallerSlot now carries the whole Caller (scheme, id, subject) rather than just the subject string.
4 daystokens: short opaque API tokens in a single tokens.sqlitePjotr Prins
A second credential format beside gn-auth's JWT, for agents and scripts: `gn4ai_` + 56 body characters + 2 checksum characters = 64, from an alphabet with no look-alikes, so a typo is refused before the registry is consulted. Opaque by design -- nothing is stored in the token; every fact lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode 0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row, takes effect on the next request, and a token in a log line reveals nothing; lose the registry and every short token stops working. Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and GET /tokens/who are behind a router layer that refuses any caller whose TCP peer is not 127.x.x.x/::1, and any request claiming to come through a proxy. A layer rather than a per-handler check, so nothing of a remote request is parsed before it is refused; there is no configuration switch, because minting credentials must never be reachable from the network. A short token reads public data only: every token-only endpoint, and `#[gnauth(secured)]` endpoints for public rows; a confidential dataset is refused with a 403 that names the JWT route, since gn-auth cannot validate an opaque token (the on-the-fly-bridge and registry-grant alternatives were both rejected for v1). scripts/gn4-token.py create|list|revoke|who drives the routes, so there is one implementation of the format and the revocation rules; the token is printed exactly once. Tests: the localhost guard over all four routes (remote peer, no peer, proxied header), and mint -> authenticate -> who -> revoke -> 401.
5 daysravanan+gemma: call gemma itself, not gemma-wrapperPjotr Prins
gemma-wrapper was doing two things ravanan now does or GEMMA already does: the parallel fan-out (ravanan's CWL scatter replaced it long ago, which is why every job passed --no-parallel) and LOCO orchestration -- gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and takes the annotation file it needs. What was left was a content-hash K cache and a --json record with results packed into .tar.xz archives. So cwl/gemma/gemma-loco.scm now runs two gemma invocations per chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix> -lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat, deterministic names. The kinship matrix travels from one step to the other as a CWL File output, and each job's .assoc.txt and .log.txt are the workflow's outputs -- so compute_ravanan reads files instead of parsing a wrapper JSON record and unpacking archives, and the pve note comes straight out of the jobs' logs. Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows -- 636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper also needed Ruby and pfff on PATH, and named its outputs with the extension doubled (...1.assoc.txt.assoc.txt). /compute/gemma keeps gemma-wrapper, deliberately: that is what makes the two endpoints comparable rather than two copies of one stack. scripts/gemma-compare.py runs the same body through both and reports row-level differences and timings. The gemma-wrapper content-hash K cache is the one thing given up; a repeat request recomputes (271 ms of kinship on this fixture).
5 daysgroups: the design in /groups/{group}, and a drift check against GN2's ParInfoPjotr
Two follow-ups to the parental-design endpoint. /groups/{group} now carries the headline in its notes -- "Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default" -- plus a next link to the full design. "Which cross is this?" is two strain names, and needing a second request to see them was the only reason the endpoint had to be called first. A group with no metadata says so in the same note, pointing at parents_and_f1s.json and at `individuals.exclude`, rather than leaving the reader to wonder why nothing was dropped. The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it drifted once already, silently: the paternal/maternal labels were swapped against GN2's unpacking, and nothing noticed because the exclusion rule takes both strains regardless of role. scripts/check-parental-metadata.py parses ParInfo with ast (no gn2 import) and compares it group by group, reporting disagreements per role and treating groups the JSON knows but GN2 does not as informational. tests/parental_metadata.rs runs it while a GN2 checkout is present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at which point the JSON is the only copy and there is nothing to drift from. It reports 23 groups reproduced, 1 extra (BXD-Longevity).
5 daysplot/bnw: two SVG figures, drawn the way /plot/gemma draws onePjotr
Post the reply from /compute/bnw back and get an SVG: the same plumbing as /plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/ svg+xml, the plotter's line in x-plot-summary), nothing recomputed. Two figures, because one cannot say what the other can: * the network (default): nodes on a circle, one arrow per edge that survived the threshold, arrow width and label = the edge's posterior, and the below-threshold edges as faint dotted lines -- so a reader sees what was *considered* as well as what survived, which is half of why the posterior table exists; * ?kind=posterior: every candidate edge as a cell in a matrix, dark = high posterior, outlined cells are the ones above the threshold. The diagram cannot show that an edge came within 0.02 of the threshold; this can. Pure-Python SVG, standard library only: no graphviz in the deployment. A circular layout is less pretty than a force-directed one and is deterministic, which matters more for a figure that accompanies a JSON result. Node labels keep the distinguishing tail of a long id, with the full name beside the node. Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the second figure. tests/plot.rs covers both figures plus the 422s (not a BNW result, unknown kind).
7 daysplot/bulklmm: the 0.05 and 0.01 lines, never 0.10 unless askedPjotr
Checked what the plot actually draws: it has never drawn the alpha 0.10 line -- the plotter matches 0.05 and 0.01 only. What lists all three is the reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"), which is where the 0.10 came from. Still, the point stands on its own: 0.10 is the loosest bar in the set and drawing it invites reading it as the line to clear. So the defaults are stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request (?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's Bonferroni, which is opt-in for the same reason: a reference line is an annotation, not a result. tests/bulklmm.rs counts the lines: exactly two by default, and no alpha 0.10 label.
7 daysplot/gemma: no reference line by default; Bonferroni is one more labelled linePjotr
A line on a plot is an annotation, not part of the result, so the default is now no line at all. GEMMA's Bonferroni becomes opt-in like any other: ?bonferroni=true GEMMA's own line, labelled ?thresholds=3.65&threshold_labels=… another engine's, labelled (nothing) just the scan Which means a reader sees exactly the lines someone chose, each with its source in the label -- and the accidental "this plot says significant" impression of a default convention line is gone. The plotter's summary and its <desc> no longer mention Bonferroni unless it was drawn. tests/plot.rs: the plain plot has neither line; with thresholds the two BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true GEMMA's own line appears.
7 daysplot/gemma: draw thresholds computed elsewhere, labelled with their sourcePjotr
GEMMA has no permutation test here, so its plot carries a Bonferroni line. A threshold from /compute/bulklmm belongs to a different null and must not look like GEMMA's, so it is passed in and named: POST /plot/gemma?thresholds=3.50,3.65 &threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05 Each is drawn dashed in the series colour with "label: value" beside the line, with GEMMA's Bonferroni line still there, and the plotter's x-plot-summary names the reference lines it drew. A threshold that is not a positive number is a 422 invalid_threshold rather than a silent omission. /compute/bulklmm's reply offers that href already, with its alpha 0.05 threshold filled in -- so "same values, two models, one picture" is one link from the scan, and the labels keep the two nulls apart. tests/plot.rs: the plain plot has no bulklmm line; with thresholds both are drawn and labelled; a non-numeric threshold is refused.
7 daysbulklmm: report the variance components, and say what a boundary meansPjotr
"is the kinship term doing anything?" should be answerable from the reply. The driver now prints h2 and the residual variance under the null model, and the reply carries both: Variance components: h2 = 0.000000, sigma2_e = 0.402171, estimated once under the null (intercept) model from y and K, not per marker. With a few dozen phenotyped individuals these estimates pile up at the boundaries 0 and 1 -- the likelihood is flat there -- so read h2 as "kinship matters a lot / not at all" rather than as a precise proportion. At a boundary (h2 ~ 0 the model is ordinary regression, which is why its LODs can match R/qtl's to the digit). Checked with a strain-ID-driven trait (values = rank of strain), which has h2 ~ 1 by construction: BulkLMM reports h2 = 1.000000, sigma2_e = 3e-06 and a peak of LOD 3.1 -- the kinship term absorbs the strain structure -- while R/qtl, which has no kinship term at all, reports a spurious peak of 5.06 on the same values. That is the discrimination the h2 column was missing.
7 daysdocs: what BulkLMM models, what its cores do, and how to package itPjotr
docs/bulklmm.md records the three things that are easy to get wrong when a BulkLMM result is quoted: * where it stands against the other engines -- Haley-Knott with no kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA), LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship LMM whose permutation thresholds are rotation-based and therefore cheap -- the only engine here that can say "genome-wide significant at alpha 0.05" against a null that respects relatedness; * what it is not: global kinship rather than LOCO, so its p-values are not GEMMA's; h2_null says how much relatedness it found (near zero here, which is why it reproduces R/qtl's 9.59 where GEMMA differs); one degree of freedom, so multiparent scans stay with /compute/qtl2; and no standard errors in 1.2.2's scan(), so the rows are marker, chr, pos, lod, beta, h2; * the cores, measured: Threads.@threads is in bulkscan.jl and bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is serial by design and the visible parallelism was BLAS (96 threads 10.5 s against 4 threads 7.7 s). What the driver does about it -- exact marker blocks, cached matrix and kinship, capped BLAS, and the breakdown printed into the reply -- with the numbers (20.9 s at one block against 18.3 s at four, same LOD 9.589938). Also the packaging picture, since it is not obvious from the outside: guix-bioinformatics has a working julia-bin and a julia-build-system, but only three of BulkLMM's closure are packaged, so a pure-Guix build is 35-45 recipes; the pragmatic route is julia-bin plus a depot built once from the pinned Manifest. scripts/prepare-julia-depot.sh is that step, written down, and the depot stays small and GPU-free (79 packages, 139 MB) with JULIA_PKG_OFFLINE=true at request time. The plot test learns that both thresholds are drawn, not one.
7 daysbulklmm: exact marker-block parallelism, and what the cores are actually doingPjotr
Reported: a BulkLMM run does not look multicore. It is not a misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint calls -- has no `Threads.@threads` at all; the package threads `bulkscan()` (many traits) and bulkscan_helpers, not scan. The parallelism that was visible before was BLAS (OpenBLAS defaulting to all 96 cores here), which is why `user` time ran 4x `real` while Julia threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s. * The driver splits the markers into independent blocks and scans them on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact: the LOD for each marker is independent given the null variance components, which depend on y and K only. Verified identical -- peak LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks -- and permutation thresholds combine by taking each permutation's maximum across blocks before the quantiles, which is what a genome-wide threshold is. * The parsed matrix and the kinship are cached per panel (read 1.3 s -> 0.4 s, kinship 0.9 s -> 0), because both depend only on the cross. * The summary in `notes` now says what ran: julia threads, blocks, BLAS threads, and the read/scan split. * The hand parser is back: CSV.jl into a DataFrame is an order of magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s). * BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for one scan). Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks 18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of the run and each block re-estimates its variance components. Where the cores pay is bulkscan() -- many traits in one process, threaded over trait blocks -- which is the shape the next endpoint wants; documented in the README rather than left as folklore.
8 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.
8 daystraits: genotypes at a confirmed peak, and the plot of themPjotr
Returning genotypes is the one thing this API otherwise never does, so it is fenced and the fence is the feature: GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker> * the server re-runs the R/qtl scan for the trait -- the same machinery and gate as /compute/rqtl -- and takes the peak from that run, rather than trusting a client that names a marker; * the marker must lie within ?window= Mb of that peak (default 10), or the answer is 422 not_the_peak quoting the peak's marker, chromosome, Mb and LOD; * the peak must reach ?min_lod= (default 3), or 422 not_a_hit; * only animals that have a value for the trait are listed, and only this marker's calls. Nothing else of the matrix leaves. On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals across A 110 / B 8 / H 74 -- the two a class of eight being exactly the thing worth seeing. POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's `animals` (or its `genotypes` map) with the trait's values and get an SVG with one point per animal, genotype class across, value up, class means marked. Nothing is recomputed -- the plot is of the numbers the client sends, like the other plots. New script scripts/plot-genotypes.py (stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key. tests/peakgenotypes.rs: the peak's calls for animals with values (with the counts agreeing with the rows), the SVG, and a marker on another chromosome refused with the peak quoted. Also keeps /compute/rqtl's error order: naming both a group and a file is "ambiguous" even when the request also forgot its values.
8 daysuploads: map an uploaded genotype by name, and clean up after itPjotr
Three follow-ups from putting an upload through the whole loop. * /compute/qtl2's `genotype_file` branch only accepted positional `trait_values`, i.e. it demanded the file's column order -- which a caller with an uploaded population does not have. It now takes `samples` (name and value) or a `basket` and aligns them onto the cross by name, leaving individuals without a value as missing. Positional `trait_values` still work for a file whose order is known. * DELETE /uploads/genotypes/{name} now removes the .geno it wrote on demand: a file nobody can name is a file that confuses the next operator. * scripts/example-genotype-upload.py spreads its markers across chromosomes. One chromosome of markers is a poor example -- a leave-one-chromosome-out kinship has nothing to leave out, so the scan came back with no rows at all, which looked like a broken upload and was really a property of the example. 20 markers over the genome map cleanly (19 rows, the 20th used for the HMM), and the README says to use "kinship": "overall" if a file really has one chromosome.
8 daysuploads: a genotype matrix that updates an existing populationPjotr
Tab-delimited, header names the individuals from column 4, then marker, chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own SQLite file -- nothing goes near MariaDB -- and written out on demand as a GN .geno carrying the base's @-metadata, so the R scripts read exactly what was uploaded. An upload updates a population, it does not invent one, and the checks say so: * the base version must exist (resolved through the same genotypes::resolve_named the rest of the API uses); * at most 50 individuals the base does not have, and at least one that it does -- no overlap is a different population, not an update; * at most 200 000 markers, and the refusal says the sysadmin has to organize a file that size, because this is a request path, not an ingest; * the values must be the same kind as the base's: letters, whole numbers or probabilities. A letters panel holding 0/1 would be read as the wrong thing silently, so it is refused naming both kinds. GET /uploads/genotypes lists them, /{name} its metadata (overlap, new individuals, kind, digest), /{name}/markers pages the marker table, and /{name}/geno writes the file and returns the path for `genotype_file` on the compute endpoints. DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update -- a subset of markers, a few individuals dropped and added -- and tests/genoupload.rs builds its own from the deployment's BXD.4 for the same reason: upload, page the markers, write the .geno and check the layout, delete, then break each rule (no overlap, numbers in a letters panel, commas) and check the refusal says which. Unit tests cover the three value kinds, missing cells, ragged rows, a non-numeric position, the 50-new-individual limit, and the .geno layout (Chr first, then the marker -- GN's order, not the upload's).
8 daysmatrices: upload new traits and covariates as one named tablePjotr
The other half of changing data: bringing new columns in. A matrix is a delimited table -- a header row, then one row per individual, the first column naming them -- stored in the uploader's own SQLite file and checked against an existing genotype version's panel before it is accepted. * POST /matrices {name, genotype_version, contents, kinds?}: parse the table (tab, comma or semicolon, guessed from the header), then check it against that version's individuals. The count must match and every name must be in the panel; 92 of 93, or a name the panel does not have, is a 422 that says which. A mapping that silently drops individuals is worse than an error. * kinds marks columns that are covariates; everything else is a trait. * Each column becomes a basket named <matrix>.<column> -- a dot, not a slash, because a basket name has to survive a URL path segment -- so a column maps on any engine with {"basket": "my-traits.bw"} and inherits the revision history and digest without a line of new mapping code. * GET /matrices lists them, GET /matrices/{name} their columns and basket names, GET /matrices/{name}/values the table back (parsed values, null where missing), DELETE removes the matrix and its columns. Two bugs this found in the basket store, both real: * SQLite has foreign keys OFF by default, so the ON DELETE CASCADE never fired: deleting a basket left its rows behind, rowids were reused, and the next create hit "UNIQUE constraint failed: basket_version.basket_id, revision". The pragma is now set, and deletion removes the dependants explicitly, because files written before it have orphans. * The first attempt named a column's basket "<matrix>/<column>", which the basket-name rule (rightly) refuses. The dot keeps both rules. scripts/example-matrix.py writes a table from a genotype file's own header with deterministic values, so the fixture passes the panel check by construction -- tests/matrices.rs builds its own for the same reason and walks the loop: upload, read back, map a column with R/qtl2, delete, plus the three refusals (too few individuals, a name not in the panel, an unknown version).
8 dayscompute/qtl2: multiparent crosses, by namePjotr
A multiparent population cannot come from a .geno. That file carries two codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and the collapse is not invertible, so DOL's 194 DO animals cannot be fitted as a DO however much reference data appears. What can be is a cross that ships its own qtl2 control file with founder genotypes, so the endpoint now takes one by name: {"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...} * resolve_control_file looks in genotype/rqtl2/{name}.json, then {name}/{name}.json (do/do.json), then {name}.json under GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps and founder_geno headers without converting anything. It reports crosstype, founders and whether the cross is multiparent, and refuses a multiparent crosstype with no founder_geno rather than silently scanning it as biallelic. * Samples are matched to the cross's own population by name and the strays are counted in notes; rqtl2_cross cannot be combined with group or genotype_file. * The gate is given individuals x founders, because a multiparent scan fits one coefficient per founder allele at every marker. * Autosomes only now applies to shipped crosses as well: the DO's X is left out (369 markers) instead of being fitted without a sex covariate, and the script subsets with cross[, chr = autosomal]. Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s (cached), scan 21 s. scan1 runs on one core for a multiparent cross because its forked path stalls on allele probabilities here -- 25+ minutes at 8 cores with no CPU consumed, against 21 s serial. Two bugs found by running an actual published trait through it: * do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was reported at 0.0 Mb. The map's unit is now detected from its largest position and printed. * The phenotype file was read with the default whitespace separator, so a label with a space ("DO::10001 (WBC)") split into three fields and read.table stopped with "line 1 did not have 3 elements". Both the pheno and covariate reads are tab-separated now. The R error extraction also joins R's wrapped continuation line, so a complaint reads as a sentence rather than "Error in scan(...) :".
9 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
9 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
9 daysqtl2: the engine script and its packagingPjotr
Phase 0 of the R/qtl2 work, so the engine can be exercised before any endpoint exists. `r-qtl2` (0.42 in this channel) plus `r-digest`/`r-jsonlite` go into the dev shell's propagated inputs; without them R has to be handed a list of 594 store paths by hand to load qtl2 at all. `scripts/qtl2_scan.R` is written for this API rather than copied from GeneNetwork3's `scripts/rqtl2_wrapper.R`, which cannot be trusted as a reference: its `"LMM"` and `"LOCO"` branches call `scan1` with identical arguments, its covariates are a `covar <- NULL; # TODO` stub, and its only tests mock the file-writing helpers -- nothing ever reads a cross. The script reads a qtl2 control file, a two-column phenotype file and optional covariates, and writes `marker,chr,pos,lod` (Mb, so the shape matches /compute/rqtl and /plot/rqtl draws it unchanged). Genotype probabilities and the kinship matrices are cached in --cache-dir keyed by the control file's content, because a DO cross costs 536 s of genotype probabilities on four cores against 11 s of kinship. Four things the cross-file plumbing cost, all found by running it: * `read_cross2` joins the control file's directory onto whatever paths it finds, so an absolute path becomes `<dir>/<absolute>` and dies. The server has to write the control file *beside* its data; * our generated genotypes are markers-in-rows, which needs `"geno_transposed": true` -- without it qtl2 reads the individuals as markers (93 individuals appear as 93 extra markers, and every marker is "not in both genotypes and maps"); * the genotype strings need a `"genotypes"` map (the GN letters @mat/@het/@pat -> A/H/B -> 1/2/3); without it every value is "treated as missing" and the scan comes back flat (peak LOD 0.14); * with those fixed, a cross converted from `genotype/BXD.4.geno` scans in ~1 s and agrees with /compute/rqtl on the same panel and trait: qtl2's LMM peaks on chr8 at ~111 Mb, LOD 3.50, where R/qtl's HK `scanone` peaks on chr8 at ~69 Mb, LOD 2.93 -- same chromosome, different model, which is the cross-check the plan asks for. Not yet: the Rust side (`src/qtl2.rs`), the `/compute/qtl2` route, the genotypes-listing `qtl2` flag and the converter that writes the control file. Those are Phase 1-3.
9 daysrqtl: map the cM-only genotype filesPjotr
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and none of them could be scanned. GeneNetwork3's wrapper (copied verbatim here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts the samples at a hardcoded column 5. On a cM-only file that is one column too far, so the genotype block loses its first individual and the whole scan dies in `do.call(rbind, ...)` with "number of columns of result is not a multiple of vector length" -- a message that says nothing about the cause. The wrapper now picks the map column from whichever of the requested scale and the file's own columns exists (falling back, and saying so), and starts the samples right after the position columns; the sex default and the sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same assumption and got the same fix. This is the second deliberate deviation from GeneNetwork3's script, after the sample-count guard, and both are commented as such at the code. * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the first sample column are computed, not assumed. * tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb` dropped -- same panel, cM only. * tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six markers come back (the failure mode was five-vs-six columns, not an error message that pointed at the genotype file). * src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards `--scale`; it does, and now falls back, so the request field says what `pos` is in. Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499 markers) and BXD.4 (3 811) are unchanged. 307 tests pass; the same four as before do not.
9 dayscompute: fit covariates in /compute/fitqtl tooPjotr
`fitqtl` takes a `covar` argument, so the effect it reports can be the one *after* conditioning -- which is usually the number worth quoting for a peak that overlaps a known confounder. `covariates` now works on /compute/fitqtl exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute by name, inline values, explicit map, same coding and same note), so the two endpoints report a covariate identically. The R side does what R/qtl needs: the phenotype file carries one column per covariate after the trait, `--covarstruct` says which are categorical, and the script dummy-expands those (k-1 against the first sorted level, missing values left missing). Two things it has to get right: * `model.matrix()` drops rows with a missing value, which silently misaligns the covariate frame against the cross -- so the dummies are built directly (`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`; * a covariate can be perfectly spread across the panel and **constant among the individuals a trait covers** -- BXD's `Status` is exactly that -- and R/qtl then stops with `Lapack routine dgesv: system is exactly singular`. Columns that do not vary among the phenotyped individuals are left out, and the reply names them, the same rule the GEMMA covariate file applies. Measured on BXD trait 10001 at its peak, `BXD.4`: no covariate LOD 2.9310 additive +2.394 32.8% var with Status LOD 3.3602 additive +2.463 36.6% var with an inline numeric covariate LOD 2.9373 additive +2.361 The LOD is no longer the scan's once a covariate is fitted -- it is a different model -- which the docs now say, next to the no-covariate parity check that still holds. * scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate column rule, and the dropped names in the summary line. * src/fitqtl.rs: writes the covariates into the phenotype file and the struct file, and passes both. * src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so the coding and the complete-case count match /compute/rqtl. * tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD) and an unknown attribute refused with the list of valid ones. * README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD stops matching the scan's. 306 tests pass; the same four as before do not.
9 dayscompute: add /compute/fitqtl for effect sizes at named markersPjotr
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not "how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits one, and reports the QTL's effect, its standard error, the model LOD and the variance explained. That is what a table of effects needs, and it is a separate endpoint because it is a separate question, asked about a handful of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model per marker, so a genome-wide version would be thousands of fits. Cap: 100 markers per request. Rows are positional: marker, chr, pos, lod, pct_var, additive, se, t, p `additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL genotype +-1, so it is half the difference between the two homozygote classes, the same convention GeneNetwork's published `additive` uses. **Two independent checks that the numbers are right.** On BXD trait 10001 (`BXD.4`), at the scan's peak: * `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same marker -- identical to every digit the CSV carries, because they are the same cross and the same phenotype. That parity is now an integration test; * `additive` = 2.394472, against the value genenetwork.org publishes for that trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away -- and that published number came from qtlreaper, a different engine. * scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's script and so not extended), builds the cross the same way, fits `fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV row per marker -- plus a one-line summary naming the strongest fit. `calc.genoprob(step=0)`: marker-level probabilities are all a marker-level fit needs. * src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr` stays text, it can be `X`), keeps the command line for `notes`. * src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as `/compute/rqtl` (by name onto the panel, parents/F1s excluded unless asked), the same `normalize`/`winsor_*` pipeline (effects come out on that scale), and the same R/qtl gate, so a fitqtl request and a scan queue against one another rather than competing for cores. * src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`). * tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the LOD parity against scanone, and the refusals (no markers, both genotype inputs). * README.md, docs/www/SKILL.md: the endpoint, what each column means, and when to reach for it instead of the scan. 305 tests pass; the same four as before do not.
11 daysplot: draw a mapping, and suggest it from the mappingPjotr
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a saved mapping response into an SVG Manhattan panel. They are now endpoints, and the mapping that produces the response suggests them: POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml POST /plot/gemma same body as /compute/gemma -> image/svg+xml Taking the mapping's own body rather than the mapping's output means one call gets you the picture, and the plot cannot describe a run you did not do: the handlers call the compute handlers and hand the resulting envelope to the script, exactly as the scripts' docstrings describe (they read `data.markers`, `data.permutation` and the notes' strain counts). The mapping is re-run, but through the engines' own caches -- seconds on BXD. The reply is an SVG, not the JSON envelope: an image is not JSON, and the plotter's one-line peak summary rides back in `x-plot-summary` so a caller that cannot see the picture still learns where the peak is. Both compute responses gain the matching suggestion: `actions` carries "the same request, plotted" (built from the request that ran, like the permutation suggestion), so an assistant can map, look, and only then decide whether the peak is worth the 100-permutation run. * src/plot.rs: run the configured script over the envelope, read the SVG back, clean up; a missing script is a 502 naming interpreter and script, not a panic. * src/routes/plot.rs: the two handlers (token-gated), delegating to compute_rqtl/compute_gemma so validation and alignment stay in one place; `x-plot-summary` sanitised into a header. * src/routes/{rqtl,gemma}.rs: the plot suggestion, before the permutation one. * src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER. * tests/plot.rs: the suggestion resolves to a complete SVG with a peak summary, and an invalid mapping request is the mapping's own 422. * tests/{rqtl,phewas}.rs: find suggestions by href rather than position. * README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md (new "Plotting": python3 must be on the server's PATH).
11 daysrqtl: refuse a phenotype whose sample count is not the cross'sPjotr
Asked to map a 9-strain trait, the wrapper scanned it against the 6-individual fixture cross and returned six markers with LOD scores and `state: complete` -- a plausible answer belonging to nobody. R recycled and truncated the values silently, and nothing downstream could tell. GeneNetwork3's copy of this script has the same hole. The cross-file format carries no per-sample names, so the count is the only check available; order remains the caller's responsibility (and the reason values and genotype file must come from the same panel). With the check, the same request is a 502 rqtl_failed quoting both counts, and the correctly-shaped request still returns the documented LODs. * scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of the cross, and stop. * README.md (/compute/rqtl): the second fix relative to gn3, and what stays the caller's job. * docs/www/ai-instruction.md (4a): values and genotypes must describe the same individuals, one for one, in the same order. * docs/www/operator-checklist.md (Compute): what genotype files actually exist on this box (test fixtures, 198 BXD individuals) and why a database trait, with its own strain subset, cannot be mapped against them.
12 daysai-instruction: issue a token per user without gn-authPjotr
The page told the operator to mint with POST /get-test-token, which is gn-auth's dev password grant and does not exist in the deployment being set up. Issuing a second user's token also had no good answer: the stub minted exactly one token per process, so a new user meant starting a throwaway instance, grabbing --token-file and killing it. So add --mint-only: mint for --subject, print, write --token-file and exit without listening. Verification only depends on the key (committed, so shared by every instance) and on iss, which now falls back to AUTH_SERVER_URL -- a token minted this way verifies against the long-running stub untouched. * scripts/gn-auth-stub.py (main): --mint-only; --issuer defaults to AUTH_SERVER_URL before the port-derived default. Docstring says how a second user's token is issued. * docs/www/ai-instruction.md (1. Get a token): the operator's command is the stub, not /get-test-token; the token lasts whatever --expires-in said rather than an hour, and there is no early revocation. (4. What is public): minting is operator-side. Appendix: "Token issuance" rewritten around the stub -- public-data-only, expiry as the only revocation, sub as attribution, one token per user -- with gn-auth kept as what replaces it later; the stub is no longer listed as a dev leftover to delete, but as the auth server that must never be routed publicly. * README.md (Tests): mention --mint-only next to the stub's other flags.
12 daysgn-auth-stub: --expires-in, document the public-data-only tokenPjotr
A stub token had a fixed 24h lifetime, and nothing said what an ungranted one can reach. Both matter for the case that is not a test run: handing a caller or an agent a long-lived token that may only read public data. An ungranted stub token already is exactly that -- the restriction lives in the authorisation lookup genenetwork4 consults for anything not public, not in the token's scope -- so only the lifetime was missing. * scripts/gn-auth-stub.py (duration): new helper, "30d"/"24h"/"60m"/s. (sign): take expires_in instead of a hard-coded 86400. (main): --expires-in, defaulting to 24h as before; docstring says what an ungranted token can reach and why the key's fixity makes a month-long token safe. * README.md (Tests): the public-data-only token, what stays visible (dataset metadata), the --expires-in example and the iss/port coupling.
12 daysgn-auth-stub: bind before writing the token filePjotr
A second instance started while one is already listening used to write its own token to --token-file (and print it) before failing to bind, leaving a token nothing was serving in the place the first instance's token was expected -- tests then failed as if the stub had issued it. Bind first, then write and print. * scripts/gn-auth-stub.py (main): construct the server before writing --token-file or printing the token.
12 daystests: run against a real database and index, with a gn-auth stubPjotr
The suites skip rather than fail whatever needs a database, an index or a token they cannot get, which is right in the build sandbox but leaves most endpoints unexercised. With a live MariaDB, a full Xapian index (GN4_FULL_INDEX) and a token, the same commands run the whole suite: 225 Rust tests pass with 0 failures, and the Python suite runs all 94 with no skips -- as against 77 skipped without them. The token was the only piece with no local stand-in, so add scripts/gn-auth-stub.py: it serves the JWKS matching the committed test key, answers /auth/token (so POST /get-test-token works) and answers both authorisation endpoints with "no privileges", the stranger case the non-public 403 tests assert. --grant-resource-id grants one resource id to the fixture token for the owner-side tests. * scripts/gn-auth-stub.py: new file. * README.md (Tests): "Against a real database and index", what the three inputs are, the full run, and what still skips and why.
2026-10-14gemma/rqtl: three fixes from the endpoint reviewAlexander_Kabui
- rqtl: per-call unique output filename. The content hash alone was deterministic across processes, so two concurrent identical requests shared one output file; R truncates on open, so one request could read the other's half-written results with no error. The per-call unique (pid + nanos, already computed for the pheno file) now names the output file too. - streaming/gemma/rqtl: append a failure marker to the run log on every error path, mirroring the completion marker. A status poller can now tell a failed run from a still-running one. - rqtl: --scale is now honored. The wrapper selects the geno file's cM/Mb map column from it, and the route rejects values other than mb/cm with 422 instead of silently ignoring them. Both new behaviors are unit-tested; the live Rscript tests pass.
2026-10-14indexer delta: fall back to FULL on a case-only dataset renameAlexander_Kabui
* src/indexer/delta.rs (scan): enumerate dataset labels under COLLATE utf8mb4_bin so a case-only rename's old and new labels both enter scope; route to FULL when a case-variant label pair is in the affected scope. * src/indexer/run.rs (run_delta): seed the duplicate-check map with manifest labels not in scope; route to FULL when a manifest carries an unknown versioning flavour. * scripts/gn-indexer-lag.sh: honour the table-prefix seam only with the explicit test opt-in; treat a literal NULL MAX as no change; guard an empty lag value. * scripts/gn-indexer-run.sh: --index=PATH-aware argument scan.
2026-10-14indexer: add versioning-based DELTA modeAlexander_Kabui
* src/indexer/delta.rs: change detection over MariaDB system versioning (ROW_START/ROW_END): coverage marker, versioning probe, purge check, per-table candidate queries and identity-term fan-out. * src/indexer/run.rs (run_delta): reuse the streaming crawl and shard diff, scoped to the affected datasets. * src/indexer/sidecar.rs: manifest v3 with versioning_marker/versioning_flavour. * src/bin/gn-indexer.rs: add the delta mode. * scripts/gn-indexer-run.sh, scripts/gn-indexer-lag.sh: flock wrapper and lag SLA.
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>