| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
`"pca": true` on the matrix request now folds the correlation matrix into
principal components: eigenvalues descending, explained variance ratios,
loadings per kept trait, and the eigenvalues' sum reported so a caller can
check the decomposition is sane instead of trusting it.
The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no
new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and
this is for a collection, not a dataset.
Two decisions from ai-memory #103, both visible in the reply:
* **Traits with any null cell are dropped and named**, with the reason. A
component from a matrix with holes is not defined; GN2 silently mutates its
shared-sample list instead, and nobody can reproduce that component.
* **The sign is fixed**: within a component the largest absolute loading is
positive. Eigenvectors are defined up to sign, so without a rule two correct
implementations disagree and no parity test could be exact.
Verified against numpy.linalg.eigh on the same r the reply carries
(scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007):
eigenvalues: worst relative difference 3.2e-14
loadings: worst absolute difference 9.99e-16
trace: 5.000000000000 for 5 traits
Unit tests cover the identity (unit eigenvalues), the closed form for 2x2
(1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The
monitor gained `compute-correlation-pca`, whose judge checks that the
eigenvalues sum to the trait count and the ratios to 1.
|
|
The script could not have caught anything: it posted without an Authorization
header (401 against a token-protected server), asked for a trait's values as
`{"name": n, "values": [<trait name>]}` (422: samples need numbers), read
`columns`/`rows` from the envelope's top level (they are under `meta` and
`data`), and compared relatively -- which calls beta 0.0131 against 0.0157 a
16% disagreement on a marker where both runs agree to three decimals.
Now: `--token`/`$GN4_TOKEN`, trait values fetched from the API, the envelope
shape handled, and an ABSOLUTE `--tolerance` (default 0 = exact). It fetches,
compares and reports through the server as deployed.
Verdict on the current deployment, BXDPublish 10001 over 34 individuals:
rows: gemma-wrapper 21055, ravanan+gemma 21055, shared 21055
worst absolute difference: rs230640887 beta -1.015599 vs -1.018739 (0.00314)
--tolerance 0.005 -> agree; --tolerance 0 -> 63 163 values beyond it
So the row difference this started from is gone: same markers, same count. The
residual is that the two paths run *different GEMMA builds* -- the wrapper's
log names gemma-gn2-0.98.5-8cd4cdb, the ravanan manifest gemma 0.98.6 -- which
moves beta/se/p_lrt in the third decimal while leaving af, n_miss, l_mle and
the marker set identical (and the top hits in the same order).
|
|
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply
gains meta.permutation {n, seed, significant, suggestive, values} -- the
genome-wide maximum of every permutation, so a histogram needs no second
call. `data` stays the positional rows array on purpose: /plot/rqtl posts
it straight back and the monitor counts its rows, so the null went beside
the rows rather than around them (a generic Envelope::with_meta/Meta::extra,
skip_serializing_if, so no other endpoint's response changes).
Thresholds come from R/qtl2's scan1perm over the same genotype
probabilities, kinship and covariates as the scan, with chr_lengths from
the map -- without those it derives chromosome lengths from marker
positions and the numbers stop being comparable with R/qtl's. The
permutation CSV is written in R/qtl's two-column shape so one parser and
one percentile routine serve both engines: PermutationResult::from_values
is now shared with /compute/rqtl instead of duplicated.
Semantics, each with the note that says it:
* scan1perm shuffles phenotypes across individuals -- right for an
outbred cross, only approximate for a selfing RIL panel, where qtl2perm
is the correct package (not installed here, and not packaged);
* perm_strata takes a case attribute name (sex, Epoch, ...) and permutes
within it, because shuffling across sexes inflates the null;
* fewer than 100 permutations gets a note that the 0.05 quantile is a
quantile of too few values; more than 10 000 is refused, since each
permutation is a whole genome scan;
* a seed is reported always -- without one, two identical requests would
give different thresholds.
Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO
kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s,
1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds
converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at
1000 on the raw probe) and are reproducible: the same seed gives the same
numbers on 1 and 4 cores.
Four things that only surfaced by running it, all commented where they
bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so
opt$perm_out was always NULL and the file landed somewhere the caller did
not look); perm_strata must be a named character vector, not a factor;
the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the
permutation index first, and a first-field parser reads that as the LOD);
and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not
0.67.
bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging
meta.permutation) so a deploy that loses the R path shows up as a red
check rather than a surprise.
|
|
The access log said who (`user`) but not with what: with short tokens a
request can be made by a shared credential, so "which person" and "which
credential" are now separate fields -- `scheme` ("jwt" | "opaque") and,
for an opaque token, `token` (its registry handle, e.g.
`agent-ravanan-3`). Never the token or its hash, and the path still
carries no query string.
The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is
self-describing rather than relying on journald's prefix. Hand-rolled
civil-from-days rather than a new crate: one format, and no Cargo.lock
drift.
scripts/gn4-log.py reads such a log and shows it as a table with the
handle joined against tokens.sqlite: the subject as the registry has it
now (flagging a handle whose subject has been repointed), and the token's
status today -- [revoked], [expired], [unknown] -- which is how "an agent
was working at 14:02" is told apart from "an agent was working at 14:02
with a credential revoked since". It reads the registry read-only, passes
non-JSON lines through in table mode (the process log interleaves
backtraces), and offers --json and --summary for jq and for counting.
CallerSlot now carries the whole Caller (scheme, id, subject) rather than
just the subject string.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
Two follow-ups to the parental-design endpoint.
/groups/{group} now carries the headline in its notes -- "Parental design:
C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes
the parents and F1s by default" -- plus a next link to the full design. "Which
cross is this?" is two strain names, and needing a second request to see them
was the only reason the endpoint had to be called first. A group with no
metadata says so in the same note, pointing at parents_and_f1s.json and at
`individuals.exclude`, rather than leaving the reader to wonder why nothing was
dropped.
The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it
drifted once already, silently: the paternal/maternal labels were swapped
against GN2's unpacking, and nothing noticed because the exclusion rule takes
both strains regardless of role. scripts/check-parental-metadata.py parses
ParInfo with ast (no gn2 import) and compares it group by group, reporting
disagreements per role and treating groups the JSON knows but GN2 does not as
informational. tests/parental_metadata.rs runs it while a GN2 checkout is
present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at
which point the JSON is the only copy and there is nothing to drift from. It
reports 23 groups reproduced, 1 extra (BXD-Longevity).
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
A line on a plot is an annotation, not part of the result, so the default
is now no line at all. GEMMA's Bonferroni becomes opt-in like any other:
?bonferroni=true GEMMA's own line, labelled
?thresholds=3.65&threshold_labels=… another engine's, labelled
(nothing) just the scan
Which means a reader sees exactly the lines someone chose, each with its
source in the label -- and the accidental "this plot says significant"
impression of a default convention line is gone. The plotter's summary and
its <desc> no longer mention Bonferroni unless it was drawn.
tests/plot.rs: the plain plot has neither line; with thresholds the two
BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true
GEMMA's own line appears.
|
|
GEMMA has no permutation test here, so its plot carries a Bonferroni line.
A threshold from /compute/bulklmm belongs to a different null and must not
look like GEMMA's, so it is passed in and named:
POST /plot/gemma?thresholds=3.50,3.65
&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
Each is drawn dashed in the series colour with "label: value" beside the
line, with GEMMA's Bonferroni line still there, and the plotter's
x-plot-summary names the reference lines it drew. A threshold that is not
a positive number is a 422 invalid_threshold rather than a silent omission.
/compute/bulklmm's reply offers that href already, with its alpha 0.05
threshold filled in -- so "same values, two models, one picture" is one
link from the scan, and the labels keep the two nulls apart.
tests/plot.rs: the plain plot has no bulklmm line; with thresholds both
are drawn and labelled; a non-numeric threshold is refused.
|
|
"is the kinship term doing anything?" should be answerable from the reply.
The driver now prints h2 and the residual variance under the null model, and
the reply carries both:
Variance components: h2 = 0.000000, sigma2_e = 0.402171, estimated once
under the null (intercept) model from y and K, not per marker. With a few
dozen phenotyped individuals these estimates pile up at the boundaries 0
and 1 -- the likelihood is flat there -- so read h2 as "kinship matters a
lot / not at all" rather than as a precise proportion. At a boundary
(h2 ~ 0 the model is ordinary regression, which is why its LODs can match
R/qtl's to the digit).
Checked with a strain-ID-driven trait (values = rank of strain), which has
h2 ~ 1 by construction: BulkLMM reports h2 = 1.000000, sigma2_e = 3e-06 and
a peak of LOD 3.1 -- the kinship term absorbs the strain structure -- while
R/qtl, which has no kinship term at all, reports a spurious peak of 5.06 on
the same values. That is the discrimination the h2 column was missing.
|
|
docs/bulklmm.md records the three things that are easy to get wrong when a
BulkLMM result is quoted:
* where it stands against the other engines -- Haley-Knott with no
kinship (R/qtl), LOCO LMM whose permutations are refused (GEMMA),
LOCO LMM on HMM probabilities (R/qtl2), and this: a global-kinship
LMM whose permutation thresholds are rotation-based and therefore
cheap -- the only engine here that can say "genome-wide significant
at alpha 0.05" against a null that respects relatedness;
* what it is not: global kinship rather than LOCO, so its p-values are
not GEMMA's; h2_null says how much relatedness it found (near zero
here, which is why it reproduces R/qtl's 9.59 where GEMMA differs);
one degree of freedom, so multiparent scans stay with /compute/qtl2;
and no standard errors in 1.2.2's scan(), so the rows are
marker, chr, pos, lod, beta, h2;
* the cores, measured: Threads.@threads is in bulkscan.jl and
bulkscan_helpers.jl, never in scan.jl, so a single-trait scan is
serial by design and the visible parallelism was BLAS (96 threads
10.5 s against 4 threads 7.7 s). What the driver does about it --
exact marker blocks, cached matrix and kinship, capped BLAS, and the
breakdown printed into the reply -- with the numbers (20.9 s at one
block against 18.3 s at four, same LOD 9.589938).
Also the packaging picture, since it is not obvious from the outside:
guix-bioinformatics has a working julia-bin and a julia-build-system, but
only three of BulkLMM's closure are packaged, so a pure-Guix build is
35-45 recipes; the pragmatic route is julia-bin plus a depot built once
from the pinned Manifest. scripts/prepare-julia-depot.sh is that step,
written down, and the depot stays small and GPU-free (79 packages,
139 MB) with JULIA_PKG_OFFLINE=true at request time.
The plot test learns that both thresholds are drawn, not one.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|
|
Three follow-ups from putting an upload through the whole loop.
* /compute/qtl2's `genotype_file` branch only accepted positional
`trait_values`, i.e. it demanded the file's column order -- which a
caller with an uploaded population does not have. It now takes
`samples` (name and value) or a `basket` and aligns them onto the
cross by name, leaving individuals without a value as missing.
Positional `trait_values` still work for a file whose order is known.
* DELETE /uploads/genotypes/{name} now removes the .geno it wrote on
demand: a file nobody can name is a file that confuses the next
operator.
* scripts/example-genotype-upload.py spreads its markers across
chromosomes. One chromosome of markers is a poor example -- a
leave-one-chromosome-out kinship has nothing to leave out, so the
scan came back with no rows at all, which looked like a broken upload
and was really a property of the example. 20 markers over the genome
map cleanly (19 rows, the 20th used for the HMM), and the README says
to use "kinship": "overall" if a file really has one chromosome.
|
|
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.
An upload updates a population, it does not invent one, and the checks
say so:
* the base version must exist (resolved through the same
genotypes::resolve_named the rest of the API uses);
* at most 50 individuals the base does not have, and at least one that
it does -- no overlap is a different population, not an update;
* at most 200 000 markers, and the refusal says the sysadmin has to
organize a file that size, because this is a request path, not an
ingest;
* the values must be the same kind as the base's: letters, whole
numbers or probabilities. A letters panel holding 0/1 would be read
as the wrong thing silently, so it is refused naming both kinds.
GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.
Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
|
|
The other half of changing data: bringing new columns in. A matrix is a
delimited table -- a header row, then one row per individual, the first
column naming them -- stored in the uploader's own SQLite file and checked
against an existing genotype version's panel before it is accepted.
* POST /matrices {name, genotype_version, contents, kinds?}: parse the
table (tab, comma or semicolon, guessed from the header), then check
it against that version's individuals. The count must match and every
name must be in the panel; 92 of 93, or a name the panel does not
have, is a 422 that says which. A mapping that silently drops
individuals is worse than an error.
* kinds marks columns that are covariates; everything else is a trait.
* Each column becomes a basket named <matrix>.<column> -- a dot, not a
slash, because a basket name has to survive a URL path segment -- so
a column maps on any engine with {"basket": "my-traits.bw"} and
inherits the revision history and digest without a line of new
mapping code.
* GET /matrices lists them, GET /matrices/{name} their columns and
basket names, GET /matrices/{name}/values the table back (parsed
values, null where missing), DELETE removes the matrix and its
columns.
Two bugs this found in the basket store, both real:
* SQLite has foreign keys OFF by default, so the ON DELETE CASCADE
never fired: deleting a basket left its rows behind, rowids were
reused, and the next create hit "UNIQUE constraint failed:
basket_version.basket_id, revision". The pragma is now set, and
deletion removes the dependants explicitly, because files written
before it have orphans.
* The first attempt named a column's basket "<matrix>/<column>", which
the basket-name rule (rightly) refuses. The dot keeps both rules.
scripts/example-matrix.py writes a table from a genotype file's own header
with deterministic values, so the fixture passes the panel check by
construction -- tests/matrices.rs builds its own for the same reason and
walks the loop: upload, read back, map a column with R/qtl2, delete, plus
the three refusals (too few individuals, a name not in the panel, an
unknown version).
|
|
A multiparent population cannot come from a .geno. That file carries two
codes per marker (B/D, A/H/B) -- a collapse, not a founder allele -- and
the collapse is not invertible, so DOL's 194 DO animals cannot be fitted
as a DO however much reference data appears. What can be is a cross that
ships its own qtl2 control file with founder genotypes, so the endpoint
now takes one by name:
{"rqtl2_cross": "do", "trait_name": "DO::10001 (WBC)", "samples": ...}
* resolve_control_file looks in genotype/rqtl2/{name}.json, then
{name}/{name}.json (do/do.json), then {name}.json under
GENOTYPE_FILES; prepare_control_cross reads the cross's pheno, maps
and founder_geno headers without converting anything. It reports
crosstype, founders and whether the cross is multiparent, and
refuses a multiparent crosstype with no founder_geno rather than
silently scanning it as biallelic.
* Samples are matched to the cross's own population by name and the
strays are counted in notes; rqtl2_cross cannot be combined with
group or genotype_file.
* The gate is given individuals x founders, because a multiparent scan
fits one coefficient per founder allele at every marker.
* Autosomes only now applies to shipped crosses as well: the DO's X is
left out (369 markers) instead of being fitted without a sex
covariate, and the script subsets with cross[, chr = autosomal].
Measured on the DO (WBC, 742 individuals, 8 founders, 6044 autosomal
markers): probabilities 174 s / 1.39 GB (cached), LOCO kinship 15 s
(cached), scan 21 s. scan1 runs on one core for a multiparent cross
because its forked path stalls on allele probabilities here -- 25+
minutes at 8 cores with no CPU consumed, against 21 s serial.
Two bugs found by running an actual published trait through it:
* do_pmap.csv is in Mb, not the bp qtl2 documents, so every DO peak was
reported at 0.0 Mb. The map's unit is now detected from its largest
position and printed.
* The phenotype file was read with the default whitespace separator, so
a label with a space ("DO::10001 (WBC)") split into three fields and
read.table stopped with "line 1 did not have 3 elements". Both the
pheno and covariate reads are tab-separated now.
The R error extraction also joins R's wrapped continuation line, so a
complaint reads as a sentence rather than "Error in scan(...) :".
|
|
R/qtl sits between the existing two scans: Haley-Knott on marker
genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with
mean-imputed genotypes. Neither can analyse a multiparent population,
and neither models relatedness from the actual genotype data. R/qtl2
does both -- HMM genotype probabilities, then an LMM with
leave-one-chromosome-out kinship -- and it is what names the third peak
on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24,
against GEMMA's -log10 p 7.15 there).
* src/qtl2.rs converts this deployment's own .geno into a qtl2 cross
under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with
the genotype-code map, the orientation and the paths written out
explicitly. The shipped genotype/rqtl2/ control files cannot be
used -- they point outside the data directory, declare no code map
(every genotype reads as missing) and no orientation (individuals
read as markers) -- and each of those produces a plausible flat
scan instead of an error, which is why the conversion is ours.
* Heterozygous calls are dropped on a riself panel, as R/qtl's
convert2riself does. Left in, qtl2 reads the cross as multiparent
and its scan1 takes a path the panel cannot satisfy ("Error in
scan1_pg ... : No individuals in common").
* Autosomes only: X needs a sex covariate to be fitted properly and
Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is
restricted the same way. The note says how many markers were left
out.
* scripts/qtl2_scan.R caches genotype probabilities and kinship by a
hash of the control file *and its directory*; two conversions of
the same panel can have identical control files and identical file
sizes, and reusing the first one's probabilities silently scans the
wrong cross. It also takes the individual names from whichever
shape the cross's pheno slot has -- a single-column pheno file
leaves it a plain matrix, and the resulting empty phenotype reads
as scan1's "No individuals in common".
* Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply
unchanged -- post the same samples as /compute/rqtl.
* tests/qtl2.rs: the fixture scans and its notes name the cross; BXD
trait 10001 peaks on chr8 under both scans; refusals carry
missing_genotype / invalid_kinship.
cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like
R/qtl; a DO cross costs minutes of genotype probabilities, paid once
per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
|
|
DOL (200 individuals, 112 913 markers, one genotype version) failed on
R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare
wrapper bail-out. Two unrelated causes, both in our own code.
**R/qtl: a genotype code collided with a chromosome name.** DOL's header
says `@unk:X` -- X is the code for a missing genotype -- and the wrapper
passed that to `read.csv(na.strings=...)` for the *whole* file, so every
marker on the X chromosome lost its chromosome ID and `read.cross.csv`
refused the map. The unknown code now applies to the genotype columns
only. DOL scans: 112 913 markers in 42 s, where before it never got past
reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read.
**GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the
DOL panel is scanned per chromosome, and the Y run dies inside BLAS --
`Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking
the whole request with it (a 502 whose detail is the wrapper's banner).
The mitochondrial chromosome has the same shape of problem. `prepare` for
the GEMMA engine now writes the bimbam pair without the Y and M markers
(cached beside the individual subset, line-aligned so the geno and snps
stay in step), and the note says what was left out: "46 Y/M markers left
out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s.
**And the four tests that had been failing since before this work**, all
stale expectations rather than product bugs:
* `benchmark_questions` pinned two traits to "page 2" of a 100-per-page
search when they rank ~190th and ~219th of 246 -- one re-index pushes
them over. It now walks up to three pages and asserts presence and LOD,
not the page; and the helper `search()` pages, because a benchmark row
can name a trait that ranks outside the first hundred of its own query.
* `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol"
from the index alone. The index grew; the behaviour under test is that
the search still answers, within the limit, with the database-backed
fields null. That is what it asserts now.
* `search_proxy_scenarios` and `search_parity_replay` never pinned the
engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy
scenarios answer from the local index: a 200 where a 422 was expected,
and "expected 2 upstream calls, got 0" for every replayed query. Both now
set `SEARCH_ENGINE=forward`.
* src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`,
`Prepared.markers_dropped`.
* src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting
parents/F1s from `panel - keep` instead of subtracting the exclusion
*set*, whose length can exceed the panel (subtract overflow).
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the
genotype columns.
* tests/{benchmark,no_db,search,search_parity}.rs: as above.
Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA
106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
|
|
Phase 0 of the R/qtl2 work, so the engine can be exercised before any
endpoint exists. `r-qtl2` (0.42 in this channel) plus `r-digest`/`r-jsonlite`
go into the dev shell's propagated inputs; without them R has to be handed a
list of 594 store paths by hand to load qtl2 at all.
`scripts/qtl2_scan.R` is written for this API rather than copied from
GeneNetwork3's `scripts/rqtl2_wrapper.R`, which cannot be trusted as a
reference: its `"LMM"` and `"LOCO"` branches call `scan1` with identical
arguments, its covariates are a `covar <- NULL; # TODO` stub, and its only
tests mock the file-writing helpers -- nothing ever reads a cross.
The script reads a qtl2 control file, a two-column phenotype file and
optional covariates, and writes `marker,chr,pos,lod` (Mb, so the shape
matches /compute/rqtl and /plot/rqtl draws it unchanged). Genotype
probabilities and the kinship matrices are cached in --cache-dir keyed by
the control file's content, because a DO cross costs 536 s of genotype
probabilities on four cores against 11 s of kinship.
Four things the cross-file plumbing cost, all found by running it:
* `read_cross2` joins the control file's directory onto whatever paths it
finds, so an absolute path becomes `<dir>/<absolute>` and dies. The server
has to write the control file *beside* its data;
* our generated genotypes are markers-in-rows, which needs
`"geno_transposed": true` -- without it qtl2 reads the individuals as
markers (93 individuals appear as 93 extra markers, and every marker is
"not in both genotypes and maps");
* the genotype strings need a `"genotypes"` map (the GN letters
@mat/@het/@pat -> A/H/B -> 1/2/3); without it every value is "treated as
missing" and the scan comes back flat (peak LOD 0.14);
* with those fixed, a cross converted from `genotype/BXD.4.geno` scans in
~1 s and agrees with /compute/rqtl on the same panel and trait: qtl2's LMM
peaks on chr8 at ~111 Mb, LOD 3.50, where R/qtl's HK `scanone` peaks on
chr8 at ~69 Mb, LOD 2.93 -- same chromosome, different model, which is the
cross-check the plan asks for.
Not yet: the Rust side (`src/qtl2.rs`), the `/compute/qtl2` route, the
genotypes-listing `qtl2` flag and the converter that writes the control
file. Those are Phase 1-3.
|
|
44 of this deployment's `.geno` files carry `cM` and no `Mb` column, and
none of them could be scanned. GeneNetwork3's wrapper (copied verbatim
here) assumes the position columns are `Chr, Locus, cM, Mb` and then starts
the samples at a hardcoded column 5. On a cM-only file that is one column
too far, so the genotype block loses its first individual and the whole scan
dies in `do.call(rbind, ...)` with "number of columns of result is not a
multiple of vector length" -- a message that says nothing about the cause.
The wrapper now picks the map column from whichever of the requested scale
and the file's own columns exists (falling back, and saying so), and starts
the samples right after the position columns; the sex default and the
sample-count guard use the same count. `scripts/qtl_fitqtl.R` had the same
assumption and got the same fix.
This is the second deliberate deviation from GeneNetwork3's script, after
the sample-count guard, and both are commented as such at the code.
* scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: the position column and the
first sample column are computed, not assumed.
* tests/test_data/rqtl_sample1_cm.geno: the existing fixture with `Mb`
dropped -- same panel, cM only.
* tests/rqtl.rs: `a_cm_only_genotype_file_maps` scans it and asserts all six
markers come back (the failure mode was five-vs-six columns, not an
error message that pointed at the genotype file).
* src/rqtl.rs, src/routes/rqtl.rs: the docs said the script never forwards
`--scale`; it does, and now falls back, so the request field says what
`pos` is in.
Verified through the API: DGRP (cM-only) now maps (3 markers), and AXB (499
markers) and BXD.4 (3 811) are unchanged.
307 tests pass; the same four as before do not.
|
|
`fitqtl` takes a `covar` argument, so the effect it reports can be the one
*after* conditioning -- which is usually the number worth quoting for a peak
that overlaps a known confounder. `covariates` now works on /compute/fitqtl
exactly as on /compute/rqtl: the same `covariates::resolve` (case attribute
by name, inline values, explicit map, same coding and same note), so the two
endpoints report a covariate identically.
The R side does what R/qtl needs: the phenotype file carries one column per
covariate after the trait, `--covarstruct` says which are categorical, and
the script dummy-expands those (k-1 against the first sorted level, missing
values left missing). Two things it has to get right:
* `model.matrix()` drops rows with a missing value, which silently misaligns
the covariate frame against the cross -- so the dummies are built directly
(`ifelse(is.na(x), NA, x == level)`), not through `model.matrix`;
* a covariate can be perfectly spread across the panel and **constant among
the individuals a trait covers** -- BXD's `Status` is exactly that -- and
R/qtl then stops with `Lapack routine dgesv: system is exactly singular`.
Columns that do not vary among the phenotyped individuals are left out,
and the reply names them, the same rule the GEMMA covariate file applies.
Measured on BXD trait 10001 at its peak, `BXD.4`:
no covariate LOD 2.9310 additive +2.394 32.8% var
with Status LOD 3.3602 additive +2.463 36.6% var
with an inline numeric covariate LOD 2.9373 additive +2.361
The LOD is no longer the scan's once a covariate is fitted -- it is a
different model -- which the docs now say, next to the no-covariate parity
check that still holds.
* scripts/qtl_fitqtl.R: `--covarstruct`, dummy expansion, the degenerate
column rule, and the dropped names in the summary line.
* src/fitqtl.rs: writes the covariates into the phenotype file and the
struct file, and passes both.
* src/routes/fitqtl.rs: the `covariates` field, resolved after alignment so
the coding and the complete-case count match /compute/rqtl.
* tests/fitqtl.rs: a covariate run (the note, the fitted row, a changed LOD)
and an unknown attribute refused with the list of valid ones.
* README.md, docs/www/SKILL.md: covariates on this endpoint, and why its LOD
stops matching the scan's.
306 tests pass; the same four as before do not.
|
|
`/compute/rqtl` answers "where" -- scanone gives a LOD per marker -- but not
"how much": a scan statistic is not a fitted model. R/qtl's `fitqtl` fits
one, and reports the QTL's effect, its standard error, the model LOD and the
variance explained. That is what a table of effects needs, and it is a
separate endpoint because it is a separate question, asked about a handful
of markers (a scan's peaks) rather than a genome: `fitqtl` fits one model
per marker, so a genome-wide version would be thousands of fits. Cap: 100
markers per request.
Rows are positional:
marker, chr, pos, lod, pct_var, additive, se, t, p
`additive` is fitqtl's estimate for the QTL term -- R/qtl codes the QTL
genotype +-1, so it is half the difference between the two homozygote
classes, the same convention GeneNetwork's published `additive` uses.
**Two independent checks that the numbers are right.** On BXD trait 10001
(`BXD.4`), at the scan's peak:
* `lod` = 2.930985, and `/compute/rqtl`'s scanone gives 2.930985 at the same
marker -- identical to every digit the CSV carries, because they are the
same cross and the same phenotype. That parity is now an integration test;
* `additive` = 2.394472, against the value genenetwork.org publishes for that
trait (`/api/v_pre1/trait/BXD/10001` -> 2.394444), which is 28 ppm away --
and that published number came from qtlreaper, a different engine.
* scripts/qtl_fitqtl.R (new): reads the same `.geno` and two-column
phenotype file as `rqtl_wrapper.R` (which is a verbatim copy of GN3's
script and so not extended), builds the cross the same way, fits
`fitqtl` at each named marker with `get.ests=TRUE`, and writes one CSV
row per marker -- plus a one-line summary naming the strongest fit.
`calc.genoprob(step=0)`: marker-level probabilities are all a
marker-level fit needs.
* src/fitqtl.rs (new): runs it, parses the CSV into positional rows (`chr`
stays text, it can be `X`), keeps the command line for `notes`.
* src/routes/fitqtl.rs (new): the endpoint -- same alignment rules as
`/compute/rqtl` (by name onto the panel, parents/F1s excluded unless
asked), the same `normalize`/`winsor_*` pipeline (effects come out on
that scale), and the same R/qtl gate, so a fitqtl request and a scan
queue against one another rather than competing for cores.
* src/config.rs: `FITQTL_SCRIPT_PATH` (default `scripts/qtl_fitqtl.R`).
* tests/fitqtl.rs: the fixture (columns, sign of the effect, notes), the
LOD parity against scanone, and the refusals (no markers, both genotype
inputs).
* README.md, docs/www/SKILL.md: the endpoint, what each column means, and
when to reach for it instead of the scan.
305 tests pass; the same four as before do not.
|
|
The two plotters Pjotr added (scripts/plot-rqtl.py, plot-gemma.py) turn a
saved mapping response into an SVG Manhattan panel. They are now
endpoints, and the mapping that produces the response suggests them:
POST /plot/rqtl same body as /compute/rqtl -> image/svg+xml
POST /plot/gemma same body as /compute/gemma -> image/svg+xml
Taking the mapping's own body rather than the mapping's output means one
call gets you the picture, and the plot cannot describe a run you did not
do: the handlers call the compute handlers and hand the resulting
envelope to the script, exactly as the scripts' docstrings describe
(they read `data.markers`, `data.permutation` and the notes' strain
counts). The mapping is re-run, but through the engines' own caches --
seconds on BXD.
The reply is an SVG, not the JSON envelope: an image is not JSON, and the
plotter's one-line peak summary rides back in `x-plot-summary` so a
caller that cannot see the picture still learns where the peak is.
Both compute responses gain the matching suggestion: `actions` carries
"the same request, plotted" (built from the request that ran, like the
permutation suggestion), so an assistant can map, look, and only then
decide whether the peak is worth the 100-permutation run.
* src/plot.rs: run the configured script over the envelope, read the
SVG back, clean up; a missing script is a 502 naming interpreter and
script, not a panic.
* src/routes/plot.rs: the two handlers (token-gated), delegating to
compute_rqtl/compute_gemma so validation and alignment stay in one
place; `x-plot-summary` sanitised into a header.
* src/routes/{rqtl,gemma}.rs: the plot suggestion, before the
permutation one.
* src/config.rs: PLOT_PYTHON, RQTL_PLOTTER, GEMMA_PLOTTER.
* tests/plot.rs: the suggestion resolves to a complete SVG with a peak
summary, and an invalid mapping request is the mapping's own 422.
* tests/{rqtl,phewas}.rs: find suggestions by href rather than position.
* README.md, docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md
(new "Plotting": python3 must be on the server's PATH).
|
|
Asked to map a 9-strain trait, the wrapper scanned it against the
6-individual fixture cross and returned six markers with LOD scores and
`state: complete` -- a plausible answer belonging to nobody. R recycled
and truncated the values silently, and nothing downstream could tell.
GeneNetwork3's copy of this script has the same hole.
The cross-file format carries no per-sample names, so the count is the
only check available; order remains the caller's responsibility (and the
reason values and genotype file must come from the same panel). With the
check, the same request is a 502 rqtl_failed quoting both counts, and the
correctly-shaped request still returns the documented LODs.
* scripts/rqtl_wrapper.R: compare the pheno file's rows with nind() of
the cross, and stop.
* README.md (/compute/rqtl): the second fix relative to gn3, and what
stays the caller's job.
* docs/www/ai-instruction.md (4a): values and genotypes must describe the
same individuals, one for one, in the same order.
* docs/www/operator-checklist.md (Compute): what genotype files actually
exist on this box (test fixtures, 198 BXD individuals) and why a database
trait, with its own strain subset, cannot be mapped against them.
|
|
The page told the operator to mint with POST /get-test-token, which is
gn-auth's dev password grant and does not exist in the deployment being
set up. Issuing a second user's token also had no good answer: the stub
minted exactly one token per process, so a new user meant starting a
throwaway instance, grabbing --token-file and killing it.
So add --mint-only: mint for --subject, print, write --token-file and
exit without listening. Verification only depends on the key (committed,
so shared by every instance) and on iss, which now falls back to
AUTH_SERVER_URL -- a token minted this way verifies against the
long-running stub untouched.
* scripts/gn-auth-stub.py (main): --mint-only; --issuer defaults to
AUTH_SERVER_URL before the port-derived default. Docstring says how a
second user's token is issued.
* docs/www/ai-instruction.md (1. Get a token): the operator's command is
the stub, not /get-test-token; the token lasts whatever --expires-in said
rather than an hour, and there is no early revocation. (4. What is
public): minting is operator-side. Appendix: "Token issuance" rewritten
around the stub -- public-data-only, expiry as the only revocation, sub
as attribution, one token per user -- with gn-auth kept as what replaces
it later; the stub is no longer listed as a dev leftover to delete, but
as the auth server that must never be routed publicly.
* README.md (Tests): mention --mint-only next to the stub's other flags.
|
|
A stub token had a fixed 24h lifetime, and nothing said what an
ungranted one can reach. Both matter for the case that is not a test
run: handing a caller or an agent a long-lived token that may only read
public data. An ungranted stub token already is exactly that -- the
restriction lives in the authorisation lookup genenetwork4 consults for
anything not public, not in the token's scope -- so only the lifetime
was missing.
* scripts/gn-auth-stub.py (duration): new helper, "30d"/"24h"/"60m"/s.
(sign): take expires_in instead of a hard-coded 86400.
(main): --expires-in, defaulting to 24h as before; docstring says what
an ungranted token can reach and why the key's fixity makes a month-long
token safe.
* README.md (Tests): the public-data-only token, what stays visible
(dataset metadata), the --expires-in example and the iss/port coupling.
|
|
A second instance started while one is already listening used to write
its own token to --token-file (and print it) before failing to bind,
leaving a token nothing was serving in the place the first instance's
token was expected -- tests then failed as if the stub had issued it.
Bind first, then write and print.
* scripts/gn-auth-stub.py (main): construct the server before writing
--token-file or printing the token.
|
|
The suites skip rather than fail whatever needs a database, an index or
a token they cannot get, which is right in the build sandbox but leaves
most endpoints unexercised. With a live MariaDB, a full Xapian index
(GN4_FULL_INDEX) and a token, the same commands run the whole suite:
225 Rust tests pass with 0 failures, and the Python suite runs all 94
with no skips -- as against 77 skipped without them.
The token was the only piece with no local stand-in, so add
scripts/gn-auth-stub.py: it serves the JWKS matching the committed test
key, answers /auth/token (so POST /get-test-token works) and answers
both authorisation endpoints with "no privileges", the stranger case
the non-public 403 tests assert. --grant-resource-id grants one
resource id to the fixture token for the owner-side tests.
* scripts/gn-auth-stub.py: new file.
* README.md (Tests): "Against a real database and index", what the
three inputs are, the full run, and what still skips and why.
|
|
- rqtl: per-call unique output filename. The content hash alone was
deterministic across processes, so two concurrent identical
requests shared one output file; R truncates on open, so one
request could read the other's half-written results with no error.
The per-call unique (pid + nanos, already computed for the pheno
file) now names the output file too.
- streaming/gemma/rqtl: append a failure marker to the run log on
every error path, mirroring the completion marker. A status
poller can now tell a failed run from a still-running one.
- rqtl: --scale is now honored. The wrapper selects the geno file's
cM/Mb map column from it, and the route rejects values other
than mb/cm with 422 instead of silently ignoring them.
Both new behaviors are unit-tested; the live Rscript tests pass.
|
|
* src/indexer/delta.rs (scan): enumerate dataset labels under COLLATE
utf8mb4_bin so a case-only rename's old and new labels both enter scope;
route to FULL when a case-variant label pair is in the affected scope.
* src/indexer/run.rs (run_delta): seed the duplicate-check map with manifest
labels not in scope; route to FULL when a manifest carries an unknown
versioning flavour.
* scripts/gn-indexer-lag.sh: honour the table-prefix seam only with the
explicit test opt-in; treat a literal NULL MAX as no change; guard an
empty lag value.
* scripts/gn-indexer-run.sh: --index=PATH-aware argument scan.
|
|
* src/indexer/delta.rs: change detection over MariaDB system versioning
(ROW_START/ROW_END): coverage marker, versioning probe, purge check,
per-table candidate queries and identity-term fan-out.
* src/indexer/run.rs (run_delta): reuse the streaming crawl and shard diff,
scoped to the affected datasets.
* src/indexer/sidecar.rs: manifest v3 with versioning_marker/versioning_flavour.
* src/bin/gn-indexer.rs: add the delta mode.
* scripts/gn-indexer-run.sh, scripts/gn-indexer-lag.sh: flock wrapper and lag SLA.
|
|
Same request/response shape as /compute/gemma: synchronous (no job
queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming.
Extracted that streaming infrastructure (run_streaming, create_run_log/
RunLogError, status/StatusError, RunStatus/RunStatusParams,
is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new
src/streaming.rs so both compute endpoints share it instead of
duplicating it; src/routes/gemma.rs updated to the shared names,
confirmed its own 8 tests still pass unchanged.
scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim
and run via Rscript -- same reasoning as reusing gemma-wrapper rather
than reimplementing GEMMA. Scope: a single scan (scanone) plus, if
nperm > 0, its permutation thresholds (67th/95th percentile, via a
Rust port of numpy.percentile's default linear interpolation) -- the
common case. Not implemented: covariates, interval mapping,
permutation strata, pair-scan (scantwo) -- scantwo in particular needs
cross-referencing the genotype file for nearest-marker lookups and a
second (MAP_) output file, a materially bigger port than this covers.
One fix relative to GeneNetwork3, found by reading the code (no test
covers the real caller's keys): generate_rqtl_cmd's content hash only
ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller
uses keys "geno"/"pheno" -- they never match, so the genotype/
phenotype file contents never actually enter the cache-key hash in
production, only the other options. Two different phenotypes mapped
with the same other options would get the same cached output filename
and silently reuse each other's result. Fixed here: both files'
contents are always hashed (std DefaultHasher -- a cache key, not a
security boundary, so no need for GeneNetwork3's md5).
guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as
propagated-inputs (same category as gemma-wrapper, and same reasoning:
invoked as a subprocess, not linked). All stock Guix -- confirmed the
existing -L (needed only for gemma-wrapper, from guix-bioinformatics)
is not needed for these; the dev shell has Rscript/R-qtl on PATH from
guix.scm's own propagated-inputs alone.
Test fixture: a small, hand-written, valid BXD riset genotype/
phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's
own tests/unit/db/.../genotype_sample1.geno -- that fixture has a
genotype code outside its own header's declared set and duplicate
marker names across chromosomes, harmless for the unrelated parsing
unit it's written for there, but both would break a real R/qtl run.
Verified end to end, not just unit tested: tests/rqtl.rs runs the real
Rscript/R-qtl against that fixture (a real scanone scan, real
permutation thresholds with suggestive <= significant, the run-id-
collision 409 pre-seeded with no Rscript needed, and the completion
marker landing in a real log after the run finishes). cargo test
passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|