| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
`"pca": true` on the matrix request now folds the correlation matrix into
principal components: eigenvalues descending, explained variance ratios,
loadings per kept trait, and the eigenvalues' sum reported so a caller can
check the decomposition is sane instead of trusting it.
The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no
new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and
this is for a collection, not a dataset.
Two decisions from ai-memory #103, both visible in the reply:
* **Traits with any null cell are dropped and named**, with the reason. A
component from a matrix with holes is not defined; GN2 silently mutates its
shared-sample list instead, and nobody can reproduce that component.
* **The sign is fixed**: within a component the largest absolute loading is
positive. Eigenvectors are defined up to sign, so without a rule two correct
implementations disagree and no parity test could be exact.
Verified against numpy.linalg.eigh on the same r the reply carries
(scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007):
eigenvalues: worst relative difference 3.2e-14
loadings: worst absolute difference 9.99e-16
trace: 5.000000000000 for 5 traits
Unit tests cover the identity (unit eigenvalues), the closed form for 2x2
(1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The
monitor gained `compute-correlation-pca`, whose judge checks that the
eigenvalues sum to the trait count and the ratios to 1.
|
|
GEMMA's reply offered R/qtl as the way to get significance thresholds,
which is the wrong advice for a relatedness-structured panel: haley-knott's
permutation shuffles phenotypes without regard to the kinship, so its null
is anti-conservative exactly where an LMM was chosen in the first place.
BulkLMM is the same model family as this scan and permutes by rotating the
data through the kinship decomposition, so its thresholds do respect
relatedness -- and it is fast.
The 200 reply now carries three actions in order: draw it (/plot/gemma),
POST /compute/bulklmm with a ready-to-post body (group, trait_name,
samples, nperm 500, seed 1), then POST /compute/rqtl relabelled as a
non-LMM cross-check with a warning that its threshold is anti-conservative
here.
Measured on BXD trait 10001, so the text asserts nothing it has not seen:
GEMMA's best marker is rsm10000005700 on chr8 (p_lrt 5.9e-07, -log10 p
6.23) and BulkLMM's best is the same marker (LOD 3.866, beta -0.0615) --
at the peak the two LMM families agree on the locus. Their thresholds are
not interchangeable: with 500 rotations BulkLMM's 0.05 threshold is LOD
3.78 (the peak clears it, and a second seed gave 3.62), while R/qtl2's
LOCO scan1perm on the same trait and permutation count gives 2.62, so a
global kinship is the stricter null; and LOD 3.87 and -log10 p 6.23 are
different scales for one marker, so a BulkLMM threshold must never be
compared with a GEMMA p-value directly.
The 422 permutations_not_supported text quotes the same measurements
instead of prose, and the suggested body was run verbatim to check it:
POST /compute/bulklmm with it returns 200, 21056 rows, and the threshold
note "500 rotations (seed 1): alpha 0.01 LOD 4.10, alpha 0.05 LOD 3.62".
|
|
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply
gains meta.permutation {n, seed, significant, suggestive, values} -- the
genome-wide maximum of every permutation, so a histogram needs no second
call. `data` stays the positional rows array on purpose: /plot/rqtl posts
it straight back and the monitor counts its rows, so the null went beside
the rows rather than around them (a generic Envelope::with_meta/Meta::extra,
skip_serializing_if, so no other endpoint's response changes).
Thresholds come from R/qtl2's scan1perm over the same genotype
probabilities, kinship and covariates as the scan, with chr_lengths from
the map -- without those it derives chromosome lengths from marker
positions and the numbers stop being comparable with R/qtl's. The
permutation CSV is written in R/qtl's two-column shape so one parser and
one percentile routine serve both engines: PermutationResult::from_values
is now shared with /compute/rqtl instead of duplicated.
Semantics, each with the note that says it:
* scan1perm shuffles phenotypes across individuals -- right for an
outbred cross, only approximate for a selfing RIL panel, where qtl2perm
is the correct package (not installed here, and not packaged);
* perm_strata takes a case attribute name (sex, Epoch, ...) and permutes
within it, because shuffling across sexes inflates the null;
* fewer than 100 permutations gets a note that the 0.05 quantile is a
quantile of too few values; more than 10 000 is refused, since each
permutation is a whole genome scan;
* a seed is reported always -- without one, two identical requests would
give different thresholds.
Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO
kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s,
1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds
converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at
1000 on the raw probe) and are reproducible: the same seed gives the same
numbers on 1 and 4 cores.
Four things that only surfaced by running it, all commented where they
bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so
opt$perm_out was always NULL and the file landed somewhere the caller did
not look); perm_strata must be a named character vector, not a factor;
the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the
permutation index first, and a first-field parser reads that as the LOD);
and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not
0.67.
bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging
meta.permutation) so a deploy that loses the R path shows up as a red
check rather than a surprise.
|
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
The implementation had already landed in 698e6ab by accident -- a `git add
-A` swept it in with the operator-checklist docs, which is why its message
says nothing about partial correlation. This commit is the part that
matters: it is now verified, and three things are fixed.
**Verification.** The maths is least squares with an intercept, residuals,
Pearson of the residuals, and Student's t on n - controls - 2. Checked two
ways against independent code:
* BXDPublish 10001 with 10002 held constant, target 17164, n=6:
ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq +
scipy.stats.t.sf r=0.9960864631522379 p=0.00029372.
Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567.
* The same for 17084: identical to 16 digits, r=0.9840718340719924.
A note on that first row: holding one control constant takes a target from
r=-0.11 to r=+0.996 on six points. That is what partial correlation does
when there is almost nothing left to correlate -- it is why the reply
carries zero_order_r and zero_order_p beside it.
**Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp`
term sign-flipped, which still yields a smooth monotone function, so
nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out
as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12
relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both
checkpoints are now tests. A perfect partial r (self-correlation) is
clamped to +-1 with p=0.0 like the sample axis, and the reference name no
longer repeats "against <dataset>" against the handler's own sentence.
The monitor now exercises the axis: `compute-correlation-partial`.
|
|
work yet
Writing down what the monitor found and what fixed it, because it is config
that lives outside the repository and is therefore easy to lose at the next
deploy: with only the cache dirs set, every engine that shells out answers
502 while the whole read surface and search look healthy.
Measured on production after the fix (bin/monitor-gn4 --tier compute):
8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453),
fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed,
1 busy.
The remaining failure is BulkLMM and it is not a config line: the Julia
project is registry-based and no depot on this box has its packages, so
it stops at "Run Pkg.instantiate()". Populating a depot downloads from the
Julia registry, which should be a decision rather than a side effect.
|
|
`type` on /compute/correlation now selects GN2's `corr_type`: sample
(default), tissue or literature. Partial is refused by name rather than
silently treated as sample.
**tissue** reduces each gene symbol to its highest-Mean probe in
TissueProbeSetXRef and pairs the two value vectors **by TissueID**. That
last word is the difference from GN2, which keeps each vector as a bare
list and pairs by position on the assumption that every probe covers the
same tissues. On this database it does not: every probe carries exactly
26 values (liver, kidney, spleen, ... peripheral) or exactly 32
(striatum, hippocampus, neocortex, ... brain regions), the two sets are
disjoint, and 45 281 probes are in each. Pairing those positionally
correlates liver expression with striatum expression because both happen
to be entry one. Pairing by tissue means a cross-panel pair has no
correlation at all, which is the honest answer and makes the panel
boundary visible instead of inventing a number.
Verified independently for Shh vs Hcrtr1: 0.7222718302270719 from the
endpoint, 0.7222719072696666 from a separate Pearson over the same
TissueID-aligned pairs (the residue is MySQL's single-precision `float`
column as printed by the client).
**literature** looks the pair up in LCorrRamin3 in both directions after
translating rat/human ids through GeneIDXRef, in chunks of 4 000 genes
(the first attempt died on MySQL's 65 535-placeholder limit). It is a
precomputed value, not a statistic: p and n are null. A trait with no
gene id or no row is left out and counted in the note, where GN2 reports
0.0 and fills the tail of its top-N with them -- that reads as "measured,
no relationship", which is not what happened.
Gene ids come from ProbeSet.GeneId, not ProbeSetXRef.GeneId: the latter
has no such column here, which GN3's own mapping also records.
|
|
Recorded next to `Method` so the next comparison does not chase a 0.005
disagreement in r for an afternoon.
GN2's `get_sample_r_and_p_values` takes a parameter called `type` and is
called with `corr_params['type']` -- "sample" -- where the method belongs,
so `if type == 'pearson'` is never true and `scipy.stats.spearmanr` always
runs. The method is used only on the tissue path.
Evidence, from the live site with `method=pearson&type=sample`:
1427571_at vs 1447043_at reported -0.8446202819617522
spearman -0.8446202819617523
pearson -0.8393085803732605
1427571_at vs 1427641_at reported -0.8479321530363810
spearman -0.8479321530363810
pearson -0.8451666639589470
BXDPublish 10001 vs 17084 reported 1.0
spearman 1.0000000000000000
pearson 0.9757237677298747
Asked for spearman, our endpoint reproduces the live site's whole reply:
the same five traits in the same order, r to 2.2e-16, p to 5e-10
relative. Asked for pearson it reproduces the Pearson r of the values the
same site publishes, to 1.4e-8 (their sample_data endpoint rounds to three
decimals).
The live site stays the reference for the sample axis, the six-pair
overlap, the |r|-descending order and the reply shape. The statistic it is
asked for is the one place where it is not: it ignores the request.
|
|
correlation as 0.0
Two divergences from the live genenetwork.org, both measured against it.
**The sample axis.** GN2's `get_sample_r_and_p_values` walks
`target_dataset.group.samplelist` and takes the reference value by name:
the correlation runs over the group's animals, not over every individual
the trait happens to have a value for. Trait 1427571_at of HC_M2_0606_P
carries 99 values, the BXD panel holds 67 of those individuals, and the
answer changes materially -- GN2's top hit is a -0.845 probe that does
not appear in our top five at all. `corr_samples_group` now takes
`samples_primary` (default, the group's genotype panel, which is where
GN2's list comes from) or `samples_all` (the old behaviour); the panel is
read with `genotypes::resolve`, falling back to the group's
used-for-mapping strains where a group has no genotype file on this
machine. `samples_other` is refused rather than guessed.
**A perfect correlation.** The engine's r can land at 1.0000000000000002,
which makes 1 - r^2 negative and its p-value NaN. The live API answers
0.0 for that row (scipy's pearsonr), so the reply now clamps: r to +-1 and
p to 0.0. That is the reference's *output* being reproduced, not a
computation being redone -- #96 stays resolved in the same direction.
With the axis fixed, n agrees with the live site exactly (67) and r
agrees with GN2's own published per-sample values to 1.4e-8. What does
not yet agree is GN2's reported r itself: over the same 67 individuals,
its `/correlation` endpoint says -0.8446202819617522 for 1447043_at while
correlating the values its own `/sample_data` endpoint publishes for the
same two traits gives -0.8393085803732605; on a second pair the gap is
-0.8479321530363810 against -0.8451666639589470. Both pairs are off in
the same direction and by a similar amount, so the two GN2 endpoints are
reading different values for the same samples -- not a difference of
sample sets (all four BXD genotype files give the same 67) and not a
transform (2**v, 10**v, exp, v**2 and log were all tried). Until that is
settled, our numbers are the reference's published values correlated
correctly, and the ranking cannot be identical to a number we cannot
reproduce from the reference's own data.
|
|
The cases genenetwork2 and genenetwork3 assert, copied so GN4 answers the
same questions the same way:
- genenetwork3 test_normalize_values: only pairs where both sides have a
value survive;
- genenetwork3 test_generate_input: the dataset row keeps its positions,
a missing value is an empty field rather than a skipped one;
- genenetwork3 test_parse_results, with its own fixture file
(data/correlation/sorted_results.txt): |r| descending, positives and
negatives interleaved;
- genenetwork3 test_get_samples: an unparseable value never reaches a
correlation, and only shared sample keys are used;
- the overlap boundary GN3 encodes as `num_overlap > 5`, which is where
MIN_OVERLAP = 6 comes from.
What is *not* portable is most of their suites, and that is worth writing
down rather than discovering again: GN2's correlation unit tests are all
mocks (compute_corr_coeff_p_value patched out, header strings, a
three-value tuple), GN3's test_correlation is plumbing around the same
mock, and test_rust_correlation tests the file/JSON layer that this
implementation deliberately does not have -- it calls the engine in
memory. None of them assert a single r or p, because those come from
scipy at run time.
One port needed widening: GN3's get_samples fixture has four shared
samples, which is below its own minimum overlap, so asserting a
correlation on it is impossible in both implementations; the fixture is
widened and the assertion is that an unparseable value costs exactly its
own pair.
|
|
Either side of a correlation may now be a Publish (phenotype) dataset, not
just a ProbeSet one: a pheno trait correlated against an expression
dataset is what GN2's mapping page is usually used for, and returning 404
for BXDPublish was the wrong answer.
A publish reference trait is addressed by its numeric PublishXRef.Id --
that is how GN2 addresses phenotype traits, and a name there is refused
with a message saying so rather than a confusing "no such trait". The
bulk fetch gains the publish equivalent, keyed on the freeze's
InbredSetId, so the same individuals line up on both sides.
Measured live: BXDPublish/10001 (34 observations) against HC_M2_0606_P,
top 5, HTTP 200 in 5.0 s; 1440079_at at r -0.757 (p 3.2e-6) leads.
|
|
POST /compute/correlation now computes instead of returning canned data:
one reference trait against every trait of a target dataset, observations
being individuals, pearson or spearman, top-N by |r|.
The reply is positional -- trait, r, p, n (column names in meta.columns) --
with the method, the minimum overlap and the top-N stated in the notes.
`bicor` is refused with the methods that exist; a bad top_n is refused
rather than clamped; a dataset or trait with nothing to correlate says so
instead of returning an empty list.
src/db.rs gains the bulk fetch the sample axis needs: every value of every
trait of a dataset restricted to the reference trait's individuals, one
record per (trait, individual, value), ordered by trait so the caller
groups as it streams. It never loads a dataset whole -- ProbeSetData is
5.3e9 rows, so the query narrows by dataset and by individuals, which is
also why the sweep, not the arithmetic, is the whole cost.
Measured on the live db_webqtl: HC_M2_0606_P / 1427571_at against its own
dataset, top 5, HTTP 200 in 16 s; the trait correlates with itself at
1.0 over 99 observations, then 0.802 (p 1.9e-23), 0.790, -0.789, -0.788.
Synchronous is affordable at this size; GN2's job indirection exists for
top-N 20000 across datasets, which is the next thing to time.
Known and deliberate for now: the publish/genotype axes are not wired (the
fetch is probeset-only), and a perfect correlation reports p as null --
GSL's t statistic is undefined at |r| = 1, whose r comes out as
1.0000000000000002, so 1 - r^2 is slightly negative and the engine returns
NaN. Reproducing the engine exactly, rather than "fixing" the p-value
here, is the decision recorded in docs/correlations.md.
|
|
src/correlation.rs now holds the part the engine does not offer in memory,
and nothing of the arithmetic: `Pearson`/`Spearman` and their p-values are
correlation_rust's (and GSL's). What is ours is the alignment into the
engine's parser, the minimum-overlap gate, the ordering and top-N:
- alignment is by individual, never by position: an individual the target
trait lacks becomes "NA", which the engine's parser drops together with
its reference value, so a missing observation costs one pair, not the
row;
- minimum overlap 6, so a two- or four-point +-1 cannot appear;
- ordering reuses correlation_rust's own `custom_float_sorter` -- |r|
descending, NaNs last -- so GN2, GN3 and GN4 sort identically;
- top-N after the sort, clamped to GN2's 20000 ceiling;
- bicor is refused by name with the methods that exist, rather than
answered with something else.
Tests cover both methods, the six-pair minimum, name alignment across a
reordered and sparse candidate, the ordering with top-N, and the method
parsing. The route is still the stub; wiring it to the bulk fetch is next.
|
|
Ravanan's reuse key is (script, inputs) and both live in its --store under
RAVANAN_CACHE_DIR, so the cache's lifetime is that directory's, not the
container's. Measured: 13.6 s cold, then 3.6 s for the identical request
after restarting the container, with the store unchanged at 14 files --
Slurm's own spool state is lost and irrelevant to reuse.
The default cache dir is under /tmp, which is neither durable nor visible
inside a container that bind-mounts one host path, so a job cannot read
the interned inputs. The reply now carries a note when the cache dir is
under /tmp, naming the fix, rather than leaving it to be discovered as a
slow or failing run.
docs/ravanan-slurm.md gains a "What survives a restart" table (ravanan
store, scratch, token registry, gemma-wrapper cache, Slurm state) and a
path checklist; the README points at it.
|
|
stale tests
`ccwl compile` emits CWL as YAML (its only other target is dot), so the
new input check must read that shape -- it was parsing JSON and would
have turned a working request into a 502. A hand-written scan of the
`inputs:` block, deliberately not a YAML dependency: ccwl's output is
machine-generated and unvaried, and adding serde_yaml would mean
regenerating the crate list in guix.scm for one small check.
Two tests were stale in ways only the package's own check phase shows:
- tests/auth.rs built AuthToken literals without the new `scheme`/`id`
fields, so the crate's integration tests did not compile;
- test/python/test_envelope.py asserted that /version carries no `next`
links, which stopped being true when every read endpoint gained them
("always offer the next step"). The deployed build fails that test
too, so it is not a regression from this work -- but it did block
`guix build -f guix.scm` at the python-rest-test phase.
Verified with the packaged binary (guix build -f guix.scm, all phases
including both test suites): POST /compute/gemma/ravanan through ravanan
and the Slurm container returned 200 with 636 rows in 13.6 s, on a
request authenticated with a gn4ai_ short token -- the log line naming
user zachs, scheme opaque and token smoke-2.
|
|
cwl/gemma/gemma-loco.scm is read from the deployment's checkout at
request time (GEMMA_RAVANAN_WORKFLOW), not compiled into the binary, so
a checkout and a binary from different commits can meet. When they do,
cwltool answers "Type mismatch for input `kinship_pheno_file'; expected
`File' but got `null'" -- which says nothing about the cause. Found by
pointing the deployed f64a708 binary at the current workflow.
The compiled CWL now has its declared inputs compared with what this
build is about to send, before anything is staged for ravanan: a missing
input or an undeclared one is refused with both lists named, plus the
instruction to take the .scm and the binary from the same commit.
|
|
The access log said who (`user`) but not with what: with short tokens a
request can be made by a shared credential, so "which person" and "which
credential" are now separate fields -- `scheme` ("jwt" | "opaque") and,
for an opaque token, `token` (its registry handle, e.g.
`agent-ravanan-3`). Never the token or its hash, and the path still
carries no query string.
The line also gains `ts` (RFC 3339 UTC), so a redirected stderr file is
self-describing rather than relying on journald's prefix. Hand-rolled
civil-from-days rather than a new crate: one format, and no Cargo.lock
drift.
scripts/gn4-log.py reads such a log and shows it as a table with the
handle joined against tokens.sqlite: the subject as the registry has it
now (flagging a handle whose subject has been repointed), and the token's
status today -- [revoked], [expired], [unknown] -- which is how "an agent
was working at 14:02" is told apart from "an agent was working at 14:02
with a credential revoked since". It reads the registry read-only, passes
non-JSON lines through in table mode (the process log interleaves
backtraces), and offers --json and --summary for jq and for counting.
CallerSlot now carries the whole Caller (scheme, id, subject) rather than
just the subject string.
|
|
A second credential format beside gn-auth's JWT, for agents and scripts:
`gn4ai_` + 56 body characters + 2 checksum characters = 64, from an
alphabet with no look-alikes, so a typo is refused before the registry is
consulted. Opaque by design -- nothing is stored in the token; every fact
lives in one SQLite database (`tokens.sqlite`, GN4_API_TOKENS_DB, mode
0600, gitignored) keyed by the SHA-256 of the secret. Revoking is a row,
takes effect on the next request, and a token in a log line reveals
nothing; lose the registry and every short token stops working.
Administration is local-only: GET/POST /tokens, DELETE /tokens/{id} and
GET /tokens/who are behind a router layer that refuses any caller whose
TCP peer is not 127.x.x.x/::1, and any request claiming to come through a
proxy. A layer rather than a per-handler check, so nothing of a remote
request is parsed before it is refused; there is no configuration switch,
because minting credentials must never be reachable from the network.
A short token reads public data only: every token-only endpoint, and
`#[gnauth(secured)]` endpoints for public rows; a confidential dataset is
refused with a 403 that names the JWT route, since gn-auth cannot validate
an opaque token (the on-the-fly-bridge and registry-grant alternatives
were both rejected for v1).
scripts/gn4-token.py create|list|revoke|who drives the routes, so there is
one implementation of the format and the revocation rules; the token is
printed exactly once.
Tests: the localhost guard over all four routes (remote peer, no peer,
proxied header), and mint -> authenticate -> who -> revoke -> 401.
|
|
gemma -gk reads the phenotype file only to learn which individuals are
analysed -- the matrix comes from the genotypes -- so the workflow now
gives the kinship step a constant-valued pheno file carrying just the
trait's missing-data pattern, and gives gwa the real values. Verified
byte-for-byte: -gk on a real trait and on its constant-valued twin
produce identical .cXX.txt files, and trait 2's assoc checksums are
identical whether K came from the constant pheno or the real one.
Consequence, measured on the BXD fixture: a second trait over the same
animals costs +1.0 MB of store and 7.5 s (20 GWA jobs) instead of
+13 MB and 14.6 s (a second kinship scatter), because the 20 K matrices
are identical inputs for both traits. Three keys now: workflow (the
ccwl source), kinship (genotype + snps + lmm + the NA pattern) and
trait (kinship key + the values), so the compiled CWL and the kinship
pheno file are shared and only the trait's pheno file and inputs YAML
are per-trait.
This is also stricter than gemma-wrapper's own key, which ignores the
phenotype entirely and would reuse a K computed for a different set of
animals.
|
|
The proxy logs the caller through x-gn4-user, but the process log -- the one an
operator tails while a job runs -- named nobody, so "who started this GEMMA
run" meant going to the nginx log or guessing. genenetwork4 now writes one
compact JSON line per request on stderr:
{"event":"request","method":"GET","path":"/groups/BXD","status":200,
"took_ms":10,"request_id":"…","user":"zachs"}
Decisions in it: `user` is absent, not "-", when no token was verified (a dash
is a value a reader can mistake for a user name); the path is logged without
its query string, because a query can carry a credential and has no business
in a log; the token value itself is never logged, only its subject; and the
line carries the `request_id`, so it sits beside the existing `event: error`
lines for 5xx and one grep joins a failure to the request that caused it.
`GN4_ACCESS_LOG=false` turns it off where the proxy's log is the record of
truth.
Verified on a probe instance: with the test token the line says
user "gn4-test-stub", with zachs's token user "zachs", and an unauthenticated
401 has no user field at all.
docs/www/operator-checklist.md claimed "genenetwork4 does not log the token's
sub" -- true until now, which is exactly the kind of sentence that rots; it is
replaced by the line above, what it contains and what it deliberately omits.
README's x-gn4-user paragraph gains the same, next to the identity it already
described.
|
|
compute_ravanan wrote <pid>-<nanos>-pheno.txt, -ravanan-inputs.yaml and
-gemma-loco.cwl, so ravanan's (script, inputs) work-reuse key was new on
every call and a repeat of the identical request recomputed all 40
steps. Naming them by a hash of the request's inputs (genotype path and
size, snps contents, trait values, lmm) makes an identical request hit
the store instead.
Measured on the BXD fixture, full 20-chromosome LOCO mapping,
single-machine: 14.3 s cold, 1.6 s warm, identical .assoc.txt checksums.
For comparison the gemma-wrapper stack costs 11.9 s cold and 12.1 s
warm -- its own cache did not hit on a repeat here -- so the ravanan
path is now the cheaper one for any repeated request and the slightly
dearer one for a first request.
The run log stays per-call: it is what a caller streams by run_id. The
three shared files are written aside and renamed, so two concurrent
identical requests cannot read a half-written file.
|
|
gemma-wrapper was doing two things ravanan now does or GEMMA already
does: the parallel fan-out (ravanan's CWL scatter replaced it long ago,
which is why every job passed --no-parallel) and LOCO orchestration --
gemma 0.98.6 documents `-loco [chr]` natively for both -gk and -lmm and
takes the annotation file it needs. What was left was a content-hash K
cache and a --json record with results packed into .tar.xz archives.
So cwl/gemma/gemma-loco.scm now runs two gemma invocations per
chromosome: `-gk -loco <chr> -o <chr>-kin`, then `-k <that matrix>
-lmm <n> -loco <chr> -pve -o <chr>-gwa`, with `-outdir .` for flat,
deterministic names. The kinship matrix travels from one step to the
other as a CWL File output, and each job's .assoc.txt and .log.txt are
the workflow's outputs -- so compute_ravanan reads files instead of
parsing a wrapper JSON record and unpacking archives, and the pve note
comes straight out of the jobs' logs.
Measured on tests/test_data/BXD (chr 1 and chr 2): identical rows --
636/636 and 583/583 on beta, se, l_mle, p_lrt. One chromosome end to
end: 496 ms (gemma) vs 1140 ms (wrapper), 2.3x per job. The wrapper
also needed Ruby and pfff on PATH, and named its outputs with the
extension doubled (...1.assoc.txt.assoc.txt).
/compute/gemma keeps gemma-wrapper, deliberately: that is what makes
the two endpoints comparable rather than two copies of one stack.
scripts/gemma-compare.py runs the same body through both and reports
row-level differences and timings.
The gemma-wrapper content-hash K cache is the one thing given up;
a repeat request recomputes (271 ms of kinship on this fixture).
|
|
Two follow-ups to the parental-design endpoint.
/groups/{group} now carries the headline in its notes -- "Parental design:
C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes
the parents and F1s by default" -- plus a next link to the full design. "Which
cross is this?" is two strain names, and needing a second request to see them
was the only reason the endpoint had to be called first. A group with no
metadata says so in the same note, pointing at parents_and_f1s.json and at
`individuals.exclude`, rather than leaving the reader to wonder why nothing was
dropped.
The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it
drifted once already, silently: the paternal/maternal labels were swapped
against GN2's unpacking, and nothing noticed because the exclusion rule takes
both strains regardless of role. scripts/check-parental-metadata.py parses
ParInfo with ast (no gn2 import) and compares it group by group, reporting
disagreements per role and treating groups the JSON knows but GN2 does not as
informational. tests/parental_metadata.rs runs it while a GN2 checkout is
present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at
which point the JSON is the only copy and there is nothing to drift from. It
reports 23 groups reproduced, 1 extra (BXD-Longevity).
|
|
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only
implied. It reports the F1 strains, the maternal and paternal strain (from the
deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups),
the @type of the group's genotype file, the individuals each role names where
the metadata names them (BXD-Longevity does, most do not), and -- the part a
caller needs -- which members of the current panel the default rule drops and
which the `F1` name heuristic catches.
Two things that came out of building it and comparison with GN2:
* the JSON's paternal/maternal labels were INVERTED against GN2's reading.
GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal
strain is C57BL/6J; our file said paternal. No mapping was affected
(non_strains takes both regardless of role) but the metadata was wrong, and
anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat
note -- would have been backwards. The file is rewritten from ParInfo itself
(ast-parsed, not retyped), 23 groups' roles filled in and the 24th
(BXD-Longevity, which GN2 does not have) left as it was.
* an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's
own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports
excluded_from_panel and excluded_by_name separately and says which is doing
the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1
names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming
exclusion entries that match nothing.
genotypes::non_strains is rebuilt on the same typed parser (DesignRole /
ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit
test for the roles, empty-name handling and the unlisted-group source, and an
integration test asserting BXD's maternal/paternal orientation and the note.
|
|
Parents and F1s are excluded by default (as in GeneNetwork2/3), with
include_parents_f1 to keep them all. That boolean was not enough for what was
asked -- "include these, drop that substrain" -- and the exclusion list itself
is data (parents_and_f1s.json, seven groups) that no request can reach. F1
names also differ per group (B6D2F1 for BXD, BHF1/HBF1 for BXH,
C57BL/6JxBXD037F1 for the intercross F1s), so pattern matching is the wrong
tool.
Every group-mapping endpoint -- /compute/gemma, /compute/gemma/ravanan,
/compute/rqtl, /compute/qtl2, /compute/fitqtl, /compute/bulklmm -- now takes
"individuals": {"include": [...], "exclude": [...]}
applied after the parent/F1 rule, exclude winning over include. An unknown name
is a 422 invalid_individuals whose message lists the panel: a silently dropped
animal changes a kinship matrix without anyone noticing, so the failure has to
be loud. genotypes::select_panel is the one implementation, and it returns what
it did (dropped parents/F1s, re-included by name, excluded by name) so the
Genotype note can say it: "197 individuals, 38 parents/F1s dropped; 1 excluded
by name".
The note now also spells out the distinction that makes this option
treacherous: the kinship uses *every* individual in the genotype file, the
association uses only those with a trait value -- GEMMA reads no phenotypes for
the kinship. So "include the F1s" changes the relatedness structure without
putting the F1s in the fit, and a request can legitimately have 234 individuals
in the genotype file and 34 in the association. Verified on BXD.8: excluding
BXD1 gives 197 individuals / 33 values; include_parents_f1 with an F1
re-included and a substrain excluded gives 234 / 34.
A basket supplies values by name, so a selection applies to a basket-fed
mapping the same way: the individuals excluded are simply not in the panel any
more, and their basket values are ignored.
GenotypeError gains Invalid for this (422) so a caller's typo is not reported
as a missing genotype file; the read-only endpoints that can never see one map
it the same way for exhaustiveness.
|
|
Two additions that make the endpoint usable rather than merely correct.
POST /compute/bnw/features -- from a network to a mappable trait. Every node
with at least one parent at the threshold is a module (the node plus its
parents), scored per sample by the first principal component of the module's
standardised members, sign-corrected so the score rises with the node. Without
?module= the reply is one row per module -- module, members, size,
variance_explained -- where variance_explained says whether the score is a fair
summary at all. With ?module=NAME it is that module's per-sample scores in the
{name, value} shape every mapping endpoint already takes, so the chain runs
network -> module -> /compute/rqtl or /compute/gemma with no translation layer.
That is what BNW is for in GeneNetwork; the diagram is a by-product. The notes
name the construction (a citable module score must say what it is) and repeat
the caveat: a module is a correlation structure, not a QTL.
GET /compute/bnw/examples and /compute/bnw/examples/{name} -- BNW's own worked
examples, for an agent to explore instead of inventing data. The five datasets
its tutorial and examples pages present are listed with their sizes, and each is
served as a matrix in the values shape /compute/bnw takes, with a ready-to-post
action carrying the body. ?columns= restricts the matrix (the spleen example
has 14 variables and a caller may want fewer). genotype_column warns that a
column is a genotype code: this driver is continuous-only, so it enters the
network as a number, the way the web application's GeneNetwork path treats it.
Files come from BNW_EXAMPLES_DIR (a gn-bnw checkout's sourcecodes/
example_datasets); a deployment without it answers 404 no_examples.
Measured on the dev instance: features over five BXD traits gives two modules
(BXD::10001 with BXD::10003 and BXD::10041, 60.4% of variance; BXD::10044 with
BXD::10001, 27.2%), and BXD::10001's 19 scores posted straight to /compute/rqtl
and /compute/gemma both return 200 -- gemma peaks on chromosome 6. tests/bnw.rs
covers the module list, one module's scores in mapping shape, the unknown-module
422, the example list, one example's matrix and its ready-to-post action, and a
404 for a name that does not exist.
|
|
Post the reply from /compute/bnw back and get an SVG: the same plumbing as
/plot/gemma and /plot/bulklmm (plot::render, PLOT_PYTHON + BNW_PLOTTER, image/
svg+xml, the plotter's line in x-plot-summary), nothing recomputed.
Two figures, because one cannot say what the other can:
* the network (default): nodes on a circle, one arrow per edge that survived
the threshold, arrow width and label = the edge's posterior, and the
below-threshold edges as faint dotted lines -- so a reader sees what was
*considered* as well as what survived, which is half of why the posterior
table exists;
* ?kind=posterior: every candidate edge as a cell in a matrix, dark = high
posterior, outlined cells are the ones above the threshold. The diagram
cannot show that an edge came within 0.02 of the threshold; this can.
Pure-Python SVG, standard library only: no graphviz in the deployment. A
circular layout is less pretty than a force-directed one and is deterministic,
which matters more for a figure that accompanies a JSON result. Node labels
keep the distinguishing tail of a long id, with the full name beside the node.
Verified live on the dev instance: POST /plot/bnw with a saved /compute/bnw
reply returns 200, image/svg+xml, 5 256 bytes, and x-plot-summary naming the
strongest edge (Trait2 -> Trait1, posterior 1.000); ?kind=posterior returns the
second figure. tests/plot.rs covers both figures plus the 422s (not a BNW
result, unknown kind).
|
|
POST /compute/bnw learns a network over a set of variables and returns one row
per candidate edge: from, to, posterior, in_structure. Inputs are published
traits of a group (group + traits) or an explicit matrix (values: one entry per
variable with per-sample values), which covers the case the database cannot --
a basket's columns, a handful of probe sets.
BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x
samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and
parses the reply: nothing about the three C programs, the private work
directory or the type line in the input file lives here, so BNW's pipeline can
change without a GN4 release. The driver's failures map onto the API's own:
bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed
computation 502 bnw_failed in the driver's words.
Decisions worth recording:
* parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold
0.5, read from its create_tiers_gom_part1.php rather than chosen here --
and the reply names the values used, because a network at 0.5 and one at
0.9 are different answers to the same question;
* at most 40 variables and at least 3 samples complete across every variable.
The first is the structure search's exponential in-degree; the second is
BNW modelling a complete multivariate normal, so a sample missing any value
is dropped -- and the note says how many survived rather than silently
shrinking the panel;
* its own gate, always claimed (threshold 0): there is no such thing as a
small BNW run, the cost is the search, not the number of individuals;
* the notes say plainly that a posterior is not a p-value and that a network
is not a QTL, because that is the easiest thing to over-read.
tests/bnw.rs compares the endpoint against the answer the BNW *web
application* gives for its own six-trait example (the same fixture as the
driver's parity check, gn-bnw/test/golden/): the five edges must match, and the
notes must name the parameters and carry the caveat. It skips when the driver
or its binaries are absent.
Not deployed to production.
|
|
cannot do
Two gaps, both invisible until a real request ran against a cluster.
The endpoint read only genotype_file, snps_file and trait_values from the
request; group, genotype_version and samples were accepted by the shared
request type and silently dropped. A /compute/gemma-shaped request
(group + samples) therefore arrived as empty paths and empty values and
died with "I/O error: No such file or directory (os error 2)" from
reading "". Running it exposed the second gap in the same breath: the
gate sized the request from trait_values.len(), so a samples-only request
was sized 0 and took no concurrency slot at all.
resolve_inputs is now one function in routes/gemma.rs, called by both
handlers: upload materialised into the pair the engine reads, basket
resolved to named values, group prepared with the parent/F1 and
empty-genotype rules, named samples aligned by name onto whatever
individuals the file has, positional values checked against the panel
size. That block was ~250 lines of duplicated rules inside compute_gemma,
which is how it drifted in the first place; the direct runner now calls
the same function and keeps only its covariates, normalization and gate.
The ravanan reply also carries the input notes now (which genotype
version, how many values aligned, what was dropped), so a run says what it
mapped as well as where the peak is.
Refused rather than ignored, because the CWL workflow has no step for
either: covariates (422 covariates_not_supported) and normalize (422
normalization_not_supported), each naming /compute/gemma as the endpoint
that can do it. A new test asserts both refusals happen before any job is
submitted, so it needs no ravanan, ccwl or cluster.
Verified against the Slurm container from gn-machines: group: BXD became
40 Slurm jobs (20 kinship + 20 GWA, scattered over the snps file's
chromosomes), all completed, 21 055 rows in ~3.5 min. README and
docs/ravanan-slurm.md record the four requirements that run turned up,
three of them not obvious: ccwl on PATH, a ravanan build with the
--slurm-api-endpoint fix, the *server's* HOME writable by the job user
inside the cluster (gemma-wrapper locks $HOME/.<hash>.lck there), and
GUIX_PACKAGE_PATH for the manifest.
|
|
The ravanan path asks for the same variance components as the direct one:
gemma-loco.scm's gwa step passes -pve, so each chromosome's job writes
pve/se(pve)/vg/ve into its own .log.txt, and compute_ravanan turns them
into the same note (every chromosome, plus the extremes with vg/ve) and
the same unrounded "variance:" lines in the ?id= run log.
It reads them from the extracted GWA archive rather than from the copy
gemma-wrapper keeps beside it, and that is the point of doing it here:
the copy beside the archive is a 5eb53fa feature, while a ravanan job
runs whatever gemma-wrapper its own Guix manifest resolves, which need
not be that new. The .log.txt has always been inside the archive, and
the archive is unpacked anyway for the chromosome's .assoc.txt -- so
read_variance_components_from_logs labels each log by the "-loco <chr>"
in its command line, falling back to the <hash>.<chr>.assoc.txt.log.txt
file name, and sorts chr1..chrX.
Verification, precisely: the CWL compiles with -pve (ccwl compile, checked
by hand), the archive reader is unit-tested (labelling from the command
line, from the file name, numeric ordering, a log without the block
dropped), and a real archive from this deployment does carry the block.
An end-to-end ravanan request could NOT be completed on the dev box, for
reasons that have nothing to do with -pve and are now written down in
docs/ravanan-slurm.md: ravanan's own Guix evaluation of the manifest
fails ("rust-pyo3-0.26.0: unbound variable" -- the ambient channel set,
since the production process does not set GUIX_PACKAGE_PATH at all, so
this endpoint cannot resolve gemma-wrapper there), its intern of inputs
hard-links and so needs store and inputs on one filesystem ("Invalid
cross-device link"), and the one job that did start sat "Waiting for
lock" on gemma-wrapper's $HOME/<hash>.lck until the call timed out.
|
|
The note reported the range and named the two chromosomes that bounded it,
which is not enough to answer the question the numbers exist for. Under
--loco each pve belongs to a *different* kinship matrix, and GEMMA's
per-chromosome .log.txt files live on the server -- so a caller told "0.31
to 0.95" could not ask which chromosome was which, or compare a single
chromosome against BulkLMM.
The note now lists all twenty, compactly, with the range and the two
extremes keeping their vg/ve:
GEMMA -pve, null model, per chromosome (LOCO): chr1 0.6674±0.3451,
chr2 0.8426±0.3668, ..., chrX 0.8563±0.3525. That is pve 0.3118 to
0.9549 over 20 chromosomes; lowest chr8 pve 0.3118 ± 0.5009 (vg
23.3659, ve 12.3389), highest chr7 pve 0.9549 ± 0.3570 (vg 76.0933,
ve 0.8661).
and streaming::append_lines puts the unrounded values in the run log as
variance: chr7 pve=0.954938 se_pve=0.357032 vg=76.0933 ve=0.86612
so a ?id= run's /compute/gemma/status stream carries them while the run is
still in flight, rather than only in a reply someone has to keep. Both
measured on BXD::10001 through the dev server before committing.
The values are printed with four significant figures (scientific when the
magnitude needs it), unchanged: rounding variance components to four
decimals was the earlier bug, and these are the same numbers a reader
compares with BulkLMM's h2.
Docs: README's /compute/gemma section, SKILL's gemma bullet ("quote
per-chromosome values, not an average: they are twenty different kinship
matrices") and docs/bulklmm.md's example.
|
|
A failing scan came back as
BulkLMM failed: in expression starting at scripts/bulklmm_scan.jl:26
which names neither the package nor the reason. Julia had said it plainly,
two lines up:
ERROR: LoadError: ArgumentError: Package CSV not found in current path.
- Run `import Pkg; Pkg.add("CSV")` to install the CSV package.
The complaint finder looked for a line starting "Error" -- Julia writes
"ERROR:" -- and fell back to the last non-empty line, which is the line
number. That is the same defect as gemma-wrapper's Ruby backtrace, in our
own error handling: the wrapped tool said why, and we threw it away.
julia_complaint takes the first ERROR:/LoadError line plus Julia's hint
line when there is one, with the last line kept only as a fallback for a
driver that fails silently. Three unit tests cover the three shapes.
The integration tests also stop failing on a half-instantiated Julia
project: bulklmm_available used to accept any directory with a
Project.toml, and /tmp/gj-env has one naming BulkLMM and nothing else, so
the scan died on `using CSV` instead of skipping. It now requires the
project to name the three packages the driver loads, which is what a
usable env is.
|
|
tests/envelope.rs follows every href in every reply, and running it end to
end found six that were not addresses:
- /species, /species/{name}/datasets and /groups/{group}/datasets offered
/species/{species} and /datasets/{dataset} -- templates, which 404 when
followed. Each now points at a real row from the listing in hand, and
the description says any other name can go in its place.
- /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the
listing it already links is the callable one, so the shape moved into
that link's description.
- /traits/{dataset}/{trait}/values offered a GET
/traits/{ds}/{trait}/correlation, which does not exist: correlation is
POST /compute/correlation and that reply already carries it in
"actions". Removed, with the reason written where it was.
- the same reply built its /groups/... links from the *dataset* name.
BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is
a 404 (no genotype files for it). db::get_group_name resolves the group
from InbredSet.Id and the links use it; when a dataset has no group the
group links are simply absent.
- /groups/{g}/phewas with no parameters is a 422 ("give either locus, or
chr and mb"), so that link is now a shape with placeholders.
Placeholders stay where the caller must supply a value
(.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips
them only when the description says what to fill in -- otherwise a client
reading the next list cannot tell a call from a template.
Two stale expectations alongside: /version carries the root links now (it
is the natural first call), and /search's results put its own follow-ups
first and the root links after them.
Tests are honest about their environment too: links skips when no search
index is configured (an empty XAPIAN_DB_PATH answers 502), and the
envelope crawl exempts /openapi.json, the API document, which is
deliberately not enveloped.
|
|
The GWA step now passes -pve, and the reply gains a note with the
variance components GEMMA computed for the null model. Under --loco
there is one per chromosome, so the note gives their range and names
the two that bound it:
GEMMA -pve, null model, per chromosome (LOCO): pve 0.3118 to 0.9549
over 20 chromosomes -- lowest 8 pve 0.3118 +/- 0.5009 (vg 23.3659, ve
12.3389), highest 7 pve 0.9549 +/- 0.3570 (vg 76.0933, ve 0.8661).
That is BXD::10001 measured, not assumed. pve is the proportion of
variance the kinship explains -- the same quantity as /compute/bulklmm's
h2, per chromosome rather than genome-wide -- which is what the
16627/16769 comparison against BulkLMM needed.
Reading the numbers took three decisions worth stating:
- Four decimals is wrong for variance components. The first live run
reported 'vg 0.0000, ve 0.0000' for a BXD::16769 null fit whose real
components are 3.411e-5 and 3.411e-10 -- both rounded to zero, which
reads as 'no variance' when the split is 1.0. Components are printed
with four significant figures, scientific when the magnitude needs it,
the way GEMMA's own logs do with %g.
- se(pve) is -nan on a boundary, and Rust parses -nan as a NaN f64, so
the numeric parse alone let it through as NaN. Non-finite values are
now 'not available': a boundary estimate has no standard error.
- Twenty chromosomes of values bury the other notes, and every value is
in the .log.txt beside the archive anyway, so the note names the ends.
Requires gemma-wrapper 5eb53fa, which keeps each run's .log.txt beside
its archive instead of only inside the .tar.xz.
Docs: README's /compute/gemma section, SKILL's gemma bullet (compare pve
with h2, never vg with sigma2_e -- pve is scale-free), and
docs/bulklmm.md's 'what would settle it', which now says the plumbing is
in rather than pending.
|
|
The GTEx problem -- it is human-only, and a mouse symbol there is an empty
report -- is answered by the endpoint that already knows which entity it
matched: /genes/{symbol}/aliases now returns
label the entity's canonical symbol in that species ("Shh" for mouse,
"SHH" for ?species=human) -- the symbol a human-only service needs
links [{name, href, description}] built for that symbol and species:
RGD, MGI, BioAlliance, NCBI Gene, Wikidata (the QID), UniProt,
STRING, GeneMANIA and GeneCup, plus GTEx/Open Targets/Protein
Atlas/GWAS Catalog for a human entity or BioGPS/Allen Brain Atlas
for a mouse one
So a mouse Shh reaches GTEx through the human ortholog's label rather than a
guessed one, and a client no longer holds the URL vocabulary. Only links
that answer are built: HomoloGene, UniGene, Genotation, GeneBridge and
PhenoGen are gone, and Panther is browser-only (see SKILL.md "Gene
link-outs", where each URL was checked).
A cached entity written before the label was kept is refreshed rather than
served -- otherwise the link-outs would be built on a symbol it does not
have -- and `stale: true` still says when Wikidata was unreachable and the
cached answer was used.
tests/wikidata.rs: a mouse gene gets the mouse services and no GTEx; the
human ortholog gets label SHH and a GTEx href built from it. Both halves
skip when the reply is stale, since a Wikidata timeout is not a failure.
|
|
Checked what the plot actually draws: it has never drawn the alpha 0.10
line -- the plotter matches 0.05 and 0.01 only. What lists all three is the
reply's notes ("alpha 0.01: LOD 4.58, alpha 0.05: 3.65, alpha 0.1: 3.50"),
which is where the 0.10 came from.
Still, the point stands on its own: 0.10 is the loosest bar in the set and
drawing it invites reading it as the line to clear. So the defaults are
stated and pinned -- 0.05 and 0.01 on the plot, 0.10 only on request
(?all_thresholds=true, or an explicit ?thresholds=) -- alongside GEMMA's
Bonferroni, which is opt-in for the same reason: a reference line is an
annotation, not a result.
tests/bulklmm.rs counts the lines: exactly two by default, and no
alpha 0.10 label.
|
|
The OpenAPI description still promised "a Bonferroni reference line by
default", and GEMMA's own suggested plot call said the same. Both now say
what the code does, and where the number comes from: the plotter computes
0.05 over the analysed markers -- conservative under linkage and unrelated
to relatedness -- and ?thresholds= takes one computed elsewhere, each line
labelled with its source.
Also corrected in the README's /plot section and in SKILL.md, where the
same promise had been copied.
|
|
A line on a plot is an annotation, not part of the result, so the default
is now no line at all. GEMMA's Bonferroni becomes opt-in like any other:
?bonferroni=true GEMMA's own line, labelled
?thresholds=3.65&threshold_labels=… another engine's, labelled
(nothing) just the scan
Which means a reader sees exactly the lines someone chose, each with its
source in the label -- and the accidental "this plot says significant"
impression of a default convention line is gone. The plotter's summary and
its <desc> no longer mention Bonferroni unless it was drawn.
tests/plot.rs: the plain plot has neither line; with thresholds the two
BulkLMM lines are labelled and Bonferroni is absent; with ?bonferroni=true
GEMMA's own line appears.
|
|
GEMMA has no permutation test here, so its plot carries a Bonferroni line.
A threshold from /compute/bulklmm belongs to a different null and must not
look like GEMMA's, so it is passed in and named:
POST /plot/gemma?thresholds=3.50,3.65
&threshold_labels=bulklmm%20alpha%200.10,bulklmm%20alpha%200.05
Each is drawn dashed in the series colour with "label: value" beside the
line, with GEMMA's Bonferroni line still there, and the plotter's
x-plot-summary names the reference lines it drew. A threshold that is not
a positive number is a 422 invalid_threshold rather than a silent omission.
/compute/bulklmm's reply offers that href already, with its alpha 0.05
threshold filled in -- so "same values, two models, one picture" is one
link from the scan, and the labels keep the two nulls apart.
tests/plot.rs: the plain plot has no bulklmm line; with thresholds both
are drawn and labelled; a non-numeric threshold is refused.
|
|
"is the kinship term doing anything?" should be answerable from the reply.
The driver now prints h2 and the residual variance under the null model, and
the reply carries both:
Variance components: h2 = 0.000000, sigma2_e = 0.402171, estimated once
under the null (intercept) model from y and K, not per marker. With a few
dozen phenotyped individuals these estimates pile up at the boundaries 0
and 1 -- the likelihood is flat there -- so read h2 as "kinship matters a
lot / not at all" rather than as a precise proportion. At a boundary
(h2 ~ 0 the model is ordinary regression, which is why its LODs can match
R/qtl's to the digit).
Checked with a strain-ID-driven trait (values = rank of strain), which has
h2 ~ 1 by construction: BulkLMM reports h2 = 1.000000, sigma2_e = 3e-06 and
a peak of LOD 3.1 -- the kinship term absorbs the strain structure -- while
R/qtl, which has no kinship term at all, reports a spurious peak of 5.06 on
the same values. That is the discrimination the h2 column was missing.
|
|
The marker blocks a single-trait scan is split into were the driver's
default (Threads.nthreads()) and changed with --threads. They are a
separate knob now:
BULKLMM_THREADS Julia threads (--threads) default 4
BULKLMM_BLOCKS independent marker blocks, scanned default 4
in parallel (1 = serial scan)
Blocks are exact either way -- LOD 9.589938 and the same permutation
thresholds at 1 and 4 -- so the knob only buys wall clock. The reply says
what ran ("split into 4 independent blocks and scanned on 4 Julia
thread(s) (BULKLMM_BLOCKS / BULKLMM_THREADS ...)"), because the answer to
"is it multicore?" should be in the response and not in the log.
|
|
We do not run GEMMA permutations -- each repeats the whole LOCO scan and the
gate gives GEMMA one slot -- but "not supported here" is not the same as
"not available". The refusal now says both:
1. POST /compute/bulklmm with nperm: 1000 -- an LMM whose thresholds come
from rotations that keep the kinship (R/qtl's nperm does not, and is
anti-conservative on a structured panel);
2. run GEMMA's own permutation locally. The reply's notes already name the
bimbam pair and the phenotype file, so `gemma-wrapper --permutate` or
`gemma -lmm 9 --permutate` on those inputs reproduces exactly the null
GEMMA would have used -- hours instead of seconds, which is the honest
price of that particular null.
docs/bulklmm.md gains the table this belongs in: three ways to get a
threshold, ordered by how honest the null is for a structured panel
(rotation with kinship; GEMMA's own LOCO permutation run locally; R/qtl's
nperm as a sanity check only). SKILL.md says the same in two lines.
tests/gemma.rs now asserts both halves of the refusal, and the earlier
assertion on /compute/rqtl -- which stopped being true when the refusal was
rewritten -- is replaced rather than left passing by accident.
|
|
The threads/blocks/read/scan breakdown and the permutation thresholds are
two separate lines on stderr, and the reply only carried the last one --
so the answer to "is it multicore?" was in the log rather than in the
response.
|
|
Reported: a BulkLMM run does not look multicore. It is not a
misconfiguration. BulkLMM's single-trait `scan()` -- what this endpoint
calls -- has no `Threads.@threads` at all; the package threads
`bulkscan()` (many traits) and bulkscan_helpers, not scan. The
parallelism that was visible before was BLAS (OpenBLAS defaulting to all
96 cores here), which is why `user` time ran 4x `real` while Julia
threads made no difference: --threads 1 13.7 s, --threads 4 12.3 s.
* The driver splits the markers into independent blocks and scans them
on Julia threads (--blocks, default = BULKLMM_THREADS). It is exact:
the LOD for each marker is independent given the null variance
components, which depend on y and K only. Verified identical -- peak
LOD 9.589938 and the same permutation thresholds at 1 and 4 blocks --
and permutation thresholds combine by taking each permutation's
maximum across blocks before the quantiles, which is what a
genome-wide threshold is.
* The parsed matrix and the kinship are cached per panel (read 1.3 s ->
0.4 s, kinship 0.9 s -> 0), because both depend only on the cross.
* The summary in `notes` now says what ran: julia threads, blocks, BLAS
threads, and the read/scan split.
* The hand parser is back: CSV.jl into a DataFrame is an order of
magnitude slower for a 21056 x 235 matrix (14-17 s against 2 s).
* BLAS is capped at 4 threads -- 96 was slower (10.5 s against 7.7 s for
one scan).
Measured on BXD.8 trait 16627 (21056 markers): 1 block 20.9 s, 4 blocks
18.3 s, same answer. The gain is modest because the LMM loop is ~8 s of
the run and each block re-estimates its variance components. Where the
cores pay is bulkscan() -- many traits in one process, threaded over
trait blocks -- which is the shape the next endpoint wants; documented in
the README rather than left as folklore.
|
|
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values
have no threshold we can compute: a GEMMA permutation repeats the whole
scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual
stand-in and it is the wrong null -- it shuffles phenotypes as if the
individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global
kinship matrix and permutes by rotating the data through the kinship's
eigen decomposition, so a threshold costs seconds.
POST /compute/bulklmm {group, genotype_version, samples|basket,
nperm, seed, reml, assumption}
Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8,
35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2
4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81,
0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's
published locus and LOD, and a significant peak against a threshold this
API could not compute before.
Two differences from GEMMA are in every reply, not implied: a global
kinship matrix rather than leave-one-chromosome-out, and h2_null saying
how much relatedness was found (here ~0, so the fit is effectively
fixed-effects -- which is why it agrees with R/qtl where GEMMA differs).
* src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a
strains x markers matrix (our BIMBAM pair is already the allele
dosages BulkLMM wants), a marker map, the phenotype and covariates
-- runs the driver through streaming::run_streaming_with_env, and
reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid
f64 in Rust and would otherwise be reported as a value.
* scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds),
with the seed recorded so a threshold set is reproducible.
* gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is
cheap, unlike R/qtl's.
* Julia comes from the deployment's own store binary with a prepared
depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request
runs JULIA_PKG_OFFLINE=true, so a scan can never download anything.
`--project=<path>`, not `--project <path>`: Julia 1.8 takes the
space-separated form as a program argument.
tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and
seed are in the notes, and a scan without permutations says there is no
threshold rather than leaving a bare LOD to be read as significant.
|
|
Two engines reading one riset panel can fit different predictors, and the
reply now says which one it used and what it cost -- with this panel's own
numbers.
* `genotype_model: "riself" | "f2"` on /compute/qtl2 (`het` kept as the
older spelling: drop/keep). riself is the default: R/qtl's
convert2riself coding, the H calls become missing and the HMM
re-imputes them. f2 keeps them as a third genotype -- a diagnostic.
* The conversion now counts the calls it drops and keeps the count
beside the cross (conversion.json), so a reply can say
"610684 heterozygous call(s) (12.7 %) were dropped". A cross
converted before that file existed measures the count once, on first
use, rather than never.
* Crossed with the note that already explained the H drop, so the two
read together: what riself does, how many calls it cost, and what f2
would give.
Documented in the README as "Why R/qtl and R/qtl2 disagree on a riset
panel", with the measurements (BXD trait 16627, BXD.8: R/qtl 9.59 at
rs33299937; qtl2 riself 3.73; qtl2 f2 9.68), R/qtl's adaptive
convert2riself rule quoted from its own source, and qtl2's side: riself
has two states, f2 three, and `genril2` -- from qtl2 0.18, "the progeny
are inbred lines", cross_info first column the number of generations --
is the honest model for residual heterozygosity and is available in the
0.42 we run. What is missing for it is recorded precisely: a founder-based
cross expects individual genotypes coded as founder alleles rather than
1/2/3 (a prototype with numeric codes read every genotype as missing), so
founder_geno from the two parental alleles and an allele-coded het remain
to be done, with DOL as the counter-example (its H is a collapsed DO, not
residual heterozygosity).
tests/qtl2.rs: the model is reported (Cross: riself / Cross: f2), the
dropped count is in the reply, keeping the calls finds more, and an
unknown model is refused rather than ignored.
|
|
R/qtl's convert2riself drops the third genotype code (see the note below),
and so does this converter -- but qtl2 then never sees the call, the HMM
re-imputes it from flanking markers, and on a marker that is 14% H that is
a different predictor from the one R/qtl fits.
{"group":"BXD","genotype_version":"BXD.8","het":"keep"} -> crosstype "f2"
On BXD trait 16627 (BXD.8, 35 phenotyped animals):
marker Mb R/qtl qtl2 drop qtl2 keep
rsm10000000889 5.652 9.59 3.79 9.98
rs33299937 5.653 9.59 3.73 9.68
rs13476332 6.311 9.59 3.70 ...
rsm10000000890 6.312 9.59 3.67 ...
So dropping the H calls is what costs qtl2 the peak: kept as a third
genotype it recovers R/qtl's number (9.68 against 9.59). The option is a
DIAGNOSTIC: an inbred line has no heterozygotes, so an f2-shaped cross is
wrong for inference even when it reproduces the other engine -- it is how
one finds out that the conversion, not the model, is the difference.
The policy is part of the conversion cache key (v3), and the reply says
when it is not the default. `het` applies only to a panel this server
converts: a shipped control-file cross carries its own coding and is
refused rather than silently ignored.
|
|
Returning genotypes is the one thing this API otherwise never does, so it
is fenced and the fence is the feature:
GET /groups/{group}/traits/{trait_id}/peak-genotypes?marker=<marker>
* the server re-runs the R/qtl scan for the trait -- the same machinery
and gate as /compute/rqtl -- and takes the peak from that run, rather
than trusting a client that names a marker;
* the marker must lie within ?window= Mb of that peak (default 10), or
the answer is 422 not_the_peak quoting the peak's marker, chromosome,
Mb and LOD;
* the peak must reach ?min_lod= (default 3), or 422 not_a_hit;
* only animals that have a value for the trait are listed, and only
this marker's calls. Nothing else of the matrix leaves.
On DOL 10003 that is UNCHS026345 (chr9 75.76 Mb, LOD 5.86) with 192 animals
across A 110 / B 8 / H 74 -- the two a class of eight being exactly the
thing worth seeing.
POST /plot/genotypes draws R/qtl's Fig 4.1 from it: post the reply's
`animals` (or its `genotypes` map) with the trait's values and get an SVG
with one point per animal, genotype class across, value up, class means
marked. Nothing is recomputed -- the plot is of the numbers the client
sends, like the other plots. New script scripts/plot-genotypes.py
(stdlib-only SVG, like its siblings) and the GENOTYPE_PLOTTER key.
tests/peakgenotypes.rs: the peak's calls for animals with values (with the
counts agreeing with the rows), the SVG, and a marker on another
chromosome refused with the peak quoted.
Also keeps /compute/rqtl's error order: naming both a group and a file is
"ambiguous" even when the request also forgot its values.
|