| Age | Commit message (Collapse) | Author |
|
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix
reply and `POST /plot/heatmap` draws it: the r matrix in that order, the
dendrogram that produced it, and a diverging scale with blue negative and red
positive. A reader can see *why* two traits are neighbours instead of trusting
an ordering.
Clustering decisions, all recorded because an unrepeatable clustering is worse
than none:
* distance `1 - r` -- the usual choice for correlations, and it keeps
anti-correlated traits apart, which is what a heatmap of correlations should
show;
* **average linkage (UPGMA)** -- single linkage chains along a gradient and
complete linkage makes one flat blob out of noisy traits; average is the
compromise GN1's R heatmaps used too;
* **deterministic ties and leaf order** (lower index first), so two runs on one
matrix produce the same picture and plots stay diffable;
* a matrix with a **gap is refused** rather than clustered: an unknown distance
treated as 1 would invent a structure, and the reply's note says to drop
those traits and ask again.
Pairing with the PCA work this is the visual half of a collection: scree plot
for how much structure there is, clustered heatmap for where it is.
Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation
with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at
all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back.
Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
|
|
GN2 renders a scree plot with Plotly in the browser, from the explained
variance ratios its matrix page computed. This is the same figure as a file:
`POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns
image/svg+xml, like every other plot endpoint -- the plot is of the numbers
that were computed, nothing is recomputed.
scripts/plot-pca.py draws a bar per component (its share of the variance,
GN2's number), a cumulative line with the 80%-of-variance crossing annotated
(what a reader actually uses to choose how many components to keep), the
eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps
in the matrix -- the one thing GN2 hides by mutating its shared-sample list.
`kind=N` on the request caps how many components are drawn, since a
200-trait matrix would otherwise draw 200 bars.
Also: the monitor's SVG judge now understands a binary reply (the client hands
back content type and size, not the bytes), and `plot-pca` is a check of its
own: it asks the matrix for a PCA and posts the reply to the plotter.
Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative
path, which only resolves when the service's working directory is the
checkout). plot-pca and plot-genotypes both ok.
|
|
`"pca": true` on the matrix request now folds the correlation matrix into
principal components: eigenvalues descending, explained variance ratios,
loadings per kept trait, and the eigenvalues' sum reported so a caller can
check the decomposition is sane instead of trusting it.
The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no
new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and
this is for a collection, not a dataset.
Two decisions from ai-memory #103, both visible in the reply:
* **Traits with any null cell are dropped and named**, with the reason. A
component from a matrix with holes is not defined; GN2 silently mutates its
shared-sample list instead, and nobody can reproduce that component.
* **The sign is fixed**: within a component the largest absolute loading is
positive. Eigenvectors are defined up to sign, so without a rule two correct
implementations disagree and no parity test could be exact.
Verified against numpy.linalg.eigh on the same r the reply carries
(scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007):
eigenvalues: worst relative difference 3.2e-14
loadings: worst absolute difference 9.99e-16
trace: 5.000000000000 for 5 traits
Unit tests cover the identity (unit eigenvalues), the closed form for 2x2
(1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The
monitor gained `compute-correlation-pca`, whose judge checks that the
eigenvalues sum to the trait count and the ratios to 1.
|
|
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply
gains meta.permutation {n, seed, significant, suggestive, values} -- the
genome-wide maximum of every permutation, so a histogram needs no second
call. `data` stays the positional rows array on purpose: /plot/rqtl posts
it straight back and the monitor counts its rows, so the null went beside
the rows rather than around them (a generic Envelope::with_meta/Meta::extra,
skip_serializing_if, so no other endpoint's response changes).
Thresholds come from R/qtl2's scan1perm over the same genotype
probabilities, kinship and covariates as the scan, with chr_lengths from
the map -- without those it derives chromosome lengths from marker
positions and the numbers stop being comparable with R/qtl's. The
permutation CSV is written in R/qtl's two-column shape so one parser and
one percentile routine serve both engines: PermutationResult::from_values
is now shared with /compute/rqtl instead of duplicated.
Semantics, each with the note that says it:
* scan1perm shuffles phenotypes across individuals -- right for an
outbred cross, only approximate for a selfing RIL panel, where qtl2perm
is the correct package (not installed here, and not packaged);
* perm_strata takes a case attribute name (sex, Epoch, ...) and permutes
within it, because shuffling across sexes inflates the null;
* fewer than 100 permutations gets a note that the 0.05 quantile is a
quantile of too few values; more than 10 000 is refused, since each
permutation is a whole genome scan;
* a seed is reported always -- without one, two identical requests would
give different thresholds.
Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO
kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s,
1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds
converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at
1000 on the raw probe) and are reproducible: the same seed gives the same
numbers on 1 and 4 cores.
Four things that only surfaced by running it, all commented where they
bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so
opt$perm_out was always NULL and the file landed somewhere the caller did
not look); perm_strata must be a named character vector, not a factor;
the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the
permutation index first, and a first-field parser reads that as the LOD);
and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not
0.67.
bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging
meta.permutation) so a deploy that loses the R path shows up as a red
check rather than a surprise.
|
|
A warm run answers from caches the previous binary filled, so it can pass
while the cold path -- what a new caller's first request actually does -- is
broken. `--cold` sends each cached engine a payload with one sample value
nudged by a thousandth, then repeats the original: every engine here keys its
cache on content (gemma-wrapper's hash, ravanan's interned inputs,
rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both
runs are judged; the line reports "cold X / warm Y".
Measured on production with the 30-sample fixture:
compute-ravanan 102.0 s cold / 3.7 s warm
compute-gemma 3.1 s cold / 2.0 s warm
compute-rqtl 7.5 s cold / 7.3 s warm
compute-qtl2 3.3 s cold / 3.3 s warm
compute-fitqtl 6.7 s cold / 6.9 s warm
compute-bulklmm 19.6 s cold / 19.0 s warm
ravanan is the one where the cache is the difference between minutes and
seconds; the rest are compute-bound at this size, and the honest thing is to
print the pair rather than assert a speedup. A check whose payload cannot be
perturbed skips with that reason instead of reporting two warm runs as cold
and warm.
The operator checklist gains the cold run as part of deploying, with both
costs named: ravanan's cold run takes minutes, and every cold run adds cache
entries.
|
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
The check was written but never built: discovery kept one trait id, and the
control comes from a second one. With this, --tier compute is 10 ok, 0 failed
and 'compute-correlation-partial' answers 5 rows in 774 ms.
|
|
The implementation had already landed in 698e6ab by accident -- a `git add
-A` swept it in with the operator-checklist docs, which is why its message
says nothing about partial correlation. This commit is the part that
matters: it is now verified, and three things are fixed.
**Verification.** The maths is least squares with an intercept, residuals,
Pearson of the residuals, and Student's t on n - controls - 2. Checked two
ways against independent code:
* BXDPublish 10001 with 10002 held constant, target 17164, n=6:
ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq +
scipy.stats.t.sf r=0.9960864631522379 p=0.00029372.
Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567.
* The same for 17084: identical to 16 digits, r=0.9840718340719924.
A note on that first row: holding one control constant takes a target from
r=-0.11 to r=+0.996 on six points. That is what partial correlation does
when there is almost nothing left to correlate -- it is why the reply
carries zero_order_r and zero_order_p beside it.
**Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp`
term sign-flipped, which still yields a smooth monotone function, so
nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out
as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12
relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both
checkpoints are now tests. A perfect partial r (self-correlation) is
clamped to +-1 with p=0.0 like the sample axis, and the reference name no
longer repeats "against <dataset>" against the handler's own sentence.
The monitor now exercises the axis: `compute-correlation-partial`.
|
|
Two separate faults behind one red line. The check posted `{group, markers: []}`,
which is not this endpoint's request at all -- /plot/genotypes draws the numbers
it is sent (`marker`, `genotypes`, `values`), it looks nothing up -- and the
trait it picked (BXDPublish 10001, LOD 2.93) is *below* the fence that
/peak-genotypes enforces (LOD >= 3), so even a correct payload would earn a
422. So: checks can now build their body from earlier replies (`body` may be a
callable), and can decline with a reason (`Skip`) instead of going red --
calling a deliberately fenced endpoint's 422 a failure trains a reader to
ignore red. The check now walks the documented chain: a published trait whose
LRS is at least 14, its peak-genotypes calls, its values, then the plot.
Full monitor after: 38 ok, 0 failed, 1 skipped, 0 busy.
|
|
`--only` needs exact check names, which is no help when what you know is the
dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring
against the check's name, the sentence describing what it needs, and its
hint -- so `--filter julia` finds compute-bulklmm even though its name says
nothing about Julia. It selects tiers like `--only` does, and says so when
nothing matches *because* discovery has not run (the compute checks do not
exist until a group with samples is known).
Also un-ignores bin/: the monitor is source, and an ops tool that is not
versioned is the problem ai-memory #38 already named. Bytecode stays out.
Co-report: the whole compute tier is green now (9 ok, 0 failed), including
BulkLMM -- the recipe and the depot quirk are in the operator checklist.
|