about summary refs log tree commit diff
path: root/bin/monitor-gn4
AgeCommit message (Collapse)Author
2 dayscorrelation matrix: clustered heatmap and its plot HEAD mainPjotr Prins
`"cluster": true` adds `heatmap: {order, merges, max_height}` to the matrix reply and `POST /plot/heatmap` draws it: the r matrix in that order, the dendrogram that produced it, and a diverging scale with blue negative and red positive. A reader can see *why* two traits are neighbours instead of trusting an ordering. Clustering decisions, all recorded because an unrepeatable clustering is worse than none: * distance `1 - r` -- the usual choice for correlations, and it keeps anti-correlated traits apart, which is what a heatmap of correlations should show; * **average linkage (UPGMA)** -- single linkage chains along a gradient and complete linkage makes one flat blob out of noisy traits; average is the compromise GN1's R heatmaps used too; * **deterministic ties and leaf order** (lower index first), so two runs on one matrix produce the same picture and plots stay diffable; * a matrix with a **gap is refused** rather than clustered: an unknown distance treated as 1 would invent a structure, and the reply's note says to drop those traits and ask again. Pairing with the PCA work this is the visual half of a collection: scree plot for how much structure there is, clustered heatmap for where it is. Monitor, as asked: `compute-correlation-heatmap` (order must be a permutation with n-1 merges), `plot-heatmap`, and the plot endpoints that had no check at all -- `plot-gemma` and `plot-bnw`, each posting a real compute result back. Plots tier is now five checks, all ok; full run 47 ok, 0 failed, 1 skipped.
2 daysplot: the scree plot of a matrix PCA, which GN2 draws client-sidePjotr Prins
GN2 renders a scree plot with Plotly in the browser, from the explained variance ratios its matrix page computed. This is the same figure as a file: `POST /plot/pca` takes the matrix reply (with `"pca": true`) as-is and returns image/svg+xml, like every other plot endpoint -- the plot is of the numbers that were computed, nothing is recomputed. scripts/plot-pca.py draws a bar per component (its share of the variance, GN2's number), a cumulative line with the 80%-of-variance crossing annotated (what a reader actually uses to choose how many components to keep), the eigenvalue in each bar's tooltip, and the traits the PCA had to drop for gaps in the matrix -- the one thing GN2 hides by mutating its shared-sample list. `kind=N` on the request caps how many components are drawn, since a 200-trait matrix would otherwise draw 200 bars. Also: the monitor's SVG judge now understands a binary reply (the client hands back content type and size, not the bytes), and `plot-pca` is a check of its own: it asks the matrix for a PCA and posts the reply to the plotter. Deployed: PCA_PLOTTER is pinned in env.sh (the code default is a relative path, which only resolves when the service's working directory is the checkout). plot-pca and plot-genotypes both ok.
2 dayscorrelation matrix: PCA, behind its parity testPjotr Prins
`"pca": true` on the matrix request now folds the correlation matrix into principal components: eigenvalues descending, explained variance ratios, loadings per kept trait, and the eigenvalues' sum reported so a caller can check the decomposition is sane instead of trusting it. The eigendecomposition is cyclic Jacobi -- symmetric matrices, no LAPACK, no new dependency -- capped at 300 traits because Jacobi is O(n^3) per sweep and this is for a collection, not a dataset. Two decisions from ai-memory #103, both visible in the reply: * **Traits with any null cell are dropped and named**, with the reason. A component from a matrix with holes is not defined; GN2 silently mutates its shared-sample list instead, and nobody can reproduce that component. * **The sign is fixed**: within a component the largest absolute loading is positive. Eigenvectors are defined up to sign, so without a rule two correct implementations disagree and no parity test could be exact. Verified against numpy.linalg.eigh on the same r the reply carries (scripts/pca-parity.py, BXDPublish 10001/10002/10003/10005/10007): eigenvalues: worst relative difference 3.2e-14 loadings: worst absolute difference 9.99e-16 trace: 5.000000000000 for 5 traits Unit tests cover the identity (unit eigenvalues), the closed form for 2x2 (1 +/- rho), the trace, a non-symmetric matrix being refused, and the cap. The monitor gained `compute-correlation-pca`, whose judge checks that the eigenvalues sum to the trait count and the ratios to 1.
3 daysqtl2: permutations for the LOD thresholdsPjotr
POST /compute/qtl2 gains nperm, perm_seed and perm_strata, and the reply gains meta.permutation {n, seed, significant, suggestive, values} -- the genome-wide maximum of every permutation, so a histogram needs no second call. `data` stays the positional rows array on purpose: /plot/rqtl posts it straight back and the monitor counts its rows, so the null went beside the rows rather than around them (a generic Envelope::with_meta/Meta::extra, skip_serializing_if, so no other endpoint's response changes). Thresholds come from R/qtl2's scan1perm over the same genotype probabilities, kinship and covariates as the scan, with chr_lengths from the map -- without those it derives chromosome lengths from marker positions and the numbers stop being comparable with R/qtl's. The permutation CSV is written in R/qtl's two-column shape so one parser and one percentile routine serve both engines: PermutationResult::from_values is now shared with /compute/rqtl instead of duplicated. Semantics, each with the note that says it: * scan1perm shuffles phenotypes across individuals -- right for an outbred cross, only approximate for a selfing RIL panel, where qtl2perm is the correct package (not installed here, and not packaged); * perm_strata takes a case attribute name (sex, Epoch, ...) and permutes within it, because shuffling across sexes inflates the null; * fewer than 100 permutations gets a note that the 0.05 quantile is a quantile of too few values; more than 10 000 is refused, since each permutation is a whole genome scan; * a seed is reported always -- without one, two identical requests would give different thresholds. Cost, measured on BXD trait 10001 (34 phenotyped, 20 chromosomes, LOCO kinship, 4 cores): 20 permutations 0.6 s, 100 in 2.2 s, 500 in 14 s, 1000 in 24 s through the API; 0.022-0.03 s per permutation. Thresholds converge as expected (0.05: 4.331 at 100 and 200 permutations, 4.374 at 1000 on the raw probe) and are reproducible: the same seed gives the same numbers on 1 and 4 cores. Four things that only surfaced by running it, all commented where they bit: optparse keeps the dash (--perm-out is opt[["perm-out"]], so opt$perm_out was always NULL and the file landed somewhere the caller did not look); perm_strata must be a named character vector, not a factor; the LOD in the CSV is the LAST numeric field (R/qtl's shape puts the permutation index first, and a first-field parser reads that as the LOD); and percentiles interpolate at (n-1)p/100, so 100 maxima give 0.6733, not 0.67. bin/monitor-gn4 gains compute-qtl2-perm (nperm 10, judging meta.permutation) so a deploy that loses the R path shows up as a red check rather than a surprise.
3 daysmonitor: run the cached engines cold, not just warmPjotr Prins
A warm run answers from caches the previous binary filled, so it can pass while the cold path -- what a new caller's first request actually does -- is broken. `--cold` sends each cached engine a payload with one sample value nudged by a thousandth, then repeats the original: every engine here keys its cache on content (gemma-wrapper's hash, ravanan's interned inputs, rqtl/qtl2/bulklmm's own), so the first is a miss and the second a hit. Both runs are judged; the line reports "cold X / warm Y". Measured on production with the 30-sample fixture: compute-ravanan 102.0 s cold / 3.7 s warm compute-gemma 3.1 s cold / 2.0 s warm compute-rqtl 7.5 s cold / 7.3 s warm compute-qtl2 3.3 s cold / 3.3 s warm compute-fitqtl 6.7 s cold / 6.9 s warm compute-bulklmm 19.6 s cold / 19.0 s warm ravanan is the one where the cache is the difference between minutes and seconds; the rest are compute-bound at this size, and the honest thing is to print the pair rather than assert a speedup. A check whose payload cannot be perturbed skips with that reason instead of reporting two warm runs as cold and warm. The operator checklist gains the cold run as part of deploying, with both costs named: ravanan's cold run takes minutes, and every cold run adds cache entries.
3 dayscorrelation: the matrix endpoint, and the rule that a deploy ends with a ↵Pjotr Prins
monitor run `POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}` references -- a collection, as GN2's matrix form posts it, which spans datasets -- and answers an N x N matrix of r, p and n. The five decisions that are not obvious, each recorded where a reader will look for them: * **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to Spearman mid-row once a Pearson r passes 0.999, which the self-pair always does, so its matrix is asymmetric by construction. Here the method is chosen once and r[i][j] == r[j][i] exactly. * **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine calls, not N^2. * **The diagonal is by construction**: r = 1, p = null, n = the trait's own count, so half the traits cannot come back as 1.0000000000000002 with a NaN p. * **Per-cell n is its own matrix**, and a cell under `min_overlap` (default 6) is null in r and p while its n still says how many points there were. GN2 accepts 2, where |r| = 1 by definition. * **The sample axis is the first trait's group** (#106), resolved once and stated in the reply with the number of individuals it kept. Fetches are one query per dataset (`get_dataset_values_for_traits`, `get_publish_values_for_traits`), not one per trait: a matrix chooses a handful out of twenty thousand. Capped at 1 000 traits, because the cost is quadratic in the correlations and in the reply. No PCA -- the r matrix has to be trusted on its own first. Verified twice: cells agree with scipy to the last bit (BXDPublish 10001/10002/10005: 0.6756270426459695, 0.6950450737755965, 0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964, 0.8819879006700073), and a cell equals what the row endpoint says for the same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix, `min_overlap` above every cell's n, an unknown trait, `bicor`, and no traits at all. The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier all`, and says why: `/version` can report a commit and a reachable database while every engine that shells out is dead. The monitor gained a matrix check whose judge enforces square, symmetric and a unit diagonal.
3 daysmonitor: discover a second published trait, so the partial axis has a controlPjotr Prins
The check was written but never built: discovery kept one trait id, and the control comes from a second one. With this, --tier compute is 10 ok, 0 failed and 'compute-correlation-partial' answers 5 rows in 774 ms.
3 dayscorrelation: verify the partial axis against numpy/scipy, and clamp a perfect rPjotr Prins
The implementation had already landed in 698e6ab by accident -- a `git add -A` swept it in with the operator-checklist docs, which is why its message says nothing about partial correlation. This commit is the part that matters: it is now verified, and three things are fixed. **Verification.** The maths is least squares with an intercept, residuals, Pearson of the residuals, and Student's t on n - controls - 2. Checked two ways against independent code: * BXDPublish 10001 with 10002 held constant, target 17164, n=6: ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq + scipy.stats.t.sf r=0.9960864631522379 p=0.00029372. Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567. * The same for 17084: identical to 16 digits, r=0.9840718340719924. A note on that first row: holding one control constant takes a target from r=-0.11 to r=+0.996 on six points. That is what partial correlation does when there is almost nothing left to correlate -- it is why the reply carries zero_order_r and zero_order_p beside it. **Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp` term sign-flipped, which still yields a smooth monotone function, so nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12 relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both checkpoints are now tests. A perfect partial r (self-correlation) is clamped to +-1 with p=0.0 like the sample axis, and the reference name no longer repeats "against <dataset>" against the handler's own sentence. The monitor now exercises the axis: `compute-correlation-partial`.
3 daysmonitor: a check may decline to run, and the genotype plot follows the real flowPjotr Prins
Two separate faults behind one red line. The check posted `{group, markers: []}`, which is not this endpoint's request at all -- /plot/genotypes draws the numbers it is sent (`marker`, `genotypes`, `values`), it looks nothing up -- and the trait it picked (BXDPublish 10001, LOD 2.93) is *below* the fence that /peak-genotypes enforces (LOD >= 3), so even a correct payload would earn a 422. So: checks can now build their body from earlier replies (`body` may be a callable), and can decline with a reason (`Skip`) instead of going red -- calling a deliberately fenced endpoint's 422 a failure trains a reader to ignore red. The check now walks the documented chain: a published trait whose LRS is at least 14, its peak-genotypes calls, its values, then the plot. Full monitor after: 38 ok, 0 failed, 1 skipped, 0 busy.
3 daysmonitor: a --filter for "just retest this", and track the toolPjotr Prins
`--only` needs exact check names, which is no help when what you know is the dependency ("julia", "Rscript", "slurmrestd"). `--filter` matches a substring against the check's name, the sentence describing what it needs, and its hint -- so `--filter julia` finds compute-bulklmm even though its name says nothing about Julia. It selects tiers like `--only` does, and says so when nothing matches *because* discovery has not run (the compute checks do not exist until a group with samples is known). Also un-ignores bin/: the monitor is source, and an ops tool that is not versioned is the problem ai-memory #38 already named. Bytecode stays out. Co-report: the whole compute tier is green now (9 ok, 0 failed), including BulkLMM -- the recipe and the depot quirk are in the operator checklist.