about summary refs log tree commit diff
path: root/src/partial.rs
AgeCommit message (Collapse)Author
2 dayscorrelation: verify the partial axis against numpy/scipy, and clamp a perfect rPjotr Prins
The implementation had already landed in 698e6ab by accident -- a `git add -A` swept it in with the operator-checklist docs, which is why its message says nothing about partial correlation. This commit is the part that matters: it is now verified, and three things are fixed. **Verification.** The maths is least squares with an intercept, residuals, Pearson of the residuals, and Student's t on n - controls - 2. Checked two ways against independent code: * BXDPublish 10001 with 10002 held constant, target 17164, n=6: ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq + scipy.stats.t.sf r=0.9960864631522379 p=0.00029372. Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567. * The same for 17084: identical to 16 digits, r=0.9840718340719924. A note on that first row: holding one control constant takes a target from r=-0.11 to r=+0.996 on six points. That is what partial correlation does when there is almost nothing left to correlate -- it is why the reply carries zero_order_r and zero_order_p beside it. **Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp` term sign-flipped, which still yields a smooth monotone function, so nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12 relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both checkpoints are now tests. A perfect partial r (self-correlation) is clamped to +-1 with p=0.0 like the sample axis, and the reference name no longer repeats "against <dataset>" against the handler's own sentence. The monitor now exercises the axis: `compute-correlation-partial`.
3 daysoperator checklist: the tool block env.sh must keep, and why BulkLMM cannot ↵Pjotr Prins
work yet Writing down what the monitor found and what fixed it, because it is config that lives outside the repository and is therefore easy to lose at the next deploy: with only the cache dirs set, every engine that shells out answers 502 while the whole read surface and search look healthy. Measured on production after the fix (bin/monitor-gn4 --tier compute): 8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453), fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed, 1 busy. The remaining failure is BulkLMM and it is not a config line: the Julia project is registry-based and no depot on this box has its packages, so it stops at "Run Pkg.instantiate()". Populating a depot downloads from the Julia registry, which should be a decision rather than a side effect.