diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-15 01:37:51 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-15 01:37:51 +0000 |
| commit | c8ce261f2bf499ab3260ff5ce186c801491b9fea (patch) | |
| tree | a5d26ae5871df85fc13d7ec75e9aaccbdaebc171 /test/python/test_dev_token.py | |
| parent | 41c70f042731e36fb91b8fb3f4f393a4aa654e9d (diff) | |
| download | genenetwork4-c8ce261f2bf499ab3260ff5ce186c801491b9fea.tar.gz | |
correlation: the sample-axis endpoint, synchronous
POST /compute/correlation now computes instead of returning canned data: one reference trait against every trait of a target dataset, observations being individuals, pearson or spearman, top-N by |r|. The reply is positional -- trait, r, p, n (column names in meta.columns) -- with the method, the minimum overlap and the top-N stated in the notes. `bicor` is refused with the methods that exist; a bad top_n is refused rather than clamped; a dataset or trait with nothing to correlate says so instead of returning an empty list. src/db.rs gains the bulk fetch the sample axis needs: every value of every trait of a dataset restricted to the reference trait's individuals, one record per (trait, individual, value), ordered by trait so the caller groups as it streams. It never loads a dataset whole -- ProbeSetData is 5.3e9 rows, so the query narrows by dataset and by individuals, which is also why the sweep, not the arithmetic, is the whole cost. Measured on the live db_webqtl: HC_M2_0606_P / 1427571_at against its own dataset, top 5, HTTP 200 in 16 s; the trait correlates with itself at 1.0 over 99 observations, then 0.802 (p 1.9e-23), 0.790, -0.789, -0.788. Synchronous is affordable at this size; GN2's job indirection exists for top-N 20000 across datasets, which is the next thing to time. Known and deliberate for now: the publish/genotype axes are not wired (the fetch is probeset-only), and a perfect correlation reports p as null -- GSL's t statistic is undefined at |r| = 1, whose r comes out as 1.0000000000000002, so 1 - r^2 is slightly negative and the engine returns NaN. Reproducing the engine exactly, rather than "fixing" the p-value here, is the decision recorded in docs/correlations.md.
Diffstat (limited to 'test/python/test_dev_token.py')
0 files changed, 0 insertions, 0 deletions
