about summary refs log tree commit diff
path: root/docs/correlations.md
AgeCommit message (Collapse)Author
3 dayscorrelations: the design decisions, and the reference is reachable after allPjotr Prins
Synchronous, no redis, no job: one POST computes and answers. Use the correlation_rust engine in-process (already packaged, rev 473ad82), reuse its maths rather than rewriting the p-value, and implement only the sort and top-N. Minimum overlap 6 (gn3/python, not the engine's 4), pearson and spearman only, the engine's exact |r|-descending/NaN-last ordering, top-N as a request option applied after the sort, and the sample axis first. Verification is against the live genenetwork.org mapping page: it answers HTTP 200 in ~31 s, so the earlier "-m 20 curl" made it look unreachable when it is merely slow. Budget ~31 s per reference page in the parity loop.
3 daysdocs: what GeneNetwork correlates, and what correlation_rust doesPjotr
Answers the question that blocks duplicating the correlation_rust path: what is actually correlated. Four kinds over three axes -- sample (one trait vs every trait of a dataset, over individuals), tissue (gene vs gene over 26-32 tissue systems), literature (a table lookup in LCorrRamin3, no n at all) and partial (conditioned, GN3 only, async). Records the engine's contract as it is: JSON in, CSV out, one trait per line positionally aligned to x_vals, non-floats dropped pairwise, rows under 4 pairs refused, output sorted by |r| descending with NaNs last, top-N applied afterwards in Python, and no bicor in the engine at all -- which matters because GN3's Python path uses a minimum of 6 and the engine uses 4. Measured read-only against db_webqtl: ProbeSetData 5 314 694 464 rows, PublishData 3 288 638, TissueProbeSetData 90 562 DataIds with 26-32 values each (avg 29.0), LCorrRamin3 127 368 780. Hence no path loads a dataset whole, and the value fetch -- not the arithmetic -- is the cost. Ends with the two shapes a GN4 integration can take (thin: keep the file dance; direct: reuse parser::parse_rows_with_names and Pearson/Spearman in memory) and the four semantics that must be decided either way.