about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-11-14 23:21:24 +0000
committerPjotr Prins <pjotr.public01@thebird.nl>2026-11-14 23:21:24 +0000
commit2900aa07dfbb857382088ab4148f29dfd04f0123 (patch)
treeb4160e2d20e86af7f121e0bb2127cd211aa650c2 /docs
parent4d554b02daaea23b60826aa7b3c644baa55376d2 (diff)
downloadgenenetwork4-2900aa07dfbb857382088ab4148f29dfd04f0123.tar.gz
correlations: the design decisions, and the reference is reachable after all
Synchronous, no redis, no job: one POST computes and answers.  Use the
correlation_rust engine in-process (already packaged, rev 473ad82), reuse
its maths rather than rewriting the p-value, and implement only the sort
and top-N.  Minimum overlap 6 (gn3/python, not the engine's 4), pearson
and spearman only, the engine's exact |r|-descending/NaN-last ordering,
top-N as a request option applied after the sort, and the sample axis
first.

Verification is against the live genenetwork.org mapping page: it answers
HTTP 200 in ~31 s, so the earlier "-m 20 curl" made it look unreachable
when it is merely slow.  Budget ~31 s per reference page in the parity
loop.
Diffstat (limited to 'docs')
-rw-r--r--docs/correlations.md62
1 files changed, 39 insertions, 23 deletions
diff --git a/docs/correlations.md b/docs/correlations.md
index 1ff3770..05c6781 100644
--- a/docs/correlations.md
+++ b/docs/correlations.md
@@ -148,29 +148,45 @@ Literature never touches the engine: it is `compute_all_lit_correlation` over
 ## Duplicating it into GN4
 
 `src/routes/correlation.rs` is a stub returning canned
-`{trait_id, correlation}`, and its doc comment already names the target:
-*"correlation_rust in-process, rev-pinned"*. Because the crate is a library,
-"in-process" can mean two quite different amounts of code:
-
-- **Thin**: depend on the git rev `473ad82`, keep the file dance, call
-  `analysis::Analysis::compute(json_path)`, read the CSV. Least code and exact
-  upstream semantics — including the `unwrap()`/`panic!` on file I/O and bad
-  JSON, which in a request thread aborts it (that is why GN2 and GN3 use a
-  subprocess at all).
-- **Direct**: skip the files. `parser::parse_rows_with_names`,
-  `Pearson`/`Spearman` and the `Correlation` trait are all `pub`, so a caller
-  can build the vectors in memory, reuse the maths (the p-value is the subtle
-  part — reuse it, do not rewrite it) and implement only the sort and top-N.
-  No temporary files, no CSV parsing, no `unwrap()` on I/O in the request path.
-
-Either way GN4 must supply what it does not have today: the **bulk per-dataset
-value fetch** (the expensive half), the trait metadata for the reply, and a
-decision on four open semantics:
-
-1. minimum overlap: 4 (engine) or 6 (GN3 Python);
-2. pearson and spearman only — bicor needs another engine;
-3. top-N before or after enrichment;
-4. whether to reproduce the engine's exact NaN/tie ordering.
+`{trait_id, correlation}`. What to build, decided on 2026-11-14:
+
+**Synchronous, no redis, no job.** One POST computes and answers. GN2's
+`/corr_compute` writes a form to a file and runs a subprocess job, and GN3's
+partial path goes through redis; neither is wanted for the first version. That
+also means the reply carries the result, not a `?id=`/`/status` pair.
+
+**Use the `correlation_rust` engine, in-process.** `gn-rust-correlation` is
+already packaged (`gn/packages/genenetwork-rust.scm`, rev `473ad82`), so this is
+a dependency to wire in rather than a package to write. Its `parser::parse_rows_with_names`,
+`Pearson`/`Spearman` and the `Correlation` trait are `pub`: build the vectors in
+memory, reuse the maths — **the p-value is the subtle part; reuse it, do not
+rewrite it** — and implement only the sort and top-N. No temporary files, no CSV
+round trip, and none of upstream's `unwrap()`/`panic!` on file I/O in a request
+thread (which is why GN2 and GN3 shell out at all).
+
+**Semantics, settled:**
+
+| decision | choice |
+|---|---|
+| minimum overlap | **6** (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant |
+| methods | pearson and spearman only; bicor does not exist in the engine and stays out |
+| ordering | reproduce the engine exactly: \|r\| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference |
+| top-N | a request option (GN2's menu is 100…20000, 500 by default); applied *after* the sort |
+| first axis | **sample** only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2 |
+
+**Verification is against the live site**, which turns out to be reachable: it
+answers `show_trait` with HTTP 200 in ~31 s (a `-m 20` curl had made it look
+unreachable). Each reference page therefore costs ~31 s, which needs budgeting in
+any parity loop.
+
+What GN4 still has to supply, and does not have today:
+
+- the **bulk per-dataset value fetch** — the expensive half; see the measured
+  sizes above, and note the shape the existing queries use: the caller's sample
+  subset for the caller's trait list, never a whole dataset;
+- trait metadata for the reply (the id, and the name GN2 shows);
+- the sample-group knob (`samples_primary` / `other` / `all`) and the target
+  dataset, both of which change *which* rows are fetched.
 
 ## References