diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-14 23:21:24 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-14 23:21:24 +0000 |
| commit | 2900aa07dfbb857382088ab4148f29dfd04f0123 (patch) | |
| tree | b4160e2d20e86af7f121e0bb2127cd211aa650c2 /docs | |
| parent | 4d554b02daaea23b60826aa7b3c644baa55376d2 (diff) | |
| download | genenetwork4-2900aa07dfbb857382088ab4148f29dfd04f0123.tar.gz | |
correlations: the design decisions, and the reference is reachable after all
Synchronous, no redis, no job: one POST computes and answers. Use the correlation_rust engine in-process (already packaged, rev 473ad82), reuse its maths rather than rewriting the p-value, and implement only the sort and top-N. Minimum overlap 6 (gn3/python, not the engine's 4), pearson and spearman only, the engine's exact |r|-descending/NaN-last ordering, top-N as a request option applied after the sort, and the sample axis first. Verification is against the live genenetwork.org mapping page: it answers HTTP 200 in ~31 s, so the earlier "-m 20 curl" made it look unreachable when it is merely slow. Budget ~31 s per reference page in the parity loop.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/correlations.md | 62 |
1 files changed, 39 insertions, 23 deletions
diff --git a/docs/correlations.md b/docs/correlations.md index 1ff3770..05c6781 100644 --- a/docs/correlations.md +++ b/docs/correlations.md @@ -148,29 +148,45 @@ Literature never touches the engine: it is `compute_all_lit_correlation` over ## Duplicating it into GN4 `src/routes/correlation.rs` is a stub returning canned -`{trait_id, correlation}`, and its doc comment already names the target: -*"correlation_rust in-process, rev-pinned"*. Because the crate is a library, -"in-process" can mean two quite different amounts of code: - -- **Thin**: depend on the git rev `473ad82`, keep the file dance, call - `analysis::Analysis::compute(json_path)`, read the CSV. Least code and exact - upstream semantics — including the `unwrap()`/`panic!` on file I/O and bad - JSON, which in a request thread aborts it (that is why GN2 and GN3 use a - subprocess at all). -- **Direct**: skip the files. `parser::parse_rows_with_names`, - `Pearson`/`Spearman` and the `Correlation` trait are all `pub`, so a caller - can build the vectors in memory, reuse the maths (the p-value is the subtle - part — reuse it, do not rewrite it) and implement only the sort and top-N. - No temporary files, no CSV parsing, no `unwrap()` on I/O in the request path. - -Either way GN4 must supply what it does not have today: the **bulk per-dataset -value fetch** (the expensive half), the trait metadata for the reply, and a -decision on four open semantics: - -1. minimum overlap: 4 (engine) or 6 (GN3 Python); -2. pearson and spearman only — bicor needs another engine; -3. top-N before or after enrichment; -4. whether to reproduce the engine's exact NaN/tie ordering. +`{trait_id, correlation}`. What to build, decided on 2026-11-14: + +**Synchronous, no redis, no job.** One POST computes and answers. GN2's +`/corr_compute` writes a form to a file and runs a subprocess job, and GN3's +partial path goes through redis; neither is wanted for the first version. That +also means the reply carries the result, not a `?id=`/`/status` pair. + +**Use the `correlation_rust` engine, in-process.** `gn-rust-correlation` is +already packaged (`gn/packages/genenetwork-rust.scm`, rev `473ad82`), so this is +a dependency to wire in rather than a package to write. Its `parser::parse_rows_with_names`, +`Pearson`/`Spearman` and the `Correlation` trait are `pub`: build the vectors in +memory, reuse the maths — **the p-value is the subtle part; reuse it, do not +rewrite it** — and implement only the sort and top-N. No temporary files, no CSV +round trip, and none of upstream's `unwrap()`/`panic!` on file I/O in a request +thread (which is why GN2 and GN3 shell out at all). + +**Semantics, settled:** + +| decision | choice | +|---|---| +| minimum overlap | **6** (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant | +| methods | pearson and spearman only; bicor does not exist in the engine and stays out | +| ordering | reproduce the engine exactly: \|r\| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference | +| top-N | a request option (GN2's menu is 100…20000, 500 by default); applied *after* the sort | +| first axis | **sample** only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2 | + +**Verification is against the live site**, which turns out to be reachable: it +answers `show_trait` with HTTP 200 in ~31 s (a `-m 20` curl had made it look +unreachable). Each reference page therefore costs ~31 s, which needs budgeting in +any parity loop. + +What GN4 still has to supply, and does not have today: + +- the **bulk per-dataset value fetch** — the expensive half; see the measured + sizes above, and note the shape the existing queries use: the caller's sample + subset for the caller's trait list, never a whole dataset; +- trait metadata for the reply (the id, and the name GN2 shows); +- the sample-group knob (`samples_primary` / `other` / `all`) and the target + dataset, both of which change *which* rows are fetched. ## References |
