about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-14 22:58:18 +0000
committerPjotr <wrk@bosa.lan>2026-11-14 22:58:18 +0000
commit4d554b02daaea23b60826aa7b3c644baa55376d2 (patch)
treeb6f3e879d0509bcdad1abd5207b76afa5291925e /docs
parent3a62ef34c127d36fd4e04583349f64eee5de2c6f (diff)
downloadgenenetwork4-4d554b02daaea23b60826aa7b3c644baa55376d2.tar.gz
docs: what GeneNetwork correlates, and what correlation_rust does
Answers the question that blocks duplicating the correlation_rust path:
what is actually correlated. Four kinds over three axes -- sample (one
trait vs every trait of a dataset, over individuals), tissue (gene vs gene
over 26-32 tissue systems), literature (a table lookup in LCorrRamin3, no
n at all) and partial (conditioned, GN3 only, async).

Records the engine's contract as it is: JSON in, CSV out, one trait per
line positionally aligned to x_vals, non-floats dropped pairwise, rows
under 4 pairs refused, output sorted by |r| descending with NaNs last,
top-N applied afterwards in Python, and no bicor in the engine at all --
which matters because GN3's Python path uses a minimum of 6 and the engine
uses 4.

Measured read-only against db_webqtl: ProbeSetData 5 314 694 464 rows,
PublishData 3 288 638, TissueProbeSetData 90 562 DataIds with 26-32 values
each (avg 29.0), LCorrRamin3 127 368 780. Hence no path loads a dataset
whole, and the value fetch -- not the arithmetic -- is the cost.

Ends with the two shapes a GN4 integration can take (thin: keep the file
dance; direct: reuse parser::parse_rows_with_names and Pearson/Spearman in
memory) and the four semantics that must be decided either way.
Diffstat (limited to 'docs')
-rw-r--r--docs/correlations.md189
1 files changed, 189 insertions, 0 deletions
diff --git a/docs/correlations.md b/docs/correlations.md
new file mode 100644
index 0000000..1ff3770
--- /dev/null
+++ b/docs/correlations.md
@@ -0,0 +1,189 @@
+# What GeneNetwork correlates, and how
+
+GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers
+four different things measured over three different axes. This note records
+what is actually correlated, which engine computes it, and where. It exists
+because the answer to "duplicate the correlation_rust path" depends entirely
+on knowing which of the four you are duplicating.
+
+Written from the code, with row counts measured read-only against the
+production `db_webqtl` (2026-11-14).
+
+## The four correlations
+
+| kind | **x** | **y** | observations | computed? |
+|---|---|---|---|---|
+| **sample** | one trait, per individual (BXD strain, animal, plant, human case) | each trait of a target dataset, same individuals | individuals (~34–40 for BXD) | every request |
+| **tissue** | one gene's expression across tissue systems | another gene's expression across the same tissues | tissues (**26–32** per gene) | every request |
+| **literature** | a gene | another gene | published studies | **read from a table** |
+| **partial** (GN3 only) | one trait | one trait, conditioned on other traits/genotypes | individuals | every request, async |
+
+Nothing here correlates markers with traits — that is QTL mapping. In the
+sample and partial cases the observations are **individuals**; in the tissue
+case they are **tissues**; literature has no n at all.
+
+### Sample correlation: the one that matters most
+
+`POST /compute/correlation` for trait X against "BXD Liver" is not one
+correlation: it is X against *every* trait in that dataset, one r each, all
+over the same individual axis — about 20k r's over ~34–40 strains.
+
+Which individuals is a request knob (`corr_samples_group` in GN2):
+
+- **primary only** — the primary trait's own group members (BXD strains only);
+- **other only** — everything in the data *except* those, which is how you
+  correlate across other strains or species;
+- **all** — every sample the trait data carries.
+
+Two rules decide correctness, both easy to get wrong:
+
+- **Missing data is pairwise.** A sample counts only when *both* sides have a
+  numeric value; `"x"` and empty are dropped along with their counterpart on
+  the other side. So `n` (`num_overlap`) varies per trait, and the engine
+  refuses rows with **fewer than 4** pairs (`correlations.rs`, the
+  `ty.x_vals.len() < 4 || ty.y_vals.len() < 4` guard). GN3's pure-Python
+  `compute_correlation` (`gn3/computations/correlations2.py`) uses **6**
+  instead — the two paths disagree on the minimum.
+- **Pairing is positional.** `x_vals[i]` is compared with field *i* of each
+  dataset line. The alignment is established earlier and by name: the cached
+  path re-selects columns by matching the header (`read_text_file` →
+  `__fetch_id_positions__`, `gn2/wqflask/correlation/pre_computes.py`), the
+  live path keys a dict by sample name. Get that order wrong and the answer is
+  plausible and silently wrong.
+
+### Tissue correlation
+
+Genes are joined on lowercased **symbol**, one probeset per symbol chosen in
+SQL by `max(Mean)` within `TissueProbeSetFreezeId = 1`
+(`gn2/base/mrna_assay_tissue_data.py`). The vectors are tuples built in row
+order from `SELECT Id, value FROM TissueProbeSetData`, so two genes with
+different numbers of rows produce unequal vectors: Pearson takes `n` from x
+and silently truncates, Spearman panics.
+
+### Literature correlation
+
+The value for a gene pair comes out of `LCorrRamin3` in **both directions**
+(`GeneId1, GeneId2, value` and `GeneId2, GeneId1, value`), with mouse → rat and
+mouse → human handled by `GeneIDXRef` — `gn3/db/correlations.py:40-92`. A
+per-request join, not a stored per-request result.
+
+## Measured sizes (read-only, `db_webqtl`, 2026-11-14)
+
+| table | rows | note |
+|---|---|---|
+| `ProbeSetData` | 5 314 694 464 | trait × individual × value — why nothing loads a dataset whole |
+| `PublishData` | 3 288 638 | phenotype values |
+| `TissueProbeSetData` | 90 562 DataIds, 26–32 values each (avg 29.0) | the tissue axis |
+| `LCorrRamin3` | 127 368 780 | the only precomputed correlation values anywhere |
+
+Consequence: no path fetches a dataset's traits whole. It fetches *your*
+sample subset for *your* trait list
+(`StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))`),
+which is also why the top-N bulk path exists: the arithmetic on 34 values ×
+20k traits is noise, the value fetch is the whole cost.
+
+## The `correlation_rust` engine
+
+`gn-rust-correlation-0.1.4-1.473ad82` (Guix: `guix-bioinformatics/gn/packages/genenetwork-rust.scm`;
+upstream `Alexanderlacuna/correlation_rust`). It is **a library with a 17-line
+binary bolted on**: `main.rs` takes one argument — a JSON path — calls
+`Analysis::compute`, prints the result and an elapsed time. `lib.rs` exposes
+`analysis`, `correlations`, `parser`, `reader`, `sorter`.
+
+### Contract: JSON in, CSV out
+
+Input JSON (`gn3/computations/rust_correlation.py`, `generate_json_file`):
+
+```json
+{"method": "pearson", "file_path": "<dataset.csv>", "x_vals": [...],
+ "sample_values": "bxd1", "output_file": "<out.csv>", "file_delimiter": ","}
+```
+
+`sample_values` is **dead**: it is in the `JsonData` struct and nothing reads
+it (hardcoded `"bxd1"` in both callers).
+
+The dataset file is one trait per line, `name,v1,v2,…,vn`, where position *i*
+corresponds to `x_vals[i]`. Per row (`correlations.rs:135`),
+`parse_rows_with_names` splits the name, pairs `x_vals[i]` with field *i*, and
+**drops a field that does not parse as `f64` together with its x value** —
+which is how missing data survives the alignment.
+
+Statistics: `rgsl`'s `correlation`/`spearman`; p two-sided from the
+t-distribution with `df = n - 2`, mirroring R's `cor.test`. **Bicor does not
+exist in the engine** — only `"pearson"` and `"spearman"` are matched; `bicor`
+exists solely on GN3's Python path.
+
+Output (`sorter.rs`): `name,r,p,n` per line, **sorted by |r| descending with
+NaNs last**, and that is the whole file — no trait id, no dataset, no
+filtering. Top-N is applied afterwards in Python
+(`parse_correlation_output`, `idx < top_n`), after the engine has sorted
+everything.
+
+### The Python around it
+
+`compute_correlation_rust` (`gn2/wqflask/correlation/rust_correlation.py:390`)
+is the entry point; `correlation_gn3_api.compute_correlation:185` its only
+caller. Five stages:
+
+1. **Resolve** — `create_target_this_trait(start_vars)` builds this_dataset,
+   this_trait, target_dataset.
+2. **Get the target data**, one of three ways: a per-dataset text dump
+   (`pre_computes.py`, columns re-selected by name); live
+   (`target_dataset.get_trait_data(sample_keys)`); or the top-N bulk SQL path
+   (`compute_top_n_sample`, `ProbeSetData` joined to xref/freeze, chunked by
+   the number of samples, one line per trait).
+3. **Write** the two files into a temporary directory and
+   `subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True)`.
+   `CORRELATION_COMMAND` is `<GN2_PROFILE>/bin/correlation_rust`
+   (`gn2/default_settings.py`); GN3 reads it from Flask config with no default.
+4. **Parse** the CSV into `{trait: {corr_coefficient, p_value, num_overlap}}`
+   (or `tissue_corr`/`tissue_number`/`tissue_p_val`).
+5. **Optionally enrich** (`should_compute_all`): take the top hits and run the
+   other two kinds over just those traits, then merge per trait — so one row
+   can carry sample, tissue and literature r at once.
+
+Literature never touches the engine: it is `compute_all_lit_correlation` over
+`LCorrRamin3`.
+
+## Duplicating it into GN4
+
+`src/routes/correlation.rs` is a stub returning canned
+`{trait_id, correlation}`, and its doc comment already names the target:
+*"correlation_rust in-process, rev-pinned"*. Because the crate is a library,
+"in-process" can mean two quite different amounts of code:
+
+- **Thin**: depend on the git rev `473ad82`, keep the file dance, call
+  `analysis::Analysis::compute(json_path)`, read the CSV. Least code and exact
+  upstream semantics — including the `unwrap()`/`panic!` on file I/O and bad
+  JSON, which in a request thread aborts it (that is why GN2 and GN3 use a
+  subprocess at all).
+- **Direct**: skip the files. `parser::parse_rows_with_names`,
+  `Pearson`/`Spearman` and the `Correlation` trait are all `pub`, so a caller
+  can build the vectors in memory, reuse the maths (the p-value is the subtle
+  part — reuse it, do not rewrite it) and implement only the sort and top-N.
+  No temporary files, no CSV parsing, no `unwrap()` on I/O in the request path.
+
+Either way GN4 must supply what it does not have today: the **bulk per-dataset
+value fetch** (the expensive half), the trait metadata for the reply, and a
+decision on four open semantics:
+
+1. minimum overlap: 4 (engine) or 6 (GN3 Python);
+2. pearson and spearman only — bicor needs another engine;
+3. top-N before or after enrichment;
+4. whether to reproduce the engine's exact NaN/tie ordering.
+
+## References
+
+- `genenetwork2/gn2/wqflask/correlation/rust_correlation.py` — the wrapper and
+  the three top-N paths.
+- `genenetwork2/gn2/wqflask/correlation/pre_computes.py` — the per-dataset text
+  dump and its name-keyed column selection.
+- `genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py` — the single
+  entry point, delegating to GN3's functions in-process.
+- `genenetwork3/gn3/api/correlation.py` — `/sample_x`, `/sample_r`, `/lit_corr`,
+  `/tissue_corr`, `/partial`.
+- `genenetwork3/gn3/computations/rust_correlation.py` — `run_correlation` and
+  the output parser.
+- `genenetwork3/gn3/db/correlations.py` — the literature and tissue SQL.
+- `genenetwork3/scripts/sample_correlations.py` — the pickled-file subprocess
+  used by the non-Rust sample path.