# What GeneNetwork correlates, and how GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers four different things measured over three different axes. This note records what is actually correlated, which engine computes it, and where. It exists because the answer to "duplicate the correlation_rust path" depends entirely on knowing which of the four you are duplicating. Written from the code, with row counts measured read-only against the production `db_webqtl` (2026-11-14). ## The four correlations | kind | **x** | **y** | observations | computed? | |---|---|---|---|---| | **sample** | one trait, per individual (BXD strain, animal, plant, human case) | each trait of a target dataset, same individuals | individuals (~34–40 for BXD) | every request | | **tissue** | one gene's expression across tissue systems | another gene's expression across the same tissues | tissues (**26–32** per gene) | every request | | **literature** | a gene | another gene | published studies | **read from a table** | | **partial** (GN3 only) | one trait | one trait, conditioned on other traits/genotypes | individuals | every request, async | Nothing here correlates markers with traits — that is QTL mapping. In the sample and partial cases the observations are **individuals**; in the tissue case they are **tissues**; literature has no n at all. ### Sample correlation: the one that matters most `POST /compute/correlation` for trait X against "BXD Liver" is not one correlation: it is X against *every* trait in that dataset, one r each, all over the same individual axis — about 20k r's over ~34–40 strains. Which individuals is a request knob (`corr_samples_group` in GN2): - **primary only** — the primary trait's own group members (BXD strains only); - **other only** — everything in the data *except* those, which is how you correlate across other strains or species; - **all** — every sample the trait data carries. Two rules decide correctness, both easy to get wrong: - **Missing data is pairwise.** A sample counts only when *both* sides have a numeric value; `"x"` and empty are dropped along with their counterpart on the other side. So `n` (`num_overlap`) varies per trait, and the engine refuses rows with **fewer than 4** pairs (`correlations.rs`, the `ty.x_vals.len() < 4 || ty.y_vals.len() < 4` guard). GN3's pure-Python `compute_correlation` (`gn3/computations/correlations2.py`) uses **6** instead — the two paths disagree on the minimum. - **Pairing is positional.** `x_vals[i]` is compared with field *i* of each dataset line. The alignment is established earlier and by name: the cached path re-selects columns by matching the header (`read_text_file` → `__fetch_id_positions__`, `gn2/wqflask/correlation/pre_computes.py`), the live path keys a dict by sample name. Get that order wrong and the answer is plausible and silently wrong. ### Tissue correlation Genes are joined on lowercased **symbol**, one probeset per symbol chosen in SQL by `max(Mean)` within `TissueProbeSetFreezeId = 1` (`gn2/base/mrna_assay_tissue_data.py`). The vectors are tuples built in row order from `SELECT Id, value FROM TissueProbeSetData`, so two genes with different numbers of rows produce unequal vectors: Pearson takes `n` from x and silently truncates, Spearman panics. ### Literature correlation The value for a gene pair comes out of `LCorrRamin3` in **both directions** (`GeneId1, GeneId2, value` and `GeneId2, GeneId1, value`), with mouse → rat and mouse → human handled by `GeneIDXRef` — `gn3/db/correlations.py:40-92`. A per-request join, not a stored per-request result. ## Measured sizes (read-only, `db_webqtl`, 2026-11-14) | table | rows | note | |---|---|---| | `ProbeSetData` | 5 314 694 464 | trait × individual × value — why nothing loads a dataset whole | | `PublishData` | 3 288 638 | phenotype values | | `TissueProbeSetData` | 90 562 DataIds, 26–32 values each (avg 29.0) | the tissue axis | | `LCorrRamin3` | 127 368 780 | the only precomputed correlation values anywhere | Consequence: no path fetches a dataset's traits whole. It fetches *your* sample subset for *your* trait list (`StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))`), which is also why the top-N bulk path exists: the arithmetic on 34 values × 20k traits is noise, the value fetch is the whole cost. ## The `correlation_rust` engine `gn-rust-correlation-0.1.4-1.473ad82` (Guix: `guix-bioinformatics/gn/packages/genenetwork-rust.scm`; upstream `Alexanderlacuna/correlation_rust`). It is **a library with a 17-line binary bolted on**: `main.rs` takes one argument — a JSON path — calls `Analysis::compute`, prints the result and an elapsed time. `lib.rs` exposes `analysis`, `correlations`, `parser`, `reader`, `sorter`. ### Contract: JSON in, CSV out Input JSON (`gn3/computations/rust_correlation.py`, `generate_json_file`): ```json {"method": "pearson", "file_path": "", "x_vals": [...], "sample_values": "bxd1", "output_file": "", "file_delimiter": ","} ``` `sample_values` is **dead**: it is in the `JsonData` struct and nothing reads it (hardcoded `"bxd1"` in both callers). The dataset file is one trait per line, `name,v1,v2,…,vn`, where position *i* corresponds to `x_vals[i]`. Per row (`correlations.rs:135`), `parse_rows_with_names` splits the name, pairs `x_vals[i]` with field *i*, and **drops a field that does not parse as `f64` together with its x value** — which is how missing data survives the alignment. Statistics: `rgsl`'s `correlation`/`spearman`; p two-sided from the t-distribution with `df = n - 2`, mirroring R's `cor.test`. **Bicor does not exist in the engine** — only `"pearson"` and `"spearman"` are matched; `bicor` exists solely on GN3's Python path. Output (`sorter.rs`): `name,r,p,n` per line, **sorted by |r| descending with NaNs last**, and that is the whole file — no trait id, no dataset, no filtering. Top-N is applied afterwards in Python (`parse_correlation_output`, `idx < top_n`), after the engine has sorted everything. ### The Python around it `compute_correlation_rust` (`gn2/wqflask/correlation/rust_correlation.py:390`) is the entry point; `correlation_gn3_api.compute_correlation:185` its only caller. Five stages: 1. **Resolve** — `create_target_this_trait(start_vars)` builds this_dataset, this_trait, target_dataset. 2. **Get the target data**, one of three ways: a per-dataset text dump (`pre_computes.py`, columns re-selected by name); live (`target_dataset.get_trait_data(sample_keys)`); or the top-N bulk SQL path (`compute_top_n_sample`, `ProbeSetData` joined to xref/freeze, chunked by the number of samples, one line per trait). 3. **Write** the two files into a temporary directory and `subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True)`. `CORRELATION_COMMAND` is `/bin/correlation_rust` (`gn2/default_settings.py`); GN3 reads it from Flask config with no default. 4. **Parse** the CSV into `{trait: {corr_coefficient, p_value, num_overlap}}` (or `tissue_corr`/`tissue_number`/`tissue_p_val`). 5. **Optionally enrich** (`should_compute_all`): take the top hits and run the other two kinds over just those traits, then merge per trait — so one row can carry sample, tissue and literature r at once. Literature never touches the engine: it is `compute_all_lit_correlation` over `LCorrRamin3`. ## Duplicating it into GN4 `src/routes/correlation.rs` is a stub returning canned `{trait_id, correlation}`. What to build, decided on 2026-11-14: **Synchronous, no redis, no job.** One POST computes and answers. GN2's `/corr_compute` writes a form to a file and runs a subprocess job, and GN3's partial path goes through redis; neither is wanted for the first version. That also means the reply carries the result, not a `?id=`/`/status` pair. **Use the `correlation_rust` engine, in-process.** `gn-rust-correlation` is already packaged (`gn/packages/genenetwork-rust.scm`, rev `473ad82`), so this is a dependency to wire in rather than a package to write. Its `parser::parse_rows_with_names`, `Pearson`/`Spearman` and the `Correlation` trait are `pub`: build the vectors in memory, reuse the maths — **the p-value is the subtle part; reuse it, do not rewrite it** — and implement only the sort and top-N. No temporary files, no CSV round trip, and none of upstream's `unwrap()`/`panic!` on file I/O in a request thread (which is why GN2 and GN3 shell out at all). **Semantics, settled:** | decision | choice | |---|---| | minimum overlap | **6** (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant | | methods | pearson and spearman only; bicor does not exist in the engine and stays out | | ordering | reproduce the engine exactly: \|r\| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference | | top-N | a request option (GN2's menu is 100…20000, 500 by default); applied *after* the sort | | first axis | **sample** only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2 | **Verification is against the live site**, which turns out to be reachable: it answers `show_trait` with HTTP 200 in ~31 s (a `-m 20` curl had made it look unreachable). Each reference page therefore costs ~31 s, which needs budgeting in any parity loop. What GN4 still has to supply, and does not have today: - the **bulk per-dataset value fetch** — the expensive half; see the measured sizes above, and note the shape the existing queries use: the caller's sample subset for the caller's trait list, never a whole dataset; - trait metadata for the reply (the id, and the name GN2 shows); - the sample-group knob (`samples_primary` / `other` / `all`) and the target dataset, both of which change *which* rows are fetched. ## References - `genenetwork2/gn2/wqflask/correlation/rust_correlation.py` — the wrapper and the three top-N paths. - `genenetwork2/gn2/wqflask/correlation/pre_computes.py` — the per-dataset text dump and its name-keyed column selection. - `genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py` — the single entry point, delegating to GN3's functions in-process. - `genenetwork3/gn3/api/correlation.py` — `/sample_x`, `/sample_r`, `/lit_corr`, `/tissue_corr`, `/partial`. - `genenetwork3/gn3/computations/rust_correlation.py` — `run_correlation` and the output parser. - `genenetwork3/gn3/db/correlations.py` — the literature and tissue SQL. - `genenetwork3/scripts/sample_correlations.py` — the pickled-file subprocess used by the non-Rust sample path.