diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-14 22:58:18 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-14 22:58:18 +0000 |
| commit | 4d554b02daaea23b60826aa7b3c644baa55376d2 (patch) | |
| tree | b6f3e879d0509bcdad1abd5207b76afa5291925e /docs | |
| parent | 3a62ef34c127d36fd4e04583349f64eee5de2c6f (diff) | |
| download | genenetwork4-4d554b02daaea23b60826aa7b3c644baa55376d2.tar.gz | |
docs: what GeneNetwork correlates, and what correlation_rust does
Answers the question that blocks duplicating the correlation_rust path: what is actually correlated. Four kinds over three axes -- sample (one trait vs every trait of a dataset, over individuals), tissue (gene vs gene over 26-32 tissue systems), literature (a table lookup in LCorrRamin3, no n at all) and partial (conditioned, GN3 only, async). Records the engine's contract as it is: JSON in, CSV out, one trait per line positionally aligned to x_vals, non-floats dropped pairwise, rows under 4 pairs refused, output sorted by |r| descending with NaNs last, top-N applied afterwards in Python, and no bicor in the engine at all -- which matters because GN3's Python path uses a minimum of 6 and the engine uses 4. Measured read-only against db_webqtl: ProbeSetData 5 314 694 464 rows, PublishData 3 288 638, TissueProbeSetData 90 562 DataIds with 26-32 values each (avg 29.0), LCorrRamin3 127 368 780. Hence no path loads a dataset whole, and the value fetch -- not the arithmetic -- is the cost. Ends with the two shapes a GN4 integration can take (thin: keep the file dance; direct: reuse parser::parse_rows_with_names and Pearson/Spearman in memory) and the four semantics that must be decided either way.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/correlations.md | 189 |
1 files changed, 189 insertions, 0 deletions
diff --git a/docs/correlations.md b/docs/correlations.md new file mode 100644 index 0000000..1ff3770 --- /dev/null +++ b/docs/correlations.md @@ -0,0 +1,189 @@ +# What GeneNetwork correlates, and how + +GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers +four different things measured over three different axes. This note records +what is actually correlated, which engine computes it, and where. It exists +because the answer to "duplicate the correlation_rust path" depends entirely +on knowing which of the four you are duplicating. + +Written from the code, with row counts measured read-only against the +production `db_webqtl` (2026-11-14). + +## The four correlations + +| kind | **x** | **y** | observations | computed? | +|---|---|---|---|---| +| **sample** | one trait, per individual (BXD strain, animal, plant, human case) | each trait of a target dataset, same individuals | individuals (~34–40 for BXD) | every request | +| **tissue** | one gene's expression across tissue systems | another gene's expression across the same tissues | tissues (**26–32** per gene) | every request | +| **literature** | a gene | another gene | published studies | **read from a table** | +| **partial** (GN3 only) | one trait | one trait, conditioned on other traits/genotypes | individuals | every request, async | + +Nothing here correlates markers with traits — that is QTL mapping. In the +sample and partial cases the observations are **individuals**; in the tissue +case they are **tissues**; literature has no n at all. + +### Sample correlation: the one that matters most + +`POST /compute/correlation` for trait X against "BXD Liver" is not one +correlation: it is X against *every* trait in that dataset, one r each, all +over the same individual axis — about 20k r's over ~34–40 strains. + +Which individuals is a request knob (`corr_samples_group` in GN2): + +- **primary only** — the primary trait's own group members (BXD strains only); +- **other only** — everything in the data *except* those, which is how you + correlate across other strains or species; +- **all** — every sample the trait data carries. + +Two rules decide correctness, both easy to get wrong: + +- **Missing data is pairwise.** A sample counts only when *both* sides have a + numeric value; `"x"` and empty are dropped along with their counterpart on + the other side. So `n` (`num_overlap`) varies per trait, and the engine + refuses rows with **fewer than 4** pairs (`correlations.rs`, the + `ty.x_vals.len() < 4 || ty.y_vals.len() < 4` guard). GN3's pure-Python + `compute_correlation` (`gn3/computations/correlations2.py`) uses **6** + instead — the two paths disagree on the minimum. +- **Pairing is positional.** `x_vals[i]` is compared with field *i* of each + dataset line. The alignment is established earlier and by name: the cached + path re-selects columns by matching the header (`read_text_file` → + `__fetch_id_positions__`, `gn2/wqflask/correlation/pre_computes.py`), the + live path keys a dict by sample name. Get that order wrong and the answer is + plausible and silently wrong. + +### Tissue correlation + +Genes are joined on lowercased **symbol**, one probeset per symbol chosen in +SQL by `max(Mean)` within `TissueProbeSetFreezeId = 1` +(`gn2/base/mrna_assay_tissue_data.py`). The vectors are tuples built in row +order from `SELECT Id, value FROM TissueProbeSetData`, so two genes with +different numbers of rows produce unequal vectors: Pearson takes `n` from x +and silently truncates, Spearman panics. + +### Literature correlation + +The value for a gene pair comes out of `LCorrRamin3` in **both directions** +(`GeneId1, GeneId2, value` and `GeneId2, GeneId1, value`), with mouse → rat and +mouse → human handled by `GeneIDXRef` — `gn3/db/correlations.py:40-92`. A +per-request join, not a stored per-request result. + +## Measured sizes (read-only, `db_webqtl`, 2026-11-14) + +| table | rows | note | +|---|---|---| +| `ProbeSetData` | 5 314 694 464 | trait × individual × value — why nothing loads a dataset whole | +| `PublishData` | 3 288 638 | phenotype values | +| `TissueProbeSetData` | 90 562 DataIds, 26–32 values each (avg 29.0) | the tissue axis | +| `LCorrRamin3` | 127 368 780 | the only precomputed correlation values anywhere | + +Consequence: no path fetches a dataset's traits whole. It fetches *your* +sample subset for *your* trait list +(`StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))`), +which is also why the top-N bulk path exists: the arithmetic on 34 values × +20k traits is noise, the value fetch is the whole cost. + +## The `correlation_rust` engine + +`gn-rust-correlation-0.1.4-1.473ad82` (Guix: `guix-bioinformatics/gn/packages/genenetwork-rust.scm`; +upstream `Alexanderlacuna/correlation_rust`). It is **a library with a 17-line +binary bolted on**: `main.rs` takes one argument — a JSON path — calls +`Analysis::compute`, prints the result and an elapsed time. `lib.rs` exposes +`analysis`, `correlations`, `parser`, `reader`, `sorter`. + +### Contract: JSON in, CSV out + +Input JSON (`gn3/computations/rust_correlation.py`, `generate_json_file`): + +```json +{"method": "pearson", "file_path": "<dataset.csv>", "x_vals": [...], + "sample_values": "bxd1", "output_file": "<out.csv>", "file_delimiter": ","} +``` + +`sample_values` is **dead**: it is in the `JsonData` struct and nothing reads +it (hardcoded `"bxd1"` in both callers). + +The dataset file is one trait per line, `name,v1,v2,…,vn`, where position *i* +corresponds to `x_vals[i]`. Per row (`correlations.rs:135`), +`parse_rows_with_names` splits the name, pairs `x_vals[i]` with field *i*, and +**drops a field that does not parse as `f64` together with its x value** — +which is how missing data survives the alignment. + +Statistics: `rgsl`'s `correlation`/`spearman`; p two-sided from the +t-distribution with `df = n - 2`, mirroring R's `cor.test`. **Bicor does not +exist in the engine** — only `"pearson"` and `"spearman"` are matched; `bicor` +exists solely on GN3's Python path. + +Output (`sorter.rs`): `name,r,p,n` per line, **sorted by |r| descending with +NaNs last**, and that is the whole file — no trait id, no dataset, no +filtering. Top-N is applied afterwards in Python +(`parse_correlation_output`, `idx < top_n`), after the engine has sorted +everything. + +### The Python around it + +`compute_correlation_rust` (`gn2/wqflask/correlation/rust_correlation.py:390`) +is the entry point; `correlation_gn3_api.compute_correlation:185` its only +caller. Five stages: + +1. **Resolve** — `create_target_this_trait(start_vars)` builds this_dataset, + this_trait, target_dataset. +2. **Get the target data**, one of three ways: a per-dataset text dump + (`pre_computes.py`, columns re-selected by name); live + (`target_dataset.get_trait_data(sample_keys)`); or the top-N bulk SQL path + (`compute_top_n_sample`, `ProbeSetData` joined to xref/freeze, chunked by + the number of samples, one line per trait). +3. **Write** the two files into a temporary directory and + `subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True)`. + `CORRELATION_COMMAND` is `<GN2_PROFILE>/bin/correlation_rust` + (`gn2/default_settings.py`); GN3 reads it from Flask config with no default. +4. **Parse** the CSV into `{trait: {corr_coefficient, p_value, num_overlap}}` + (or `tissue_corr`/`tissue_number`/`tissue_p_val`). +5. **Optionally enrich** (`should_compute_all`): take the top hits and run the + other two kinds over just those traits, then merge per trait — so one row + can carry sample, tissue and literature r at once. + +Literature never touches the engine: it is `compute_all_lit_correlation` over +`LCorrRamin3`. + +## Duplicating it into GN4 + +`src/routes/correlation.rs` is a stub returning canned +`{trait_id, correlation}`, and its doc comment already names the target: +*"correlation_rust in-process, rev-pinned"*. Because the crate is a library, +"in-process" can mean two quite different amounts of code: + +- **Thin**: depend on the git rev `473ad82`, keep the file dance, call + `analysis::Analysis::compute(json_path)`, read the CSV. Least code and exact + upstream semantics — including the `unwrap()`/`panic!` on file I/O and bad + JSON, which in a request thread aborts it (that is why GN2 and GN3 use a + subprocess at all). +- **Direct**: skip the files. `parser::parse_rows_with_names`, + `Pearson`/`Spearman` and the `Correlation` trait are all `pub`, so a caller + can build the vectors in memory, reuse the maths (the p-value is the subtle + part — reuse it, do not rewrite it) and implement only the sort and top-N. + No temporary files, no CSV parsing, no `unwrap()` on I/O in the request path. + +Either way GN4 must supply what it does not have today: the **bulk per-dataset +value fetch** (the expensive half), the trait metadata for the reply, and a +decision on four open semantics: + +1. minimum overlap: 4 (engine) or 6 (GN3 Python); +2. pearson and spearman only — bicor needs another engine; +3. top-N before or after enrichment; +4. whether to reproduce the engine's exact NaN/tie ordering. + +## References + +- `genenetwork2/gn2/wqflask/correlation/rust_correlation.py` — the wrapper and + the three top-N paths. +- `genenetwork2/gn2/wqflask/correlation/pre_computes.py` — the per-dataset text + dump and its name-keyed column selection. +- `genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py` — the single + entry point, delegating to GN3's functions in-process. +- `genenetwork3/gn3/api/correlation.py` — `/sample_x`, `/sample_r`, `/lit_corr`, + `/tissue_corr`, `/partial`. +- `genenetwork3/gn3/computations/rust_correlation.py` — `run_correlation` and + the output parser. +- `genenetwork3/gn3/db/correlations.py` — the literature and tissue SQL. +- `genenetwork3/scripts/sample_correlations.py` — the pickled-file subprocess + used by the non-Rust sample path. |
