about summary refs log tree commit diff
path: root/docs/correlations.md
blob: 05c6781dabaedb725d96b835aa1f907daa4b36a5 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
# What GeneNetwork correlates, and how

GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers
four different things measured over three different axes. This note records
what is actually correlated, which engine computes it, and where. It exists
because the answer to "duplicate the correlation_rust path" depends entirely
on knowing which of the four you are duplicating.

Written from the code, with row counts measured read-only against the
production `db_webqtl` (2026-11-14).

## The four correlations

| kind | **x** | **y** | observations | computed? |
|---|---|---|---|---|
| **sample** | one trait, per individual (BXD strain, animal, plant, human case) | each trait of a target dataset, same individuals | individuals (~34–40 for BXD) | every request |
| **tissue** | one gene's expression across tissue systems | another gene's expression across the same tissues | tissues (**26–32** per gene) | every request |
| **literature** | a gene | another gene | published studies | **read from a table** |
| **partial** (GN3 only) | one trait | one trait, conditioned on other traits/genotypes | individuals | every request, async |

Nothing here correlates markers with traits — that is QTL mapping. In the
sample and partial cases the observations are **individuals**; in the tissue
case they are **tissues**; literature has no n at all.

### Sample correlation: the one that matters most

`POST /compute/correlation` for trait X against "BXD Liver" is not one
correlation: it is X against *every* trait in that dataset, one r each, all
over the same individual axis — about 20k r's over ~34–40 strains.

Which individuals is a request knob (`corr_samples_group` in GN2):

- **primary only** — the primary trait's own group members (BXD strains only);
- **other only** — everything in the data *except* those, which is how you
  correlate across other strains or species;
- **all** — every sample the trait data carries.

Two rules decide correctness, both easy to get wrong:

- **Missing data is pairwise.** A sample counts only when *both* sides have a
  numeric value; `"x"` and empty are dropped along with their counterpart on
  the other side. So `n` (`num_overlap`) varies per trait, and the engine
  refuses rows with **fewer than 4** pairs (`correlations.rs`, the
  `ty.x_vals.len() < 4 || ty.y_vals.len() < 4` guard). GN3's pure-Python
  `compute_correlation` (`gn3/computations/correlations2.py`) uses **6**
  instead — the two paths disagree on the minimum.
- **Pairing is positional.** `x_vals[i]` is compared with field *i* of each
  dataset line. The alignment is established earlier and by name: the cached
  path re-selects columns by matching the header (`read_text_file` →
  `__fetch_id_positions__`, `gn2/wqflask/correlation/pre_computes.py`), the
  live path keys a dict by sample name. Get that order wrong and the answer is
  plausible and silently wrong.

### Tissue correlation

Genes are joined on lowercased **symbol**, one probeset per symbol chosen in
SQL by `max(Mean)` within `TissueProbeSetFreezeId = 1`
(`gn2/base/mrna_assay_tissue_data.py`). The vectors are tuples built in row
order from `SELECT Id, value FROM TissueProbeSetData`, so two genes with
different numbers of rows produce unequal vectors: Pearson takes `n` from x
and silently truncates, Spearman panics.

### Literature correlation

The value for a gene pair comes out of `LCorrRamin3` in **both directions**
(`GeneId1, GeneId2, value` and `GeneId2, GeneId1, value`), with mouse → rat and
mouse → human handled by `GeneIDXRef` — `gn3/db/correlations.py:40-92`. A
per-request join, not a stored per-request result.

## Measured sizes (read-only, `db_webqtl`, 2026-11-14)

| table | rows | note |
|---|---|---|
| `ProbeSetData` | 5 314 694 464 | trait × individual × value — why nothing loads a dataset whole |
| `PublishData` | 3 288 638 | phenotype values |
| `TissueProbeSetData` | 90 562 DataIds, 26–32 values each (avg 29.0) | the tissue axis |
| `LCorrRamin3` | 127 368 780 | the only precomputed correlation values anywhere |

Consequence: no path fetches a dataset's traits whole. It fetches *your*
sample subset for *your* trait list
(`StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))`),
which is also why the top-N bulk path exists: the arithmetic on 34 values ×
20k traits is noise, the value fetch is the whole cost.

## The `correlation_rust` engine

`gn-rust-correlation-0.1.4-1.473ad82` (Guix: `guix-bioinformatics/gn/packages/genenetwork-rust.scm`;
upstream `Alexanderlacuna/correlation_rust`). It is **a library with a 17-line
binary bolted on**: `main.rs` takes one argument — a JSON path — calls
`Analysis::compute`, prints the result and an elapsed time. `lib.rs` exposes
`analysis`, `correlations`, `parser`, `reader`, `sorter`.

### Contract: JSON in, CSV out

Input JSON (`gn3/computations/rust_correlation.py`, `generate_json_file`):

```json
{"method": "pearson", "file_path": "<dataset.csv>", "x_vals": [...],
 "sample_values": "bxd1", "output_file": "<out.csv>", "file_delimiter": ","}
```

`sample_values` is **dead**: it is in the `JsonData` struct and nothing reads
it (hardcoded `"bxd1"` in both callers).

The dataset file is one trait per line, `name,v1,v2,…,vn`, where position *i*
corresponds to `x_vals[i]`. Per row (`correlations.rs:135`),
`parse_rows_with_names` splits the name, pairs `x_vals[i]` with field *i*, and
**drops a field that does not parse as `f64` together with its x value** —
which is how missing data survives the alignment.

Statistics: `rgsl`'s `correlation`/`spearman`; p two-sided from the
t-distribution with `df = n - 2`, mirroring R's `cor.test`. **Bicor does not
exist in the engine** — only `"pearson"` and `"spearman"` are matched; `bicor`
exists solely on GN3's Python path.

Output (`sorter.rs`): `name,r,p,n` per line, **sorted by |r| descending with
NaNs last**, and that is the whole file — no trait id, no dataset, no
filtering. Top-N is applied afterwards in Python
(`parse_correlation_output`, `idx < top_n`), after the engine has sorted
everything.

### The Python around it

`compute_correlation_rust` (`gn2/wqflask/correlation/rust_correlation.py:390`)
is the entry point; `correlation_gn3_api.compute_correlation:185` its only
caller. Five stages:

1. **Resolve** — `create_target_this_trait(start_vars)` builds this_dataset,
   this_trait, target_dataset.
2. **Get the target data**, one of three ways: a per-dataset text dump
   (`pre_computes.py`, columns re-selected by name); live
   (`target_dataset.get_trait_data(sample_keys)`); or the top-N bulk SQL path
   (`compute_top_n_sample`, `ProbeSetData` joined to xref/freeze, chunked by
   the number of samples, one line per trait).
3. **Write** the two files into a temporary directory and
   `subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True)`.
   `CORRELATION_COMMAND` is `<GN2_PROFILE>/bin/correlation_rust`
   (`gn2/default_settings.py`); GN3 reads it from Flask config with no default.
4. **Parse** the CSV into `{trait: {corr_coefficient, p_value, num_overlap}}`
   (or `tissue_corr`/`tissue_number`/`tissue_p_val`).
5. **Optionally enrich** (`should_compute_all`): take the top hits and run the
   other two kinds over just those traits, then merge per trait — so one row
   can carry sample, tissue and literature r at once.

Literature never touches the engine: it is `compute_all_lit_correlation` over
`LCorrRamin3`.

## Duplicating it into GN4

`src/routes/correlation.rs` is a stub returning canned
`{trait_id, correlation}`. What to build, decided on 2026-11-14:

**Synchronous, no redis, no job.** One POST computes and answers. GN2's
`/corr_compute` writes a form to a file and runs a subprocess job, and GN3's
partial path goes through redis; neither is wanted for the first version. That
also means the reply carries the result, not a `?id=`/`/status` pair.

**Use the `correlation_rust` engine, in-process.** `gn-rust-correlation` is
already packaged (`gn/packages/genenetwork-rust.scm`, rev `473ad82`), so this is
a dependency to wire in rather than a package to write. Its `parser::parse_rows_with_names`,
`Pearson`/`Spearman` and the `Correlation` trait are `pub`: build the vectors in
memory, reuse the maths — **the p-value is the subtle part; reuse it, do not
rewrite it** — and implement only the sort and top-N. No temporary files, no CSV
round trip, and none of upstream's `unwrap()`/`panic!` on file I/O in a request
thread (which is why GN2 and GN3 shell out at all).

**Semantics, settled:**

| decision | choice |
|---|---|
| minimum overlap | **6** (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant |
| methods | pearson and spearman only; bicor does not exist in the engine and stays out |
| ordering | reproduce the engine exactly: \|r\| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference |
| top-N | a request option (GN2's menu is 100…20000, 500 by default); applied *after* the sort |
| first axis | **sample** only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2 |

**Verification is against the live site**, which turns out to be reachable: it
answers `show_trait` with HTTP 200 in ~31 s (a `-m 20` curl had made it look
unreachable). Each reference page therefore costs ~31 s, which needs budgeting in
any parity loop.

What GN4 still has to supply, and does not have today:

- the **bulk per-dataset value fetch** — the expensive half; see the measured
  sizes above, and note the shape the existing queries use: the caller's sample
  subset for the caller's trait list, never a whole dataset;
- trait metadata for the reply (the id, and the name GN2 shows);
- the sample-group knob (`samples_primary` / `other` / `all`) and the target
  dataset, both of which change *which* rows are fetched.

## References

- `genenetwork2/gn2/wqflask/correlation/rust_correlation.py` — the wrapper and
  the three top-N paths.
- `genenetwork2/gn2/wqflask/correlation/pre_computes.py` — the per-dataset text
  dump and its name-keyed column selection.
- `genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py` — the single
  entry point, delegating to GN3's functions in-process.
- `genenetwork3/gn3/api/correlation.py` — `/sample_x`, `/sample_r`, `/lit_corr`,
  `/tissue_corr`, `/partial`.
- `genenetwork3/gn3/computations/rust_correlation.py` — `run_correlation` and
  the output parser.
- `genenetwork3/gn3/db/correlations.py` — the literature and tissue SQL.
- `genenetwork3/scripts/sample_correlations.py` — the pickled-file subprocess
  used by the non-Rust sample path.