What GeneNetwork correlates, and how
GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers four different things measured over three different axes. This note records what is actually correlated, which engine computes it, and where. It exists because the answer to "duplicate the correlation_rust path" depends entirely on knowing which of the four you are duplicating.
Written from the code, with row counts measured read-only against the
production db_webqtl (2026-11-14).
The four correlations
| kind | x | y | observations | computed? |
|---|---|---|---|---|
| sample | one trait, per individual (BXD strain, animal, plant, human case) | each trait of a target dataset, same individuals | individuals (~34–40 for BXD) | every request |
| tissue | one gene's expression across tissue systems | another gene's expression across the same tissues | tissues (26–32 per gene) | every request |
| literature | a gene | another gene | published studies | read from a table |
| partial (GN3 only) | one trait | one trait, conditioned on other traits/genotypes | individuals | every request, async |
Nothing here correlates markers with traits — that is QTL mapping. In the sample and partial cases the observations are individuals; in the tissue case they are tissues; literature has no n at all.
Sample correlation: the one that matters most
POST /compute/correlation for trait X against "BXD Liver" is not one
correlation: it is X against every trait in that dataset, one r each, all
over the same individual axis — about 20k r's over ~34–40 strains.
Which individuals is a request knob (corr_samples_group in GN2):
- primary only — the primary trait's own group members (BXD strains only);
- other only — everything in the data except those, which is how you correlate across other strains or species;
- all — every sample the trait data carries.
Two rules decide correctness, both easy to get wrong:
- Missing data is pairwise. A sample counts only when both sides have a
numeric value;
"x"and empty are dropped along with their counterpart on the other side. Son(num_overlap) varies per trait, and the engine refuses rows with fewer than 4 pairs (correlations.rs, thety.x_vals.len() < 4 || ty.y_vals.len() < 4guard). GN3's pure-Pythoncompute_correlation(gn3/computations/correlations2.py) uses 6 instead — the two paths disagree on the minimum. - Pairing is positional.
x_vals[i]is compared with field i of each dataset line. The alignment is established earlier and by name: the cached path re-selects columns by matching the header (read_text_file→__fetch_id_positions__,gn2/wqflask/correlation/pre_computes.py), the live path keys a dict by sample name. Get that order wrong and the answer is plausible and silently wrong.
Tissue correlation
Genes are joined on lowercased symbol, one probeset per symbol chosen in
SQL by max(Mean) within TissueProbeSetFreezeId = 1
(gn2/base/mrna_assay_tissue_data.py). The vectors are tuples built in row
order from SELECT Id, value FROM TissueProbeSetData, so two genes with
different numbers of rows produce unequal vectors: Pearson takes n from x
and silently truncates, Spearman panics.
Literature correlation
The value for a gene pair comes out of LCorrRamin3 in both directions
(GeneId1, GeneId2, value and GeneId2, GeneId1, value), with mouse → rat and
mouse → human handled by GeneIDXRef — gn3/db/correlations.py:40-92. A
per-request join, not a stored per-request result.
Measured sizes (read-only, db_webqtl, 2026-11-14)
| table | rows | note |
|---|---|---|
ProbeSetData |
5 314 694 464 | trait × individual × value — why nothing loads a dataset whole |
PublishData |
3 288 638 | phenotype values |
TissueProbeSetData |
90 562 DataIds, 26–32 values each (avg 29.0) | the tissue axis |
LCorrRamin3 |
127 368 780 | the only precomputed correlation values anywhere |
Consequence: no path fetches a dataset's traits whole. It fetches your
sample subset for your trait list
(StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))),
which is also why the top-N bulk path exists: the arithmetic on 34 values ×
20k traits is noise, the value fetch is the whole cost.
The correlation_rust engine
gn-rust-correlation-0.1.4-1.473ad82 (Guix: guix-bioinformatics/gn/packages/genenetwork-rust.scm;
upstream Alexanderlacuna/correlation_rust). It is a library with a 17-line
binary bolted on: main.rs takes one argument — a JSON path — calls
Analysis::compute, prints the result and an elapsed time. lib.rs exposes
analysis, correlations, parser, reader, sorter.
Contract: JSON in, CSV out
Input JSON (gn3/computations/rust_correlation.py, generate_json_file):
{"method": "pearson", "file_path": "<dataset.csv>", "x_vals": [...],
"sample_values": "bxd1", "output_file": "<out.csv>", "file_delimiter": ","}
sample_values is dead: it is in the JsonData struct and nothing reads
it (hardcoded "bxd1" in both callers).
The dataset file is one trait per line, name,v1,v2,…,vn, where position i
corresponds to x_vals[i]. Per row (correlations.rs:135),
parse_rows_with_names splits the name, pairs x_vals[i] with field i, and
drops a field that does not parse as f64 together with its x value —
which is how missing data survives the alignment.
Statistics: rgsl's correlation/spearman; p two-sided from the
t-distribution with df = n - 2, mirroring R's cor.test. Bicor does not
exist in the engine — only "pearson" and "spearman" are matched; bicor
exists solely on GN3's Python path.
Output (sorter.rs): name,r,p,n per line, sorted by |r| descending with
NaNs last, and that is the whole file — no trait id, no dataset, no
filtering. Top-N is applied afterwards in Python
(parse_correlation_output, idx < top_n), after the engine has sorted
everything.
The Python around it
compute_correlation_rust (gn2/wqflask/correlation/rust_correlation.py:390)
is the entry point; correlation_gn3_api.compute_correlation:185 its only
caller. Five stages:
- Resolve —
create_target_this_trait(start_vars)builds this_dataset, this_trait, target_dataset. - Get the target data, one of three ways: a per-dataset text dump
(
pre_computes.py, columns re-selected by name); live (target_dataset.get_trait_data(sample_keys)); or the top-N bulk SQL path (compute_top_n_sample,ProbeSetDatajoined to xref/freeze, chunked by the number of samples, one line per trait). - Write the two files into a temporary directory and
subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True).CORRELATION_COMMANDis<GN2_PROFILE>/bin/correlation_rust(gn2/default_settings.py); GN3 reads it from Flask config with no default. - Parse the CSV into
{trait: {corr_coefficient, p_value, num_overlap}}(ortissue_corr/tissue_number/tissue_p_val). - Optionally enrich (
should_compute_all): take the top hits and run the other two kinds over just those traits, then merge per trait — so one row can carry sample, tissue and literature r at once.
Literature never touches the engine: it is compute_all_lit_correlation over
LCorrRamin3.
Duplicating it into GN4
src/routes/correlation.rs is a stub returning canned
{trait_id, correlation}. What to build, decided on 2026-11-14:
Synchronous, no redis, no job. One POST computes and answers. GN2's
/corr_compute writes a form to a file and runs a subprocess job, and GN3's
partial path goes through redis; neither is wanted for the first version. That
also means the reply carries the result, not a ?id=//status pair.
Use the correlation_rust engine, in-process. gn-rust-correlation is
already packaged (gn/packages/genenetwork-rust.scm, rev 473ad82), so this is
a dependency to wire in rather than a package to write. Its parser::parse_rows_with_names,
Pearson/Spearman and the Correlation trait are pub: build the vectors in
memory, reuse the maths — the p-value is the subtle part; reuse it, do not
rewrite it — and implement only the sort and top-N. No temporary files, no CSV
round trip, and none of upstream's unwrap()/panic! on file I/O in a request
thread (which is why GN2 and GN3 shell out at all).
Semantics, settled:
| decision | choice |
|---|---|
| minimum overlap | 6 (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant |
| methods | pearson and spearman only; bicor does not exist in the engine and stays out |
| ordering | reproduce the engine exactly: |r| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference |
| top-N | a request option (GN2's menu is 100…20000, 500 by default); applied after the sort |
| first axis | sample only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2 |
Verification is against the live site, which turns out to be reachable: it
answers show_trait with HTTP 200 in ~31 s (a -m 20 curl had made it look
unreachable). Each reference page therefore costs ~31 s, which needs budgeting in
any parity loop.
What GN4 still has to supply, and does not have today:
- the bulk per-dataset value fetch — the expensive half; see the measured sizes above, and note the shape the existing queries use: the caller's sample subset for the caller's trait list, never a whole dataset;
- trait metadata for the reply (the id, and the name GN2 shows);
- the sample-group knob (
samples_primary/other/all) and the target dataset, both of which change which rows are fetched.
References
genenetwork2/gn2/wqflask/correlation/rust_correlation.py— the wrapper and the three top-N paths.genenetwork2/gn2/wqflask/correlation/pre_computes.py— the per-dataset text dump and its name-keyed column selection.genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py— the single entry point, delegating to GN3's functions in-process.genenetwork3/gn3/api/correlation.py—/sample_x,/sample_r,/lit_corr,/tissue_corr,/partial.genenetwork3/gn3/computations/rust_correlation.py—run_correlationand the output parser.genenetwork3/gn3/db/correlations.py— the literature and tissue SQL.genenetwork3/scripts/sample_correlations.py— the pickled-file subprocess used by the non-Rust sample path.
