about summary refs log tree commit diff

What GeneNetwork correlates, and how

GeneNetwork2 and GeneNetwork3 both offer "correlation", and the word covers four different things measured over three different axes. This note records what is actually correlated, which engine computes it, and where. It exists because the answer to "duplicate the correlation_rust path" depends entirely on knowing which of the four you are duplicating.

Written from the code, with row counts measured read-only against the production db_webqtl (2026-11-14).

The four correlations

kind x y observations computed?
sample one trait, per individual (BXD strain, animal, plant, human case) each trait of a target dataset, same individuals individuals (~34–40 for BXD) every request
tissue one gene's expression across tissue systems another gene's expression across the same tissues tissues (26–32 per gene) every request
literature a gene another gene published studies read from a table
partial (GN3 only) one trait one trait, conditioned on other traits/genotypes individuals every request, async

Nothing here correlates markers with traits — that is QTL mapping. In the sample and partial cases the observations are individuals; in the tissue case they are tissues; literature has no n at all.

Sample correlation: the one that matters most

POST /compute/correlation for trait X against "BXD Liver" is not one correlation: it is X against every trait in that dataset, one r each, all over the same individual axis — about 20k r's over ~34–40 strains.

Which individuals is a request knob (corr_samples_group in GN2):

  • primary only — the primary trait's own group members (BXD strains only);
  • other only — everything in the data except those, which is how you correlate across other strains or species;
  • all — every sample the trait data carries.

Two rules decide correctness, both easy to get wrong:

  • Missing data is pairwise. A sample counts only when both sides have a numeric value; "x" and empty are dropped along with their counterpart on the other side. So n (num_overlap) varies per trait, and the engine refuses rows with fewer than 4 pairs (correlations.rs, the ty.x_vals.len() < 4 || ty.y_vals.len() < 4 guard). GN3's pure-Python compute_correlation (gn3/computations/correlations2.py) uses 6 instead — the two paths disagree on the minimum.
  • Pairing is positional. x_vals[i] is compared with field i of each dataset line. The alignment is established earlier and by name: the cached path re-selects columns by matching the header (read_text_file → __fetch_id_positions__, gn2/wqflask/correlation/pre_computes.py), the live path keys a dict by sample name. Get that order wrong and the answer is plausible and silently wrong.

Tissue correlation

Genes are joined on lowercased symbol, one probeset per symbol chosen in SQL by max(Mean) within TissueProbeSetFreezeId = 1 (gn2/base/mrna_assay_tissue_data.py). The vectors are tuples built in row order from SELECT Id, value FROM TissueProbeSetData, so two genes with different numbers of rows produce unequal vectors: Pearson takes n from x and silently truncates, Spearman panics.

Literature correlation

The value for a gene pair comes out of LCorrRamin3 in both directions (GeneId1, GeneId2, value and GeneId2, GeneId1, value), with mouse → rat and mouse → human handled by GeneIDXRef — gn3/db/correlations.py:40-92. A per-request join, not a stored per-request result.

Measured sizes (read-only, db_webqtl, 2026-11-14)

table rows note
ProbeSetData 5 314 694 464 trait × individual × value — why nothing loads a dataset whole
PublishData 3 288 638 phenotype values
TissueProbeSetData 90 562 DataIds, 26–32 values each (avg 29.0) the tissue axis
LCorrRamin3 127 368 780 the only precomputed correlation values anywhere

Consequence: no path fetches a dataset's traits whole. It fetches your sample subset for your trait list (StrainID IN (…) AND Id IN (SELECT … WHERE ProbeSetFreeze.Name = ? AND ProbeSet.Name IN (…))), which is also why the top-N bulk path exists: the arithmetic on 34 values × 20k traits is noise, the value fetch is the whole cost.

The correlation_rust engine

gn-rust-correlation-0.1.4-1.473ad82 (Guix: guix-bioinformatics/gn/packages/genenetwork-rust.scm; upstream Alexanderlacuna/correlation_rust). It is a library with a 17-line binary bolted on: main.rs takes one argument — a JSON path — calls Analysis::compute, prints the result and an elapsed time. lib.rs exposes analysis, correlations, parser, reader, sorter.

Contract: JSON in, CSV out

Input JSON (gn3/computations/rust_correlation.py, generate_json_file):

{"method": "pearson", "file_path": "<dataset.csv>", "x_vals": [...],
 "sample_values": "bxd1", "output_file": "<out.csv>", "file_delimiter": ","}

sample_values is dead: it is in the JsonData struct and nothing reads it (hardcoded "bxd1" in both callers).

The dataset file is one trait per line, name,v1,v2,…,vn, where position i corresponds to x_vals[i]. Per row (correlations.rs:135), parse_rows_with_names splits the name, pairs x_vals[i] with field i, and drops a field that does not parse as f64 together with its x value — which is how missing data survives the alignment.

Statistics: rgsl's correlation/spearman; p two-sided from the t-distribution with df = n - 2, mirroring R's cor.test. Bicor does not exist in the engine — only "pearson" and "spearman" are matched; bicor exists solely on GN3's Python path.

Output (sorter.rs): name,r,p,n per line, sorted by |r| descending with NaNs last, and that is the whole file — no trait id, no dataset, no filtering. Top-N is applied afterwards in Python (parse_correlation_output, idx < top_n), after the engine has sorted everything.

The Python around it

compute_correlation_rust (gn2/wqflask/correlation/rust_correlation.py:390) is the entry point; correlation_gn3_api.compute_correlation:185 its only caller. Five stages:

  1. Resolve — create_target_this_trait(start_vars) builds this_dataset, this_trait, target_dataset.
  2. Get the target data, one of three ways: a per-dataset text dump (pre_computes.py, columns re-selected by name); live (target_dataset.get_trait_data(sample_keys)); or the top-N bulk SQL path (compute_top_n_sample, ProbeSetData joined to xref/freeze, chunked by the number of samples, one line per trait).
  3. Write the two files into a temporary directory and subprocess.run([CORRELATION_COMMAND, json, tmpdir], check=True). CORRELATION_COMMAND is <GN2_PROFILE>/bin/correlation_rust (gn2/default_settings.py); GN3 reads it from Flask config with no default.
  4. Parse the CSV into {trait: {corr_coefficient, p_value, num_overlap}} (or tissue_corr/tissue_number/tissue_p_val).
  5. Optionally enrich (should_compute_all): take the top hits and run the other two kinds over just those traits, then merge per trait — so one row can carry sample, tissue and literature r at once.

Literature never touches the engine: it is compute_all_lit_correlation over LCorrRamin3.

Duplicating it into GN4

src/routes/correlation.rs is a stub returning canned {trait_id, correlation}. What to build, decided on 2026-11-14:

Synchronous, no redis, no job. One POST computes and answers. GN2's /corr_compute writes a form to a file and runs a subprocess job, and GN3's partial path goes through redis; neither is wanted for the first version. That also means the reply carries the result, not a ?id=//status pair.

Use the correlation_rust engine, in-process. gn-rust-correlation is already packaged (gn/packages/genenetwork-rust.scm, rev 473ad82), so this is a dependency to wire in rather than a package to write. Its parser::parse_rows_with_names, Pearson/Spearman and the Correlation trait are pub: build the vectors in memory, reuse the maths — the p-value is the subtle part; reuse it, do not rewrite it — and implement only the sort and top-N. No temporary files, no CSV round trip, and none of upstream's unwrap()/panic! on file I/O in a request thread (which is why GN2 and GN3 shell out at all).

Semantics, settled:

decision choice
minimum overlap 6 (the gn3/python value, not the engine's 4) — a row needs six paired observations, which also removes the 2–4 point correlations that cannot be significant
methods pearson and spearman only; bicor does not exist in the engine and stays out
ordering reproduce the engine exactly: |r| descending, NaNs last — so GN2, GN3 and GN4 sort identically and a diff means a real difference
top-N a request option (GN2's menu is 100…20000, 500 by default); applied after the sort
first axis sample only — one trait against every trait of a target dataset, observations are individuals. Tissue, literature and partial stay out until this row matches GN2

Verification is against the live site, which turns out to be reachable: it answers show_trait with HTTP 200 in ~31 s (a -m 20 curl had made it look unreachable). Each reference page therefore costs ~31 s, which needs budgeting in any parity loop.

What GN4 still has to supply, and does not have today:

  • the bulk per-dataset value fetch — the expensive half; see the measured sizes above, and note the shape the existing queries use: the caller's sample subset for the caller's trait list, never a whole dataset;
  • trait metadata for the reply (the id, and the name GN2 shows);
  • the sample-group knob (samples_primary / other / all) and the target dataset, both of which change which rows are fetched.

References

  • genenetwork2/gn2/wqflask/correlation/rust_correlation.py — the wrapper and the three top-N paths.
  • genenetwork2/gn2/wqflask/correlation/pre_computes.py — the per-dataset text dump and its name-keyed column selection.
  • genenetwork2/gn2/wqflask/correlation/correlation_gn3_api.py — the single entry point, delegating to GN3's functions in-process.
  • genenetwork3/gn3/api/correlation.py — /sample_x, /sample_r, /lit_corr, /tissue_corr, /partial.
  • genenetwork3/gn3/computations/rust_correlation.py — run_correlation and the output parser.
  • genenetwork3/gn3/db/correlations.py — the literature and tissue SQL.
  • genenetwork3/scripts/sample_correlations.py — the pickled-file subprocess used by the non-Rust sample path.