about summary refs log tree commit diff
path: root/src/correlation.rs
AgeCommit message (Collapse)Author
3 daysoperator checklist: the tool block env.sh must keep, and why BulkLMM cannot ↵Pjotr Prins
work yet Writing down what the monitor found and what fixed it, because it is config that lives outside the repository and is therefore easy to lose at the next deploy: with only the cache dirs set, every engine that shells out answers 502 while the whole read surface and search look healthy. Measured on production after the fix (bin/monitor-gn4 --tier compute): 8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453), fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed, 1 busy. The remaining failure is BulkLMM and it is not a config line: the Julia project is registry-based and no depot on this box has its packages, so it stops at "Run Pkg.instantiate()". Populating a depot downloads from the Julia registry, which should be a decision rather than a side effect.
3 dayscorrelation: the live site answers Spearman whatever method it is asked forPjotr Prins
Recorded next to `Method` so the next comparison does not chase a 0.005 disagreement in r for an afternoon. GN2's `get_sample_r_and_p_values` takes a parameter called `type` and is called with `corr_params['type']` -- "sample" -- where the method belongs, so `if type == 'pearson'` is never true and `scipy.stats.spearmanr` always runs. The method is used only on the tissue path. Evidence, from the live site with `method=pearson&type=sample`: 1427571_at vs 1447043_at reported -0.8446202819617522 spearman -0.8446202819617523 pearson -0.8393085803732605 1427571_at vs 1427641_at reported -0.8479321530363810 spearman -0.8479321530363810 pearson -0.8451666639589470 BXDPublish 10001 vs 17084 reported 1.0 spearman 1.0000000000000000 pearson 0.9757237677298747 Asked for spearman, our endpoint reproduces the live site's whole reply: the same five traits in the same order, r to 2.2e-16, p to 5e-10 relative. Asked for pearson it reproduces the Pearson r of the values the same site publishes, to 1.4e-8 (their sample_data endpoint rounds to three decimals). The live site stays the reference for the sample axis, the six-pair overlap, the |r|-descending order and the reply shape. The statistic it is asked for is the one place where it is not: it ignores the request.
3 dayscorrelation: run over the group's sample axis, and report a perfect ↵Pjotr Prins
correlation as 0.0 Two divergences from the live genenetwork.org, both measured against it. **The sample axis.** GN2's `get_sample_r_and_p_values` walks `target_dataset.group.samplelist` and takes the reference value by name: the correlation runs over the group's animals, not over every individual the trait happens to have a value for. Trait 1427571_at of HC_M2_0606_P carries 99 values, the BXD panel holds 67 of those individuals, and the answer changes materially -- GN2's top hit is a -0.845 probe that does not appear in our top five at all. `corr_samples_group` now takes `samples_primary` (default, the group's genotype panel, which is where GN2's list comes from) or `samples_all` (the old behaviour); the panel is read with `genotypes::resolve`, falling back to the group's used-for-mapping strains where a group has no genotype file on this machine. `samples_other` is refused rather than guessed. **A perfect correlation.** The engine's r can land at 1.0000000000000002, which makes 1 - r^2 negative and its p-value NaN. The live API answers 0.0 for that row (scipy's pearsonr), so the reply now clamps: r to +-1 and p to 0.0. That is the reference's *output* being reproduced, not a computation being redone -- #96 stays resolved in the same direction. With the axis fixed, n agrees with the live site exactly (67) and r agrees with GN2's own published per-sample values to 1.4e-8. What does not yet agree is GN2's reported r itself: over the same 67 individuals, its `/correlation` endpoint says -0.8446202819617522 for 1447043_at while correlating the values its own `/sample_data` endpoint publishes for the same two traits gives -0.8393085803732605; on a second pair the gap is -0.8479321530363810 against -0.8451666639589470. Both pairs are off in the same direction and by a similar amount, so the two GN2 endpoints are reading different values for the same samples -- not a difference of sample sets (all four BXD genotype files give the same 67) and not a transform (2**v, 10**v, exp, v**2 and log were all tried). Until that is settled, our numbers are the reference's published values correlated correctly, and the ranking cannot be identical to a number we cannot reproduce from the reference's own data.
3 dayscorrelation: port the reference suites' cases, and note what is not portablePjotr Prins
The cases genenetwork2 and genenetwork3 assert, copied so GN4 answers the same questions the same way: - genenetwork3 test_normalize_values: only pairs where both sides have a value survive; - genenetwork3 test_generate_input: the dataset row keeps its positions, a missing value is an empty field rather than a skipped one; - genenetwork3 test_parse_results, with its own fixture file (data/correlation/sorted_results.txt): |r| descending, positives and negatives interleaved; - genenetwork3 test_get_samples: an unparseable value never reaches a correlation, and only shared sample keys are used; - the overlap boundary GN3 encodes as `num_overlap > 5`, which is where MIN_OVERLAP = 6 comes from. What is *not* portable is most of their suites, and that is worth writing down rather than discovering again: GN2's correlation unit tests are all mocks (compute_corr_coeff_p_value patched out, header strings, a three-value tuple), GN3's test_correlation is plumbing around the same mock, and test_rust_correlation tests the file/JSON layer that this implementation deliberately does not have -- it calls the engine in memory. None of them assert a single r or p, because those come from scipy at run time. One port needed widening: GN3's get_samples fixture has four shared samples, which is below its own minimum overlap, so asserting a correlation on it is impossible in both implementations; the fixture is widened and the assertion is that an unparseable value costs exactly its own pair.
3 dayscorrelation: the sample-axis engine path, with the decided semanticsPjotr Prins
src/correlation.rs now holds the part the engine does not offer in memory, and nothing of the arithmetic: `Pearson`/`Spearman` and their p-values are correlation_rust's (and GSL's). What is ours is the alignment into the engine's parser, the minimum-overlap gate, the ordering and top-N: - alignment is by individual, never by position: an individual the target trait lacks becomes "NA", which the engine's parser drops together with its reference value, so a missing observation costs one pair, not the row; - minimum overlap 6, so a two- or four-point +-1 cannot appear; - ordering reuses correlation_rust's own `custom_float_sorter` -- |r| descending, NaNs last -- so GN2, GN3 and GN4 sort identically; - top-N after the sort, clamped to GN2's 20000 ceiling; - bicor is refused by name with the methods that exist, rather than answered with something else. Tests cover both methods, the six-pair minimum, name alignment across a reordered and sparse candidate, the ordering with top-N, and the method parsing. The route is still the stub; wiring it to the bulk fetch is next.