| Age | Commit message (Collapse) | Author |
|
monitor run
`POST /compute/correlation/matrix` takes a list of `{dataset, trait_name}`
references -- a collection, as GN2's matrix form posts it, which spans
datasets -- and answers an N x N matrix of r, p and n.
The five decisions that are not obvious, each recorded where a reader will
look for them:
* **One method for the whole matrix.** GN2's `show_corr_matrix.py` flips to
Spearman mid-row once a Pearson r passes 0.999, which the self-pair always
does, so its matrix is asymmetric by construction. Here the method is
chosen once and r[i][j] == r[j][i] exactly.
* **Each unordered pair is computed once and mirrored** -- N(N-1)/2 engine
calls, not N^2.
* **The diagonal is by construction**: r = 1, p = null, n = the trait's own
count, so half the traits cannot come back as 1.0000000000000002 with a
NaN p.
* **Per-cell n is its own matrix**, and a cell under `min_overlap` (default
6) is null in r and p while its n still says how many points there were.
GN2 accepts 2, where |r| = 1 by definition.
* **The sample axis is the first trait's group** (#106), resolved once and
stated in the reply with the number of individuals it kept.
Fetches are one query per dataset (`get_dataset_values_for_traits`,
`get_publish_values_for_traits`), not one per trait: a matrix chooses a
handful out of twenty thousand. Capped at 1 000 traits, because the cost is
quadratic in the correlations and in the reply. No PCA -- the r matrix has
to be trusted on its own first.
Verified twice: cells agree with scipy to the last bit (BXDPublish
10001/10002/10005: 0.6756270426459695, 0.6950450737755965,
0.8819879006700077 against scipy 0.6756270426459695, 0.6950450737755964,
0.8819879006700073), and a cell equals what the row endpoint says for the
same pair (0.6950450737755965 in both). Edge cases exercised: a 1x1 matrix,
`min_overlap` above every cell's n, an unknown trait, `bicor`, and no
traits at all.
The operator checklist now says a deploy ends with `bin/monitor-gn4 --tier
all`, and says why: `/version` can report a commit and a reachable database
while every engine that shells out is dead. The monitor gained a matrix
check whose judge enforces square, symmetric and a unit diagonal.
|
|
The implementation had already landed in 698e6ab by accident -- a `git add
-A` swept it in with the operator-checklist docs, which is why its message
says nothing about partial correlation. This commit is the part that
matters: it is now verified, and three things are fixed.
**Verification.** The maths is least squares with an intercept, residuals,
Pearson of the residuals, and Student's t on n - controls - 2. Checked two
ways against independent code:
* BXDPublish 10001 with 10002 held constant, target 17164, n=6:
ours r=0.9960864631522379 p=0.00029371957431469716, numpy lstsq +
scipy.stats.t.sf r=0.9960864631522379 p=0.00029372.
Zero-order: ours -0.10577609106505677, scipy pearsonr -0.1057760910650567.
* The same for 17084: identical to 16 digits, r=0.9840718340719924.
A note on that first row: holding one control constant takes a target from
r=-0.11 to r=+0.996 on six points. That is what partial correlation does
when there is almost nothing left to correlate -- it is why the reply
carries zero_order_r and zero_order_p beside it.
**Fixes.** `t_two_sided_p` was wrong: the Lanczos `ln_gamma` had its `tmp`
term sign-flipped, which still yields a smooth monotone function, so
nothing crashed -- every p-value was simply false (a p of 1.8e-24 came out
as 1.0). The corrected version matches `scipy.stats.t.sf` to 1.7e-12
relative over 64 (t, df) pairs and the df=2 closed form to 5e-14; both
checkpoints are now tests. A perfect partial r (self-correlation) is
clamped to +-1 with p=0.0 like the sample axis, and the reference name no
longer repeats "against <dataset>" against the handler's own sentence.
The monitor now exercises the axis: `compute-correlation-partial`.
|
|
work yet
Writing down what the monitor found and what fixed it, because it is config
that lives outside the repository and is therefore easy to lose at the next
deploy: with only the cache dirs set, every engine that shells out answers
502 while the whole read surface and search look healthy.
Measured on production after the fix (bin/monitor-gn4 --tier compute):
8 ok, 1 failed -- correlation, gemma (21 055 rows), rqtl, qtl2 (20 453),
fitqtl, ravanan (21 055 rows), bnw, bnw-examples. Before: 1 ok, 7 failed,
1 busy.
The remaining failure is BulkLMM and it is not a config line: the Julia
project is registry-based and no depot on this box has its packages, so
it stops at "Run Pkg.instantiate()". Populating a depot downloads from the
Julia registry, which should be a decision rather than a side effect.
|
|
`type` on /compute/correlation now selects GN2's `corr_type`: sample
(default), tissue or literature. Partial is refused by name rather than
silently treated as sample.
**tissue** reduces each gene symbol to its highest-Mean probe in
TissueProbeSetXRef and pairs the two value vectors **by TissueID**. That
last word is the difference from GN2, which keeps each vector as a bare
list and pairs by position on the assumption that every probe covers the
same tissues. On this database it does not: every probe carries exactly
26 values (liver, kidney, spleen, ... peripheral) or exactly 32
(striatum, hippocampus, neocortex, ... brain regions), the two sets are
disjoint, and 45 281 probes are in each. Pairing those positionally
correlates liver expression with striatum expression because both happen
to be entry one. Pairing by tissue means a cross-panel pair has no
correlation at all, which is the honest answer and makes the panel
boundary visible instead of inventing a number.
Verified independently for Shh vs Hcrtr1: 0.7222718302270719 from the
endpoint, 0.7222719072696666 from a separate Pearson over the same
TissueID-aligned pairs (the residue is MySQL's single-precision `float`
column as printed by the client).
**literature** looks the pair up in LCorrRamin3 in both directions after
translating rat/human ids through GeneIDXRef, in chunks of 4 000 genes
(the first attempt died on MySQL's 65 535-placeholder limit). It is a
precomputed value, not a statistic: p and n are null. A trait with no
gene id or no row is left out and counted in the note, where GN2 reports
0.0 and fills the tail of its top-N with them -- that reads as "measured,
no relationship", which is not what happened.
Gene ids come from ProbeSet.GeneId, not ProbeSetXRef.GeneId: the latter
has no such column here, which GN3's own mapping also records.
|
|
correlation as 0.0
Two divergences from the live genenetwork.org, both measured against it.
**The sample axis.** GN2's `get_sample_r_and_p_values` walks
`target_dataset.group.samplelist` and takes the reference value by name:
the correlation runs over the group's animals, not over every individual
the trait happens to have a value for. Trait 1427571_at of HC_M2_0606_P
carries 99 values, the BXD panel holds 67 of those individuals, and the
answer changes materially -- GN2's top hit is a -0.845 probe that does
not appear in our top five at all. `corr_samples_group` now takes
`samples_primary` (default, the group's genotype panel, which is where
GN2's list comes from) or `samples_all` (the old behaviour); the panel is
read with `genotypes::resolve`, falling back to the group's
used-for-mapping strains where a group has no genotype file on this
machine. `samples_other` is refused rather than guessed.
**A perfect correlation.** The engine's r can land at 1.0000000000000002,
which makes 1 - r^2 negative and its p-value NaN. The live API answers
0.0 for that row (scipy's pearsonr), so the reply now clamps: r to +-1 and
p to 0.0. That is the reference's *output* being reproduced, not a
computation being redone -- #96 stays resolved in the same direction.
With the axis fixed, n agrees with the live site exactly (67) and r
agrees with GN2's own published per-sample values to 1.4e-8. What does
not yet agree is GN2's reported r itself: over the same 67 individuals,
its `/correlation` endpoint says -0.8446202819617522 for 1447043_at while
correlating the values its own `/sample_data` endpoint publishes for the
same two traits gives -0.8393085803732605; on a second pair the gap is
-0.8479321530363810 against -0.8451666639589470. Both pairs are off in
the same direction and by a similar amount, so the two GN2 endpoints are
reading different values for the same samples -- not a difference of
sample sets (all four BXD genotype files give the same 67) and not a
transform (2**v, 10**v, exp, v**2 and log were all tried). Until that is
settled, our numbers are the reference's published values correlated
correctly, and the ranking cannot be identical to a number we cannot
reproduce from the reference's own data.
|
|
Either side of a correlation may now be a Publish (phenotype) dataset, not
just a ProbeSet one: a pheno trait correlated against an expression
dataset is what GN2's mapping page is usually used for, and returning 404
for BXDPublish was the wrong answer.
A publish reference trait is addressed by its numeric PublishXRef.Id --
that is how GN2 addresses phenotype traits, and a name there is refused
with a message saying so rather than a confusing "no such trait". The
bulk fetch gains the publish equivalent, keyed on the freeze's
InbredSetId, so the same individuals line up on both sides.
Measured live: BXDPublish/10001 (34 observations) against HC_M2_0606_P,
top 5, HTTP 200 in 5.0 s; 1440079_at at r -0.757 (p 3.2e-6) leads.
|
|
POST /compute/correlation now computes instead of returning canned data:
one reference trait against every trait of a target dataset, observations
being individuals, pearson or spearman, top-N by |r|.
The reply is positional -- trait, r, p, n (column names in meta.columns) --
with the method, the minimum overlap and the top-N stated in the notes.
`bicor` is refused with the methods that exist; a bad top_n is refused
rather than clamped; a dataset or trait with nothing to correlate says so
instead of returning an empty list.
src/db.rs gains the bulk fetch the sample axis needs: every value of every
trait of a dataset restricted to the reference trait's individuals, one
record per (trait, individual, value), ordered by trait so the caller
groups as it streams. It never loads a dataset whole -- ProbeSetData is
5.3e9 rows, so the query narrows by dataset and by individuals, which is
also why the sweep, not the arithmetic, is the whole cost.
Measured on the live db_webqtl: HC_M2_0606_P / 1427571_at against its own
dataset, top 5, HTTP 200 in 16 s; the trait correlates with itself at
1.0 over 99 observations, then 0.802 (p 1.9e-23), 0.790, -0.789, -0.788.
Synchronous is affordable at this size; GN2's job indirection exists for
top-N 20000 across datasets, which is the next thing to time.
Known and deliberate for now: the publish/genotype axes are not wired (the
fetch is probeset-only), and a perfect correlation reports p as null --
GSL's t statistic is undefined at |r| = 1, whose r comes out as
1.0000000000000002, so 1 - r^2 is slightly negative and the engine returns
NaN. Reproducing the engine exactly, rather than "fixing" the p-value
here, is the decision recorded in docs/correlations.md.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
The endpoint is a documented stub returning canned data; the
request fields are part of the validation and openapi contract and
stay unread until the correlation engine integration reads them.
|
|
/compute/correlation.
API version prefixes are deployment concerns -- the v2 lives in the main URL (site/v2/compute/correlation) -- so code paths carry functional names only.
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Line-wrapping and canonical import ordering only (verified with
git diff -w). Establishes rustfmt's output as the canonical style
before the search/correlation work lands.
|
|
Summaries appear in the generated /openapi.json and make the spec
readable on its own. The correlation endpoint is a stub: it validates
the request shape and returns an envelope with canned data and a null
next pointer, establishing the contract ahead of the real engine.
|