| Age | Commit message (Collapse) | Author |
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
/accessions.
Answers four more questions of aisearch's full_benchmark.csv from
tables the API did not expose.
* src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain
`mapping_method`, the name of the MappingMethod row their
mapping_method_id points to (qtlreaper, Rqtl, ...), or null.
* src/db.rs (Dataset): datasets gain `tissue`, the Tissue their
ProbeFreeze points to (e.g. "Brain mRNA").
* src/routes/mapping_methods.rs: GET /mapping-methods lists the
MappingMethod table, including tools no group uses (PLINK, FastMap).
* src/routes/accessions.rs: GET /accessions/{accession} resolves a
GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the
dataset's name, title, category (publish/probeset) and group.
* docs/genenetwork_datatypes.md: MappingMethodId is a key into the
MappingMethod table (the authoritative reading); GN2's code-based reading
is noted as superseded.
* tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new
full-benchmark rows that pass (full rows 0..114; search rows run against
recorded production gn3, the rest need the database), and full rows 33
(mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only
GTEx v8 has TPM datasets). The untested rows are listed in the file's
header comment with the reason for each.
* tests, test/python: tests for the new fields and endpoints; both
endpoints added to the OpenAPI and next-link crawl tests.
* README.md, RELEASE_NOTES.md: document the new fields and endpoints.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Lists the published traits (across groups) a publication reports.
Datasets the caller may not view are left out, using the same check as
GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)).
* src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`,
`abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and
`url` (the GeneNetwork show_trait page). The three TraitInfo queries
now share one SELECT. New get_group_traits_by_abbreviation and
get_publication_traits.
* src/routes/groups.rs (group_traits): add ?abbreviation= filter.
* src/db.rs (Group), src/routes/groups.rs (groups): groups carry
`family` and `species`; add family, species and q filters.
* src/routes/search.rs (search): add `page` (gn3's page unsorted, a
page of the ranked list with `sort`); declare `pubmed_id`; hits gain
`abbreviation` next to `locus` (get_publish_annotations).
* src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the
benchmark's RDF data uses (was 4.61).
* tests, test/python, test/data/benchmark_search.json, README.md: cover
all of the above; tests/benchmark.rs now also answers rows 11, 16, 25,
30, 34 and 37 of small_benchmark.csv, and compares LOD exactly.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and
Publication; LEFT JOIN on the latter two since a handful of
PublishXRef rows have no matching row. SQL column aliases are written
to match mysql_common's UpperCamelCase conversion exactly, so no field
needs a #[mysql(rename = "...")] override.
(get_group_trait_info): New function.
(pool): New function. mysql::Pool::new eagerly opens its default 10
connections before returning (about 7s against this database); every
query function here used to call it per request, paying that cost on
every single API call. Builds the Pool once behind a
OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle
from then on (OnceLock::get_or_init can't be used alone since
Pool::new is fallible and get_or_try_init is still unstable).
(get_all_species, get_species_by_name, get_all_groups,
get_group_by_name, get_group_strains, get_group_trait_values,
get_group_trait_info): Use it instead of building their own Pool.
* src/lib.rs (GroupTraitLookup): Generalize over the found value (was
Vec<db::TraitValue>-specific), so group_trait_info can share it with
group_trait_values.
(get_group_trait_info_result): New function.
(group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}.
(app): Register it.
* tests/common/mod.rs: New file. get() issues a GET against
genenetwork4::app() and warns on stderr if it takes over a second --
this is what surfaced the Pool::new cost above.
* tests/species.rs, tests/species_by_name.rs, tests/groups.rs,
tests/group_by_name.rs, tests/group_strains.rs,
tests/group_trait_values.rs: Use tests/common::get instead of building
each request by hand.
* tests/group_trait_info.rs, test/python/test_group_trait_info.py: New
files.
* tests/openapi.rs, test/python/test_openapi.py: Check the new path is
in the generated spec.
* README.md (GET /groups/{group}/traits/{trait_id}): Document it.
(Tests): List the new test files; document tests/common::get's timing
warnings and how to see them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Line-wrapping and canonical import ordering only (verified with
git diff -w). Establishes rustfmt's output as the canonical style
before the search/correlation work lands.
|
|
The old skip check connected to 127.0.0.1:3306, so tests ran (and
failed with 500) whenever a server was listening but the configured
credentials were rejected. tests/common/mod.rs now attempts a real
connection with the configured SQL_URI; tests skip only when the
database is genuinely unusable, and run for real otherwise.
|
|
* src/db.rs (Group): New struct. Summary of one InbredSet row,
matching the shape of GeneNetwork2's GET /api/v_pre1/groups response.
(get_all_groups): New function. Mirrors GeneNetwork2's
wqflask.api.router.get_groups_list (no species filter yet).
(GroupDetail): New struct, deriving FromRow instead of being built
from a tuple like SpeciesDetail, since InbredSet has 14 columns, more
than mysql_common's FromRow is implemented for tuples (up to 12).
FromRow matches by column name (PascalCase of the field name via
#[mysql(rename_all = "PascalCase")]), except public, which is
lowercase in the schema and needs its own #[mysql(rename = "public")].
(get_group_by_name): New function. Mirrors GeneNetwork2's
wqflask.api.router.get_group_info, but returns every column instead of
a fixed subset.
* src/lib.rs (app): Add /groups and /groups/{group} routes.
(groups, group_by_name): New handlers, following the same
ApiError/request-id/spawn_blocking pattern as species/species_by_name.
(group_by_name): On no match, set ApiError's next to /groups.
* tests/groups.rs: New file. Checks BXD is present (used throughout
the GeneNetwork2 and GeneNetwork3 test suites as a fixture group,
always present in the database); skips rather than fails when no
database is reachable, same as species.rs.
* test/python/test_groups.py: New file. Likewise.
* tests/group_by_name.rs: New file. Checks BXD's full row and an
unknown-group 404 case, same pattern as species_by_name.rs.
* test/python/test_group_by_name.py: New file. Likewise.
* tests/openapi.rs (openapi_json_describes_all_endpoints): Check for
/groups and /groups/{group} among the documented paths.
* test/python/test_openapi.py
(test_openapi_json_describes_all_endpoints): Likewise.
* README.md: Document /groups and /groups/{group}.
|