about summary refs log tree commit diff
path: root/tests/groups.rs
AgeCommit message (Collapse)Author
9 daysapi: normalize trait values, and winsorise the tails, before mappingPjotr
GeneNetwork2's trait page offers a set of normalisations, and its mapping page warns about outliers without ever fixing them. This adds both, under one spelling that `/compute/rqtl`, `/compute/gemma` and `/groups/{g}/traits/{id}/values` all share: normalize none | log2 | log10 | sqrt | negate | zscore | robust_zscore | rank (GN2's names `invert` and `qnorm` accepted) winsor_quantile clip each tail at a fraction, e.g. 0.05 winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule They run in a fixed, reported order -- transform, then winsorise, then standardise -- so clipping acts on the scale being analysed and the result lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The note gives the steps *in the order they ran* (`winsorized at Tukey fences, then zscore`), the count clipped in each tail, and how many values sit outside Tukey fences -- the warning GN2 gives before mapping and this API gave nowhere. **Checked against genenetwork.org itself.** GN2's public trait page embeds its own reference values (`js_data.qnorm_values` / `zscore_values`, the `data-qnorm`/`data-zscore` attributes), so there is an oracle for the two transforms whose definition matters. On BXD trait 10010 (28 samples): `rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's three-decimal rounding. Getting there found a real disagreement: GN2 uses `scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use the **population** sd; ours used the sample sd and came out `sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2. Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based inverse-normal (Blom) transform, so it is called `rank` here; and its `sqrt` silently adds 1 when a value is in [0,1], which shifts every effect size -- a negative value is refused instead. * src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom scores over an Acklam inverse-normal, `Params` for query/body use, `to_json` for suggestions, and eleven unit tests -- the quantile table, ties sharing an average rank, missing values never imputed, GN2's own `find_outliers` example, and the note's wording. * src/routes/{rqtl,gemma}.rs: the three fields, applied **after** alignment (over the samples that enter the fit) and before anything reads the values, with the note; the R/qtl permutation suggestion and GEMMA's R/qtl suggestion carry the pipeline. * src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values endpoint, and the mapping bodies in `actions` carry the same fields -- a caller who fetches z-scores must map those numbers, not the raw ones. * tests/groups.rs: values are mean 0 / population sd 1, the note says so, every `/compute/*` suggestion in `actions` carries it, and an impossible pipeline is a 422 `invalid_normalization`. * README.md, docs/www/SKILL.md: what each name means, the order, the population-sd parity, and that effects come out on the transformed scale. 298 tests pass; the same four as before do not.
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-08* src/db.rs, src/routes: add mapping_method, tissue, /mapping-methods, ↵Pjotr
/accessions. Answers four more questions of aisearch's full_benchmark.csv from tables the API did not expose. * src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain `mapping_method`, the name of the MappingMethod row their mapping_method_id points to (qtlreaper, Rqtl, ...), or null. * src/db.rs (Dataset): datasets gain `tissue`, the Tissue their ProbeFreeze points to (e.g. "Brain mRNA"). * src/routes/mapping_methods.rs: GET /mapping-methods lists the MappingMethod table, including tools no group uses (PLINK, FastMap). * src/routes/accessions.rs: GET /accessions/{accession} resolves a GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the dataset's name, title, category (publish/probeset) and group. * docs/genenetwork_datatypes.md: MappingMethodId is a key into the MappingMethod table (the authoritative reading); GN2's code-based reading is noted as superseded. * tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new full-benchmark rows that pass (full rows 0..114; search rows run against recorded production gn3, the rest need the database), and full rows 33 (mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only GTEx v8 has TPM datasets). The untested rows are listed in the file's header comment with the reason for each. * tests, test/python: tests for the new fields and endpoints; both endpoints added to the OpenAPI and next-link crawl tests. * README.md, RELEASE_NOTES.md: document the new fields and endpoints. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}; time DB tests; share pool.Pjotr Prins
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and Publication; LEFT JOIN on the latter two since a handful of PublishXRef rows have no matching row. SQL column aliases are written to match mysql_common's UpperCamelCase conversion exactly, so no field needs a #[mysql(rename = "...")] override. (get_group_trait_info): New function. (pool): New function. mysql::Pool::new eagerly opens its default 10 connections before returning (about 7s against this database); every query function here used to call it per request, paying that cost on every single API call. Builds the Pool once behind a OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle from then on (OnceLock::get_or_init can't be used alone since Pool::new is fallible and get_or_try_init is still unstable). (get_all_species, get_species_by_name, get_all_groups, get_group_by_name, get_group_strains, get_group_trait_values, get_group_trait_info): Use it instead of building their own Pool. * src/lib.rs (GroupTraitLookup): Generalize over the found value (was Vec<db::TraitValue>-specific), so group_trait_info can share it with group_trait_values. (get_group_trait_info_result): New function. (group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}. (app): Register it. * tests/common/mod.rs: New file. get() issues a GET against genenetwork4::app() and warns on stderr if it takes over a second -- this is what surfaced the Pool::new cost above. * tests/species.rs, tests/species_by_name.rs, tests/groups.rs, tests/group_by_name.rs, tests/group_strains.rs, tests/group_trait_values.rs: Use tests/common::get instead of building each request by hand. * tests/group_trait_info.rs, test/python/test_group_trait_info.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}): Document it. (Tests): List the new test files; document tests/common::get's timing warnings and how to see them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Format all Rust sources with cargo fmt (no behavior change)Alexander_Kabui
Line-wrapping and canonical import ordering only (verified with git diff -w). Establishes rustfmt's output as the canonical style before the search/correlation work lands.
2026-10-04Fix DB tests: probe SQL_URI credentials, not just the portAlexander_Kabui
The old skip check connected to 127.0.0.1:3306, so tests ran (and failed with 500) whenever a server was listening but the configured credentials were rejected. tests/common/mod.rs now attempts a real connection with the configured SQL_URI; tests skip only when the database is genuinely unusable, and run for real otherwise.
2026-10-03Add /groups and /groups/{group} endpoints.Pjotr Prins
* src/db.rs (Group): New struct. Summary of one InbredSet row, matching the shape of GeneNetwork2's GET /api/v_pre1/groups response. (get_all_groups): New function. Mirrors GeneNetwork2's wqflask.api.router.get_groups_list (no species filter yet). (GroupDetail): New struct, deriving FromRow instead of being built from a tuple like SpeciesDetail, since InbredSet has 14 columns, more than mysql_common's FromRow is implemented for tuples (up to 12). FromRow matches by column name (PascalCase of the field name via #[mysql(rename_all = "PascalCase")]), except public, which is lowercase in the schema and needs its own #[mysql(rename = "public")]. (get_group_by_name): New function. Mirrors GeneNetwork2's wqflask.api.router.get_group_info, but returns every column instead of a fixed subset. * src/lib.rs (app): Add /groups and /groups/{group} routes. (groups, group_by_name): New handlers, following the same ApiError/request-id/spawn_blocking pattern as species/species_by_name. (group_by_name): On no match, set ApiError's next to /groups. * tests/groups.rs: New file. Checks BXD is present (used throughout the GeneNetwork2 and GeneNetwork3 test suites as a fixture group, always present in the database); skips rather than fails when no database is reachable, same as species.rs. * test/python/test_groups.py: New file. Likewise. * tests/group_by_name.rs: New file. Checks BXD's full row and an unknown-group 404 case, same pattern as species_by_name.rs. * test/python/test_group_by_name.py: New file. Likewise. * tests/openapi.rs (openapi_json_describes_all_endpoints): Check for /groups and /groups/{group} among the documented paths. * test/python/test_openapi.py (test_openapi_json_describes_all_endpoints): Likewise. * README.md: Document /groups and /groups/{group}.