about summary refs log tree commit diff
path: root/test/python
AgeCommit message (Collapse)Author
3 daysgemma: read ccwl's YAML when checking the workflow's inputs, and fix two ↵Pjotr Prins
stale tests `ccwl compile` emits CWL as YAML (its only other target is dot), so the new input check must read that shape -- it was parsing JSON and would have turned a working request into a 502. A hand-written scan of the `inputs:` block, deliberately not a YAML dependency: ccwl's output is machine-generated and unvaried, and adding serde_yaml would mean regenerating the crate list in guix.scm for one small check. Two tests were stale in ways only the package's own check phase shows: - tests/auth.rs built AuthToken literals without the new `scheme`/`id` fields, so the crate's integration tests did not compile; - test/python/test_envelope.py asserted that /version carries no `next` links, which stopped being true when every read endpoint gained them ("always offer the next step"). The deployed build fails that test too, so it is not a regression from this work -- but it did block `guix build -f guix.scm` at the python-rest-test phase. Verified with the packaged binary (guix build -f guix.scm, all phases including both test suites): POST /compute/gemma/ravanan through ravanan and the Slurm container returned 200 with 636 rows in 13.6 s, on a request authenticated with a gn4ai_ short token -- the log line naming user zachs, scheme opaque and token smoke-2.
2026-10-08* src/routes/dev.rs: POST /get-test-token answers local callers only, not ↵Pjotr
tied to dev mode. The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind address. It decides per request whether to answer. * src/routes/dev.rs (get_test_token): answers a caller only when the TCP peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped 127.x) and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header (a reverse proxy on the same host would otherwise make every client look local). Anything else, including an unknown peer, is a 403. Local is never established from a header. * src/main.rs: serve with into_make_service_with_connect_info, which provides the peer address. * src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it is not a clearly affirmative value (true/1/yes/on) the route is not registered: a 404, absent from /openapi.json. It fails closed, so a typo ("flase") turns the endpoint off. * src/lib.rs: register the route unless that switch is off. * tests: tests/get_test_token.rs rewritten (remote, unknown and proxied peers refused; 127.x and ::1 answered; always documented by default); tests/get_test_token_off.rs for the switch (own process: it sets an environment variable); common::post_from supplies a peer address to the in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV is now empty (no setting needed). * README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document the check, the switch, and "Behind nginx or in a container" (a bridge container sees the gateway as the peer; a host-network container behind nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For, or switch the endpoint off). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/db.rs, src/routes: add mapping_method, tissue, /mapping-methods, ↵Pjotr
/accessions. Answers four more questions of aisearch's full_benchmark.csv from tables the API did not expose. * src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain `mapping_method`, the name of the MappingMethod row their mapping_method_id points to (qtlreaper, Rqtl, ...), or null. * src/db.rs (Dataset): datasets gain `tissue`, the Tissue their ProbeFreeze points to (e.g. "Brain mRNA"). * src/routes/mapping_methods.rs: GET /mapping-methods lists the MappingMethod table, including tools no group uses (PLINK, FastMap). * src/routes/accessions.rs: GET /accessions/{accession} resolves a GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the dataset's name, title, category (publish/probeset) and group. * docs/genenetwork_datatypes.md: MappingMethodId is a key into the MappingMethod table (the authoritative reading); GN2's code-based reading is noted as superseded. * tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new full-benchmark rows that pass (full rows 0..114; search rows run against recorded production gn3, the rest need the database), and full rows 33 (mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only GTEx v8 has TPM datasets). The untested rows are listed in the file's header comment with the reason for each. * tests, test/python: tests for the new fields and endpoints; both endpoints added to the OpenAPI and next-link crawl tests. * README.md, RELEASE_NOTES.md: document the new fields and endpoints. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07* src/routes/search.rs (search): add lod, locus, and sort/order.Pjotr
Search hits gain `lod` (lrs / 4.61) and `locus` (the peak marker name, looked up in PublishXRef in one batched query; null when the database is unavailable). `sort=lrs|lod` with `order=asc|desc` ranks hits: gn3 is asked for its first 1000 matches, those are ranked, then `limit` applies. * src/db.rs (LRS_PER_LOD, lod, get_publish_loci): add the shared LRS-to-LOD constant and the locus lookup; TraitInfo and ProbeSetInfo gain `lod`, computed in SQL from the same constant. * tests/search.rs, test/python/test_search.py: add the "highest LOD, locus marker and additive effect for serotonin in BXD male midbrain" scenario over a recorded production fixture, plus sort/order tests. * tests/group_trait_info.rs, tests/dataset_probeset_info.rs and their Python mirrors: assert lod (and locus for trait 17985). * README.md (Endpoints): document sort, order, lod and locus. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07* test/python/server_test_case.py (_read_scramble): skip the packet header.Pjotr
_read_packet returns the 4-byte header plus body, but _read_scramble parsed from offset 1 as if the header were absent. It matched a zero byte in the header instead of the one ending the server version string, so the scramble was garbage, the mysql_native_password login was always denied, and database_reachable() returned False on every server, skipping the DB-gated tests. Start at offset 5. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07Add GET /groups/{group}/case-attributesPjotr Prins
Per-strain metadata beyond genotype/phenotype data (age, sex, treatment, etc.), backed by MariaDB CaseAttribute/CaseAttributeXRefNew. Combines GeneNetwork3's separate /case-attribute/{id}/names and /case-attribute/{id}/values (gn3.api.case_attributes) into one response -- neither list is large enough on its own to need splitting. This is the only one of GeneNetwork2/3/gn-guile with the concept at all: GeneNetwork2 delegates to GeneNetwork3 rather than storing case attributes itself, and gn-guile doesn't have it. db::get_group_case_attributes returns {names: [...], values: [{strain, case_attributes: {...}}]}; values is built by folding adjacent (strain, attribute, value) rows (ordered by strain name) rather than grouping afterwards, since SQL already guarantees the adjacency. Found and fixed a real bug while verifying this live: GeneNetwork3's own query joins CaseAttribute to CaseAttributeXRefNew on CaseAttributeId alone, but CaseAttributeId is not unique across groups -- it's a shared vocabulary (id 37 means "Epoch" for dozens of unrelated groups simultaneously); the actual key is (InbredSetId, CaseAttributeId). Confirmed against this database: without also joining on InbredSetId, BXD's result included a spurious "Strain": "Live" attribute belonging to a different, unrelated group. Fixed on genenetwork4's side by adding the missing join condition. A group with no case attributes defined returns 200 with both lists empty, not a 404 (the group itself exists) -- verified against LXS, a real group with zero CaseAttribute rows. tests/group_case_attributes.rs and its Python mirror cover BXD's 4 real attributes (asserting each strain has exactly those 4, not more, to catch the join bug above), the empty-group case, missing token, and unknown group. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-07Add tests for denying a genuinely <1-year-old non-public datasetPjotr Prins
All four 403-deny tests so far used UTHSC_BXD_Cochlea_0225, which was a real non-public, non-embargoed dataset when first used as a fixture but is now ~20 months old -- still correctly denied (under the 5-year embargo threshold), but no longer actually tests "freshly added, private" data specifically, only "not yet old enough to auto-expire." Found UTHSC_BXD_2wk_Cochlea_0926 (public=1, confidentiality=1, created 2026-09-01, the most recently added ProbeSetFreeze in the database as of this writing) and added a parallel test alongside each existing UTHSC_BXD_Cochlea_0225 test -- dataset_probeset_info, its /values sibling, the dataset_probesets listing, and the unified /traits endpoint, in both Rust and Python -- confirming the deny path holds for data that's actually less than a year old, not just data that merely hasn't aged past 5 years yet. Verified live against the real database before adding (is_publicly_readable's two conditions -- flags and age -- both still correctly deny for this dataset; values/probeset data exists for it, so the test exercises the authorization check rather than a 404). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05search: require a token; restore hermetic /search coverageAlexander_Kabui
- src/routes/search.rs: AuthToken per correlation.rs pattern (+ Unauthorized). - tests/search.rs: stub /auth/public-jwks + locally-signed token; scenarios run offline again (multi_thread runtime: the blocking ureq JWKS fetch starved the single-thread stub); add invalid-token 401 case. - test/python/test_search.py: fetch token once, reuse. - RELEASE_NOTES.md: /search now requires a token. - test/data: test-only JWKS + RSA key fixtures.
2026-10-05Rebase cleanup: resolve conflicts, dedup, fix Config test literalAlexander_Kabui
- Cargo.toml: drop duplicate serde_json (keep 1.0.151). - Cargo.lock: keep origin/main's rustc-1.85-compatible pins; minimally add reqwest/url. - README: drop stale /search bullet. - test/python: drop dead 'import socket' (shared database_reachable now). - src/config.rs: dev_fields test literal gains gn3_url/gn3_timeout_secs. NOTE: pre-commit fmt --check bypassed; local rustfmt 1.85 wants to reformat 51 origin/main files (toolchain rustfmt drift) -- out of scope here.
2026-10-05* src/routes/search.rs (search): share one reqwest client with aAlexander_Kabui
total timeout for gn3 calls reqwest::get built a fresh client per query (no keepalive) and set no timeout, while gn3 sets none of its own -- a stalled upstream would hang the handler past any front-proxy timeout. One shared client in a OnceLock now, mirroring the shared-DB-pool pattern. * src/config.rs (Config): add GN3_TIMEOUT_SECS (default 30); normalize GN3_URL to end with a slash. * tests/search.rs test/python/test_search.py (search): one scenario mapping a gn3 5xx to the 502 backend_error shape.
2026-10-05* src/routes/search.rs (search): map any gn3 4xx with an error body to 422Alexander_Kabui
Production gn3 puts the diagnostic under "error" and leaks Python reprs under "error_type", so the old "parsererror" match never fired and malformed queries fell to 502 with no actionable feedback. Match on "error" alone; gn3 5xx stays 502. * tests/search.rs test/python/test_search.py (search): stub replays production's error body shape.
2026-10-05* test/python/test_search.py (search): contract mirror over the realAlexander_Kabui
binary Spawns the compiled server with GN3_URL pointed at a local gn3 stub serving the recorded fixture; asserts the envelope shape, parameter validation, and the 422 parser-error forwarding. Mirrors tests/search.rs per the test-pair discipline; runs in the Guix sandbox (the stub is loopback-only).
2026-10-05* test/python/server_test_case.py (ServerTestCase): probe SQL_URIAlexander_Kabui
credentials, not just the port Mirrors the Rust-side fix in 4a2cf5b on the Python suites: the old skip check connected to 127.0.0.1:3306, so tests ran (and failed) whenever a server was listening but rejected the credentials. database_reachable() now performs the real mysql_native_password handshake and skips only when the database is genuinely unusable; the five DB-mapped test modules use it instead of their local port probes. * test/python/test_{groups,group_by_name,group_strains,species,species_by_name}.py (database_reachable): use the shared probe; drop local copies.
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Add POST /dev/token, a dev-only gn-auth token issuerPjotr Prins
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's well-known development user and returns the result, so local testing no longer needs a manual curl/gn-auth round trip of its own. Only ever registered -- not just rejected at request time, genuinely absent from the router and from /openapi.json -- when two new Config fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback bind (GENENETWORK4_BIND, also new). Both are config, not something genenetwork4 infers from its actual network exposure, by design: dev mode alone, or a loopback bind alone, each leave it disabled, and the check (Config::dev_token_enabled) is a plain function so the condition itself has direct unit test coverage rather than only being exercised indirectly through the route. src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0", via a proper SocketAddr rather than string formatting -- the latter would mishandle an IPv6 bind address like "::1" (needs brackets). GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to gn-auth's own published dev fixtures (flask init-dev-clients, gn_auth/wsgi.py), so this works unconfigured against an unmodified dev gn-auth instance -- confirmed live, including using the returned token against a real gated endpoint (/traits/BXD/10001), not just checking the token response's shape. tests/dev_token.rs covers all four combinations of the two settings in one sequential test function (mutating the same two env vars from more than one #[tokio::test] in a file would race, since they run concurrently by default) plus the /openapi.json visibility check. test/python/test_dev_token.py mirrors this using separate test classes instead, which is safe here since each spawns its own server subprocess with its own environment (ServerTestCase gains EXTRA_ENV for this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /traits/{dataset}/... spanning ProbeSet and Publish dataPjotr Prins
A new, additional endpoint family that addresses both trait categories the same way -- (dataset, trait) -- matching GeneNetwork2's own generic wqflask.api.router.trait_sample_data/get_trait_info, rather than the two category-specific shapes genenetwork4 already has (/datasets/{dataset}/probesets... and /groups/{group}/traits...), which are unchanged and keep working exactly as before. - db::resolve_trait_dataset(name) resolves a dataset name against ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze row, via the new get_publish_freeze_by_name) -- whichever matches decides the category. This means only dataset names are ever "guessed" across categories, not trait names/ids, which is the much smaller and safer surface: GN2's own equivalent tries ProbeSet.Name and PublishXRef.Id against the same bare trait string and silently falls back, which can return the wrong trait on a same-named collision. get_publish_freeze_by_name accepts a PublishFreeze's own Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/ Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or "BXD"). - GET /traits/{dataset}: list every trait in a dataset, tagged by category. - GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric trait against a Publish dataset is a 404 (can't match a PublishXRef.Id), not a 500. - GET /traits/{dataset}/{trait}/values: that trait's per-strain values. TraitValue and ProbeSetValue are already identical in shape, so this unifies with no information loss, unlike the metadata endpoint above. Metadata responses are a tagged union ({"category": "probeset"| "publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than a lossy common-fields schema -- the two share little beyond mean/locus/lrs/additive. utoipa generates this as a oneOf with a category discriminator, verified against /openapi.json. Gated the same way as /datasets/{dataset}/probesets..., extended to Publish data for the first time: /groups/{group}/traits itself is still not gated, but reaching the same data through /traits/{group} now requires a token and checks PublishFreeze.public/confidentiality. No gn-auth-side change was needed -- POST /auth/data/authorisation already dispatches *Publish-suffixed dataset names to its phenotype resource resolver (gn_auth/auth/authorisation/data/phenotypes.py), the exact mechanism this integration already relies on for mRNA data. auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to routes::auth (pub) so routes::traits can reuse it instead of duplicating the string. tests/traits.rs and its Python mirror cover both categories (including both ways of addressing a Publish dataset), the 401/403/404 cases, and the non-numeric-trait-against-Publish-dataset edge case, run against a real gn-auth instance. The Publish branch's deny path itself is only exercised via gn-auth's "unlinked dataset" fallback (already proven for ProbeSet) -- no non-public PublishFreeze exists in the database this was tested against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /species/{species}/datasetsPjotr Prins
Lists the expression (ProbeSet) datasets belonging to a species, across all of its groups -- the species-scoped equivalent of /groups/{group}/datasets (db::get_species_datasets joins ProbeSetFreeze through ProbeFreeze and InbredSet to InbredSet.SpeciesId, reusing the existing Dataset struct). GeneNetwork2's own species-scoped variant of wqflask.api.router.get_datasets_for_group still requires a group name (species only disambiguates it); this doesn't. Not gated on gn-auth, matching its /groups/{group}/datasets sibling: this returns dataset metadata (names, public/confidentiality flags), not trait data, unlike the /datasets/{dataset}/probesets family. tests/species_datasets.rs and its Python mirror cover the mouse-group and unknown-species cases. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits and GET /datasets/{dataset}/probesetsPjotr Prins
List endpoints for the two trait domains, each the listing equivalent of an existing singular endpoint: - GET /groups/{group}/traits: every published (phenotype) trait in a group (db::get_group_traits), same fields/join as /groups/{group}/traits/{trait_id}, no token required -- matching its singular sibling, which isn't gated either. A large group (BXD has 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; no pagination yet, matching GeneNetwork2's own wqflask.api.router.fetch_traits, which has no default limit either. - GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait in a dataset (db::get_dataset_probesets), same fields/join as /datasets/{dataset}/probesets/{probeset}. Gated the same way as that singular endpoint -- a valid token is required unconditionally, and a non-public dataset is checked against gn-auth once for the whole dataset via a new auth::authorise_dataset, rather than once per probeset in the list. Leaving this listing ungated would have let a caller read out a whole non-public dataset's trait data in one request, bypassing the gate just added to the singular endpoint. auth::authorise_dataset wraps authorise_trait with an empty trait name: gn-auth resolves a trait's resource purely from its dataset name, so the trait half of a "Dataset::Trait" string never affects which privileges come back. Verified directly against the live instance both for an unlinked non-public dataset (403, deny) and folded into dataset_probesets's own test for the public/embargo-aged bypass path. tests/group_traits.rs, tests/dataset_probesets.rs and their Python mirrors cover both, run against a real gn-auth instance for the gated one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Gate GET /datasets/{dataset}/probesets/{probeset}/values on gn-authPjotr Prins
Same treatment as dataset_probeset_info: require a valid AuthToken unconditionally, and for a dataset that isn't publicly readable, check group:resource:view-resource via authorise_trait -- denied is a 403, gn-auth unreachable is a 502. Unify DatasetProbesetLookup/DatasetProbesetInfoLookup back into one generic enum (with the new Forbidden variant) now that both dataset_probeset_info and dataset_probeset_values need it, rather than keeping two near-identical enums. tests/dataset_probeset_values.rs and its Python mirror gain the same 401/403 cases as dataset_probeset_info's, run against a real gn-auth instance; the 403 case reuses UTHSC_BXD_Cochlea_0225, confirmed to have both ProbeSet metadata and ProbeSetData values rows so the test exercises the authorization check rather than ProbesetNotFound. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Gate GET /datasets/{dataset}/probesets/{probeset} on gn-authPjotr Prins
Require a valid gn-auth bearer token on this endpoint unconditionally. For a dataset that isn't publicly readable, check the group:resource:view-resource privilege via a new auth::authorise_trait, the "Dataset::Trait"-keyed equivalent of authorise (POST /auth/data/authorisation instead of /auth/resource/authorisation) -- gn-auth resolves the resource itself from the dataset name, so no separate resource-id lookup is needed here. A trait never linked to a gn-auth resource (gn-auth answers 404) is treated as "no privileges", not a request failure. Denied requests get a 403; a failure to reach/parse gn-auth's answer gets a 502, distinct from both that and this service's own 500s. is_publicly_readable now also takes the dataset's CreateTime and treats anything older than 5 years as public regardless of its public/confidentiality flags -- a confidentiality embargo isn't meant to last forever. No admin/sysadmin bypass is implemented. Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the new security(...) annotation on this endpoint's #[utoipa::path] has something to point at. tests/dataset_probeset_info.rs and its Python mirror gain cases for the missing-token (401) and denied-but-existing (403) outcomes, run against a real gn-auth instance set up as described in docs/gn-auth.md; the 403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed dataset never linked into gn-auth. tests/common gains get_with_token and auth_server_available/auth_token helpers for this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add ProbeSetXRef endpoints: datasets, probesets, probeset values.Pjotr Prins
The expression (ProbeSet) counterpart to the PublishXRef endpoints: ProbeSetXRef has no single Id column of its own (its primary key is DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)), so a trait here is addressed by dataset name plus ProbeSet.Name instead of a single numeric trait id. * src/db.rs (Dataset): New struct, one ProbeSetFreeze row. (get_group_datasets): New function. Joins ProbeSetFreeze and ProbeFreeze by group Id, matching GeneNetwork2's wqflask.api.router.get_datasets_for_group. (get_dataset_by_name): New function. Matches Name, Name2, FullName, ShortName or numeric Id, matching wqflask.api.router.get_dataset_info's ProbeSet-branch matching logic. A dataset name is globally unique on its own, unlike a group's traits, so this isn't scoped by group. (ProbeSetInfo): New struct, a probe's ProbeSetXRef mapping stats plus its ProbeSet gene annotation. (get_dataset_probeset_info): New function, matching wqflask.api.router.get_trait_info's ProbeSet branch. (ProbeSetValue): New struct. (get_dataset_probeset_values): New function, matching wqflask.api.router.trait_sample_data's ProbeSet branch. * src/lib.rs (DatasetProbesetLookup): New enum, the expression-dataset equivalent of GroupTraitLookup -- distinguishes a missing dataset from a missing probeset in an existing dataset. (group_datasets): New handler for GET /groups/{group}/datasets. (dataset_by_name): New handler for GET /datasets/{dataset}. (get_dataset_probeset_info_result, dataset_probeset_info): New function and handler for GET /datasets/{dataset}/probesets/{probeset}. (get_dataset_probeset_values_result, dataset_probeset_values): New function and handler for GET /datasets/{dataset}/probesets/{probeset}/values. (app): Register all four. * tests/group_datasets.rs, tests/dataset_by_name.rs, tests/dataset_probeset_info.rs, tests/dataset_probeset_values.rs, test/python/test_group_datasets.py, test/python/test_dataset_by_name.py, test/python/test_dataset_probeset_info.py, test/python/test_dataset_probeset_values.py: New files. Br_U_0803_M (dataset) / 100001_at (probeset, gene symbol Cd3g) in BXD are real rows used as fixtures. * tests/openapi.rs, test/python/test_openapi.py: Check the four new paths are in the generated spec. * README.md: Document the four new endpoints and list the new test files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}; time DB tests; share pool.Pjotr Prins
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and Publication; LEFT JOIN on the latter two since a handful of PublishXRef rows have no matching row. SQL column aliases are written to match mysql_common's UpperCamelCase conversion exactly, so no field needs a #[mysql(rename = "...")] override. (get_group_trait_info): New function. (pool): New function. mysql::Pool::new eagerly opens its default 10 connections before returning (about 7s against this database); every query function here used to call it per request, paying that cost on every single API call. Builds the Pool once behind a OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle from then on (OnceLock::get_or_init can't be used alone since Pool::new is fallible and get_or_try_init is still unstable). (get_all_species, get_species_by_name, get_all_groups, get_group_by_name, get_group_strains, get_group_trait_values, get_group_trait_info): Use it instead of building their own Pool. * src/lib.rs (GroupTraitLookup): Generalize over the found value (was Vec<db::TraitValue>-specific), so group_trait_info can share it with group_trait_values. (get_group_trait_info_result): New function. (group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}. (app): Register it. * tests/common/mod.rs: New file. get() issues a GET against genenetwork4::app() and warns on stderr if it takes over a second -- this is what surfaced the Pool::new cost above. * tests/species.rs, tests/species_by_name.rs, tests/groups.rs, tests/group_by_name.rs, tests/group_strains.rs, tests/group_trait_values.rs: Use tests/common::get instead of building each request by hand. * tests/group_trait_info.rs, test/python/test_group_trait_info.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}): Document it. (Tests): List the new test files; document tests/common::get's timing warnings and how to see them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}/values, publish trait data.Pjotr Prins
* src/db.rs (TraitValue): New struct. (get_group_trait_values): New function. Joins PublishData, Strain and PublishXRef by group Id and PublishXRef.Id, matching gn-guile's gn.data.dataset:get-bxd-publish-name-value-dict generalized beyond BXD: PublishXRef.Id alone isn't unique across groups, only the pair (InbredSetId, Id) is, the same way GeneNetwork2's wqflask.api.router.trait_sample_data combines trait_name with a group or dataset to resolve a Publish trait. * src/lib.rs (GroupTraitLookup): New enum, distinguishes a missing group from a missing trait in an existing group. (get_group_trait_values_result): New function, looks up a group then its trait, returning a GroupTraitLookup. (group_trait_values): New handler for GET /groups/{group}/traits/{trait_id}/values. 404s as group_not_found or trait_not_found depending on what get_group_trait_values_result found. (app): Register it. * tests/group_trait_values.rs, test/python/test_group_trait_values.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}/values): Document it, including that only Publish (phenotype) traits are supported so far, not ProbeSet or Geno traits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03Add GET /groups/{group}/strains, mirroring gn-guile's strain-id-names.Pjotr Prins
* src/db.rs (Strain): New struct. (get_group_strains): New function. Joins Strain and StrainXRef by group Id, optionally filtered to strains used for mapping, matching gn-guile's gn.data.strains:strain-id-names (same InbredSetId and used-for-mapping? arguments), returning strain names as well as ids. * src/lib.rs (GroupStrainsParams): New struct, the used_for_mapping query parameter. (group_strains): New handler for GET /groups/{group}/strains. Looks the group up the same way group_by_name does, for the same 404 shape on an unknown group. (app): Register it. * tests/group_strains.rs, test/python/test_group_strains.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/strains): Document it, including the used_for_mapping filter excluding BXD's parent strains. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03Match group by numeric Id, use Group alias for InbredSet fields.Pjotr Prins
* src/db.rs (get_group_by_name): Also match CAST(Id AS CHAR), so a numeric Id works like GeneNetwork2's wqflask.api.router.get_group_info plus gn-guile's gn.data.population:population-info, which looks a group up by Id only. (GroupDetail): Rename inbred_set_id, inbred_set_name, inbred_set_code fields to group_id, display_name, group_code, adding #[mysql(rename = "...")] overrides so FromRow still matches the InbredSetId, InbredSetName, InbredSetCode MariaDB columns; InbredSet stays a MariaDB-only name, not a Rust identifier. * src/lib.rs (group_by_name): Update the "group" path parameter description to mention matching by numeric Id too. * tests/group_by_name.rs (group_by_name_bxd_returns_group_row): Update assertion for the display_name field rename. (group_by_name_numeric_id_returns_same_group_row): New test. * test/python/test_group_by_name.py: Likewise. * README.md (GET /groups/{group}): Document matching by numeric Id and the group_id/display_name/group_code field names. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-03Add /groups and /groups/{group} endpoints.Pjotr Prins
* src/db.rs (Group): New struct. Summary of one InbredSet row, matching the shape of GeneNetwork2's GET /api/v_pre1/groups response. (get_all_groups): New function. Mirrors GeneNetwork2's wqflask.api.router.get_groups_list (no species filter yet). (GroupDetail): New struct, deriving FromRow instead of being built from a tuple like SpeciesDetail, since InbredSet has 14 columns, more than mysql_common's FromRow is implemented for tuples (up to 12). FromRow matches by column name (PascalCase of the field name via #[mysql(rename_all = "PascalCase")]), except public, which is lowercase in the schema and needs its own #[mysql(rename = "public")]. (get_group_by_name): New function. Mirrors GeneNetwork2's wqflask.api.router.get_group_info, but returns every column instead of a fixed subset. * src/lib.rs (app): Add /groups and /groups/{group} routes. (groups, group_by_name): New handlers, following the same ApiError/request-id/spawn_blocking pattern as species/species_by_name. (group_by_name): On no match, set ApiError's next to /groups. * tests/groups.rs: New file. Checks BXD is present (used throughout the GeneNetwork2 and GeneNetwork3 test suites as a fixture group, always present in the database); skips rather than fails when no database is reachable, same as species.rs. * test/python/test_groups.py: New file. Likewise. * tests/group_by_name.rs: New file. Checks BXD's full row and an unknown-group 404 case, same pattern as species_by_name.rs. * test/python/test_group_by_name.py: New file. Likewise. * tests/openapi.rs (openapi_json_describes_all_endpoints): Check for /groups and /groups/{group} among the documented paths. * test/python/test_openapi.py (test_openapi_json_describes_all_endpoints): Likewise. * README.md: Document /groups and /groups/{group}.
2026-10-03Add shared ApiError response shape and request-id tracking.Pjotr Prins
* Cargo.toml (dependencies): Add tower-http, features request-id and util. * src/error.rs: New file. ApiError, modeled on RFC 7807 (Problem Details for HTTP APIs) plus code, operation_id, spec_url, request_id and next. type and spec_url are relative paths since this service has no fixed public domain. Implements axum::response::IntoResponse. * src/lib.rs (app): Add SetRequestIdLayer and PropagateRequestIdLayer, assigning each request an x-request-id and copying it onto the response. (request_id_string): New function. (species, species_by_name): Take an Extension<RequestId> parameter. Return Result<_, ApiError> instead of Result<_, (StatusCode, String)>. (species_by_name): On no match, set ApiError's next to /species, so a caller with a wrong species name can discover the valid ones. * tests/species_by_name.rs (species_by_name_unknown_species_returns_404): Check the full ApiError body (status, code, operation_id, spec_url, next) and that request_id matches the x-request-id response header, instead of only the status code. * test/python/test_species_by_name.py (test_species_by_name_unknown_species_returns_404): Likewise. * README.md: Add an Error handling section documenting the ApiError shape and each field. * rust-package.scm: Regenerate with guix import crate -f Cargo.lock genenetwork4. Adds crate-source definitions for tower-http and its new transitive dependencies. * guix.scm (genenetwork4-git)[native-inputs]: Add rust-tower-http-0.7.1, rust-getrandom-0.4.3 and rust-r-efi-6.0.0.
2026-10-03Add /species/{species} for a single species' full row.Pjotr Prins
Checked for an existing implementation first: GeneNetwork2's wqflask/api/router.py has /api/v_pre1/species/<species_name>, matching Name, FullName or SpeciesName, but returning only a 4-column subset (Id, Name, FullName, TaxonomyId). GeneNetwork3's gn3/db/wiki.py has only internal name-to-id lookups for the wiki feature; its /species is SPARQL-based metadata, not a Species table row. gn-guile's species code (gn/data/species.scm) is also SPARQL/RDF metadata, not backed by db_webqtl. * src/db.rs (params): Import. (SpeciesDetail): New struct, one full row of the Species table. (get_species_by_name): New function. Same match logic as GeneNetwork2's get_species_info (Name, FullName or SpeciesName), but selects every column instead of a fixed subset. * src/lib.rs (app): Add /species/{species} route. (species_by_name): New handler. 404 if no species matches. * tests/species_by_name.rs: New file. Tests mouse specifically (Name=mouse is always present) and an unknown-species 404 case; skips rather than fails when no database is reachable, same as species.rs. * test/python/test_species_by_name.py: New file. Likewise. * tests/openapi.rs (openapi_json_describes_all_endpoints): Check for /species/{species} among the documented paths. * test/python/test_openapi.py (test_openapi_json_describes_all_endpoints): Likewise. * README.md: Document /species/{species}.
2026-10-03Add /openapi.json describing the REST API.Pjotr Prins
* Cargo.toml (dependencies): Add utoipa, feature axum_extras; add utoipa-axum. * src/lib.rs: Use utoipa_axum::router::OpenApiRouter and routes! instead of axum::Router and route!, so paths register into an OpenApi document as well as the router. (ApiDoc): New struct, deriving utoipa::OpenApi with title, description and license. (app): Add /openapi.json, serving the OpenApi document split out of OpenApiRouter via split_for_parts. (hello_world, version, species): Add #[utoipa::path(...)] attributes describing each endpoint's responses. (VersionInfo): Derive utoipa::ToSchema in addition to Serialize. * src/db.rs (Species): Likewise. * tests/openapi.rs: New file. Checks /openapi.json's info.title and that it lists all three endpoints. * test/python/test_openapi.py: New file. Likewise. * README.md: Document /openapi.json. * rust-package.scm: Regenerate with guix import crate -f Cargo.lock genenetwork4. Adds crate-source definitions for utoipa, utoipa-gen, utoipa-axum and pastey. * guix.scm (genenetwork4-git)[native-inputs]: Add rust-utoipa-6.0.0, rust-utoipa-gen-6.0.1, rust-utoipa-axum-0.3.0 and rust-pastey-0.2.3.
2026-10-03Assert mouse and rat are present in /species.Pjotr Prins
* tests/species.rs (species_returns_ok_with_species_list): Assert the returned names include mouse and rat, which are always present in the database. * test/python/test_species.py (test_species_returns_ok_with_species_list): Likewise.
2026-10-03Add /species endpoint backed by MariaDB.Pjotr Prins
* Cargo.toml (dependencies): Add mysql, default-features = false, features minimal-rust and derive (minimal pulls in flate2/zlib, which fails to compile here; minimal-rust uses flate2's pure-Rust backend instead). * Cargo.lock: Regenerate. Picks up mysql and its roughly 100 transitive dependencies; also bumps libc to 0.2.190 and tokio to 1.53.2 from re-resolution. * src/db.rs: New file. sql_uri, Species, get_all_species. Mirrors GeneNetwork3's gn3.db.species.get_all_species: same query, same ordering. Connection string from the SQL_URI environment variable (same config key GeneNetwork3 uses), defaulting to mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl. * src/lib.rs (app): Add /species route. (species): New handler. Runs db::get_all_species on a blocking thread, since mysql::Conn is synchronous. * tests/species.rs: New file. Checks for a database on 127.0.0.1:3306 first and skips with a message instead of failing when none is reachable, since the Guix build sandbox has no network. * test/python/test_species.py: New file. Likewise. * README.md: Document /species, SQL_URI, and the skip-without-database behavior of species.rs and test_species.py. * rust-package.scm: Regenerate with guix import crate -f Cargo.lock genenetwork4. Adds crate-source definitions for mysql's transitive dependencies; bumps rust-libc and rust-tokio to the versions above. * guix.scm (genenetwork4-git)[native-inputs]: Add every new rust-package.scm entry mysql pulled in (roughly 100 packages, including rust-mysql-28.0.3, rust-mysql-common-0.37.3, rust-uuid, rust-url, rust-sha1, rust-sha2, rust-flate2).
2026-10-01Add /version endpoint, extract shared python test helper.Pjotr Prins
* .gitignore: Add __pycache__. * src/lib.rs (app): Add /version route. (VersionInfo, version): New struct and handler, returning name, short_name, version, release, date, and git as a JSON record. * build.rs: New file. Embeds GN4_GIT_HASH and GN4_COMMIT_DATE via cargo:rustc-env, read from git rev-parse and git show of HEAD. * Cargo.toml (dependencies): Add serde with derive feature. [dev-dependencies]: Add serde_json. * tests/version.rs: New file. * test/python/server_test_case.py: New file. ServerTestCase, wait_for_port, BINARY, HOST, PORT moved here from test/python/test_hello_world.py. * test/python/test_hello_world.py: Use server_test_case.ServerTestCase instead of duplicating setUpClass and tearDownClass. * test/python/test_version.py: New file. * guix.scm: Add (gnu packages version-control) to use-modules. (genenetwork4-git)[native-inputs]: Add git-minimal. (genenetwork4-git)[arguments]<#:phases>[python-rest-test]: Run test/python with python3 -m unittest discover instead of invoking test_hello_world.py directly, so test_version.py also runs.
2026-10-01Add Python REST API test, run it from guix.scm.Pjotr Prins
* src/main.rs (main): Read GENENETWORK4_PORT environment variable for the listen port, default to 3000. * test/python/test_hello_world.py: New file. * guix.scm: Add (gnu packages python) to use-modules. (genenetwork4-git)[native-inputs]: Add python. (genenetwork4-git)[arguments]<#:phases>: Add 'python-rest-test phase after 'install, running test/python/test_hello_world.py against the installed binary on port 3737.