| Age | Commit message (Collapse) | Author |
|
tied to dev mode.
The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind
address. It decides per request whether to answer.
* src/routes/dev.rs (get_test_token): answers a caller only when the TCP
peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped
127.x) and the request carries no X-Forwarded-For, Forwarded or
X-Real-IP header (a reverse proxy on the same host would otherwise make
every client look local). Anything else, including an unknown peer, is a
403. Local is never established from a header.
* src/main.rs: serve with into_make_service_with_connect_info, which
provides the peer address.
* src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and
dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it
is not a clearly affirmative value (true/1/yes/on) the route is not
registered: a 404, absent from /openapi.json. It fails closed, so a
typo ("flase") turns the endpoint off.
* src/lib.rs: register the route unless that switch is off.
* tests: tests/get_test_token.rs rewritten (remote, unknown and proxied
peers refused; 127.x and ::1 answered; always documented by default);
tests/get_test_token_off.rs for the switch (own process: it sets an
environment variable); common::post_from supplies a peer address to the
in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV
is now empty (no setting needed).
* README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document
the check, the switch, and "Behind nginx or in a container" (a bridge
container sees the gateway as the peer; a host-network container behind
nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For,
or switch the endpoint off).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
/accessions.
Answers four more questions of aisearch's full_benchmark.csv from
tables the API did not expose.
* src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain
`mapping_method`, the name of the MappingMethod row their
mapping_method_id points to (qtlreaper, Rqtl, ...), or null.
* src/db.rs (Dataset): datasets gain `tissue`, the Tissue their
ProbeFreeze points to (e.g. "Brain mRNA").
* src/routes/mapping_methods.rs: GET /mapping-methods lists the
MappingMethod table, including tools no group uses (PLINK, FastMap).
* src/routes/accessions.rs: GET /accessions/{accession} resolves a
GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the
dataset's name, title, category (publish/probeset) and group.
* docs/genenetwork_datatypes.md: MappingMethodId is a key into the
MappingMethod table (the authoritative reading); GN2's code-based reading
is noted as superseded.
* tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new
full-benchmark rows that pass (full rows 0..114; search rows run against
recorded production gn3, the rest need the database), and full rows 33
(mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only
GTEx v8 has TPM datasets). The untested rows are listed in the file's
header comment with the reason for each.
* tests, test/python: tests for the new fields and endpoints; both
endpoints added to the OpenAPI and next-link crawl tests.
* README.md, RELEASE_NOTES.md: document the new fields and endpoints.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Per-strain metadata beyond genotype/phenotype data (age, sex,
treatment, etc.), backed by MariaDB CaseAttribute/CaseAttributeXRefNew.
Combines GeneNetwork3's separate /case-attribute/{id}/names and
/case-attribute/{id}/values (gn3.api.case_attributes) into one
response -- neither list is large enough on its own to need splitting.
This is the only one of GeneNetwork2/3/gn-guile with the concept at
all: GeneNetwork2 delegates to GeneNetwork3 rather than storing case
attributes itself, and gn-guile doesn't have it.
db::get_group_case_attributes returns {names: [...], values: [{strain,
case_attributes: {...}}]}; values is built by folding adjacent
(strain, attribute, value) rows (ordered by strain name) rather than
grouping afterwards, since SQL already guarantees the adjacency.
Found and fixed a real bug while verifying this live: GeneNetwork3's
own query joins CaseAttribute to CaseAttributeXRefNew on
CaseAttributeId alone, but CaseAttributeId is not unique across groups
-- it's a shared vocabulary (id 37 means "Epoch" for dozens of
unrelated groups simultaneously); the actual key is
(InbredSetId, CaseAttributeId). Confirmed against this database:
without also joining on InbredSetId, BXD's result included a spurious
"Strain": "Live" attribute belonging to a different, unrelated group.
Fixed on genenetwork4's side by adding the missing join condition.
A group with no case attributes defined returns 200 with both lists
empty, not a 404 (the group itself exists) -- verified against LXS,
a real group with zero CaseAttribute rows.
tests/group_case_attributes.rs and its Python mirror cover BXD's 4
real attributes (asserting each strain has exactly those 4, not more,
to catch the join bug above), the empty-group case, missing token, and
unknown group.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
- src/routes/search.rs: AuthToken per correlation.rs pattern (+ Unauthorized).
- tests/search.rs: stub /auth/public-jwks + locally-signed token; scenarios
run offline again (multi_thread runtime: the blocking ureq JWKS fetch
starved the single-thread stub); add invalid-token 401 case.
- test/python/test_search.py: fetch token once, reuse.
- RELEASE_NOTES.md: /search now requires a token.
- test/data: test-only JWKS + RSA key fixtures.
|
|
Three claims left over from before "every endpoint requires a token"
(previous commit) were actually wrong, not just stale phrasing:
- RELEASE_NOTES.md's intro still said "not all endpoints will require a
token" -- the opposite of what the rest of the file by then described.
- README's /traits/{dataset} entry and RELEASE_NOTES.md's /traits/...
entry both still said /groups/{group}/traits "isn't gated" -- true
for resource-level authorization, false for the token requirement
added in the same commit that should have updated this.
Also fixed real staleness independent of this turn: README's Tests
section enumerated specific Rust/Python test file pairs needing a live
database, a list that had silently fallen behind several rounds of new
endpoints (missing species_datasets, group_traits, dataset_probesets,
traits, get_test_token). Replaced the enumeration with a description of
which tests need what and why, so it can't go stale the same way again.
Also documented AUTH_SERVER_URL/GENENETWORK4_DEV_MODE/GENENETWORK4_BIND
for the "run the test suite with a self-provisioned token" path the
previous commit added, which the cargo test / python -m unittest
invocations here hadn't mentioned yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Calls gn-auth's own password grant (POST /auth/token) for gn-auth's
well-known development user and returns the result, so local testing
no longer needs a manual curl/gn-auth round trip of its own.
Only ever registered -- not just rejected at request time, genuinely
absent from the router and from /openapi.json -- when two new Config
fields are both set: dev_mode (GENENETWORK4_DEV_MODE) and a loopback
bind (GENENETWORK4_BIND, also new). Both are config, not something
genenetwork4 infers from its actual network exposure, by design: dev
mode alone, or a loopback bind alone, each leave it disabled, and the
check (Config::dev_token_enabled) is a plain function so the condition
itself has direct unit test coverage rather than only being exercised
indirectly through the route.
src/main.rs now binds to config.bind instead of a hardcoded "0.0.0.0",
via a proper SocketAddr rather than string formatting -- the latter
would mishandle an IPv6 bind address like "::1" (needs brackets).
GN_AUTH_DEV_CLIENT_ID/_CLIENT_SECRET/_USERNAME/_PASSWORD default to
gn-auth's own published dev fixtures (flask init-dev-clients,
gn_auth/wsgi.py), so this works unconfigured against an unmodified dev
gn-auth instance -- confirmed live, including using the returned token
against a real gated endpoint (/traits/BXD/10001), not just checking
the token response's shape.
tests/dev_token.rs covers all four combinations of the two settings in
one sequential test function (mutating the same two env vars from more
than one #[tokio::test] in a file would race, since they run
concurrently by default) plus the /openapi.json visibility check.
test/python/test_dev_token.py mirrors this using separate test classes
instead, which is safe here since each spawns its own server
subprocess with its own environment (ServerTestCase gains EXTRA_ENV for
this).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
A new, additional endpoint family that addresses both trait categories
the same way -- (dataset, trait) -- matching GeneNetwork2's own generic
wqflask.api.router.trait_sample_data/get_trait_info, rather than the
two category-specific shapes genenetwork4 already has
(/datasets/{dataset}/probesets... and /groups/{group}/traits...), which
are unchanged and keep working exactly as before.
- db::resolve_trait_dataset(name) resolves a dataset name against
ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze
row, via the new get_publish_freeze_by_name) -- whichever matches
decides the category. This means only dataset names are ever
"guessed" across categories, not trait names/ids, which is the much
smaller and safer surface: GN2's own equivalent tries ProbeSet.Name
and PublishXRef.Id against the same bare trait string and silently
falls back, which can return the wrong trait on a same-named
collision. get_publish_freeze_by_name accepts a PublishFreeze's own
Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/
Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or
"BXD").
- GET /traits/{dataset}: list every trait in a dataset, tagged by
category.
- GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric
trait against a Publish dataset is a 404 (can't match a
PublishXRef.Id), not a 500.
- GET /traits/{dataset}/{trait}/values: that trait's per-strain values.
TraitValue and ProbeSetValue are already identical in shape, so this
unifies with no information loss, unlike the metadata endpoint above.
Metadata responses are a tagged union ({"category": "probeset"|
"publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than
a lossy common-fields schema -- the two share little beyond
mean/locus/lrs/additive. utoipa generates this as a oneOf with a
category discriminator, verified against /openapi.json.
Gated the same way as /datasets/{dataset}/probesets..., extended to
Publish data for the first time: /groups/{group}/traits itself is
still not gated, but reaching the same data through /traits/{group}
now requires a token and checks PublishFreeze.public/confidentiality.
No gn-auth-side change was needed -- POST /auth/data/authorisation
already dispatches *Publish-suffixed dataset names to its phenotype
resource resolver (gn_auth/auth/authorisation/data/phenotypes.py),
the exact mechanism this integration already relies on for mRNA data.
auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to
routes::auth (pub) so routes::traits can reuse it instead of
duplicating the string.
tests/traits.rs and its Python mirror cover both categories (including
both ways of addressing a Publish dataset), the 401/403/404 cases, and
the non-numeric-trait-against-Publish-dataset edge case, run against a
real gn-auth instance. The Publish branch's deny path itself is only
exercised via gn-auth's "unlinked dataset" fallback (already proven for
ProbeSet) -- no non-public PublishFreeze exists in the database this
was tested against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Lists the expression (ProbeSet) datasets belonging to a species, across
all of its groups -- the species-scoped equivalent of
/groups/{group}/datasets (db::get_species_datasets joins ProbeSetFreeze
through ProbeFreeze and InbredSet to InbredSet.SpeciesId, reusing the
existing Dataset struct). GeneNetwork2's own species-scoped variant of
wqflask.api.router.get_datasets_for_group still requires a group name
(species only disambiguates it); this doesn't.
Not gated on gn-auth, matching its /groups/{group}/datasets sibling:
this returns dataset metadata (names, public/confidentiality flags),
not trait data, unlike the /datasets/{dataset}/probesets family.
tests/species_datasets.rs and its Python mirror cover the mouse-group
and unknown-species cases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
List endpoints for the two trait domains, each the listing equivalent
of an existing singular endpoint:
- GET /groups/{group}/traits: every published (phenotype) trait in a
group (db::get_group_traits), same fields/join as
/groups/{group}/traits/{trait_id}, no token required -- matching its
singular sibling, which isn't gated either. A large group (BXD has
15000+ phenotype traits) can take several seconds and tens of
megabytes to return in full; no pagination yet, matching
GeneNetwork2's own wqflask.api.router.fetch_traits, which has no
default limit either.
- GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait
in a dataset (db::get_dataset_probesets), same fields/join as
/datasets/{dataset}/probesets/{probeset}. Gated the same way as that
singular endpoint -- a valid token is required unconditionally, and
a non-public dataset is checked against gn-auth once for the whole
dataset via a new auth::authorise_dataset, rather than once per
probeset in the list. Leaving this listing ungated would have let a
caller read out a whole non-public dataset's trait data in one
request, bypassing the gate just added to the singular endpoint.
auth::authorise_dataset wraps authorise_trait with an empty trait
name: gn-auth resolves a trait's resource purely from its dataset
name, so the trait half of a "Dataset::Trait" string never affects
which privileges come back. Verified directly against the live
instance both for an unlinked non-public dataset (403, deny) and
folded into dataset_probesets's own test for the public/embargo-aged
bypass path.
tests/group_traits.rs, tests/dataset_probesets.rs and their Python
mirrors cover both, run against a real gn-auth instance for the gated
one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Same treatment as dataset_probeset_info: require a valid AuthToken
unconditionally, and for a dataset that isn't publicly readable, check
group:resource:view-resource via authorise_trait -- denied is a 403,
gn-auth unreachable is a 502.
Unify DatasetProbesetLookup/DatasetProbesetInfoLookup back into one
generic enum (with the new Forbidden variant) now that both
dataset_probeset_info and dataset_probeset_values need it, rather than
keeping two near-identical enums.
tests/dataset_probeset_values.rs and its Python mirror gain the same
401/403 cases as dataset_probeset_info's, run against a real gn-auth
instance; the 403 case reuses UTHSC_BXD_Cochlea_0225, confirmed to
have both ProbeSet metadata and ProbeSetData values rows so the test
exercises the authorization check rather than ProbesetNotFound.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Require a valid gn-auth bearer token on this endpoint unconditionally.
For a dataset that isn't publicly readable, check the
group:resource:view-resource privilege via a new auth::authorise_trait,
the "Dataset::Trait"-keyed equivalent of authorise (POST
/auth/data/authorisation instead of /auth/resource/authorisation) --
gn-auth resolves the resource itself from the dataset name, so no
separate resource-id lookup is needed here. A trait never linked to a
gn-auth resource (gn-auth answers 404) is treated as "no privileges",
not a request failure. Denied requests get a 403; a failure to
reach/parse gn-auth's answer gets a 502, distinct from both that and
this service's own 500s.
is_publicly_readable now also takes the dataset's CreateTime and treats
anything older than 5 years as public regardless of its
public/confidentiality flags -- a confidentiality embargo isn't meant
to last forever. No admin/sysadmin bypass is implemented.
Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the
new security(...) annotation on this endpoint's #[utoipa::path] has
something to point at.
tests/dataset_probeset_info.rs and its Python mirror gain cases for the
missing-token (401) and denied-but-existing (403) outcomes, run against
a real gn-auth instance set up as described in docs/gn-auth.md; the
403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed
dataset never linked into gn-auth. tests/common gains get_with_token
and auth_server_available/auth_token helpers for this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* docs/gn-auth.md: Document the GN2 public>0 AND confidentiality<1
threshold (the test for whether data needs a gn-auth authorization
call at all, versus just a valid token), the now-implemented
src/auth.rs mechanism, and the four gn-auth bugs found and fixed while
building it.
* RELEASE_NOTES.md: Likewise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* RELEASE_NOTES.md: New file. Summarizes docs/gn-auth.md, notes the
Rust extractor/caching work as in progress, and points at the three
upstream gn-auth bugs found and fixed while testing against a
from-scratch gn-auth instance (gn-auth's own fix-fresh-deployment-bugs
branch).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|