| Age | Commit message (Collapse) | Author |
|
tests/envelope.rs follows every href in every reply, and running it end to
end found six that were not addresses:
- /species, /species/{name}/datasets and /groups/{group}/datasets offered
/species/{species} and /datasets/{dataset} -- templates, which 404 when
followed. Each now points at a real row from the listing in hand, and
the description says any other name can go in its place.
- /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the
listing it already links is the callable one, so the shape moved into
that link's description.
- /traits/{dataset}/{trait}/values offered a GET
/traits/{ds}/{trait}/correlation, which does not exist: correlation is
POST /compute/correlation and that reply already carries it in
"actions". Removed, with the reason written where it was.
- the same reply built its /groups/... links from the *dataset* name.
BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is
a 404 (no genotype files for it). db::get_group_name resolves the group
from InbredSet.Id and the links use it; when a dataset has no group the
group links are simply absent.
- /groups/{g}/phewas with no parameters is a 422 ("give either locus, or
chr and mb"), so that link is now a shape with placeholders.
Placeholders stay where the caller must supply a value
(.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips
them only when the description says what to fill in -- otherwise a client
reading the next list cannot tell a call from a template.
Two stale expectations alongside: /version carries the root links now (it
is the natural first call), and /search's results put its own follow-ups
first and the root links after them.
Tests are honest about their environment too: links skips when no search
index is configured (an empty XAPIAN_DB_PATH answers 502), and the
envelope crawl exempts /openapi.json, the API document, which is
deliberately not enveloped.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
A new, additional endpoint family that addresses both trait categories
the same way -- (dataset, trait) -- matching GeneNetwork2's own generic
wqflask.api.router.trait_sample_data/get_trait_info, rather than the
two category-specific shapes genenetwork4 already has
(/datasets/{dataset}/probesets... and /groups/{group}/traits...), which
are unchanged and keep working exactly as before.
- db::resolve_trait_dataset(name) resolves a dataset name against
ProbeSetFreeze first, then the new db::PublishDataset (a PublishFreeze
row, via the new get_publish_freeze_by_name) -- whichever matches
decides the category. This means only dataset names are ever
"guessed" across categories, not trait names/ids, which is the much
smaller and safer surface: GN2's own equivalent tries ProbeSet.Name
and PublishXRef.Id against the same bare trait string and silently
falls back, which can return the wrong trait on a same-named
collision. get_publish_freeze_by_name accepts a PublishFreeze's own
Name/ShortName/Id or its owning group's Name/InbredSetName/FullName/
Id, mirroring GN2's own flexible matching (e.g. "BXDPublish" or
"BXD").
- GET /traits/{dataset}: list every trait in a dataset, tagged by
category.
- GET /traits/{dataset}/{trait}: one trait's metadata. A non-numeric
trait against a Publish dataset is a 404 (can't match a
PublishXRef.Id), not a 500.
- GET /traits/{dataset}/{trait}/values: that trait's per-strain values.
TraitValue and ProbeSetValue are already identical in shape, so this
unifies with no information loss, unlike the metadata endpoint above.
Metadata responses are a tagged union ({"category": "probeset"|
"publish", ...}) wrapping ProbeSetInfo/TraitInfo unchanged, rather than
a lossy common-fields schema -- the two share little beyond
mean/locus/lrs/additive. utoipa generates this as a oneOf with a
category discriminator, verified against /openapi.json.
Gated the same way as /datasets/{dataset}/probesets..., extended to
Publish data for the first time: /groups/{group}/traits itself is
still not gated, but reaching the same data through /traits/{group}
now requires a token and checks PublishFreeze.public/confidentiality.
No gn-auth-side change was needed -- POST /auth/data/authorisation
already dispatches *Publish-suffixed dataset names to its phenotype
resource resolver (gn_auth/auth/authorisation/data/phenotypes.py),
the exact mechanism this integration already relies on for mRNA data.
auth::VIEW_RESOURCE_PRIVILEGE moves from routes::datasets (private) to
routes::auth (pub) so routes::traits can reuse it instead of
duplicating the string.
tests/traits.rs and its Python mirror cover both categories (including
both ways of addressing a Publish dataset), the 401/403/404 cases, and
the non-numeric-trait-against-Publish-dataset edge case, run against a
real gn-auth instance. The Publish branch's deny path itself is only
exercised via gn-auth's "unlinked dataset" fallback (already proven for
ProbeSet) -- no non-public PublishFreeze exists in the database this
was tested against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
List endpoints for the two trait domains, each the listing equivalent
of an existing singular endpoint:
- GET /groups/{group}/traits: every published (phenotype) trait in a
group (db::get_group_traits), same fields/join as
/groups/{group}/traits/{trait_id}, no token required -- matching its
singular sibling, which isn't gated either. A large group (BXD has
15000+ phenotype traits) can take several seconds and tens of
megabytes to return in full; no pagination yet, matching
GeneNetwork2's own wqflask.api.router.fetch_traits, which has no
default limit either.
- GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait
in a dataset (db::get_dataset_probesets), same fields/join as
/datasets/{dataset}/probesets/{probeset}. Gated the same way as that
singular endpoint -- a valid token is required unconditionally, and
a non-public dataset is checked against gn-auth once for the whole
dataset via a new auth::authorise_dataset, rather than once per
probeset in the list. Leaving this listing ungated would have let a
caller read out a whole non-public dataset's trait data in one
request, bypassing the gate just added to the singular endpoint.
auth::authorise_dataset wraps authorise_trait with an empty trait
name: gn-auth resolves a trait's resource purely from its dataset
name, so the trait half of a "Dataset::Trait" string never affects
which privileges come back. Verified directly against the live
instance both for an unlinked non-public dataset (403, deny) and
folded into dataset_probesets's own test for the public/embargo-aged
bypass path.
tests/group_traits.rs, tests/dataset_probesets.rs and their Python
mirrors cover both, run against a real gn-auth instance for the gated
one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Same treatment as dataset_probeset_info: require a valid AuthToken
unconditionally, and for a dataset that isn't publicly readable, check
group:resource:view-resource via authorise_trait -- denied is a 403,
gn-auth unreachable is a 502.
Unify DatasetProbesetLookup/DatasetProbesetInfoLookup back into one
generic enum (with the new Forbidden variant) now that both
dataset_probeset_info and dataset_probeset_values need it, rather than
keeping two near-identical enums.
tests/dataset_probeset_values.rs and its Python mirror gain the same
401/403 cases as dataset_probeset_info's, run against a real gn-auth
instance; the 403 case reuses UTHSC_BXD_Cochlea_0225, confirmed to
have both ProbeSet metadata and ProbeSetData values rows so the test
exercises the authorization check rather than ProbesetNotFound.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Require a valid gn-auth bearer token on this endpoint unconditionally.
For a dataset that isn't publicly readable, check the
group:resource:view-resource privilege via a new auth::authorise_trait,
the "Dataset::Trait"-keyed equivalent of authorise (POST
/auth/data/authorisation instead of /auth/resource/authorisation) --
gn-auth resolves the resource itself from the dataset name, so no
separate resource-id lookup is needed here. A trait never linked to a
gn-auth resource (gn-auth answers 404) is treated as "no privileges",
not a request failure. Denied requests get a 403; a failure to
reach/parse gn-auth's answer gets a 502, distinct from both that and
this service's own 500s.
is_publicly_readable now also takes the dataset's CreateTime and treats
anything older than 5 years as public regardless of its
public/confidentiality flags -- a confidentiality embargo isn't meant
to last forever. No admin/sysadmin bypass is implemented.
Register a "bearer_token" OpenAPI security scheme (src/lib.rs) so the
new security(...) annotation on this endpoint's #[utoipa::path] has
something to point at.
tests/dataset_probeset_info.rs and its Python mirror gain cases for the
missing-token (401) and denied-but-existing (403) outcomes, run against
a real gn-auth instance set up as described in docs/gn-auth.md; the
403 case uses UTHSC_BXD_Cochlea_0225, a real non-public, non-embargoed
dataset never linked into gn-auth. tests/common gains get_with_token
and auth_server_available/auth_token helpers for this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
The expression (ProbeSet) counterpart to the PublishXRef endpoints:
ProbeSetXRef has no single Id column of its own (its primary key is
DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)),
so a trait here is addressed by dataset name plus ProbeSet.Name
instead of a single numeric trait id.
* src/db.rs (Dataset): New struct, one ProbeSetFreeze row.
(get_group_datasets): New function. Joins ProbeSetFreeze and
ProbeFreeze by group Id, matching GeneNetwork2's
wqflask.api.router.get_datasets_for_group.
(get_dataset_by_name): New function. Matches Name, Name2, FullName,
ShortName or numeric Id, matching
wqflask.api.router.get_dataset_info's ProbeSet-branch matching logic.
A dataset name is globally unique on its own, unlike a group's
traits, so this isn't scoped by group.
(ProbeSetInfo): New struct, a probe's ProbeSetXRef mapping stats plus
its ProbeSet gene annotation.
(get_dataset_probeset_info): New function, matching
wqflask.api.router.get_trait_info's ProbeSet branch.
(ProbeSetValue): New struct.
(get_dataset_probeset_values): New function, matching
wqflask.api.router.trait_sample_data's ProbeSet branch.
* src/lib.rs (DatasetProbesetLookup): New enum, the expression-dataset
equivalent of GroupTraitLookup -- distinguishes a missing dataset from
a missing probeset in an existing dataset.
(group_datasets): New handler for GET /groups/{group}/datasets.
(dataset_by_name): New handler for GET /datasets/{dataset}.
(get_dataset_probeset_info_result, dataset_probeset_info): New
function and handler for GET /datasets/{dataset}/probesets/{probeset}.
(get_dataset_probeset_values_result, dataset_probeset_values): New
function and handler for
GET /datasets/{dataset}/probesets/{probeset}/values.
(app): Register all four.
* tests/group_datasets.rs, tests/dataset_by_name.rs,
tests/dataset_probeset_info.rs, tests/dataset_probeset_values.rs,
test/python/test_group_datasets.py, test/python/test_dataset_by_name.py,
test/python/test_dataset_probeset_info.py,
test/python/test_dataset_probeset_values.py: New files. Br_U_0803_M
(dataset) / 100001_at (probeset, gene symbol Cd3g) in BXD are real rows
used as fixtures.
* tests/openapi.rs, test/python/test_openapi.py: Check the four new
paths are in the generated spec.
* README.md: Document the four new endpoints and list the new test
files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|