| Age | Commit message (Collapse) | Author |
|
Two follow-ups to the parental-design endpoint.
/groups/{group} now carries the headline in its notes -- "Parental design:
C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes
the parents and F1s by default" -- plus a next link to the full design. "Which
cross is this?" is two strain names, and needing a second request to see them
was the only reason the endpoint had to be called first. A group with no
metadata says so in the same note, pointing at parents_and_f1s.json and at
`individuals.exclude`, rather than leaving the reader to wonder why nothing was
dropped.
The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it
drifted once already, silently: the paternal/maternal labels were swapped
against GN2's unpacking, and nothing noticed because the exclusion rule takes
both strains regardless of role. scripts/check-parental-metadata.py parses
ParInfo with ast (no gn2 import) and compares it group by group, reporting
disagreements per role and treating groups the JSON knows but GN2 does not as
informational. tests/parental_metadata.rs runs it while a GN2 checkout is
present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at
which point the JSON is the only copy and there is nothing to drift from. It
reports 23 groups reproduced, 1 extra (BXD-Longevity).
|
|
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only
implied. It reports the F1 strains, the maternal and paternal strain (from the
deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups),
the @type of the group's genotype file, the individuals each role names where
the metadata names them (BXD-Longevity does, most do not), and -- the part a
caller needs -- which members of the current panel the default rule drops and
which the `F1` name heuristic catches.
Two things that came out of building it and comparison with GN2:
* the JSON's paternal/maternal labels were INVERTED against GN2's reading.
GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal
strain is C57BL/6J; our file said paternal. No mapping was affected
(non_strains takes both regardless of role) but the metadata was wrong, and
anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat
note -- would have been backwards. The file is rewritten from ParInfo itself
(ast-parsed, not retyped), 23 groups' roles filled in and the 24th
(BXD-Longevity, which GN2 does not have) left as it was.
* an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's
own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports
excluded_from_panel and excluded_by_name separately and says which is doing
the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1
names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming
exclusion entries that match nothing.
genotypes::non_strains is rebuilt on the same typed parser (DesignRole /
ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit
test for the roles, empty-name handling and the unlisted-group source, and an
integration test asserting BXD's maternal/paternal orientation and the note.
|
|
tests/envelope.rs follows every href in every reply, and running it end to
end found six that were not addresses:
- /species, /species/{name}/datasets and /groups/{group}/datasets offered
/species/{species} and /datasets/{dataset} -- templates, which 404 when
followed. Each now points at a real row from the listing in hand, and
the description says any other name can go in its place.
- /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the
listing it already links is the callable one, so the shape moved into
that link's description.
- /traits/{dataset}/{trait}/values offered a GET
/traits/{ds}/{trait}/correlation, which does not exist: correlation is
POST /compute/correlation and that reply already carries it in
"actions". Removed, with the reason written where it was.
- the same reply built its /groups/... links from the *dataset* name.
BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is
a 404 (no genotype files for it). db::get_group_name resolves the group
from InbredSet.Id and the links use it; when a dataset has no group the
group links are simply absent.
- /groups/{g}/phewas with no parameters is a 422 ("give either locus, or
chr and mb"), so that link is now a shape with placeholders.
Placeholders stay where the caller must supply a value
(.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips
them only when the description says what to fill in -- otherwise a client
reading the next list cannot tell a call from a template.
Two stale expectations alongside: /version carries the root links now (it
is the natural first call), and /search's results put its own follow-ups
first and the root links after them.
Tests are honest about their environment too: links skips when no search
index is configured (an empty XAPIAN_DB_PATH answers 502), and the
envelope crawl exempts /openapi.json, the API document, which is
deliberately not enveloped.
|
|
The question that follows "why do GEMMA and R/qtl2 peak at different
markers": what does the trait look like within each genotype class at one
marker. The frequencies endpoints report the distribution of calls, not
the values behind them, and the answer must not be "serve the matrix" --
this API never does.
GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id>
-> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}}
The calls are joined to the trait's per-strain values inside the server
(the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values)
and reduced to those five numbers per class. Nothing in the payload names
an animal. A class with fewer than ten animals is called out in notes,
because a peak resting on two is a different claim from one resting on a
hundred.
It settles the DOL trait 10003 question. At GEMMA's top marker
JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean
2.53, B n=171 mean 3.95 -- the A class is two animals and they are the
trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's
convert2riself), so it keeps 173 animals there but only 2 of one class:
nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps
the 20 hets at dosage 0.5, and that is where its sharper p-value comes
from. GEMMA's next two hits are the same pattern in opposite directions
(one A among 171, then two B among 171), so the chr16 block is a haplotype
carried by a handful of extreme animals, not a population effect.
Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the
BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for
JAX00418132 is half the B-allele frequency 0.935, not the A-allele
frequency 0.065. The bimbam file and DOL.geno agree at that marker.
tests/frequencies.rs gains a route test: the class fields, A < B, the
small-class note, the "not served" note, no individual names anywhere in
the payload, and a 404 for a trait the group does not have.
|
|
A caller that knows only /version should be able to reach anything by
following links. Until now `next` was thin and uneven: the genotype
subtree offered none at all, the group record offered one, /version
offered nothing, and two links it did offer could not be followed.
* suggest:: now builds the graph: root_links (groups, species,
mapping-methods, openapi), genotype_version_links (frequencies,
marker names, a marker's location and tally, the group, the scanning
actions), unlisted_genotype_links, marker_links (location, tally,
marker list, panel tally, PheWAS at that marker), group_links and
trait_links. Envelope::with_links / with_links_except add them,
deduplicating and never suggesting the response the caller is
already reading.
* Wired through the genotype surface (listing, version detail, markers,
location), both frequencies routes, the compute endpoints, groups,
traits, both value routes, species, datasets, publications,
accessions, search, mapping methods, phewas, correlation and
/version.
* Fixed what was not followable, because `next` promises a GET that
returns 200 and a client is the one that finds out otherwise:
- /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are
gone; comput endpoints now link ?locus=<their strongest marker>
(suggest::peak_marker_of, computed before the rows move).
- /datasets and /publications are not routes; root_links offers the
listings that exist (/groups, /species, /mapping-methods) and
/openapi.json.
- /search?q= with an empty query is a 400, so it is not a link.
- the correlation endpoint's "/compute/correlation" was a POST in
`next`; it is in `actions` now.
- marker_links pointed at /markers/{marker}, which is not a route;
it points at /markers/{marker}/location.
tests/links.rs walks the graph two hops from a dozen seeds and fails on
any link that does not answer 200, plus a check that a tally leads to the
marker list and to a marker's location, and a location to that marker's
tally and its PheWAS.
|
|
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:
normalize none | log2 | log10 | sqrt | negate | zscore |
robust_zscore | rank (GN2's names `invert` and
`qnorm` accepted)
winsor_quantile clip each tail at a fraction, e.g. 0.05
winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule
They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.
**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.
Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.
* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
scores over an Acklam inverse-normal, `Params` for query/body use,
`to_json` for suggestions, and eleven unit tests -- the quantile table,
ties sharing an average rank, missing values never imputed, GN2's own
`find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
alignment (over the samples that enter the fit) and before anything reads
the values, with the note; the R/qtl permutation suggestion and GEMMA's
R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
endpoint, and the mapping bodies in `actions` carry the same fields -- a
caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
every `/compute/*` suggestion in `actions` carries it, and an impossible
pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
population-sd parity, and that effects come out on the transformed scale.
298 tests pass; the same four as before do not.
|
|
Two things that came out of reading a real response rather than a test.
**Say what a case attribute means.** `GET /groups/{group}/case-attributes`
returned names and values, and the per-trait attribute endpoint returned
values, so a caller had to guess that `SeqCvge` is a sequencing depth.
MariaDB has had the answer all along in `CaseAttribute.Description`:
SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016
Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct
Epoch BXD family subgroups. Each number with common parents. ...
Sex F= female, M= male
Both endpoints now carry it -- a `descriptions` map on the group listing, a
`description` field on the per-trait one -- and the covariate notes quote it,
so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for
1 male of each strain in 2016): categorical ..." instead of leaving the name
to be interpreted. It matters most for covariates: `SeqCvge` is a property
of the *genotype* data, its values include compound ones (`40+31L`, two
sequencing runs), and the server codes those as 20-odd levels rather than as
a depth -- a covariate fitted for the wrong reason, which the description is
the first hint against.
**Gate large computations, not every computation.** The gate was applied to
every job, so a 93-strain test request could be refused because somebody
else's 6 000-individual run held the slot -- which is how the test suite
started serialising itself behind a mutex, a workaround for the wrong
problem. What the gate is for is *large* work: a BXD panel of ~200 finishes
in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB
of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000
individuals): below that no slot is taken, nothing waits and nothing is
refused; above it the old policy holds unchanged (1 slot for gemma, 1 for
ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409
naming the run in the way).
The size is not known when the request arrives -- the panel is resolved
server-side -- so the decorator now makes a *light* claim and the handler
promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the
point where it knows, then hands it to `Gate::run` around the call that
computes, so the slot still outlives a client disconnect. A light claim
registers nothing and its drop removes nothing: only a held claim may touch
the claimant map.
* src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`,
`Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved
out of `claim_or_busy`), and `Gate::run` unchanged in spirit.
* gnauth-macros: binds the light claim; the body keeps one gate call.
* src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is
known, and the compute call wrapped in `run`; error mapping moved inside
the job so the future returns `ApiError`.
* src/rqtl.rs: `Sample` is `Clone` (the job owns its copy).
* tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are
under the threshold); rqtl keeps its lock only for the machine's sake --
concurrent `Rscript` scans would compete for cores and scratch, not for a
slot. Four new gate tests pin the threshold, that a small job never takes
the slot, that one caller cannot hold two large jobs, and that another
caller waits rather than fails.
* README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md:
what the threshold is and why it is 1 000.
Measured on :3001 after the change: two concurrent small-panel GEMMA
requests from one token both answer 200 (one was 409 before), while two
6 382-individual requests answer 502/409 -- the second gated with the usual
`compute_busy` naming the first.
277 tests pass; the same four as before do not.
|
|
The trait-values endpoint stays about trait values; a covariate needs the
study's metadata for the same samples, aligned the same way. So:
GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute}
returns one attribute's values for exactly that trait's samples, **in the
trait's own order**, which is what makes it usable without the caller
re-sorting anything:
/groups/BXD-Longevity/traits/10001/case-attributes/Sex
-> 3281 values in AgeAtDeath's order, F x 3101, M x 180
-> order identical to /traits/10001/values (asserted in the tests)
Alongside the values it reports `distinct` (each value with its count,
commonest first) and `num_values`. That is the check that decides whether
an attribute can serve as a covariate at all -- a constant one cannot --
and it saves a second request just to find out.
Attributes are per group, so an unknown one is a 404 whose `next` is the
group's attribute list: BXD has no Sex, BXD-Longevity does.
* src/db.rs (get_trait_case_attribute, get_group_trait_samples): the
group's attribute values aligned onto the trait's sample list, joining
CaseAttribute/CaseAttributeXRefNew on (InbredSetId, CaseAttributeId) as
the group endpoint does; empty text is reported as no value.
* src/routes/groups.rs: the handler, token-gated, with the response
shapes and the two `next` links (the trait's values, the group's
attributes).
* tests/case_attributes.rs: alignment against an independent /values
response, `distinct` accounting, ordering, and the 404s for BXD (no Sex)
and for a non-existent attribute.
* README.md, docs/www/ai-instruction.md (3. Case attributes): the
endpoint, and "ask per trait rather than sorting the group's attributes
yourself".
|
|
Until now /compute/{rqtl,gemma} took server paths, so the caller had to
know the deployment's layout, pick a version, and expand a trait's values
onto the panel in the file's order -- none of which the API told it. The
listing endpoint and the group-scoped request remove all three:
GET /groups/{group}/genotypes versions, newest first, no paths
POST /compute/rqtl {group, [genotype_version], samples}
POST /compute/gemma {group, [genotype_version], samples}
The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the
group's genotype/<Group>.json `genofile` list is newest-first and its
first entry is current; groups with no JSON (HET3-ITP) fall back to the
highest version number, and the listing's notes say which happened --
reads the panel from the .geno header the way GN3 does, and excludes
parents and F1s by default (GN2's `incparentsf1` off, GN3's
`get_sample_list(inc_par=False, inc_f1=False)`).
Excluding them has to subset the genotype matrix, not null-fill the
values: a null keeps an individual out of the fit but not out of GEMMA's
kinship or R/qtl's cross. So `prepare` writes a panel-subset file under
the engine's cache directory, keyed by panel, and reuses it.
Nothing in the listing or the mapping response carries a path: the
endpoint reports versions and sizes, and `notes` names the resolved
version, the panel size and how many values aligned. Explicit
`genotype_file`/`snps_file` remain as an override for files outside the
layout, and the two request shapes are mutually exclusive (422).
* src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1
exclusion, subset preparation, name alignment; 10 unit tests.
* src/routes/genotypes.rs: new endpoint, with a test that its list has no
path or file name in it.
* src/routes/{rqtl,gemma}.rs: group + genotype_version +
include_parents_f1; named `samples` aligned server-side; notes record
what resolved. gemma also accepts named samples where it used to insist
on positional trait_values.
* src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now
carry `group` and no placeholders; the dataset-scoped ones keep the
placeholder and say to add the group.
* src/routes/groups.rs: the group record points at /genotypes.
* src/config.rs: GENOTYPE_FILES (default the gn4-data directory).
* tests/genotypes.rs: new integration test (skips without the directory).
* README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the
versions, map by group" -- measured runs, the parent/F1 rule, the
subsetting requirement, and the restart list unchanged.
|
|
Fetching one trait's value per strain is usually the step before mapping
that trait, and the API can do the mapping. Those calls are POSTs with a
body, so they cannot go in `next` -- a `next` link is a GET that returns
200, which is what makes `next` safe to follow blindly and what
tests/envelope.rs relies on. Hence a fifth envelope member, `actions`,
holding method + href + description + body, and omitted entirely when
there is nothing to suggest (most responses keep the four-member shape).
The three per-trait values endpoints now carry a `POST /compute/rqtl`
and a `POST /compute/gemma` suggestion with the values already in the
body, in the response's own sample order. Genotype/SNP paths stay
bracketed placeholders: they must exist on the server, which the server
cannot know on the caller's behalf.
The instruction page says, in the same breath, never to run R/qtl or
GEMMA locally -- a local scan would use genotypes that are not the
panel's and would disagree with GeneNetwork's own results -- and that
results are cached, so repeat calls are cheap (measured: ~2 s cached
against ~20 s cold on the BXD fixtures).
Also fixes a flake the new test exposed rather than caused:
config::tests read and write process-global env vars and Config::load()
reads all of them, so the RAVANAN_* tests could fail
search_engine_keys_follow_env_over_default_layering on an unrelated
assertion. One module-level lock serialises them.
* src/response.rs (SuggestedCall, Envelope::actions/with_action/
with_actions): the new member, omitted when empty; unit tests for both.
* src/suggest.rs: new. Builds the two suggestions for either value type
(db::TraitValue, db::ProbeSetValue), with its own unit tests.
* src/routes/{traits,groups,datasets}.rs: the three values handlers.
* tests/compute_suggestions.rs: new. Suggestions present with values in
order, on all three routes; absent elsewhere; `next` still GET-only and
still returns 200.
* src/config.rs (tests): env_guard, taken by the 11 config tests.
* README.md (Success responses): the `actions` member, with an example.
* docs/www/ai-instruction.md (4a): never compute locally, why, caching,
and how to use the suggested bodies.
* docs/www/operator-checklist.md: the caching numbers.
|
|
Pure formatting sweep (import ordering, line wrapping); the tree
had accumulated rustfmt drift. Separated from any semantic change
so it stays reviewable and revertable on its own.
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Breaking: every success body is now {"data", "meta": {"state",
"took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only
`/` (plain text) and `/openapi.json` are unwrapped.
* src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers
take Ctx first and build the response with ctx.envelope(data), so
timing and the request id are never filled in by hand. `state` is
`complete` or `partial`; notes are server-authored only; `next` are
concrete relative hrefs (path_segment percent-encodes names).
* src/routes/*.rs, src/lib.rs (version): every handler returns
Envelope<..> with next links; /search is `partial` when the locus
lookup fails or a sorted 1000-hit window may be truncated, and
/publications/{pubmed_id}/traits when datasets were left out for
authorization.
* src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member.
* src/auth.rs (AuthToken): the 401 now carries request_id and took_ms
(it never had a request_id); Ctx stores RequestInfo in the request
extensions for it.
* tests: common::data / envelope_data parse every success body through
the envelope; tests/envelope.rs and test/python/test_envelope.py check
the shape, errors, and that every `next` link resolves;
tests/envelope_partial.rs checks the `partial` search without a
database; tests/openapi.rs fails if a JSON success response is not an
Envelope_* schema.
* README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document
the envelope and took_ms on errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Lists the published traits (across groups) a publication reports.
Datasets the caller may not view are left out, using the same check as
GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)).
* src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`,
`abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and
`url` (the GeneNetwork show_trait page). The three TraitInfo queries
now share one SELECT. New get_group_traits_by_abbreviation and
get_publication_traits.
* src/routes/groups.rs (group_traits): add ?abbreviation= filter.
* src/db.rs (Group), src/routes/groups.rs (groups): groups carry
`family` and `species`; add family, species and q filters.
* src/routes/search.rs (search): add `page` (gn3's page unsorted, a
page of the ranked list with `sort`); declare `pubmed_id`; hits gain
`abbreviation` next to `locus` (get_publish_annotations).
* src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the
benchmark's RDF data uses (was 4.61).
* tests, test/python, test/data/benchmark_search.json, README.md: cover
all of the above; tests/benchmark.rs now also answers rows 11, 16, 25,
30, 34 and 37 of small_benchmark.csv, and compares LOD exactly.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
Per-strain metadata beyond genotype/phenotype data (age, sex,
treatment, etc.), backed by MariaDB CaseAttribute/CaseAttributeXRefNew.
Combines GeneNetwork3's separate /case-attribute/{id}/names and
/case-attribute/{id}/values (gn3.api.case_attributes) into one
response -- neither list is large enough on its own to need splitting.
This is the only one of GeneNetwork2/3/gn-guile with the concept at
all: GeneNetwork2 delegates to GeneNetwork3 rather than storing case
attributes itself, and gn-guile doesn't have it.
db::get_group_case_attributes returns {names: [...], values: [{strain,
case_attributes: {...}}]}; values is built by folding adjacent
(strain, attribute, value) rows (ordered by strain name) rather than
grouping afterwards, since SQL already guarantees the adjacency.
Found and fixed a real bug while verifying this live: GeneNetwork3's
own query joins CaseAttribute to CaseAttributeXRefNew on
CaseAttributeId alone, but CaseAttributeId is not unique across groups
-- it's a shared vocabulary (id 37 means "Epoch" for dozens of
unrelated groups simultaneously); the actual key is
(InbredSetId, CaseAttributeId). Confirmed against this database:
without also joining on InbredSetId, BXD's result included a spurious
"Strain": "Live" attribute belonging to a different, unrelated group.
Fixed on genenetwork4's side by adding the missing join condition.
A group with no case attributes defined returns 200 with both lists
empty, not a 404 (the group itself exists) -- verified against LXS,
a real group with zero CaseAttribute rows.
tests/group_case_attributes.rs and its Python mirror cover BXD's 4
real attributes (asserting each strain has exactly those 4, not more,
to catch the join bug above), the empty-group case, missing token, and
unknown group.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs,
TestToken replacing DevToken), matching tests/get_test_token.rs and
test/python/test_get_test_token.py (renamed from
tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and
the GN_AUTH_DEV_* config keys keep their names -- they're about the
dev-token feature generically, not tied to this one route's path.
Every endpoint now requires a valid token, not just the
dataset_probeset*/traits* family: species(_by_name/_datasets),
groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all
gain an AuthToken parameter. /, /version, /openapi.json and
/get-test-token itself are the only exceptions -- a health check, a
version check, the spec describing all this, and the endpoint that
hands out a token in the first place (which would be circular if it
needed one).
Rather than repeating security(("bearer_token" = [])) on every
#[utoipa::path], ApiDoc now declares it once as the document's default
security(...), and the three exceptions besides /get-test-token (/,
/version) override it with their own empty security() -- standard
OpenAPI semantics (an operation's own security, even empty, overrides
the document default; an absent one inherits it). error::Unauthorized,
a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the
shared 401 every gated handler now references bare in its own
responses(...) list instead of writing out the same
(status = 401, description = "...", body = ApiError) tuple by hand.
Verified the resolved security per operation in a live /openapi.json.
tests/common::auth_token() (Rust) and the new
test/python/auth_helpers.auth_token() (Python, deduplicating what four
files previously each defined locally) now try this server's own
POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN --
every previously-ungated test file gains a token check and switches to
get_with_token; the whole suite, both languages, now runs with zero
manually-provisioned tokens given GENENETWORK4_DEV_MODE=true
GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains
EXTRA_ENV so each Python test class can start its own server with dev
mode on. Verified both configured (auto-fetch) and unconfigured
(GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g.
the Guix build sandbox) paths.
README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for
the rename and the new blanket token requirement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
List endpoints for the two trait domains, each the listing equivalent
of an existing singular endpoint:
- GET /groups/{group}/traits: every published (phenotype) trait in a
group (db::get_group_traits), same fields/join as
/groups/{group}/traits/{trait_id}, no token required -- matching its
singular sibling, which isn't gated either. A large group (BXD has
15000+ phenotype traits) can take several seconds and tens of
megabytes to return in full; no pagination yet, matching
GeneNetwork2's own wqflask.api.router.fetch_traits, which has no
default limit either.
- GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait
in a dataset (db::get_dataset_probesets), same fields/join as
/datasets/{dataset}/probesets/{probeset}. Gated the same way as that
singular endpoint -- a valid token is required unconditionally, and
a non-public dataset is checked against gn-auth once for the whole
dataset via a new auth::authorise_dataset, rather than once per
probeset in the list. Leaving this listing ungated would have let a
caller read out a whole non-public dataset's trait data in one
request, bypassing the gate just added to the singular endpoint.
auth::authorise_dataset wraps authorise_trait with an empty trait
name: gn-auth resolves a trait's resource purely from its dataset
name, so the trait half of a "Dataset::Trait" string never affects
which privileges come back. Verified directly against the live
instance both for an unlinked non-public dataset (403, deny) and
folded into dataset_probesets's own test for the public/embargo-aged
bypass path.
tests/group_traits.rs, tests/dataset_probesets.rs and their Python
mirrors cover both, run against a real gn-auth instance for the gated
one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* src/db.rs, src/routes/groups.rs, tests/dataset_by_name.rs,
tests/group_by_name.rs, tests/group_trait_info.rs: Run cargo fmt. No
behavior change -- this code was written before the project adopted
cargo fmt formatting in 924bcf3 and landed unformatted during the
rebase that merged the two histories.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
The expression (ProbeSet) counterpart to the PublishXRef endpoints:
ProbeSetXRef has no single Id column of its own (its primary key is
DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)),
so a trait here is addressed by dataset name plus ProbeSet.Name
instead of a single numeric trait id.
* src/db.rs (Dataset): New struct, one ProbeSetFreeze row.
(get_group_datasets): New function. Joins ProbeSetFreeze and
ProbeFreeze by group Id, matching GeneNetwork2's
wqflask.api.router.get_datasets_for_group.
(get_dataset_by_name): New function. Matches Name, Name2, FullName,
ShortName or numeric Id, matching
wqflask.api.router.get_dataset_info's ProbeSet-branch matching logic.
A dataset name is globally unique on its own, unlike a group's
traits, so this isn't scoped by group.
(ProbeSetInfo): New struct, a probe's ProbeSetXRef mapping stats plus
its ProbeSet gene annotation.
(get_dataset_probeset_info): New function, matching
wqflask.api.router.get_trait_info's ProbeSet branch.
(ProbeSetValue): New struct.
(get_dataset_probeset_values): New function, matching
wqflask.api.router.trait_sample_data's ProbeSet branch.
* src/lib.rs (DatasetProbesetLookup): New enum, the expression-dataset
equivalent of GroupTraitLookup -- distinguishes a missing dataset from
a missing probeset in an existing dataset.
(group_datasets): New handler for GET /groups/{group}/datasets.
(dataset_by_name): New handler for GET /datasets/{dataset}.
(get_dataset_probeset_info_result, dataset_probeset_info): New
function and handler for GET /datasets/{dataset}/probesets/{probeset}.
(get_dataset_probeset_values_result, dataset_probeset_values): New
function and handler for
GET /datasets/{dataset}/probesets/{probeset}/values.
(app): Register all four.
* tests/group_datasets.rs, tests/dataset_by_name.rs,
tests/dataset_probeset_info.rs, tests/dataset_probeset_values.rs,
test/python/test_group_datasets.py, test/python/test_dataset_by_name.py,
test/python/test_dataset_probeset_info.py,
test/python/test_dataset_probeset_values.py: New files. Br_U_0803_M
(dataset) / 100001_at (probeset, gene symbol Cd3g) in BXD are real rows
used as fixtures.
* tests/openapi.rs, test/python/test_openapi.py: Check the four new
paths are in the generated spec.
* README.md: Document the four new endpoints and list the new test
files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and
Publication; LEFT JOIN on the latter two since a handful of
PublishXRef rows have no matching row. SQL column aliases are written
to match mysql_common's UpperCamelCase conversion exactly, so no field
needs a #[mysql(rename = "...")] override.
(get_group_trait_info): New function.
(pool): New function. mysql::Pool::new eagerly opens its default 10
connections before returning (about 7s against this database); every
query function here used to call it per request, paying that cost on
every single API call. Builds the Pool once behind a
OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle
from then on (OnceLock::get_or_init can't be used alone since
Pool::new is fallible and get_or_try_init is still unstable).
(get_all_species, get_species_by_name, get_all_groups,
get_group_by_name, get_group_strains, get_group_trait_values,
get_group_trait_info): Use it instead of building their own Pool.
* src/lib.rs (GroupTraitLookup): Generalize over the found value (was
Vec<db::TraitValue>-specific), so group_trait_info can share it with
group_trait_values.
(get_group_trait_info_result): New function.
(group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}.
(app): Register it.
* tests/common/mod.rs: New file. get() issues a GET against
genenetwork4::app() and warns on stderr if it takes over a second --
this is what surfaced the Pool::new cost above.
* tests/species.rs, tests/species_by_name.rs, tests/groups.rs,
tests/group_by_name.rs, tests/group_strains.rs,
tests/group_trait_values.rs: Use tests/common::get instead of building
each request by hand.
* tests/group_trait_info.rs, test/python/test_group_trait_info.py: New
files.
* tests/openapi.rs, test/python/test_openapi.py: Check the new path is
in the generated spec.
* README.md (GET /groups/{group}/traits/{trait_id}): Document it.
(Tests): List the new test files; document tests/common::get's timing
warnings and how to see them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* src/db.rs (TraitValue): New struct.
(get_group_trait_values): New function. Joins PublishData, Strain and
PublishXRef by group Id and PublishXRef.Id, matching gn-guile's
gn.data.dataset:get-bxd-publish-name-value-dict generalized beyond
BXD: PublishXRef.Id alone isn't unique across groups, only the pair
(InbredSetId, Id) is, the same way GeneNetwork2's
wqflask.api.router.trait_sample_data combines trait_name with a group
or dataset to resolve a Publish trait.
* src/lib.rs (GroupTraitLookup): New enum, distinguishes a missing
group from a missing trait in an existing group.
(get_group_trait_values_result): New function, looks up a group then
its trait, returning a GroupTraitLookup.
(group_trait_values): New handler for GET
/groups/{group}/traits/{trait_id}/values. 404s as group_not_found or
trait_not_found depending on what get_group_trait_values_result
found.
(app): Register it.
* tests/group_trait_values.rs, test/python/test_group_trait_values.py:
New files.
* tests/openapi.rs, test/python/test_openapi.py: Check the new path
is in the generated spec.
* README.md (GET /groups/{group}/traits/{trait_id}/values): Document
it, including that only Publish (phenotype) traits are supported so
far, not ProbeSet or Geno traits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Line-wrapping and canonical import ordering only (verified with
git diff -w). Establishes rustfmt's output as the canonical style
before the search/correlation work lands.
|
|
Summaries appear in the generated /openapi.json and make the spec
readable on its own. The correlation endpoint is a stub: it validates
the request shape and returns an envelope with canned data and a null
next pointer, establishing the contract ahead of the real engine.
|
|
|
|
tower_http::request_id::RequestId, not axum::http::RequestId.
|
|
* src/config.rs: New module. Layered configuration,
defaults -> GN4_CONF -> GN4_SECRETS -> environment, mirroring
GeneNetwork3's SQL_URI convention so the two services can share
configuration. Missing config files are ignored.
* src/response.rs: New module. The descriptive success envelope
(data + next actions) from the foundation design note
(topics/gn4/gn4-api-foundation.gmi). Not yet adopted by existing
endpoints; that is a deliberate wire-format change to agree on
before it ships.
* src/routes/mod.rs, species.rs, groups.rs: Move the species and
groups handlers out of lib.rs into one module per domain, per the
foundation structure. Handlers are pub; lib.rs just registers them.
* src/lib.rs (app): Register the moved routes. hello_world and
version predate the domain split and stay as top-level utility
routes.
* src/db.rs (sql_uri): Resolve through config::Config instead of
reading the environment directly.
* src/main.rs: Bind the port from config::Config.
|