about summary refs log tree commit diff
path: root/src/routes/groups.rs
AgeCommit message (Collapse)Author
5 daysgroups: the design in /groups/{group}, and a drift check against GN2's ParInfoPjotr
Two follow-ups to the parental-design endpoint. /groups/{group} now carries the headline in its notes -- "Parental design: C57BL/6J (maternal) x DBA/2J (paternal), F1s B6D2F1 / D2B6F1; a mapping excludes the parents and F1s by default" -- plus a next link to the full design. "Which cross is this?" is two strain names, and needing a second request to see them was the only reason the endpoint had to be called first. A group with no metadata says so in the same note, pointing at parents_and_f1s.json and at `individuals.exclude`, rather than leaving the reader to wonder why nothing was dropped. The metadata is the same fact in two places (GN2's ParInfo, our JSON) and it drifted once already, silently: the paternal/maternal labels were swapped against GN2's unpacking, and nothing noticed because the exclusion rule takes both strains regardless of role. scripts/check-parental-metadata.py parses ParInfo with ast (no gn2 import) and compares it group by group, reporting disagreements per role and treating groups the JSON knows but GN2 does not as informational. tests/parental_metadata.rs runs it while a GN2 checkout is present (GN2_CHECKOUT, or a sibling path) and skips once GN2 is retired -- at which point the JSON is the only copy and there is nothing to drift from. It reports 23 groups reproduced, 1 extra (BXD-Longevity).
5 daysgroups: an endpoint for a family's parental designPjotr
GET /groups/{group}/parents-and-f1s answers the question the mapping notes only implied. It reports the F1 strains, the maternal and paternal strain (from the deployment's parents_and_f1s.json, ported from GN2's ParInfo -- all 24 groups), the @type of the group's genotype file, the individuals each role names where the metadata names them (BXD-Longevity does, most do not), and -- the part a caller needs -- which members of the current panel the default rule drops and which the `F1` name heuristic catches. Two things that came out of building it and comparison with GN2: * the JSON's paternal/maternal labels were INVERTED against GN2's reading. GN2 unpacks ParInfo as `f1, f12, maternal, paternal`, so BXD's maternal strain is C57BL/6J; our file said paternal. No mapping was affected (non_strains takes both regardless of role) but the metadata was wrong, and anything using "maternal" later -- reciprocal F1 direction, the @mat/@pat note -- would have been backwards. The file is rewritten from ParInfo itself (ast-parsed, not retyped), 23 groups' roles filled in and the 24th (BXD-Longevity, which GN2 does not have) left as it was. * an RI set has two kinds of F1 name: the design's (B6D2F1) and the panel's own intercross individuals (C57BL/6JxBXD037F1). The endpoint reports excluded_from_panel and excluded_by_name separately and says which is doing the work, because "0 declared, 37 by name" is otherwise confusing. Empty F1 names (C57BL-6JxC57BL-6NJF2 has two) are dropped rather than becoming exclusion entries that match nothing. genotypes::non_strains is rebuilt on the same typed parser (DesignRole / ParentalDesign), so the rule and the endpoint cannot disagree. tests: a unit test for the roles, empty-name handling and the unlisted-group source, and an integration test asserting BXD's maternal/paternal orientation and the note.
7 daysnext: every link a client is offered is a call it can makePjotr
tests/envelope.rs follows every href in every reply, and running it end to end found six that were not addresses: - /species, /species/{name}/datasets and /groups/{group}/datasets offered /species/{species} and /datasets/{dataset} -- templates, which 404 when followed. Each now points at a real row from the listing in hand, and the description says any other name can go in its place. - /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the listing it already links is the callable one, so the shape moved into that link's description. - /traits/{dataset}/{trait}/values offered a GET /traits/{ds}/{trait}/correlation, which does not exist: correlation is POST /compute/correlation and that reply already carries it in "actions". Removed, with the reason written where it was. - the same reply built its /groups/... links from the *dataset* name. BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is a 404 (no genotype files for it). db::get_group_name resolves the group from InbredSet.Id and the links use it; when a dataset has no group the group links are simply absent. - /groups/{g}/phewas with no parameters is a 422 ("give either locus, or chr and mb"), so that link is now a shape with placeholders. Placeholders stay where the caller must supply a value (.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips them only when the description says what to fill in -- otherwise a client reading the next list cannot tell a call from a template. Two stale expectations alongside: /version carries the root links now (it is the natural first call), and /search's results put its own follow-ups first and the root links after them. Tests are honest about their environment too: links skips when no search index is configured (an empty XAPIAN_DB_PATH answers 502), and the envelope crawl exempts /openapi.json, the API document, which is deliberately not enveloped.
8 daysgenotypes: a trait summarised per genotype class at one markerPjotr
The question that follows "why do GEMMA and R/qtl2 peak at different markers": what does the trait look like within each genotype class at one marker. The frequencies endpoints report the distribution of calls, not the values behind them, and the answer must not be "serve the matrix" -- this API never does. GET /groups/{group}/genotypes/{version}/markers/{marker}/classes?trait=<id> -> {"A": {"n", "mean", "sd", "min", "max"}, "H": {...}, "B": {...}} The calls are joined to the trait's per-strain values inside the server (the same lookup, authorization and 404s as /groups/{g}/traits/{id}/values) and reduced to those five numbers per class. Nothing in the payload names an animal. A class with fewer than ten animals is called out in notes, because a peak resting on two is a different claim from one resting on a hundred. It settles the DOL trait 10003 question. At GEMMA's top marker JAX00418132 (chr16 36.35 Mb) the classes are A n=2 mean 1.8, H n=20 mean 2.53, B n=171 mean 3.95 -- the A class is two animals and they are the trait's two lowest values. R/qtl2 drops H on a riself panel (R/qtl's convert2riself), so it keeps 173 animals there but only 2 of one class: nearly monomorphic, which is why its scan peaks elsewhere. GEMMA keeps the 20 hets at dosage 0.5, and that is where its sharper p-value comes from. GEMMA's next two hits are the same pattern in opposite directions (one A among 171, then two B among 171), so the chr16 block is a haplotype carried by a handful of extreme animals, not a population effect. Also fixes the reading of GEMMA's `af`: it is mean(dosage)/2 under the BIMBAM coding this deployment ships (B=1, H=0.5, A=0), so 0.469 for JAX00418132 is half the B-allele frequency 0.935, not the A-allele frequency 0.065. The bimbam file and DOL.geno agree at that marker. tests/frequencies.rs gains a route test: the class fields, A < B, the small-class note, the "not served" note, no individual names anywhere in the payload, and a 404 for a trait the group does not have.
8 daysapi: every response suggests where to go nextPjotr
A caller that knows only /version should be able to reach anything by following links. Until now `next` was thin and uneven: the genotype subtree offered none at all, the group record offered one, /version offered nothing, and two links it did offer could not be followed. * suggest:: now builds the graph: root_links (groups, species, mapping-methods, openapi), genotype_version_links (frequencies, marker names, a marker's location and tally, the group, the scanning actions), unlisted_genotype_links, marker_links (location, tally, marker list, panel tally, PheWAS at that marker), group_links and trait_links. Envelope::with_links / with_links_except add them, deduplicating and never suggesting the response the caller is already reading. * Wired through the genotype surface (listing, version detail, markers, location), both frequencies routes, the compute endpoints, groups, traits, both value routes, species, datasets, publications, accessions, search, mapping methods, phewas, correlation and /version. * Fixed what was not followable, because `next` promises a GET that returns 200 and a client is the one that finds out otherwise: - /groups/{g}/phewas without a locus is a 422. Bare PheWAS links are gone; comput endpoints now link ?locus=<their strongest marker> (suggest::peak_marker_of, computed before the rows move). - /datasets and /publications are not routes; root_links offers the listings that exist (/groups, /species, /mapping-methods) and /openapi.json. - /search?q= with an empty query is a 400, so it is not a link. - the correlation endpoint's "/compute/correlation" was a POST in `next`; it is in `actions` now. - marker_links pointed at /markers/{marker}, which is not a route; it points at /markers/{marker}/location. tests/links.rs walks the graph two hops from a dozen seeds and fails on any link that does not answer 200, plus a check that a tally leads to the marker list and to a marker's location, and a location to that marker's tally and its PheWAS.
9 daysapi: normalize trait values, and winsorise the tails, before mappingPjotr
GeneNetwork2's trait page offers a set of normalisations, and its mapping page warns about outliers without ever fixing them. This adds both, under one spelling that `/compute/rqtl`, `/compute/gemma` and `/groups/{g}/traits/{id}/values` all share: normalize none | log2 | log10 | sqrt | negate | zscore | robust_zscore | rank (GN2's names `invert` and `qnorm` accepted) winsor_quantile clip each tail at a fraction, e.g. 0.05 winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule They run in a fixed, reported order -- transform, then winsorise, then standardise -- so clipping acts on the scale being analysed and the result lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The note gives the steps *in the order they ran* (`winsorized at Tukey fences, then zscore`), the count clipped in each tail, and how many values sit outside Tukey fences -- the warning GN2 gives before mapping and this API gave nowhere. **Checked against genenetwork.org itself.** GN2's public trait page embeds its own reference values (`js_data.qnorm_values` / `zscore_values`, the `data-qnorm`/`data-zscore` attributes), so there is an oracle for the two transforms whose definition matters. On BXD trait 10010 (28 samples): `rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's three-decimal rounding. Getting there found a real disagreement: GN2 uses `scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use the **population** sd; ours used the sample sd and came out `sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2. Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based inverse-normal (Blom) transform, so it is called `rank` here; and its `sqrt` silently adds 1 when a value is in [0,1], which shifts every effect size -- a negative value is refused instead. * src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom scores over an Acklam inverse-normal, `Params` for query/body use, `to_json` for suggestions, and eleven unit tests -- the quantile table, ties sharing an average rank, missing values never imputed, GN2's own `find_outliers` example, and the note's wording. * src/routes/{rqtl,gemma}.rs: the three fields, applied **after** alignment (over the samples that enter the fit) and before anything reads the values, with the note; the R/qtl permutation suggestion and GEMMA's R/qtl suggestion carry the pipeline. * src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values endpoint, and the mapping bodies in `actions` carry the same fields -- a caller who fetches z-scores must map those numbers, not the raw ones. * tests/groups.rs: values are mean 0 / population sd 1, the note says so, every `/compute/*` suggestion in `actions` carries it, and an impossible pipeline is a 422 `invalid_normalization`. * README.md, docs/www/SKILL.md: what each name means, the order, the population-sd parity, and that effects come out on the transformed scale. 298 tests pass; the same four as before do not.
10 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
11 dayscase-attributes: one attribute for one trait, in that trait's sample orderPjotr
The trait-values endpoint stays about trait values; a covariate needs the study's metadata for the same samples, aligned the same way. So: GET /groups/{group}/traits/{trait_id}/case-attributes/{attribute} returns one attribute's values for exactly that trait's samples, **in the trait's own order**, which is what makes it usable without the caller re-sorting anything: /groups/BXD-Longevity/traits/10001/case-attributes/Sex -> 3281 values in AgeAtDeath's order, F x 3101, M x 180 -> order identical to /traits/10001/values (asserted in the tests) Alongside the values it reports `distinct` (each value with its count, commonest first) and `num_values`. That is the check that decides whether an attribute can serve as a covariate at all -- a constant one cannot -- and it saves a second request just to find out. Attributes are per group, so an unknown one is a 404 whose `next` is the group's attribute list: BXD has no Sex, BXD-Longevity does. * src/db.rs (get_trait_case_attribute, get_group_trait_samples): the group's attribute values aligned onto the trait's sample list, joining CaseAttribute/CaseAttributeXRefNew on (InbredSetId, CaseAttributeId) as the group endpoint does; empty text is reported as no value. * src/routes/groups.rs: the handler, token-gated, with the response shapes and the two `next` links (the trait's values, the group's attributes). * tests/case_attributes.rs: alignment against an independent /values response, `distinct` accounting, ordering, and the 404s for BXD (no Sex) and for a non-existent attribute. * README.md, docs/www/ai-instruction.md (3. Case attributes): the endpoint, and "ask per trait rather than sorting the group's attributes yourself".
11 daysgenotypes: list a group's genotype files, and map by group + versionPjotr
Until now /compute/{rqtl,gemma} took server paths, so the caller had to know the deployment's layout, pick a version, and expand a trait's values onto the panel in the file's order -- none of which the API told it. The listing endpoint and the group-scoped request remove all three: GET /groups/{group}/genotypes versions, newest first, no paths POST /compute/rqtl {group, [genotype_version], samples} POST /compute/gemma {group, [genotype_version], samples} The resolver (src/genotypes.rs) applies GeneNetwork2's rule -- the group's genotype/<Group>.json `genofile` list is newest-first and its first entry is current; groups with no JSON (HET3-ITP) fall back to the highest version number, and the listing's notes say which happened -- reads the panel from the .geno header the way GN3 does, and excludes parents and F1s by default (GN2's `incparentsf1` off, GN3's `get_sample_list(inc_par=False, inc_f1=False)`). Excluding them has to subset the genotype matrix, not null-fill the values: a null keeps an individual out of the fit but not out of GEMMA's kinship or R/qtl's cross. So `prepare` writes a panel-subset file under the engine's cache directory, keyed by panel, and reuses it. Nothing in the listing or the mapping response carries a path: the endpoint reports versions and sizes, and `notes` names the resolved version, the panel size and how many values aligned. Explicit `genotype_file`/`snps_file` remain as an override for files outside the layout, and the two request shapes are mutually exclusive (422). * src/genotypes.rs: new. Discovery, resolution, panel parse, parent/F1 exclusion, subset preparation, name alignment; 10 unit tests. * src/routes/genotypes.rs: new endpoint, with a test that its list has no path or file name in it. * src/routes/{rqtl,gemma}.rs: group + genotype_version + include_parents_f1; named `samples` aligned server-side; notes record what resolved. gemma also accepts named samples where it used to insist on positional trait_values. * src/suggest.rs: suggestions from /groups/{group}/traits/{id}/values now carry `group` and no placeholders; the dataset-scoped ones keep the placeholder and say to add the group. * src/routes/groups.rs: the group record points at /genotypes. * src/config.rs: GENOTYPE_FILES (default the gn4-data directory). * tests/genotypes.rs: new integration test (skips without the directory). * README.md, docs/genotypes.md, docs/www/*: the flow is now "ask for the versions, map by group" -- measured runs, the parent/F1 rule, the subsetting requirement, and the restart list unchanged.
11 daysvalues: suggest R/qtl and GEMMA in the envelope's `actions`Pjotr
Fetching one trait's value per strain is usually the step before mapping that trait, and the API can do the mapping. Those calls are POSTs with a body, so they cannot go in `next` -- a `next` link is a GET that returns 200, which is what makes `next` safe to follow blindly and what tests/envelope.rs relies on. Hence a fifth envelope member, `actions`, holding method + href + description + body, and omitted entirely when there is nothing to suggest (most responses keep the four-member shape). The three per-trait values endpoints now carry a `POST /compute/rqtl` and a `POST /compute/gemma` suggestion with the values already in the body, in the response's own sample order. Genotype/SNP paths stay bracketed placeholders: they must exist on the server, which the server cannot know on the caller's behalf. The instruction page says, in the same breath, never to run R/qtl or GEMMA locally -- a local scan would use genotypes that are not the panel's and would disagree with GeneNetwork's own results -- and that results are cached, so repeat calls are cheap (measured: ~2 s cached against ~20 s cold on the BXD fixtures). Also fixes a flake the new test exposed rather than caused: config::tests read and write process-global env vars and Config::load() reads all of them, so the RAVANAN_* tests could fail search_engine_keys_follow_env_over_default_layering on an unrelated assertion. One module-level lock serialises them. * src/response.rs (SuggestedCall, Envelope::actions/with_action/ with_actions): the new member, omitted when empty; unit tests for both. * src/suggest.rs: new. Builds the two suggestions for either value type (db::TraitValue, db::ProbeSetValue), with its own unit tests. * src/routes/{traits,groups,datasets}.rs: the three values handlers. * tests/compute_suggestions.rs: new. Suggestions present with values in order, on all three routes; absent elsewhere; `next` still GET-only and still returns 200. * src/config.rs (tests): env_guard, taken by the 11 config tests. * README.md (Success responses): the `actions` member, with an example. * docs/www/ai-instruction.md (4a): never compute locally, why, caching, and how to use the suggested bodies. * docs/www/operator-checklist.md: the caching numbers.
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-14config: GN4_DB_MODE (mariadb optional on dev) + no-DB hardeningAlexander_Kabui
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add GN4_DB_MODE=dev|production in the layered config; dev runs with no configured DB; fail-closed parse (a typo refuses). * src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db): mint a typed 503 db_unavailable at the db::pool() choke point, distinguishing Unavailable from Query. * src/main.rs: loud no-DB startup line; production refuses to start (exit non-zero) before axum::serve. * src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs: route every DB error through ApiError::from_db. * src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable. * tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503). His ask (DM 2026-10-12): "Make it an option to switch it off for development machines? I mean mariadb." The no-DB hardening is ours (review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs owner sign-off.
2026-10-13auth: add gnauth compile-time tags (secured/token_only/disabled)Pjotr
Adds a gnauth-macros proc-macro crate and a private-field Authorized capability type in src/auth.rs, so #[gnauth(secured)] is a real, compile-time-checked guarantee that a handler performs the gn-auth resource-authorisation round trip before sharing trait/probeset values, rather than just a comment. #[gnauth(token_only)] marks handlers that require a token but skip the round trip, and #[gnauth(disabled)] marks the two handlers that need no token at all. Also fixes groups.rs: group_trait_values, group_trait_info and group_traits previously performed no gn-auth round trip at all for published-trait data; they now resolve the group's PublishFreeze row and call the same auth::check used elsewhere, denying by default when no such row exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07Add GET /groups/{group}/case-attributesPjotr Prins
Per-strain metadata beyond genotype/phenotype data (age, sex, treatment, etc.), backed by MariaDB CaseAttribute/CaseAttributeXRefNew. Combines GeneNetwork3's separate /case-attribute/{id}/names and /case-attribute/{id}/values (gn3.api.case_attributes) into one response -- neither list is large enough on its own to need splitting. This is the only one of GeneNetwork2/3/gn-guile with the concept at all: GeneNetwork2 delegates to GeneNetwork3 rather than storing case attributes itself, and gn-guile doesn't have it. db::get_group_case_attributes returns {names: [...], values: [{strain, case_attributes: {...}}]}; values is built by folding adjacent (strain, attribute, value) rows (ordered by strain name) rather than grouping afterwards, since SQL already guarantees the adjacency. Found and fixed a real bug while verifying this live: GeneNetwork3's own query joins CaseAttribute to CaseAttributeXRefNew on CaseAttributeId alone, but CaseAttributeId is not unique across groups -- it's a shared vocabulary (id 37 means "Epoch" for dozens of unrelated groups simultaneously); the actual key is (InbredSetId, CaseAttributeId). Confirmed against this database: without also joining on InbredSetId, BXD's result included a spurious "Strain": "Live" attribute belonging to a different, unrelated group. Fixed on genenetwork4's side by adding the missing join condition. A group with no case attributes defined returns 200 with both lists empty, not a 404 (the group itself exists) -- verified against LXS, a real group with zero CaseAttribute rows. tests/group_case_attributes.rs and its Python mirror cover BXD's 4 real attributes (asserting each strain has exactly those 4, not more, to catch the join bug above), the empty-group case, missing token, and unknown group. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-05Rename /dev/token to /get-test-token; require a token everywherePjotr Prins
Rename: POST /dev/token -> POST /get-test-token (src/routes/dev.rs, TestToken replacing DevToken), matching tests/get_test_token.rs and test/python/test_get_test_token.py (renamed from tests/dev_token.rs/test_dev_token.py). Config::dev_token_enabled() and the GN_AUTH_DEV_* config keys keep their names -- they're about the dev-token feature generically, not tied to this one route's path. Every endpoint now requires a valid token, not just the dataset_probeset*/traits* family: species(_by_name/_datasets), groups(_by_name/_strains/_traits*/_datasets) and dataset_by_name all gain an AuthToken parameter. /, /version, /openapi.json and /get-test-token itself are the only exceptions -- a health check, a version check, the spec describing all this, and the endpoint that hands out a token in the first place (which would be circular if it needed one). Rather than repeating security(("bearer_token" = [])) on every #[utoipa::path], ApiDoc now declares it once as the document's default security(...), and the three exceptions besides /get-test-token (/, /version) override it with their own empty security() -- standard OpenAPI semantics (an operation's own security, even empty, overrides the document default; an absent one inherits it). error::Unauthorized, a new #[derive(utoipa::IntoResponses)] type wrapping ApiError, is the shared 401 every gated handler now references bare in its own responses(...) list instead of writing out the same (status = 401, description = "...", body = ApiError) tuple by hand. Verified the resolved security per operation in a live /openapi.json. tests/common::auth_token() (Rust) and the new test/python/auth_helpers.auth_token() (Python, deduplicating what four files previously each defined locally) now try this server's own POST /get-test-token first, falling back to GN4_TEST_AUTH_TOKEN -- every previously-ungated test file gains a token check and switches to get_with_token; the whole suite, both languages, now runs with zero manually-provisioned tokens given GENENETWORK4_DEV_MODE=true GENENETWORK4_BIND=127.0.0.1 for the test run. ServerTestCase gains EXTRA_ENV so each Python test class can start its own server with dev mode on. Verified both configured (auto-fetch) and unconfigured (GN4_TEST_AUTH_TOKEN fallback, and full skip when neither is set, e.g. the Guix build sandbox) paths. README.md, docs/gn-auth.md and RELEASE_NOTES.md updated throughout for the rename and the new blanket token requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits and GET /datasets/{dataset}/probesetsPjotr Prins
List endpoints for the two trait domains, each the listing equivalent of an existing singular endpoint: - GET /groups/{group}/traits: every published (phenotype) trait in a group (db::get_group_traits), same fields/join as /groups/{group}/traits/{trait_id}, no token required -- matching its singular sibling, which isn't gated either. A large group (BXD has 15000+ phenotype traits) can take several seconds and tens of megabytes to return in full; no pagination yet, matching GeneNetwork2's own wqflask.api.router.fetch_traits, which has no default limit either. - GET /datasets/{dataset}/probesets: every expression (ProbeSet) trait in a dataset (db::get_dataset_probesets), same fields/join as /datasets/{dataset}/probesets/{probeset}. Gated the same way as that singular endpoint -- a valid token is required unconditionally, and a non-public dataset is checked against gn-auth once for the whole dataset via a new auth::authorise_dataset, rather than once per probeset in the list. Leaving this listing ungated would have let a caller read out a whole non-public dataset's trait data in one request, bypassing the gate just added to the singular endpoint. auth::authorise_dataset wraps authorise_trait with an empty trait name: gn-auth resolves a trait's resource purely from its dataset name, so the trait half of a "Dataset::Trait" string never affects which privileges come back. Verified directly against the live instance both for an unlinked non-public dataset (403, deny) and folded into dataset_probesets's own test for the public/embargo-aged bypass path. tests/group_traits.rs, tests/dataset_probesets.rs and their Python mirrors cover both, run against a real gn-auth instance for the gated one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Format with cargo fmt, matching 924bcf3's project-wide pass.Pjotr Prins
* src/db.rs, src/routes/groups.rs, tests/dataset_by_name.rs, tests/group_by_name.rs, tests/group_trait_info.rs: Run cargo fmt. No behavior change -- this code was written before the project adopted cargo fmt formatting in 924bcf3 and landed unformatted during the rebase that merged the two histories. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add ProbeSetXRef endpoints: datasets, probesets, probeset values.Pjotr Prins
The expression (ProbeSet) counterpart to the PublishXRef endpoints: ProbeSetXRef has no single Id column of its own (its primary key is DataId; the natural key is the pair (ProbeSetFreezeId, ProbeSetId)), so a trait here is addressed by dataset name plus ProbeSet.Name instead of a single numeric trait id. * src/db.rs (Dataset): New struct, one ProbeSetFreeze row. (get_group_datasets): New function. Joins ProbeSetFreeze and ProbeFreeze by group Id, matching GeneNetwork2's wqflask.api.router.get_datasets_for_group. (get_dataset_by_name): New function. Matches Name, Name2, FullName, ShortName or numeric Id, matching wqflask.api.router.get_dataset_info's ProbeSet-branch matching logic. A dataset name is globally unique on its own, unlike a group's traits, so this isn't scoped by group. (ProbeSetInfo): New struct, a probe's ProbeSetXRef mapping stats plus its ProbeSet gene annotation. (get_dataset_probeset_info): New function, matching wqflask.api.router.get_trait_info's ProbeSet branch. (ProbeSetValue): New struct. (get_dataset_probeset_values): New function, matching wqflask.api.router.trait_sample_data's ProbeSet branch. * src/lib.rs (DatasetProbesetLookup): New enum, the expression-dataset equivalent of GroupTraitLookup -- distinguishes a missing dataset from a missing probeset in an existing dataset. (group_datasets): New handler for GET /groups/{group}/datasets. (dataset_by_name): New handler for GET /datasets/{dataset}. (get_dataset_probeset_info_result, dataset_probeset_info): New function and handler for GET /datasets/{dataset}/probesets/{probeset}. (get_dataset_probeset_values_result, dataset_probeset_values): New function and handler for GET /datasets/{dataset}/probesets/{probeset}/values. (app): Register all four. * tests/group_datasets.rs, tests/dataset_by_name.rs, tests/dataset_probeset_info.rs, tests/dataset_probeset_values.rs, test/python/test_group_datasets.py, test/python/test_dataset_by_name.py, test/python/test_dataset_probeset_info.py, test/python/test_dataset_probeset_values.py: New files. Br_U_0803_M (dataset) / 100001_at (probeset, gene symbol Cd3g) in BXD are real rows used as fixtures. * tests/openapi.rs, test/python/test_openapi.py: Check the four new paths are in the generated spec. * README.md: Document the four new endpoints and list the new test files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}; time DB tests; share pool.Pjotr Prins
* src/db.rs (TraitInfo): New struct. Joins PublishXRef, Phenotype and Publication; LEFT JOIN on the latter two since a handful of PublishXRef rows have no matching row. SQL column aliases are written to match mysql_common's UpperCamelCase conversion exactly, so no field needs a #[mysql(rename = "...")] override. (get_group_trait_info): New function. (pool): New function. mysql::Pool::new eagerly opens its default 10 connections before returning (about 7s against this database); every query function here used to call it per request, paying that cost on every single API call. Builds the Pool once behind a OnceLock<Mutex<Option<Pool>>> and shares the cheap, Arc-backed handle from then on (OnceLock::get_or_init can't be used alone since Pool::new is fallible and get_or_try_init is still unstable). (get_all_species, get_species_by_name, get_all_groups, get_group_by_name, get_group_strains, get_group_trait_values, get_group_trait_info): Use it instead of building their own Pool. * src/lib.rs (GroupTraitLookup): Generalize over the found value (was Vec<db::TraitValue>-specific), so group_trait_info can share it with group_trait_values. (get_group_trait_info_result): New function. (group_trait_info): New handler for GET /groups/{group}/traits/{trait_id}. (app): Register it. * tests/common/mod.rs: New file. get() issues a GET against genenetwork4::app() and warns on stderr if it takes over a second -- this is what surfaced the Pool::new cost above. * tests/species.rs, tests/species_by_name.rs, tests/groups.rs, tests/group_by_name.rs, tests/group_strains.rs, tests/group_trait_values.rs: Use tests/common::get instead of building each request by hand. * tests/group_trait_info.rs, test/python/test_group_trait_info.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}): Document it. (Tests): List the new test files; document tests/common::get's timing warnings and how to see them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Add GET /groups/{group}/traits/{trait_id}/values, publish trait data.Pjotr Prins
* src/db.rs (TraitValue): New struct. (get_group_trait_values): New function. Joins PublishData, Strain and PublishXRef by group Id and PublishXRef.Id, matching gn-guile's gn.data.dataset:get-bxd-publish-name-value-dict generalized beyond BXD: PublishXRef.Id alone isn't unique across groups, only the pair (InbredSetId, Id) is, the same way GeneNetwork2's wqflask.api.router.trait_sample_data combines trait_name with a group or dataset to resolve a Publish trait. * src/lib.rs (GroupTraitLookup): New enum, distinguishes a missing group from a missing trait in an existing group. (get_group_trait_values_result): New function, looks up a group then its trait, returning a GroupTraitLookup. (group_trait_values): New handler for GET /groups/{group}/traits/{trait_id}/values. 404s as group_not_found or trait_not_found depending on what get_group_trait_values_result found. (app): Register it. * tests/group_trait_values.rs, test/python/test_group_trait_values.py: New files. * tests/openapi.rs, test/python/test_openapi.py: Check the new path is in the generated spec. * README.md (GET /groups/{group}/traits/{trait_id}/values): Document it, including that only Publish (phenotype) traits are supported so far, not ProbeSet or Geno traits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-04Format all Rust sources with cargo fmt (no behavior change)Alexander_Kabui
Line-wrapping and canonical import ordering only (verified with git diff -w). Establishes rustfmt's output as the canonical style before the search/correlation work lands.
2026-10-04Document all endpoints with utoipa summaries; add POST /api/v2/correlation stubAlexander_Kabui
Summaries appear in the generated /openapi.json and make the spec readable on its own. The correlation endpoint is a stub: it validates the request shape and returns an envelope with canned data and a null next pointer, establishing the contract ahead of the real engine.
2026-10-03Polish: import ordering, deterministic load_file tests.Alexander_Kabui
2026-10-03Fix RequestId import in routes modules: the extractor type isAlexander_Kabui
tower_http::request_id::RequestId, not axum::http::RequestId.
2026-10-03Add foundation structure: config layering, response envelope, routes modules.Alexander_Kabui
* src/config.rs: New module. Layered configuration, defaults -> GN4_CONF -> GN4_SECRETS -> environment, mirroring GeneNetwork3's SQL_URI convention so the two services can share configuration. Missing config files are ignored. * src/response.rs: New module. The descriptive success envelope (data + next actions) from the foundation design note (topics/gn4/gn4-api-foundation.gmi). Not yet adopted by existing endpoints; that is a deliberate wire-format change to agree on before it ships. * src/routes/mod.rs, species.rs, groups.rs: Move the species and groups handlers out of lib.rs into one module per domain, per the foundation structure. Handlers are pub; lib.rs just registers them. * src/lib.rs (app): Register the moved routes. hello_world and version predate the domain split and stay as top-level utility routes. * src/db.rs (sql_uri): Resolve through config::Config instead of reading the environment directly. * src/main.rs: Bind the port from config::Config.