about summary refs log tree commit diff
path: root/tests/search.rs
AgeCommit message (Collapse)Author
8 daysnext: every link a client is offered is a call it can makePjotr
tests/envelope.rs follows every href in every reply, and running it end to end found six that were not addresses: - /species, /species/{name}/datasets and /groups/{group}/datasets offered /species/{species} and /datasets/{dataset} -- templates, which 404 when followed. Each now points at a real row from the listing in hand, and the description says any other name can go in its place. - /datasets/{name} offered /datasets/{dataset}/probesets/{probeset}: the listing it already links is the callable one, so the shape moved into that link's description. - /traits/{dataset}/{trait}/values offered a GET /traits/{ds}/{trait}/correlation, which does not exist: correlation is POST /compute/correlation and that reply already carries it in "actions". Removed, with the reason written where it was. - the same reply built its /groups/... links from the *dataset* name. BXDPublish's traits live in group BXD, and /groups/BXDPublish/phewas is a 404 (no genotype files for it). db::get_group_name resolves the group from InbredSet.Id and the links use it; when a dataset has no group the group links are simply absent. - /groups/{g}/phewas with no parameters is a 422 ("give either locus, or chr and mb"), so that link is now a shape with placeholders. Placeholders stay where the caller must supply a value (.../markers/{marker}/classes?trait={trait_id}), but the crawl now skips them only when the description says what to fill in -- otherwise a client reading the next list cannot tell a call from a template. Two stale expectations alongside: /version carries the root links now (it is the natural first call), and /search's results put its own follow-ups first and the root links after them. Tests are honest about their environment too: links skips when no search index is configured (an empty XAPIAN_DB_PATH answers 502), and the envelope crawl exempts /openapi.json, the API document, which is deliberately not enveloped.
10 daysfix: DOL maps again on both engines, and the four stale testsPjotr
DOL (200 individuals, 112 913 markers, one genotype version) failed on R/qtl with "There are missing chromosome IDs" and on GEMMA with a bare wrapper bail-out. Two unrelated causes, both in our own code. **R/qtl: a genotype code collided with a chromosome name.** DOL's header says `@unk:X` -- X is the code for a missing genotype -- and the wrapper passed that to `read.csv(na.strings=...)` for the *whole* file, so every marker on the X chromosome lost its chromosome ID and `read.cross.csv` refused the map. The unknown code now applies to the genotype columns only. DOL scans: 112 913 markers in 42 s, where before it never got past reading the file. Same fix in scripts/qtl_fitqtl.R, which shares the read. **GEMMA: its leave-one-chromosome-out run cannot fit the Y.** With LOCO the DOL panel is scanned per chromosome, and the Y run dies inside BLAS -- `Enforce failed for N>0 in src/fastblas.cpp ... fast_cblas_dgemm` -- taking the whole request with it (a 502 whose detail is the wrapper's banner). The mitochondrial chromosome has the same shape of problem. `prepare` for the GEMMA engine now writes the bimbam pair without the Y and M markers (cached beside the individual subset, line-aligned so the geno and snps stay in step), and the note says what was left out: "46 Y/M markers left out (GEMMA's LOCO cannot fit them)". DOL maps: 106 515 rows in 49 s. **And the four tests that had been failing since before this work**, all stale expectations rather than product bugs: * `benchmark_questions` pinned two traits to "page 2" of a 100-per-page search when they rank ~190th and ~219th of 246 -- one re-index pushes them over. It now walks up to three pages and asserts presence and LOD, not the page; and the helper `search()` pages, because a benchmark row can name a trait that ranks outside the first hundred of its own query. * `no_db_mode_degrades_gracefully` asserted exactly one hit for "alcohol" from the index alone. The index grew; the behaviour under test is that the search still answers, within the limit, with the database-backed fields null. That is what it asserts now. * `search_proxy_scenarios` and `search_parity_replay` never pinned the engine they test, so an ambient `SEARCH_ENGINE=native` made the proxy scenarios answer from the local index: a 200 where a 422 was expected, and "expected 2 upstream calls, got 0" for every replayed query. Both now set `SEARCH_ENGINE=forward`. * src/genotypes.rs: `DEGENERATE_CHROMOSOMES`, `drop_degenerate_chromosomes`, `Prepared.markers_dropped`. * src/routes/gemma.rs: also the 500 that `eb5320d` introduced -- counting parents/F1s from `panel - keep` instead of subtracting the exclusion *set*, whose length can exceed the panel (subtract overflow). * scripts/rqtl_wrapper.R, scripts/qtl_fitqtl.R: na.strings scoped to the genotype columns. * tests/{benchmark,no_db,search,search_parity}.rs: as above. Verified through the API on DOL trait 10003: R/qtl 112 913 rows, GEMMA 106 515 rows. gemma/rqtl/fitqtl suites green (14, 10, 4).
2026-10-14rustfmt: apply rustfmt across the treeAlexander_Kabui
Pure formatting sweep (import ordering, line wrapping); the tree had accumulated rustfmt drift. Separated from any semantic change so it stays reviewable and revertable on its own.
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07* src/routes/search.rs (search): add lod, locus, and sort/order.Pjotr
Search hits gain `lod` (lrs / 4.61) and `locus` (the peak marker name, looked up in PublishXRef in one batched query; null when the database is unavailable). `sort=lrs|lod` with `order=asc|desc` ranks hits: gn3 is asked for its first 1000 matches, those are ranked, then `limit` applies. * src/db.rs (LRS_PER_LOD, lod, get_publish_loci): add the shared LRS-to-LOD constant and the locus lookup; TraitInfo and ProbeSetInfo gain `lod`, computed in SQL from the same constant. * tests/search.rs, test/python/test_search.py: add the "highest LOD, locus marker and additive effect for serotonin in BXD male midbrain" scenario over a recorded production fixture, plus sort/order tests. * tests/group_trait_info.rs, tests/dataset_probeset_info.rs and their Python mirrors: assert lod (and locus for trait 17985). * README.md (Endpoints): document sort, order, lod and locus. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05search: require a token; restore hermetic /search coverageAlexander_Kabui
- src/routes/search.rs: AuthToken per correlation.rs pattern (+ Unauthorized). - tests/search.rs: stub /auth/public-jwks + locally-signed token; scenarios run offline again (multi_thread runtime: the blocking ureq JWKS fetch starved the single-thread stub); add invalid-token 401 case. - test/python/test_search.py: fetch token once, reuse. - RELEASE_NOTES.md: /search now requires a token. - test/data: test-only JWKS + RSA key fixtures.
2026-10-05* src/routes/search.rs (search): share one reqwest client with aAlexander_Kabui
total timeout for gn3 calls reqwest::get built a fresh client per query (no keepalive) and set no timeout, while gn3 sets none of its own -- a stalled upstream would hang the handler past any front-proxy timeout. One shared client in a OnceLock now, mirroring the shared-DB-pool pattern. * src/config.rs (Config): add GN3_TIMEOUT_SECS (default 30); normalize GN3_URL to end with a slash. * tests/search.rs test/python/test_search.py (search): one scenario mapping a gn3 5xx to the 502 backend_error shape.
2026-10-05* src/routes/search.rs (search): map any gn3 4xx with an error body to 422Alexander_Kabui
Production gn3 puts the diagnostic under "error" and leaks Python reprs under "error_type", so the old "parsererror" match never fired and malformed queries fell to 502 with no actionable feedback. Match on "error" alone; gn3 5xx stays 502. * tests/search.rs test/python/test_search.py (search): stub replays production's error body shape.
2026-10-05* src/routes/search.rs (search): add GET /search, the phase-A gn3Alexander_Kabui
search proxy. Validates q/type/limit at the boundary (400 with suggestions; limit capped at 100), forwards to {GN3_URL}/search/ with proper percent-encoding, and re-wraps hits in the descriptive envelope. Hits are typed as SearchHit (all 14 gn3 fields optional, unknown fields preserved via serde flatten) so /openapi.json documents a real schema while staying transparent to gn3 shape changes. gn3's 400+parsererror becomes 422 invalid_query carrying gn3's own diagnostic; every other backend failure is one 502 shape distinguished by code. * src/lib.rs, src/routes/mod.rs (routes): register the search route. * tests/search.rs, test/data/search_alcohol_phenotype.json (tests): in-process gn3 stub replays the recorded production fixture; four scenarios run offline -- passthrough+envelope, validation, 422 forwarding, limit cap.