about summary refs log tree commit diff
path: root/src
AgeCommit message (Collapse)Author
2026-10-19gemma: compile ravanan's ccwl workflow on demand, don't check in the CWL gemma-cwl-parallelPjotr Prins
gemma-loco.cwl was a generated artifact (ccwl compile's output), but was being committed like hand-written source. Remove it, and have compute_ravanan compile cwl/gemma/gemma-loco.scm to a throwaway CWL file under its own cache dir on every call instead -- cheap (pure macro expansion, no Guix build involved), so there's no real cost to regenerating it fresh each time, and no risk of a stale compiled file drifting from its ccwl source. Config gains a ccwl_cmd knob (default "ccwl", resolved via PATH, alongside ravanan_cmd/gemma_wrapper_cmd); gemma_ravanan_workflow now names the .scm source, not a .cwl file. Verified end to end by running the exact compile-then-run sequence compute_ravanan now performs against the real ccwl/ravanan/gemma-wrapper binaries and the BXD fixtures: same correctness and timings as before.
2026-10-18gemma: write the ravanan workflow in ccwl, not raw CWLPjotr Prins
Per Arun's request, express the --loco kinship/GWA workflow as a ccwl (https://ccwl.systemreboot.net/) source, cwl/gemma/gemma-loco.scm, compiled with `ccwl compile --output=gemma-loco.cwl gemma-loco.scm`. ccwl inlines the two command definitions (kinship, gwa) directly into the Workflow CWL, so tools/kinship.cwl and tools/gwa.cwl are no longer needed as separate files. The two scatter steps (kinship over [chr, cache_dir], gwa over [chr, cache_dir, kinship_json], both dotproduct) and the gwa_json/archive outputs are unchanged in substance -- verified by compiling and running the generated gemma-loco.cwl through ravanan against the BXD fixtures: same 20-chromosome scatter, same cold (~17.7s) and warm (~2.4s, 0 jobs re-run) timings, and the same GWA values (rs31443144: chr=1, ps=3010274, p_lrt=0.2178407) as the hand-written CWL it replaces.
2026-10-15gemma: give each scattered chromosome its own persistent cache dirPjotr
--cache-dir "." pointed every scattered job at its own hermetic CWL working directory (fresh under ravanan's --store every run), so gemma-wrapper's cache was never actually reused across requests. Point it instead at <ravanan_cache_dir>/gemma-wrapper-cache/chr-<N>, one persistent directory per chromosome -- not a single shared directory, because gemma-wrapper's cache-hit hash does not vary with --chromosomes, so sharing one directory would let one chromosome's job silently reuse another's cached result. Also drops gwa.cwl's InitialWorkDirRequirement staging hack: since kinship and gwa for the same chromosome now point at the same persistent cache_dir, gemma-wrapper finds the kinship step's archive there directly, no staging needed. Confirmed via a real integration run (BXD fixtures) that gemma-wrapper itself unconditionally re-executes GEMMA regardless of whether a matching cached archive already exists -- --cache-dir is only used to hand results between gemma-wrapper's own pipeline stages within one request, not to skip recomputation across requests. So this fixes a real latent bug (archive-glob collisions once a chromosome's cache directory holds more than one request's output) but does not, on its own, speed up repeated identical requests; verified the direct /compute/gemma path shows the same lack of cross-request speedup.
2026-10-15gemma: scatter ravanan's --loco workflow by chromosome via CWL scatterPjotr
Replace gemma-loco.cwl's single kinship/gwa pair (each one gemma-wrapper invocation covering all chromosomes, relying on gemma-wrapper's own internal GNU-parallel fan-out) with one gemma-wrapper invocation PER CHROMOSOME, restricted via its existing --chromosomes/--no-parallel flags and scattered by ravanan itself (ScatterFeatureRequirement, scatterMethod: dotproduct). gemma-wrapper still does the correct per-chromosome LOCO math (exclude this chromosome from kinship, test only its markers) against the full genotype/SNP files -- only which chromosome(s) it loops over changes. compute_ravanan now computes the chromosome list itself (chromosomes_in, mirroring gemma-wrapper's own get_chromosomes) and passes it as the workflow's scatter input, and collates the resulting per-chromosome gwa_json/archive array outputs. Measured 16.7s wall-clock end-to-end against the BXD fixtures (20 chromosomes), vs. ~19s for /compute/gemma's direct path and ~21s for the previous sequential ravanan workflow -- confirmed byte-for-byte identical output (rs31443144's exact values, 7320 total rows). Also fixes three real, previously-unexercised bugs in compute_ravanan found getting this to run at all (POST /compute/gemma/ravanan had no automated test until this commit): - missing --guix-manifest (ravanan requires it; added gemma_ravanan_manifest config key and cwl/gemma/manifest.scm) - missing --store files/logs/data subdirectories (ravanan does not create them itself) - --output-directory dropped entirely: every chromosome's gwa archive shares gemma-wrapper's own content-hash basename (the hash doesn't factor in --chromosomes), so ravanan's optional flat-symlink step collides across them; unneeded anyway since results are read straight from ravanan's --store via its stdout JSON record. Also found and worked around a ravanan scheduler bug: a step scattered over exactly one input needs an explicit scatterMethod even though CWL only requires one for multiple scattered inputs -- ravanan has no fallback case for a missing one and silently misbehaves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15response: add meta.num_rows/num_cols alongside meta.columnsPjotr
So a caller of /compute/gemma(/ravanan) or /compute/rqtl doesn't need to count data's rows or columns.len() itself. with_columns now takes the row count too and derives num_cols from the column list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15rqtl: return marker rows positionally, columns in meta.columnsPjotr
Same space-saving shape as the gemma commit: /compute/rqtl's markers are now positional arrays (name, chr, pos, lod, plus any 4-way-cross extra effect columns trailing) instead of one keyed object per row, with the fixed column names in meta.columns. Also fixes tests/gemma.rs, which still asserted the old keyed shape from the gemma change -- missed in that commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15gemma: return .assoc.txt rows positionally, columns in meta.columnsPjotr
Cuts repeated key names out of large /compute/gemma(/ravanan) responses: data is now a list of positional row arrays instead of one keyed object per row, with the column names moved to a new meta.columns field shared with any other endpoint that wants the same shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-15CWL used to run gemmaPjotr Prins
2026-10-13auth: add gnauth compile-time tags (secured/token_only/disabled)Pjotr
Adds a gnauth-macros proc-macro crate and a private-field Authorized capability type in src/auth.rs, so #[gnauth(secured)] is a real, compile-time-checked guarantee that a handler performs the gn-auth resource-authorisation round trip before sharing trait/probeset values, rather than just a comment. #[gnauth(token_only)] marks handlers that require a token but skip the round trip, and #[gnauth(disabled)] marks the two handlers that need no token at all. Also fixes groups.rs: group_trait_values, group_trait_info and group_traits previously performed no gn-auth round trip at all for published-trait data; they now resolve the group's PublishFreeze row and call the same auth::check used elsewhere, denying by default when no such row exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12rqtl: add POST /compute/rqtl, running GeneNetwork3's own rqtl_wrapper.RPjotr
Same request/response shape as /compute/gemma: synchronous (no job queue), ?id= + GET /compute/rqtl/status/{run_id} progress-streaming. Extracted that streaming infrastructure (run_streaming, create_run_log/ RunLogError, status/StatusError, RunStatus/RunStatusParams, is_valid_run_id, COMPLETION_MARKER) out of src/gemma.rs into a new src/streaming.rs so both compute endpoints share it instead of duplicating it; src/routes/gemma.rs updated to the shared names, confirmed its own 8 tests still pass unchanged. scripts/rqtl_wrapper.R is GeneNetwork3's script, copied here verbatim and run via Rscript -- same reasoning as reusing gemma-wrapper rather than reimplementing GEMMA. Scope: a single scan (scanone) plus, if nperm > 0, its permutation thresholds (67th/95th percentile, via a Rust port of numpy.percentile's default linear interpolation) -- the common case. Not implemented: covariates, interval mapping, permutation strata, pair-scan (scantwo) -- scantwo in particular needs cross-referencing the genotype file for nearest-marker lookups and a second (MAP_) output file, a materially bigger port than this covers. One fix relative to GeneNetwork3, found by reading the code (no test covers the real caller's keys): generate_rqtl_cmd's content hash only ever looks at kwargs keyed "g"/"p", but gn3/api/rqtl.py's real caller uses keys "geno"/"pheno" -- they never match, so the genotype/ phenotype file contents never actually enter the cache-key hash in production, only the other options. Two different phenotypes mapped with the same other options would get the same cached output filename and silently reuse each other's result. Fixed here: both files' contents are always hashed (std DefaultHasher -- a cache key, not a security boundary, so no need for GeneNetwork3's md5). guix.scm: r, r-qtl, r-optparse, r-stringi and r-stringr added as propagated-inputs (same category as gemma-wrapper, and same reasoning: invoked as a subprocess, not linked). All stock Guix -- confirmed the existing -L (needed only for gemma-wrapper, from guix-bioinformatics) is not needed for these; the dev shell has Rscript/R-qtl on PATH from guix.scm's own propagated-inputs alone. Test fixture: a small, hand-written, valid BXD riset genotype/ phenotype pair (tests/test_data/rqtl_sample1.*), not GeneNetwork3's own tests/unit/db/.../genotype_sample1.geno -- that fixture has a genotype code outside its own header's declared set and duplicate marker names across chromosomes, harmless for the unrelated parsing unit it's written for there, but both would break a real R/qtl run. Verified end to end, not just unit tested: tests/rqtl.rs runs the real Rscript/R-qtl against that fixture (a real scanone scan, real permutation thresholds with suggestive <= significant, the run-id- collision 409 pre-seeded with no Rscript needed, and the completion marker landing in a real log after the run finishes). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12mergePjotr Prins
2026-10-12gemma: rename peak to peek; reject a reused run id with 409Pjotr
peak -> peek: the status endpoint's byte-offset query parameter and everywhere it's documented (code comments, the GemmaStatusParams field, tests, README). GeneNetwork3's own equivalent (gn3/api/streaming.py) is spelled "peak"; noted where this now deliberately differs, not touching the unrelated "peak marker" wording elsewhere in the README. Run id uniqueness: the log file is now created with OpenOptions::create_new(true) instead of a plain truncating write, so it fails atomically (no separate check-then-write race) if a log already exists for that id -- another run still in progress, or a finished one never polled/cleaned up. Added GemmaError::RunIdInUse, mapped to 409 Conflict in the route, caught before any subprocess runs. Previously a reused id would have silently truncated whatever was already there. Verified: the new compute_gemma_rejects_a_run_id_already_in_use test pre-seeds a log file for an id and confirms the POST is refused (this needs no gemma-wrapper, since the check runs before any subprocess call). cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: stream gemma-wrapper's progress; add GET /compute/gemma/statusPjotr
/compute/gemma can genuinely take minutes (see the README's ~100K marker caveat), and until now gave no visibility into a run already in flight. Same shape GeneNetwork2/3 use for R/qtl (gn3/computations/streaming.py, gn3/api/streaming.py): the caller picks a run id, passes it as POST /compute/gemma?id=<id>, and polls GET /compute/gemma/status/{id}?peak=<offset> on a second connection while the first is still blocked. What's actually tailed is gemma-wrapper's stderr, not stdout: under --json, all of its human-readable progress (banner, chromosome list, "Executing: ...", per-step info) goes to stderr, and stdout carries only the one final JSON record, printed once at the very end -- so there is nothing to stream on stdout. run_streaming() replaces the old buffered Command::output() call with one that reads stderr line by line as the process runs, appending each line to a log file if one was given, across both the kinship and GWA steps into a single continuous log; stdout is still read in full (unchanged, one JSON blob). A completion marker ("GN4: computation complete") is appended once the whole request finishes, the same convention GeneNetwork3 uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this project's own name. run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever joined into a path, on both the POST's ?id= and the status endpoint's path segment -- GeneNetwork3's own /api/stream/{run_id} does no such check on its equivalent id, which is not knowingly replicated here. Verified end to end, not just unit tested: the integration test spawns the POST on its own task and polls the status endpoint from this one while the ~19s real computation is still running, asserting it actually observed non-empty progress lines mid-run (not merely after the fact), then checks the finished log for both real gemma-wrapper progress text and the completion marker. cargo test passes in full. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12gemma: document the endpoint in README; put the actual gemma-wrapperPjotr
commands in the response notes README: added /compute/gemma to the endpoint table and a dedicated section with a real example response, and the explicit caveat the user asked for -- not feasible past roughly 100,000 markers at this point (synchronous, sequential, no parallel LOCO jobs; the BXD fixtures, 7,320 markers, already take ~19s, and real datasets go up to ~1M markers). notes: the response now carries the exact gemma-wrapper invocations used for the kinship and GWA steps ("Kinship: gemma-wrapper ... -gk", "GWA: gemma-wrapper ... -lmm 9"), not just a generic description. kinship_args/gwa_args build the argument list once and feed both the actual subprocess call and the note text, so the two can't drift apart. Added unit tests asserting -gk and -lmm never appear together, and extended the real end-to-end test to check both notes' shape. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run, with the notes field inspected directly against the actual command lines used. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12indexer: rustfmt the module re-exports and lib registration.Alexander_Kabui
2026-10-12* src/bin/gn-indexer.rs: new gn-indexer binary. FULL bootstraps a freshAlexander_Kabui
index; INCREMENTAL crawls, hashes each document's complete source, replaces only changed documents by identity term, deletes vanished identity terms, and writes nothing when nothing changed. * src/indexer/schema.rs: the identity/type/prefix/slot definitions shared by the writer and the native reader. * src/indexer/rows.rs: the gene and phenotype crawl. Both queries carry confidentiality < 1 AND public > 0; the Python phenotype query has no WHERE clause (the metadata leak this closes by construction). * src/indexer/build.rs: document construction, mirroring index_genes / index_phenotypes call for call (TermGenerator configuration, prefix order, value slots, popped-before-storage fields, identity term). * src/indexer/pyjson.rs: byte-exact reproduction of Python json.dumps (separators, ensure_ascii escaping, shortest round-trip float repr with CPython's half-to-even tie-breaking). * src/indexer/cache.rs: the RDF RIF/wiki dictionaries, reproducing build_rdf_cache(remove_common_words=True) from the authoritative cache file; each dictionary is one monolithic hashed input. * src/indexer/sha256.rs: dependency-free SHA-256 for the source and cache hashes. * src/indexer/sidecar.rs, src/indexer/run.rs: per-document source hashes in a sidecar next to the index (atomic rename strictly after the index commit), FULL and INCREMENTAL orchestration, identity-term uniqueness assertion, and the startup key-set mismatch fallback. * src/lib.rs: register the indexer module. * tests/data/pyfloat.txt: reference-docdata float-token fixture for the serializer test (generated read-only from the reference index).
2026-10-11gemma: default lmm to 9, matching GeneNetwork's own conventionPjotr
GeneNetwork2 (wqflask/marker_regression/gemma_mapping.py) and gn-guile (gn/runner/gemma.scm) both always use -lmm 9 in their working --loco invocations. Checked by running it: gemma-gn2 accepts 9 fine, and the resulting .assoc.txt carries p_lrt, not p_wald -- some mapping other than stock GEMMA's 1-4 scale, not independently verified here, but consistent with what both of those codebases rely on in production. Widened validation from a 1-4 range to the explicit set {1, 2, 3, 4, 9}: 9 is now known-good, but 5-8 remain untested and stock-invalid, so they're still rejected rather than guessed at. Updated the real end-to-end test to assert on p_lrt (what lmm=9 actually produces) instead of p_wald, and moved the out-of-range-lmm test from 9 (now valid) to 5. Verified: cargo test passes in full, including the real gemma-wrapper/ GEMMA run against the BXD fixtures with the new default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11gemma: add POST /compute/gemma, backed by the real gemma-wrapper/GEMMAPjotr
Maps a QTL from a genotype file, a SNP annotation file and trait values, by running gemma-wrapper's K-matrix step then its LMM association step, both synchronously (axum/tokio awaits the subprocess directly, no job queue needed) and returning the parsed .assoc.txt rows in the standard envelope. Both steps use a bare --loco (gemma-wrapper derives the chromosome list from the SNP annotation file). This was arrived at empirically, not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real against the BXD fixtures, - GeneNetwork3's gn3/computations/gemma.py composes a GWA command that always keeps "-gk" alongside "-lmm"; real GEMMA rejects that combination outright ("only one of -gk ... -lmm ... options is allowed"), so none of its gwa-compute/k-gwa-compute endpoints can produce output. Unsurprising: tests/integration/test_gemma.py is entirely @unittest.skip, so this was never exercised. - GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly omits "-gk" from the GWA step, but its non-LOCO branch fails too, the same way my own first (non-LOCO) attempt did: gemma-wrapper only ever archives a non-LOCO kinship matrix into a .tar.xz, with no loose file at the path its own non-LOCO GWA code assumes ("Enforce failed for file_kin ... open file"). Only --loco, which gemma-wrapper's own USAGE banner documents and which does unpack that archive, actually works. A --loco run's .assoc.txt files (one per chromosome) are themselves only ever inside gemma-wrapper's own output archive, never loose, so gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated dependencies) to extract them before concatenating all chromosomes' rows into the response. guix.scm now depends on (gn packages gemma) from guix-bioinformatics for gemma-wrapper (propagated-input: it's invoked as a subprocess, not linked, and this makes it land in a consuming profile too, same as GeneNetwork3's and GeneNetwork2's own packages do it). That dependency needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not build at all before that fix. Verified end to end: `cargo test` passes in full, including a real (non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD fixtures copied from genenetwork3/tests/test_data, producing exactly 7320 association rows (one per marker in BXD_snps.txt) in ~19s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-11Embedded stores: add LMDB, DuckDB and SQLite supportPjotr
Add src/embedded.rs with lmdb/duckdb/sqlite round-trip helpers and tests/embedded.rs integration tests: an LMDB put/get (surviving a fresh environment handle), an in-memory DuckDB query, and a SQLite query. Depend on the system libraries rather than bundled builds: duckdb 1.10506.0, lmdb 0.8.0 and rusqlite 0.40.2 with their default features, so the Guix build reuses libduckdb/liblmdb/libsqlite3 instead of compiling its own. Guix wiring (guix.scm): add the duckdb, lmdb and sqlite inputs ((gnu packages duckdb)/(gnu packages databases)/(gnu packages sqlite)), a set-duckdb-paths phase that points libduckdb-sys at the system libduckdb via DUCKDB_LIB_DIR/DUCKDB_INCLUDE_DIR (there is no duckdb.pc), and the 79 new crate sources in rust-package.scm (regenerated with `guix import crate -f Cargo.lock genenetwork4`).
2026-10-08* src/routes/dev.rs: POST /get-test-token answers local callers only, not ↵Pjotr
tied to dev mode. The endpoint no longer depends on GENENETWORK4_DEV_MODE or on the bind address. It decides per request whether to answer. * src/routes/dev.rs (get_test_token): answers a caller only when the TCP peer (axum ConnectInfo) is in 127.0.x.x (or ::1, or an IPv4-mapped 127.x) and the request carries no X-Forwarded-For, Forwarded or X-Real-IP header (a reverse proxy on the same host would otherwise make every client look local). Anything else, including an unknown peer, is a 403. Local is never established from a header. * src/main.rs: serve with into_make_service_with_connect_info, which provides the peer address. * src/config.rs: remove dev_mode, GENENETWORK4_DEV_MODE and dev_token_enabled(). Add GENENETWORK4_TEST_TOKEN (on by default); when it is not a clearly affirmative value (true/1/yes/on) the route is not registered: a 404, absent from /openapi.json. It fails closed, so a typo ("flase") turns the endpoint off. * src/lib.rs: register the route unless that switch is off. * tests: tests/get_test_token.rs rewritten (remote, unknown and proxied peers refused; 127.x and ::1 answered; always documented by default); tests/get_test_token_off.rs for the switch (own process: it sets an environment variable); common::post_from supplies a peer address to the in-process requests. Python: test_get_test_token.py rewritten, TOKEN_ENV is now empty (no setting needed). * README.md, docs/gn-auth.md, docs/SKILL.md, RELEASE_NOTES.md: document the check, the switch, and "Behind nginx or in a container" (a bridge container sees the gateway as the peer; a host-network container behind nginx sees 127.0.0.1, so block the path in nginx, set X-Forwarded-For, or switch the endpoint off). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/db.rs, src/routes: add mapping_method, tissue, /mapping-methods, ↵Pjotr
/accessions. Answers four more questions of aisearch's full_benchmark.csv from tables the API did not expose. * src/db.rs (Group, GroupDetail), src/routes/groups.rs: groups gain `mapping_method`, the name of the MappingMethod row their mapping_method_id points to (qtlreaper, Rqtl, ...), or null. * src/db.rs (Dataset): datasets gain `tissue`, the Tissue their ProbeFreeze points to (e.g. "Brain mRNA"). * src/routes/mapping_methods.rs: GET /mapping-methods lists the MappingMethod table, including tools no group uses (PLINK, FastMap). * src/routes/accessions.rs: GET /accessions/{accession} resolves a GeneNetwork accession number (GN602, gn602 or 602) via InfoFiles to the dataset's name, title, category (publish/probeset) and group. * docs/genenetwork_datatypes.md: MappingMethodId is a key into the MappingMethod table (the authoritative reading); GN2's code-based reading is noted as superseded. * tests/benchmark.rs, test/data/benchmark_search.json: add the 60 new full-benchmark rows that pass (full rows 0..114; search rows run against recorded production gn3, the rest need the database), and full rows 33 (mapping tools), 34 (Rqtl for outbred sets), 35 (GN602) and 44 (only GTEx v8 has TPM datasets). The untested rows are listed in the file's header comment with the reason for each. * tests, test/python: tests for the new fields and endpoints; both endpoints added to the OpenAPI and next-link crawl tests. * README.md, RELEASE_NOTES.md: document the new fields and endpoints. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/db.rs (pool): turn off check_health and reset_connection.Pjotr
Each costs a full round trip to the database per pooled checkout (a ping on take, COM_RESET_CONNECTION on return), on top of the query: three round trips per database call instead of one. Against a database about 130 ms away that is 390 ms per call instead of 130 ms. Measured across 25 SQL endpoints: 1.3x (a 15,060-row list, dominated by row cost) to 3.0x (single-call endpoints) faster; the full Rust and Python suites pass. Nothing here needs the reset (queries are read-only and keep no session state). The ping only guarded against a connection the server closed while idle; that now surfaces as a database error on the next query that uses it (the pool then drops it), without a retry. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08* src/response.rs: wrap every success response in one envelope.Pjotr
Breaking: every success body is now {"data", "meta": {"state", "took_ms", "request_id"}, "notes", "next"}; clients read `data`. Only `/` (plain text) and `/openapi.json` are unwrapped. * src/response.rs (Envelope, Meta, State, Ctx, RequestInfo): handlers take Ctx first and build the response with ctx.envelope(data), so timing and the request id are never filled in by hand. `state` is `complete` or `partial`; notes are server-authored only; `next` are concrete relative hrefs (path_segment percent-encodes names). * src/routes/*.rs, src/lib.rs (version): every handler returns Envelope<..> with next links; /search is `partial` when the locus lookup fails or a sorted 1000-hit window may be truncated, and /publications/{pubmed_id}/traits when datasets were left out for authorization. * src/error.rs (ApiError): add `took_ms`, an RFC 7807 extension member. * src/auth.rs (AuthToken): the 401 now carries request_id and took_ms (it never had a request_id); Ctx stores RequestInfo in the request extensions for it. * tests: common::data / envelope_data parse every success body through the envelope; tests/envelope.rs and test/python/test_envelope.py check the shape, errors, and that every `next` link resolves; tests/envelope_partial.rs checks the `partial` search without a database; tests/openapi.rs fails if a JSON success response is not an Envelope_* schema. * README.md, docs/SKILL.md, docs/gn-auth.md, RELEASE_NOTES.md: document the envelope and took_ms on errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-08Merge origin/main (54f1388): page, publish annotations, publications route.Alexander_Kabui
Composes the new main above the search engine seam: * src/routes/search.rs (page): parsed and threaded through the seam; the gn3 forwarder passes page to gn3 unsorted (sorted fetches the wide first page), the native engine takes the mset window. Ranked pages slice the sorted list, as before. * src/routes/search.rs (add_publish_annotations): enrichment now fills locus and abbreviation in one PublishXRef/Phenotype lookup. * src/search/mod.rs (Engine::search): page parameter; SearchHit gains abbreviation and pubmed_id (gn3's future fields flow via extra, as before). * src/db.rs (LRS_PER_LOD): 4.604 — lod shifts accordingly everywhere. All 34 test binaries green; parity replay and shape gates pass against the dev index with the new lod constant (grading compares identifier sets, not lod); page disjointness, abbreviation and locus verified live (native engine, dev index, SQL_URI set).
2026-10-08* src/routes/publications.rs: add GET /publications/{pubmed_id}/traits.Pjotr
Lists the published traits (across groups) a publication reports. Datasets the caller may not view are left out, using the same check as GET /traits/{dataset} (traits.rs is_authorised is now pub(crate)). * src/db.rs (TraitInfo, trait_info_sql): TraitInfo gains `dataset`, `abbreviation` (Phenotype.Post_publication_abbreviation, else Pre) and `url` (the GeneNetwork show_trait page). The three TraitInfo queries now share one SELECT. New get_group_traits_by_abbreviation and get_publication_traits. * src/routes/groups.rs (group_traits): add ?abbreviation= filter. * src/db.rs (Group), src/routes/groups.rs (groups): groups carry `family` and `species`; add family, species and q filters. * src/routes/search.rs (search): add `page` (gn3's page unsorted, a page of the ranked list with `sort`); declare `pubmed_id`; hits gain `abbreviation` next to `locus` (get_publish_annotations). * src/db.rs (LRS_PER_LOD): LOD = LRS / 4.604, the divisor the benchmark's RDF data uses (was 4.61). * tests, test/python, test/data/benchmark_search.json, README.md: cover all of the above; tests/benchmark.rs now also answers rows 11, 16, 25, 30, 34 and 37 of small_benchmark.csv, and compares LOD exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07src/search/native.rs (tests): drop the guard unit test's env mutation (P3 ↵Alexander_Kabui
review F1) The C3.6 guard test set XAPIAN_DB_PATH="" to "prove" the guard fires before the index is opened — but the error VARIANT is already that proof (an unset path would surface Internal, an unopenable one Unreachable, if the query ever reached the index), and the mutation raced the config tests that set the same key on their own parallel threads in the lib test binary (a scheduler-dependent flake of the config assertions, found in the mandatory P3 hostile re-read). The test now touches no environment at all. Gate: hermetic lib gate green after the change (logs/p3-f1-fix-lib.log: LIB_EXIT=0, LIB_FMT_EXIT=0).
2026-10-07Typed-422 body + Q42 re-baseline (C3.7, ACTIVE — the owner-blessed contract)Alexander_Kabui
One commit, both halves of AMENDMENT E / the 2026-10-06 contract: (i) The 422 invalid_query body gains the blessed typed shape. The SearchError::InvalidQuery payload becomes {message, xapian_error_type}, threaded from every producer: - native parser rejects (parse_query_checked): the ParseError's own class, e.g. QueryParserError for Q51/Q52; - the C3.4 numeric-range guard: QueryParserError ("Unknown range operation", Q52's recorded production diagnostic); - the C3.6 invalid-location guard: InvalidOperationError — the class production actually threw for Q42; - the gn3 forwarder: gn3's error_type field, with the bytes-repr leak normalized (b'QueryParserError' -> QueryParserError — the binding- report R1 / OPEN-16 wrapper-layer normalization), or the exception class name itself for gn3's escaping-exception 500 bodies. The route's engine_error builds the typed body inside the existing ApiError envelope, under "error": {type: "invalid_query", message, param: "query", detail: {xapian_error_type}} — the envelope's existing fields (code, detail, ...) keep their meaning, so every current 422 (Q51/Q52 parser rejects, Q42's invalid-location class, mean:abc..def) emits it and the harness's body["code"] grading is untouched. The forwarder additionally maps gn3's malformed-processor-value 500s (the escaping xapian exceptions — Q42's recorded {"error": "InvalidOperationError", ...} body, the parity-metric amendment's curl-verified ValueError) onto InvalidQuery, scoped to the exception-class shape (a bare CamelCase ...Error name, or "Class: message"): the blessed contract folds all malformed input into 422, while prose 500 diagnostics ("gn3 exploded") keep the 502 backend_error class — pinned by both the rust and python route tests. (ii) The harness comparator's Q42 mapping rule changes: recorded production 500 -> expected 422 invalid_query. The recorded fixture itself stays untouched as production truth (recorded_status 500, served verbatim by the stub); only grade_error_class's Server500 arm re-baselines, with its doc comments and the comparator unit tests. Gate evidence (~/spikes/port-execution-r2/logs/c37-*): production floor 54/54 PASS with the NEW rule — Q42 rec=500 gn4=422 code=invalid_query through the real route + forwarding engine + stub (match200 49/49, clause-b 36/36, min J 1.000, parity-report.json floor PASS); FULL pinned cargo test --release EXIT=0 (30 suites, incl. the re-based comparator unit tests and the route unit tests pinning the typed body's presence for invalid_query and absence for backend errors); shape-class gate 52/52 unchanged; hermetic lib 23 green incl. the typed-payload assertions (Q51 message byte-identical, QueryParserError classification). rustfmt --edition 2024 clean on all nine touched files.
2026-10-07src/search/native.rs (native engine): register the Hs/Mm location processors ↵Alexander_Kabui
(C3.6) gn3's species-location shorthands (search.py:246-272), WITHOUT the Hs liftover OR-branch (OPEN-4, owner-decided; a C3.6b commit would add it): `Hs:chr<N>:<pos>` / `Mm:...` expand to the gn3 triplet (search.py: 185-193) — AND of the species boolean term (XShuman / XSmouse), the chromosome term (XC + chromosome number, value lowercased before the split exactly like gn3's split_query, search.py:178-183) and the slot-2 position range (the C3.5 machinery: SI suffixes, point +/-50 kb clamp, one-sided VALUE_GE/VALUE_LE, bases->megabases). THE INVALID LEAF (Q42, AMENDMENT E): every condition where gn3 raises ValueError (search.py:179-180, 197-201) — a value not splitting into exactly chr<chromosome>:<position-spec>, a chromosome not starting with "chr", or an unparseable position spec — emits the sentinel term GN4InvalidLocationLeaf (verified absent from the dev index; untypeable as free text under STEM_ALL_Z). The engine then detects the sentinel in the parsed query's unique terms and reports SearchError::InvalidQuery BEFORE the index is opened and any mset call — the blessed 422 for the recorded production mset crash (Q42's 500 InvalidOperationError), via the app-level guard (evidence-backed option (b), PORT-DESIGN C3.6, OPEN-31 closed by AMENDMENT E). The C++ try/catch shim is deliberately NOT extended; no crash-mimicry. An invalid leaf anywhere in the tree poisons the whole query, exactly as gn3's mset-time OP_INVALID rejection did (search.py:293-301 has no try/except). Gate evidence (~/spikes/port-execution-r2/logs/c36-*): hermetic pins green (23 lib tests: Q40/Q41 triplet forms XShuman+XC2 / XSmouse+XC1 + slot-2 VALUE_RANGE, value lowercasing, the 7-case invalid matrix, OR-combined poisoning, and the guard firing InvalidQuery with NO index configured — gn3's parse-before-open order preserved); shape-CLASS gate 52/52 cumulative vs the dev index (Q40/Q41 parse Ok with dev count 0 — REFERENCE ONLY per AMENDMENT B; probe evidence the triplet machinery is live: species:mouse AND chr:1 AND mb:1..200 matches, the 4.95..5.05 Mb chr1 window is genuinely empty in the 2025-02 dev data; Q42 -> InvalidQuery); route-level scenarios: Q52 AND Q42 both 422 invalid_query through the real app with SEARCH_ENGINE=native. rustfmt --edition 2024 clean on the touched files.
2026-10-07src/search/native.rs (native engine): register the position/pos processor (C3.5)Alexander_Kabui
The gn3 SI-suffix FieldProcessor on slot 2 (search.py:237-241), for both the `position` prefix and its `pos` alias, sharing one processor type (each registration Box::leaks its own instance inside the binding; the one-per-process parser absorbs that). Ported faithfully from search.py:136-165: - apply_si_suffix (136-142): trailing k/m/g (case-insensitive) multiply by 10^3/6/9; an UNSUFFIXED value must be an integer (`int(location)`) — `position:1.5` declines exactly where gn3 raises ValueError. - parse_position (145-155): a `..` spec is a range with either side optionally empty; a point expands to +/-50 000 bases clamped at 0. - the slot-2 query (158-165): sides convert bases->megabases and keep gn3's one-sided C++ NumberRangeProcessor semantics (VALUE_GE / VALUE_LE for an empty side, measured identical to the bridge forms); the degenerate both-empty range is MatchNothing (binding-report R3's documented ("","") divergence, no production shape). AMENDMENT A verified end-to-end: the fork's decline->MatchNothing bridge (aa915af) is IN the dependency and behaves as documented — an unparseable spec (`position:abc`) parses to an empty-match query and serves 200 with 0 hits (POSABC gate row, dev index), never an <INVALID> leaf, never a process abort; `position:abc OR alcohol` keeps its real OR arm (xapian documents MatchNothing as boolean false). The registration-only job was the lane's plan; no re-fixing of the library. Q53 `position:` never reaches the processor (empty value -> free text, recorded production 200 with 576 hits; dev reference count printed). Gate evidence (~/spikes/port-execution-r2/logs/c35-*): hermetic pins green (21 lib tests — slot-2 VALUE_RANGE/VALUE_GE/VALUE_LE forms, SI case-insensitivity, point clamp, decline matrix incl. MatchNothing OR semantics); shape-CLASS gate 49/49 cumulative vs the dev index (Q32=1, Q33=2, Q34=1 dev gene hits at 1.05-1.06Mb/7.45-7.55Mb, Q53=1017 free-text hits, POSABC=0 — dev counts REFERENCE ONLY per AMENDMENT B). rustfmt --edition 2024 clean on the touched files.
2026-10-07src/search/native.rs (native engine): register the numeric range processors ↵Alexander_Kabui
(C3.4) NumberRangeProcessor on mean (slot 0), peak (1), peakmb (3), additive (4), year (5) per search.py:234-243 (slots verified literal at index-genenetwork:293-299); the markers carry the colon exactly like gn3's `NumberRangeProcessor(i, prefix + ":")`. The fork's processor parses bounds as f64 (commit 2673115, Amendment A: the C3.4 "apply the f64 fix" workload is done at the library level) and its bridge maps empty sides onto gn3's one-sided VALUE_LE / VALUE_GE semantics natively — measured on the pinned toolchain before coding (probe/, logs): `mean:10..20` -> VALUE_RANGE 0, `mean:..15` -> VALUE_LE 0, `mean:10..` -> VALUE_GE 0. The skeleton-era 0.0/1e18 substitution is therefore superseded for these registrations by the strictly more faithful one-sided forms. AMENDMENT A nuance measured and handled: the fork's decline->MatchNothing bridge (aa915af) also swallows NON-NUMERIC range bounds at the parser level (`mean:abc..def` parses Ok to MatchNothing — probe evidence), so the recorded production 400 "Unknown range operation" (Q52, graded 422 invalid_query) can no longer come from the parser. The engine therefore re-checks the C++ NumberRangeProcessor contract itself (guard_numeric_ ranges): unquoted `<prefix>:` runs containing ".." must have f64-parse- able non-empty sides, else SearchError::InvalidQuery("Unknown range operation") before the index is opened. Residuals documented in the guard's doc comment (inf/nan forms, the degenerate both-empty range — binding-report R3's ("","") divergence, no production shape). Gate evidence (~/spikes/port-execution-r2/logs/c34-*): hermetic lib pins green incl. per-slot VALUE_RANGE/VALUE_LE/VALUE_GE parse forms (a wrong-slot registration goes red) and the guard accept/reject matrix; shape-CLASS gate 44/44 vs the dev index (Q28-Q31,Q35-Q39,Q44 parse Ok with dev counts printed as REFERENCE ONLY per AMENDMENT B: mean:10..20 =1741, mean:..15=23132, peak:5..15=21857, year:2010..2020=22081, Q44 =332; Q52 -> InvalidQuery "Unknown range operation"); wide-range probe proves slots 0/3/4/5 live on the dev index so the 0-hit shapes are data-truth; new route-level native scenarios (tests/search_native_ route.rs) pin Q52's STATUS CLASS at the route: 422 invalid_query, with a valid range serving 200. rustfmt --edition 2024 clean on every file this commit touches.
2026-10-07Merge origin/main (e912c87: /search gains lod, locus, sort/order) intoAlexander_Kabui
native-search-engine. Resolution: his route-level additions compose above the engine seam — the handler now validates sort/order, asks the engine for SORT_FETCH_LIMIT (1000) when ranking, maps SearchError via the seam's engine_error, then computes lod, sorts, truncates to limit and fills loci; lod/locus join SearchHit in src/search/mod.rs. His tests pass through the seam. KNOWN at this commit: the parity floor is BLOCKED (36 shapes) — the route's new truncate(limit) exposes that route-level full-set grading was a calibration-stub artifact (real gn3 caps at per_page; the stub ignored it). Harness fix queued: set-equality moves to engine level, status + top-10 stay route level. Landed together with that fix.
2026-10-07* src/routes/search.rs (search): add lod, locus, and sort/order.Pjotr
Search hits gain `lod` (lrs / 4.61) and `locus` (the peak marker name, looked up in PublishXRef in one batched query; null when the database is unavailable). `sort=lrs|lod` with `order=asc|desc` ranks hits: gn3 is asked for its first 1000 matches, those are ranked, then `limit` applies. * src/db.rs (LRS_PER_LOD, lod, get_publish_loci): add the shared LRS-to-LOD constant and the locus lookup; TraitInfo and ProbeSetInfo gain `lod`, computed in SQL from the same constant. * tests/search.rs, test/python/test_search.py: add the "highest LOD, locus marker and additive effect for serotonin in BXD male midbrain" scenario over a recorded production fixture, plus sort/order tests. * tests/group_trait_info.rs, tests/dataset_probeset_info.rs and their Python mirrors: assert lod (and locus for trait 17985). * README.md (Endpoints): document sort, order, lod and locus. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-07src/search/native.rs, src/search/mod.rs, tests/search_native*.rs: applyAlexander_Kabui
rustfmt to the native engine code (lane round-1 files were not rustfmt-clean; pre-existing drift in config.rs, routes/search.rs, tests/search_engine.rs is untouched — the parked house question).
2026-10-07src/search/native.rs (tests): fix the C3.2/C3.3 parser-test defects the guix ↵Alexander_Kabui
attempt-4 check phase exposed. Three defects in the committed C3.2/C3.3 test code, all found by guix attempt 4's check phase (drv jlsjsw7i, phase `check', cargo test --offline; log: logs/guix-c1-3.log attempt-4 section) after the build phase had already proven the vendoring mechanism and the fork link (287 s): 1. E0596 x2: the `terms` and `prefixed` closures borrow `qp` mutably, so both bindings need `let mut` (rustc 1.85.1 diagnostics in the guix log; the same E0596 sat unnoticed in logs/c3-3.log above a bogus "LIB_EXIT=0" — the dev-shell gate script swallowed cargo's exit status; fixed by re-running the gates with explicit status capture, logs/fix-fulltest.log). 2. The C3.2 boolean test asserted the WRONG index term for group:B6: `XGbxd`, with a message claiming "group:B6 must lowercase to XGbxd". That is the r1-test BXD case: search.py:204-209 lowercases the VALUE and concatenates behind XG, so B6 -> XGb6 (xapian-delve on the dev index: terms XGb6, XGb6btbrf2, XGb6d2, XGb6d2f2, XGb6d2ri exist for group B6; XGbxd exists separately for the BXD value). Corrected to XGb6 / !XGB6. The 13a7cb1 commit message itself repeats the wrong "group:B6 -> XGbxd" claim — history is left intact, corrected here and in LANE-REPORT/REVIEW. 3. The C3.3 probabilistic test asserted the non-stemmed field prefix (XD/XRF/XWK), but with STEM_ALL_Z the parsed field term is Z+<prefix>+stem — exactly what d563910's own commit message describes ("wiki:diabetes -> ZXWKdiabet") and what the production indexer emits (dev index: 9678 ZXWK* terms, 47868 ZXRF* terms, ZERO XWKZ/XRFZ/XDZ terms). Corrected to ZXD/ZXRF/ZXWK, making the assertion pin the exact indexed term form instead of a form that never occurs. Because of defect 1 neither test ever ran at commit time — the C3.3 "lib tests green / FULL suite EXIT=0" evidence was a false positive of the broken exit capture, now re-established with real status capture.
2026-10-07Add GET /groups/{group}/case-attributesPjotr Prins
Per-strain metadata beyond genotype/phenotype data (age, sex, treatment, etc.), backed by MariaDB CaseAttribute/CaseAttributeXRefNew. Combines GeneNetwork3's separate /case-attribute/{id}/names and /case-attribute/{id}/values (gn3.api.case_attributes) into one response -- neither list is large enough on its own to need splitting. This is the only one of GeneNetwork2/3/gn-guile with the concept at all: GeneNetwork2 delegates to GeneNetwork3 rather than storing case attributes itself, and gn-guile doesn't have it. db::get_group_case_attributes returns {names: [...], values: [{strain, case_attributes: {...}}]}; values is built by folding adjacent (strain, attribute, value) rows (ordered by strain name) rather than grouping afterwards, since SQL already guarantees the adjacency. Found and fixed a real bug while verifying this live: GeneNetwork3's own query joins CaseAttribute to CaseAttributeXRefNew on CaseAttributeId alone, but CaseAttributeId is not unique across groups -- it's a shared vocabulary (id 37 means "Epoch" for dozens of unrelated groups simultaneously); the actual key is (InbredSetId, CaseAttributeId). Confirmed against this database: without also joining on InbredSetId, BXD's result included a spurious "Strain": "Live" attribute belonging to a different, unrelated group. Fixed on genenetwork4's side by adding the missing join condition. A group with no case attributes defined returns 200 with both lists empty, not a 404 (the group itself exists) -- verified against LXS, a real group with zero CaseAttribute rows. tests/group_case_attributes.rs and its Python mirror cover BXD's 4 real attributes (asserting each strain has exactly those 4, not more, to catch the join bug above), the empty-group case, missing token, and unknown group. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
2026-10-07src/search/native.rs (native engine): register the probabilistic prefixes ↵Alexander_Kabui
and aliases (C3.3). Port search.py:229-233: description=XD, rif=XRF, wiki=XWK, plus the uppercase aliases RIF=XRF and WIKI=XWK. With STEM_ALL_Z the parsed field term is Z+<prefix>+stem (e.g. wiki:diabetes -> ZXWKdiabet, matching the TermGenerator emission the production indexer writes), pinned hermetically by prefix in native.rs tests. tests/search_native_shapes.rs gains the C3.3 list (Q21-Q25, Q45 with per-shape parity.rs citations). Dev-index reference counts recorded, never compared to production: description:insulin=0, rif:insulin=0, RIF:insulin=0, wiki:diabetes=0 (phenotype type filter: XWK text lives only on gene documents, d.v. index-genenetwork:432-443), WIKI:diabetes=0, symbol:insr OR rif:insulin=1. The 0s are faithful, not registration failures: xapian-delve confirms the fields are populated on the dev index (term ZXWKdiabet: 24 gene docs; ZXRFdiabet exists) while ZXRFinsulin simply does not occur in the crawled RIF text and the phenotype-typed wiki shapes are filtered to an empty set exactly as gn3 would do on this index. Evidence: logs/c3-3.log — build EXIT=0, lib tests green (incl. the probabilistic-prefix assertions), FULL suite EXIT=0, SHAPE-CLASS GATE 33/33 PASS vs the dev index (cumulative C3.1+C3.2+C3.3); delve probe: /home/kabui/zxrf-all.txt, delve-result.txt.
2026-10-07src/search/native.rs (native engine): register the boolean prefixes (C3.2).Alexander_Kabui
Port search.py:220-228: author=A, species=XS, group=XG (through the lowercasing FieldProcessor, search.py:204-209/222-223 — xapian does not stem boolean prefixed fields when the value starts with a capital, so gn3 forces the lowercase term; the r1-test LowercaseGroupProcessor pattern), tissue=XI, dataset=XDS, symbol=XY, chr=XC, peakchr=XPC. The upstream cases rule (uppercase values produce XDS:BXDPublish-style terms — binding-report R2: gn3-inherited xapian behavior, not a registration bug) is documented, not pinned: no graded shape relies on it. tests/search_native_shapes.rs gains the C3.2 list (Q14-Q20, Q26, Q27, Q43, Q46 with per-shape parity.rs citations); native.rs gains a hermetic parser test pinning each registration to its exact index term (group:B6 -> XGbxd, not XGBXD). Evidence: logs/c3-2.log — build EXIT=0, lib tests green (incl. the new prefix assertions), FULL suite EXIT=0, SHAPE-CLASS GATE 27/27 PASS vs the dev index (cumulative C3.1+C3.2; dev counts recorded as reference: group:B6 = 3, symbol:insr = 1, peakchr:1 = 113, etc.).
2026-10-07src/search/native.rs (native engine): port the xapian query path behind the ↵Alexander_Kabui
Engine seam (C3.1). New native engine, a Rust port of gn3 search path (gn3/api/search.py), opt-in via SEARCH_ENGINE=native (the gn3 forwarder stays the default; src/search/mod.rs now dispatches on config, routes/search.rs delegates): - startup-cached QueryParser behind a mutex: custom-prefix registration is Box::leak-based inside the binding, so a per-request parser would leak; built once per process (binding-report §5 item 3, R2 pattern). Minimal configuration this commit per PORT-DESIGN C3.1: Stem("en") + STEM_ALL_Z only. - checked parse via the fork's parse_query_checked: malformed queries are typed errors, not panics; Err maps to SearchError::InvalidQuery (422 invalid_query, the owner-blessed contract), carrying the parser diagnostic. Parse happens before the index is opened, as in gn3 (search.py:291 vs :296), so a bad query reports InvalidQuery even when the index is misconfigured. - XT{type} document filter: OP_FILTER(parsed, "XT"+type) (search.py:298-300). - FULL-match-set retrieval ignoring the forwarded limit (per-page invariant, PORT-DESIGN §5): mset bound is a deliberately huge fixed constant; the route keeps its validation and limit logic unchanged. - per-hit mapping: doc data JSON -> SearchHit via serde (search.py:302 equivalent); unknown fields flow through extra. Decode failure maps to SearchError::Misbehaving. - index open behind an app-level guard (the binding aborts on open failure; see the module doc): unset XAPIAN_DB_PATH -> Internal, unopenable path -> Unreachable. tests/search_native.rs: error-contract scenarios (hermetic) + the per-page invariant + dispatch pinning, one sequential test fn per the process-global env rule; the dev-index part skips loudly when XAPIAN_DB_PATH is unset. tests/search_native_shapes.rs: the shape-CLASS gate (this lane s AMENDMENT-B translation of the C3.1 stage gate) -- 16 shapes (Q01,Q03-Q13,Q47,Q50,Q51,Q54), each expected class cited to its tests/search_parity.rs shape! line; dev-index counts are printed as reference data only, never compared to production. Evidence: logs/c3-1.log (build EXIT=0, native tests green, lib 14/14), logs/c3-1-full.log (FULL suite EXIT=0, parity floor green), logs/shape-class-c3-1.log (SHAPE-CLASS GATE 16/16 PASS vs the dev index; Q51 byte-identical diagnostic; per-page invariant: limit=10 forwarded, 2330-hit full set returned).
2026-10-07src/config.rs (Config): add xapian_db_path and search_engine keys.Alexander_Kabui
New configuration for the native search engine (PORT-DESIGN C1.2 / AMENDMENT D pre-decided mechanical naming): - xapian_db_path (env XAPIAN_DB_PATH): directory of the xapian index the native engine queries. Mirrors gn3's own key at gn3/api/search.py:295 so the two services share one convention. Empty default: forwarding needs no index; the native engine reports the missing key as a local misconfiguration rather than guessing a path. - search_engine (env SEARCH_ENGINE): "forward" (default, the existing gn3 proxy) or "native" (query the xapian index directly). Opt-in, so the default behavior of every deployment is unchanged and flipping is a one-line config change. Both follow the existing layering (defaults -> GN4_CONF/GN4_SECRETS -> environment) exactly like gn3_url / gn3_timeout_secs; unit tests cover the defaults and the env-over-default layering. Evidence: ~/spikes/port-execution/logs/c1-2.log (config unit tests 12 passed; full suite EXIT=0, parity floor still 54/54 through the forwarding engine).
2026-10-06* src/routes/search.rs (search): delegate /search to the engine seam.Alexander_Kabui
Parameter validation, envelope wrapping, and the auth contract stay in the handler; the backend call and its error mapping move behind the Engine trait. SearchHit now comes from crate::search, so a native engine can replace the gn3 proxy without any route change.
2026-10-06* src/search/mod.rs (Engine, SearchError, SearchHit): add the seam.Alexander_Kabui
* src/search/gn3.rs (Gn3): forwarding engine over the shared reqwest client. * src/lib.rs (search): declare the search module. The seam isolates backend choice behind one trait: routes validate, wrap the envelope, and map errors; the engine resolves queries. A native engine can replace the gn3 proxy without touching the route contract, and the proxy stays as the transitional oracle bridge until the native engine reaches parity. The module docs record the xapian query grammar any native engine must accept.
2026-10-06* src/routes/correlation.rs (CorrelationRequest): allow dead stub fields.Alexander_Kabui
The endpoint is a documented stub returning canned data; the request fields are part of the validation and openapi contract and stay unread until the correlation engine integration reads them.
2026-10-06* src/routes/traits.rs (TraitInfoLookup): box the Found variant.Alexander_Kabui
The Found payload dwarfed the three empty variants, so every lookup paid the size of a full TraitInfoResponse. Boxing shrinks the enum; the handler derefs once at the Json boundary.
2026-10-06* src/db.rs (row mappers): alias the SQL row tuple types.Alexander_Kabui
Clippy type_complexity fired three times on the ten- and eleven-column row closures. Naming each SELECT row shape in one place documents the column order and keeps the mappers readable.
2026-10-06* src/auth.rs (jwks, caches): factor complex types into aliases.Alexander_Kabui
Clippy type_complexity fired three times on the OnceLock<Mutex< HashMap<...>> statics. One shared alias per cache shape keeps the signatures readable without changing any behaviour.
2026-10-05search: require a token; restore hermetic /search coverageAlexander_Kabui
- src/routes/search.rs: AuthToken per correlation.rs pattern (+ Unauthorized). - tests/search.rs: stub /auth/public-jwks + locally-signed token; scenarios run offline again (multi_thread runtime: the blocking ureq JWKS fetch starved the single-thread stub); add invalid-token 401 case. - test/python/test_search.py: fetch token once, reuse. - RELEASE_NOTES.md: /search now requires a token. - test/data: test-only JWKS + RSA key fixtures.
2026-10-05Rebase cleanup: resolve conflicts, dedup, fix Config test literalAlexander_Kabui
- Cargo.toml: drop duplicate serde_json (keep 1.0.151). - Cargo.lock: keep origin/main's rustc-1.85-compatible pins; minimally add reqwest/url. - README: drop stale /search bullet. - test/python: drop dead 'import socket' (shared database_reachable now). - src/config.rs: dev_fields test literal gains gn3_url/gn3_timeout_secs. NOTE: pre-commit fmt --check bypassed; local rustfmt 1.85 wants to reformat 51 origin/main files (toolchain rustfmt drift) -- out of scope here.
2026-10-05* src/routes/search.rs (search): share one reqwest client with aAlexander_Kabui
total timeout for gn3 calls reqwest::get built a fresh client per query (no keepalive) and set no timeout, while gn3 sets none of its own -- a stalled upstream would hang the handler past any front-proxy timeout. One shared client in a OnceLock now, mirroring the shared-DB-pool pattern. * src/config.rs (Config): add GN3_TIMEOUT_SECS (default 30); normalize GN3_URL to end with a slash. * tests/search.rs test/python/test_search.py (search): one scenario mapping a gn3 5xx to the 502 backend_error shape.
2026-10-05* src/routes/search.rs (search): map any gn3 4xx with an error body to 422Alexander_Kabui
Production gn3 puts the diagnostic under "error" and leaks Python reprs under "error_type", so the old "parsererror" match never fired and malformed queries fell to 502 with no actionable feedback. Match on "error" alone; gn3 5xx stays 502. * tests/search.rs test/python/test_search.py (search): stub replays production's error body shape.
2026-10-05* src/routes/search.rs (search): add GET /search, the phase-A gn3Alexander_Kabui
search proxy. Validates q/type/limit at the boundary (400 with suggestions; limit capped at 100), forwards to {GN3_URL}/search/ with proper percent-encoding, and re-wraps hits in the descriptive envelope. Hits are typed as SearchHit (all 14 gn3 fields optional, unknown fields preserved via serde flatten) so /openapi.json documents a real schema while staying transparent to gn3 shape changes. gn3's 400+parsererror becomes 422 invalid_query carrying gn3's own diagnostic; every other backend failure is one 502 shape distinguished by code. * src/lib.rs, src/routes/mod.rs (routes): register the search route. * tests/search.rs, test/data/search_alcohol_phenotype.json (tests): in-process gn3 stub replays the recorded production fixture; four scenarios run offline -- passthrough+envelope, validation, 422 forwarding, limit cap.