| Age | Commit message (Collapse) | Author |
|
The index stores every boolean term in lowercase and holds no
mixed-case variants at all: XDSbxdpublish (15 060 docs), Awilliams
(10 958), XSmouse, XCx, XYbdnf, XImrna all exist, while XDSBXDPublish,
AWilliams, XSMouse, XCX, XYBDNF and XIMRNA report "not in database" on
both the dev index and a full production one. Matching the value
exactly therefore rejected the way a caller naturally writes it --
dataset:BXDPublish returned 0 hits against an index holding 15 060
BXDPublish traits, author:Williams 0 against 10 958 williams authors.
Lowercase the value for every boolean prefix, generalising the
group-only rule gn3 already has (search.py:204-209). No graded parity
shape changes: they all use lowercase values already.
Pinned per prefix in boolean_prefixes_parse_to_exact_index_terms, so a
registration that regresses to an exact add_boolean_prefix fails there
instead of silently returning zero hits.
|
|
Composes the new main above the search engine seam:
* src/routes/search.rs (page): parsed and threaded through the seam;
the gn3 forwarder passes page to gn3 unsorted (sorted fetches the
wide first page), the native engine takes the mset window. Ranked
pages slice the sorted list, as before.
* src/routes/search.rs (add_publish_annotations): enrichment now
fills locus and abbreviation in one PublishXRef/Phenotype lookup.
* src/search/mod.rs (Engine::search): page parameter; SearchHit
gains abbreviation and pubmed_id (gn3's future fields flow via
extra, as before).
* src/db.rs (LRS_PER_LOD): 4.604 — lod shifts accordingly everywhere.
All 34 test binaries green; parity replay and shape gates pass
against the dev index with the new lod constant (grading compares
identifier sets, not lod); page disjointness, abbreviation and locus
verified live (native engine, dev index, SQL_URI set).
|
|
review F1)
The C3.6 guard test set XAPIAN_DB_PATH="" to "prove" the guard fires
before the index is opened — but the error VARIANT is already that
proof (an unset path would surface Internal, an unopenable one
Unreachable, if the query ever reached the index), and the mutation
raced the config tests that set the same key on their own parallel
threads in the lib test binary (a scheduler-dependent flake of the
config assertions, found in the mandatory P3 hostile re-read).
The test now touches no environment at all.
Gate: hermetic lib gate green after the change
(logs/p3-f1-fix-lib.log: LIB_EXIT=0, LIB_FMT_EXIT=0).
|
|
One commit, both halves of AMENDMENT E / the 2026-10-06 contract:
(i) The 422 invalid_query body gains the blessed typed shape. The
SearchError::InvalidQuery payload becomes {message, xapian_error_type},
threaded from every producer:
- native parser rejects (parse_query_checked): the ParseError's own
class, e.g. QueryParserError for Q51/Q52;
- the C3.4 numeric-range guard: QueryParserError ("Unknown range
operation", Q52's recorded production diagnostic);
- the C3.6 invalid-location guard: InvalidOperationError — the class
production actually threw for Q42;
- the gn3 forwarder: gn3's error_type field, with the bytes-repr leak
normalized (b'QueryParserError' -> QueryParserError — the binding-
report R1 / OPEN-16 wrapper-layer normalization), or the exception
class name itself for gn3's escaping-exception 500 bodies.
The route's engine_error builds the typed body inside the existing
ApiError envelope, under "error": {type: "invalid_query", message,
param: "query", detail: {xapian_error_type}} — the envelope's existing
fields (code, detail, ...) keep their meaning, so every current 422
(Q51/Q52 parser rejects, Q42's invalid-location class, mean:abc..def)
emits it and the harness's body["code"] grading is untouched.
The forwarder additionally maps gn3's malformed-processor-value 500s
(the escaping xapian exceptions — Q42's recorded
{"error": "InvalidOperationError", ...} body, the parity-metric
amendment's curl-verified ValueError) onto InvalidQuery, scoped to the
exception-class shape (a bare CamelCase ...Error name, or "Class:
message"): the blessed contract folds all malformed input into 422,
while prose 500 diagnostics ("gn3 exploded") keep the 502
backend_error class — pinned by both the rust and python route tests.
(ii) The harness comparator's Q42 mapping rule changes: recorded
production 500 -> expected 422 invalid_query. The recorded fixture
itself stays untouched as production truth (recorded_status 500,
served verbatim by the stub); only grade_error_class's Server500 arm
re-baselines, with its doc comments and the comparator unit tests.
Gate evidence (~/spikes/port-execution-r2/logs/c37-*): production
floor 54/54 PASS with the NEW rule — Q42 rec=500 gn4=422
code=invalid_query through the real route + forwarding engine + stub
(match200 49/49, clause-b 36/36, min J 1.000, parity-report.json
floor PASS); FULL pinned cargo test --release EXIT=0 (30 suites,
incl. the re-based comparator unit tests and the route unit tests
pinning the typed body's presence for invalid_query and absence for
backend errors); shape-class gate 52/52 unchanged; hermetic lib 23
green incl. the typed-payload assertions (Q51 message byte-identical,
QueryParserError classification). rustfmt --edition 2024 clean on all
nine touched files.
|
|
(C3.6)
gn3's species-location shorthands (search.py:246-272), WITHOUT the Hs
liftover OR-branch (OPEN-4, owner-decided; a C3.6b commit would add
it): `Hs:chr<N>:<pos>` / `Mm:...` expand to the gn3 triplet (search.py:
185-193) — AND of the species boolean term (XShuman / XSmouse), the
chromosome term (XC + chromosome number, value lowercased before the
split exactly like gn3's split_query, search.py:178-183) and the slot-2
position range (the C3.5 machinery: SI suffixes, point +/-50 kb clamp,
one-sided VALUE_GE/VALUE_LE, bases->megabases).
THE INVALID LEAF (Q42, AMENDMENT E): every condition where gn3 raises
ValueError (search.py:179-180, 197-201) — a value not splitting into
exactly chr<chromosome>:<position-spec>, a chromosome not starting
with "chr", or an unparseable position spec — emits the sentinel term
GN4InvalidLocationLeaf (verified absent from the dev index; untypeable
as free text under STEM_ALL_Z). The engine then detects the sentinel in
the parsed query's unique terms and reports SearchError::InvalidQuery
BEFORE the index is opened and any mset call — the blessed 422 for the
recorded production mset crash (Q42's 500 InvalidOperationError), via
the app-level guard (evidence-backed option (b), PORT-DESIGN C3.6,
OPEN-31 closed by AMENDMENT E). The C++ try/catch shim is deliberately
NOT extended; no crash-mimicry. An invalid leaf anywhere in the tree
poisons the whole query, exactly as gn3's mset-time OP_INVALID
rejection did (search.py:293-301 has no try/except).
Gate evidence (~/spikes/port-execution-r2/logs/c36-*): hermetic pins
green (23 lib tests: Q40/Q41 triplet forms XShuman+XC2 / XSmouse+XC1 +
slot-2 VALUE_RANGE, value lowercasing, the 7-case invalid matrix,
OR-combined poisoning, and the guard firing InvalidQuery with NO index
configured — gn3's parse-before-open order preserved); shape-CLASS
gate 52/52 cumulative vs the dev index (Q40/Q41 parse Ok with dev
count 0 — REFERENCE ONLY per AMENDMENT B; probe evidence the triplet
machinery is live: species:mouse AND chr:1 AND mb:1..200 matches, the
4.95..5.05 Mb chr1 window is genuinely empty in the 2025-02 dev data;
Q42 -> InvalidQuery); route-level scenarios: Q52 AND Q42 both
422 invalid_query through the real app with SEARCH_ENGINE=native.
rustfmt --edition 2024 clean on the touched files.
|
|
The gn3 SI-suffix FieldProcessor on slot 2 (search.py:237-241), for both
the `position` prefix and its `pos` alias, sharing one processor type
(each registration Box::leaks its own instance inside the binding; the
one-per-process parser absorbs that).
Ported faithfully from search.py:136-165:
- apply_si_suffix (136-142): trailing k/m/g (case-insensitive) multiply
by 10^3/6/9; an UNSUFFIXED value must be an integer (`int(location)`)
— `position:1.5` declines exactly where gn3 raises ValueError.
- parse_position (145-155): a `..` spec is a range with either side
optionally empty; a point expands to +/-50 000 bases clamped at 0.
- the slot-2 query (158-165): sides convert bases->megabases and keep
gn3's one-sided C++ NumberRangeProcessor semantics (VALUE_GE /
VALUE_LE for an empty side, measured identical to the bridge forms);
the degenerate both-empty range is MatchNothing (binding-report R3's
documented ("","") divergence, no production shape).
AMENDMENT A verified end-to-end: the fork's decline->MatchNothing bridge
(aa915af) is IN the dependency and behaves as documented — an
unparseable spec (`position:abc`) parses to an empty-match query and
serves 200 with 0 hits (POSABC gate row, dev index), never an <INVALID>
leaf, never a process abort; `position:abc OR alcohol` keeps its real
OR arm (xapian documents MatchNothing as boolean false). The
registration-only job was the lane's plan; no re-fixing of the library.
Q53 `position:` never reaches the processor (empty value -> free text,
recorded production 200 with 576 hits; dev reference count printed).
Gate evidence (~/spikes/port-execution-r2/logs/c35-*): hermetic pins
green (21 lib tests — slot-2 VALUE_RANGE/VALUE_GE/VALUE_LE forms, SI
case-insensitivity, point clamp, decline matrix incl. MatchNothing OR
semantics); shape-CLASS gate 49/49 cumulative vs the dev index
(Q32=1, Q33=2, Q34=1 dev gene hits at 1.05-1.06Mb/7.45-7.55Mb, Q53=1017
free-text hits, POSABC=0 — dev counts REFERENCE ONLY per AMENDMENT B).
rustfmt --edition 2024 clean on the touched files.
|
|
(C3.4)
NumberRangeProcessor on mean (slot 0), peak (1), peakmb (3), additive
(4), year (5) per search.py:234-243 (slots verified literal at
index-genenetwork:293-299); the markers carry the colon exactly like
gn3's `NumberRangeProcessor(i, prefix + ":")`.
The fork's processor parses bounds as f64 (commit 2673115, Amendment A:
the C3.4 "apply the f64 fix" workload is done at the library level) and
its bridge maps empty sides onto gn3's one-sided VALUE_LE / VALUE_GE
semantics natively — measured on the pinned toolchain before coding
(probe/, logs): `mean:10..20` -> VALUE_RANGE 0, `mean:..15` ->
VALUE_LE 0, `mean:10..` -> VALUE_GE 0. The skeleton-era 0.0/1e18
substitution is therefore superseded for these registrations by the
strictly more faithful one-sided forms.
AMENDMENT A nuance measured and handled: the fork's decline->MatchNothing
bridge (aa915af) also swallows NON-NUMERIC range bounds at the parser
level (`mean:abc..def` parses Ok to MatchNothing — probe evidence), so
the recorded production 400 "Unknown range operation" (Q52, graded 422
invalid_query) can no longer come from the parser. The engine therefore
re-checks the C++ NumberRangeProcessor contract itself (guard_numeric_
ranges): unquoted `<prefix>:` runs containing ".." must have f64-parse-
able non-empty sides, else SearchError::InvalidQuery("Unknown range
operation") before the index is opened. Residuals documented in the
guard's doc comment (inf/nan forms, the degenerate both-empty range —
binding-report R3's ("","") divergence, no production shape).
Gate evidence (~/spikes/port-execution-r2/logs/c34-*): hermetic lib pins
green incl. per-slot VALUE_RANGE/VALUE_LE/VALUE_GE parse forms (a
wrong-slot registration goes red) and the guard accept/reject matrix;
shape-CLASS gate 44/44 vs the dev index (Q28-Q31,Q35-Q39,Q44 parse Ok
with dev counts printed as REFERENCE ONLY per AMENDMENT B: mean:10..20
=1741, mean:..15=23132, peak:5..15=21857, year:2010..2020=22081, Q44
=332; Q52 -> InvalidQuery "Unknown range operation"); wide-range probe
proves slots 0/3/4/5 live on the dev index so the 0-hit shapes are
data-truth; new route-level native scenarios (tests/search_native_
route.rs) pin Q52's STATUS CLASS at the route: 422 invalid_query, with
a valid range serving 200. rustfmt --edition 2024 clean on every file
this commit touches.
|
|
native-search-engine.
Resolution: his route-level additions compose above the engine seam —
the handler now validates sort/order, asks the engine for
SORT_FETCH_LIMIT (1000) when ranking, maps SearchError via the seam's
engine_error, then computes lod, sorts, truncates to limit and fills
loci; lod/locus join SearchHit in src/search/mod.rs. His tests pass
through the seam.
KNOWN at this commit: the parity floor is BLOCKED (36 shapes) — the
route's new truncate(limit) exposes that route-level full-set grading
was a calibration-stub artifact (real gn3 caps at per_page; the stub
ignored it). Harness fix queued: set-equality moves to engine level,
status + top-10 stay route level. Landed together with that fix.
|
|
rustfmt to the native engine code (lane round-1 files were not
rustfmt-clean; pre-existing drift in config.rs, routes/search.rs,
tests/search_engine.rs is untouched — the parked house question).
|
|
attempt-4 check phase exposed.
Three defects in the committed C3.2/C3.3 test code, all found by guix
attempt 4's check phase (drv jlsjsw7i, phase `check', cargo test
--offline; log: logs/guix-c1-3.log attempt-4 section) after the build
phase had already proven the vendoring mechanism and the fork link
(287 s):
1. E0596 x2: the `terms` and `prefixed` closures borrow `qp` mutably,
so both bindings need `let mut` (rustc 1.85.1 diagnostics in the
guix log; the same E0596 sat unnoticed in logs/c3-3.log above a
bogus "LIB_EXIT=0" — the dev-shell gate script swallowed cargo's
exit status; fixed by re-running the gates with explicit status
capture, logs/fix-fulltest.log).
2. The C3.2 boolean test asserted the WRONG index term for group:B6:
`XGbxd`, with a message claiming "group:B6 must lowercase to
XGbxd". That is the r1-test BXD case: search.py:204-209 lowercases
the VALUE and concatenates behind XG, so B6 -> XGb6 (xapian-delve
on the dev index: terms XGb6, XGb6btbrf2, XGb6d2, XGb6d2f2,
XGb6d2ri exist for group B6; XGbxd exists separately for the BXD
value). Corrected to XGb6 / !XGB6. The 13a7cb1 commit message
itself repeats the wrong "group:B6 -> XGbxd" claim — history is
left intact, corrected here and in LANE-REPORT/REVIEW.
3. The C3.3 probabilistic test asserted the non-stemmed field prefix
(XD/XRF/XWK), but with STEM_ALL_Z the parsed field term is
Z+<prefix>+stem — exactly what d563910's own commit message
describes ("wiki:diabetes -> ZXWKdiabet") and what the production
indexer emits (dev index: 9678 ZXWK* terms, 47868 ZXRF* terms,
ZERO XWKZ/XRFZ/XDZ terms). Corrected to ZXD/ZXRF/ZXWK, making the
assertion pin the exact indexed term form instead of a form that
never occurs.
Because of defect 1 neither test ever ran at commit time — the
C3.3 "lib tests green / FULL suite EXIT=0" evidence was a false
positive of the broken exit capture, now re-established with real
status capture.
|
|
and aliases (C3.3).
Port search.py:229-233: description=XD, rif=XRF, wiki=XWK, plus the
uppercase aliases RIF=XRF and WIKI=XWK. With STEM_ALL_Z the parsed
field term is Z+<prefix>+stem (e.g. wiki:diabetes -> ZXWKdiabet,
matching the TermGenerator emission the production indexer writes),
pinned hermetically by prefix in native.rs tests.
tests/search_native_shapes.rs gains the C3.3 list (Q21-Q25, Q45 with
per-shape parity.rs citations). Dev-index reference counts recorded,
never compared to production: description:insulin=0, rif:insulin=0,
RIF:insulin=0, wiki:diabetes=0 (phenotype type filter: XWK text lives
only on gene documents, d.v. index-genenetwork:432-443), WIKI:diabetes=0,
symbol:insr OR rif:insulin=1. The 0s are faithful, not registration
failures: xapian-delve confirms the fields are populated on the dev
index (term ZXWKdiabet: 24 gene docs; ZXRFdiabet exists) while
ZXRFinsulin simply does not occur in the crawled RIF text and the
phenotype-typed wiki shapes are filtered to an empty set exactly as
gn3 would do on this index.
Evidence: logs/c3-3.log — build EXIT=0, lib tests green (incl. the
probabilistic-prefix assertions), FULL suite EXIT=0, SHAPE-CLASS GATE
33/33 PASS vs the dev index (cumulative C3.1+C3.2+C3.3); delve probe:
/home/kabui/zxrf-all.txt, delve-result.txt.
|
|
Port search.py:220-228: author=A, species=XS, group=XG (through the
lowercasing FieldProcessor, search.py:204-209/222-223 — xapian does
not stem boolean prefixed fields when the value starts with a
capital, so gn3 forces the lowercase term; the r1-test
LowercaseGroupProcessor pattern), tissue=XI, dataset=XDS, symbol=XY,
chr=XC, peakchr=XPC. The upstream cases rule (uppercase values
produce XDS:BXDPublish-style terms — binding-report R2: gn3-inherited
xapian behavior, not a registration bug) is documented, not pinned:
no graded shape relies on it.
tests/search_native_shapes.rs gains the C3.2 list (Q14-Q20, Q26, Q27,
Q43, Q46 with per-shape parity.rs citations); native.rs gains a
hermetic parser test pinning each registration to its exact index
term (group:B6 -> XGbxd, not XGBXD).
Evidence: logs/c3-2.log — build EXIT=0, lib tests green (incl. the
new prefix assertions), FULL suite EXIT=0, SHAPE-CLASS GATE 27/27
PASS vs the dev index (cumulative C3.1+C3.2; dev counts recorded as
reference: group:B6 = 3, symbol:insr = 1, peakchr:1 = 113, etc.).
|
|
Engine seam (C3.1).
New native engine, a Rust port of gn3 search path (gn3/api/search.py),
opt-in via SEARCH_ENGINE=native (the gn3 forwarder stays the default;
src/search/mod.rs now dispatches on config, routes/search.rs delegates):
- startup-cached QueryParser behind a mutex: custom-prefix registration
is Box::leak-based inside the binding, so a per-request parser would
leak; built once per process (binding-report §5 item 3, R2 pattern).
Minimal configuration this commit per PORT-DESIGN C3.1: Stem("en") +
STEM_ALL_Z only.
- checked parse via the fork's parse_query_checked: malformed queries
are typed errors, not panics; Err maps to SearchError::InvalidQuery
(422 invalid_query, the owner-blessed contract), carrying the
parser diagnostic. Parse happens before the index is opened, as in
gn3 (search.py:291 vs :296), so a bad query reports InvalidQuery
even when the index is misconfigured.
- XT{type} document filter: OP_FILTER(parsed, "XT"+type)
(search.py:298-300).
- FULL-match-set retrieval ignoring the forwarded limit (per-page
invariant, PORT-DESIGN §5): mset bound is a deliberately huge fixed
constant; the route keeps its validation and limit logic unchanged.
- per-hit mapping: doc data JSON -> SearchHit via serde
(search.py:302 equivalent); unknown fields flow through extra.
Decode failure maps to SearchError::Misbehaving.
- index open behind an app-level guard (the binding aborts on open
failure; see the module doc): unset XAPIAN_DB_PATH -> Internal,
unopenable path -> Unreachable.
tests/search_native.rs: error-contract scenarios (hermetic) + the
per-page invariant + dispatch pinning, one sequential test fn per the
process-global env rule; the dev-index part skips loudly when
XAPIAN_DB_PATH is unset. tests/search_native_shapes.rs: the shape-CLASS
gate (this lane s AMENDMENT-B translation of the C3.1 stage gate) --
16 shapes (Q01,Q03-Q13,Q47,Q50,Q51,Q54), each expected class cited to
its tests/search_parity.rs shape! line; dev-index counts are printed
as reference data only, never compared to production.
Evidence: logs/c3-1.log (build EXIT=0, native tests green, lib 14/14),
logs/c3-1-full.log (FULL suite EXIT=0, parity floor green),
logs/shape-class-c3-1.log (SHAPE-CLASS GATE 16/16 PASS vs the dev
index; Q51 byte-identical diagnostic; per-page invariant: limit=10
forwarded, 2330-hit full set returned).
|
|
* src/search/gn3.rs (Gn3): forwarding engine over the shared reqwest client.
* src/lib.rs (search): declare the search module.
The seam isolates backend choice behind one trait: routes validate,
wrap the envelope, and map errors; the engine resolves queries. A
native engine can replace the gn3 proxy without touching the route
contract, and the proxy stays as the transitional oracle bridge
until the native engine reaches parity. The module docs record the
xapian query grammar any native engine must accept.
|