Search engines: proxy and native
Everything below is about how to select and run a search engine,
not how either engine is built. Two engines implement the same
Engine seam (src/search/mod.rs): a forwarder that calls gn3's
/api/search (the default), and a native engine that searches a
xapian index directly from Rust. The default is unchanged — taking
this changes no behavior; the native engine is opt-in.
One config key selects the engine
SEARCH_ENGINE (config field search_engine, src/config.rs):
- unset or
forward: the gn3 forwarder — the default, and the parity oracle while the native engine is being verified native: the native engine
Selecting the engine is a config change; flipping back needs no
rebuild or redeploy. The native engine also needs XAPIAN_DB_PATH
pointing at a xapian index. On production that is /var/lib/xapian;
locally, an index built with the existing indexer:
gen3/scripts/index-genenetwork create-xapian-index <index-dir> <sql-uri> <sparql-uri>
Index building stays with the existing indexer -- gn3's
scripts/index-genenetwork; only searching moved into gn4.
Boolean prefix values are case-insensitive
The index stores every boolean term in lowercase (XDSbxdpublish,
Awilliams, XSmouse, XCx, XYbdnf, XImrna) and holds no
mixed-case variants at all, so the native engine lowercases the value
before forming the term: dataset:BXDPublish and dataset:bxdpublish
are the same query, as are author:Williams/author:williams and
chr:X/chr:x. gn3 only does this for group; for dataset and the
rest an uppercase value silently matched nothing (dataset:BXDPublish
returned 0 hits against an index holding 15 060 BXDPublish traits).
The native engine reads the index, not the database
Search itself never touches MariaDB — the engine opens the index
and parses the query (the grammar is gn3's: operators, boolean and
probabilistic prefixes, SI position ranges, Hs/Mm locations;
see genenetwork3/api/search.py). The database is used only to
enrich each hit with locus, the peak marker (PublishXRef), and
a database failure leaves locus null rather than failing the
search. An instance with no usable SQL_URI serves complete search
results with null loci.
What the index covers, and whose coordinates it carries
The index is a projection of MariaDB, rebuilt by src/indexer/'s two
crawls (rows.rs): genes (ProbeSet expression rows) and phenotypes
(published traits). The only filter is confidentiality < 1 AND public > 0
-- there is no species filter, no dataset whitelist and no marker cutoff --
so coverage is "every public dataset in the database" and changes only when
the database does.
Measured on the deployed index (gn4-index, 97 GB):
| documents | |
|---|---|
gene (XTgene) |
44 842 506 |
phenotype (XTphenotype) |
22 601 |
Fourteen species carry data, i.e. exactly the Species rows that have public
datasets: arabidopsis, barley, bat, drosophila, human, killifish, medaka,
monkey, mouse, nematode, rat, soybean, tomato, turquoise killifish. (The
Species table has a poplar row with no public data, which is why it has no
XS term.) Mouse, human and rat account for almost all of it -- 27.7 M,
15.2 M and 1.4 M gene documents.
Positions are GN's current build per species, not a modern assembly. The
gene crawl takes ProbeSet.Chr/Mb (probe position) and Geno.Chr/Mb
(mapped locus), which are the current columns -- not the _2016 and
_mm8 ones the schema also carries. What that amounts to per species is in
genenetwork_datatypes.md:
hg19 for human, mm10-era for mouse, rn6-era for rat. A position from a search
hit is therefore "GN's current coordinates for that species" and should be
quoted as such; a locus whose Locus has no Geno row for its species has no
locus position at all (the join is a LEFT JOIN keyed on the species), and
gene documents carry the probe's position separately from the locus's.
What the Xapian query language actually does here
The native engine is Xapian, and q is parsed by Xapian's own
QueryParser (build_query_parser in src/search/native.rs), so the
semantics are Xapian's rather than SQL's or a web search box's. The
consequences that surprise callers, all measured against the production
index:
- Juxtaposition is OR.
alcohol dependenceisalcohol OR dependence;ANDhas to be written. Counts for the phenotype index:alcohol~1 200,alcohol OR dependence~1 660,alcohol AND dependence45. - Free text is stemmed, English,
STEM_ALL_Z, case-insensitive.insulin,insulinsandINSULINare the same query. - Quoted phrases match nothing, as do
NEARandADJ: the stemmed free-text terms carry no positions in this index, so phrase queries parse and then match zero documents."central nervous system"is 0 whilecentral AND nervous AND systemis not. A hyphenated word is a phrase to Xapian, soinsulin-likealso matches nothing — writeinsulin AND like, or justinsulin. - No wildcards. Wildcard expansion is not enabled, so
alcohol*parses asalcoholandalco*matches nothing. - A prefix whose value is absent matches nothing, silently: it is not an error, and no spelling suggestion is offered.
- Boolean (
add_custom_boolean_prefix) fields are matched exactly with the value lowercased by the engine —group,dataset,species,author,tissue,symbol,chr,peakchr,position/pos,Hs,Mm. Probabilistic ones behave like free text —description,rif/RIF,wiki/WIKI. Numeric ranges (mean,peak,peakmb,additive,year) takefield:lo..hiwith either end optional.
Malformed queries: 422 with a typed body
The native engine checks the parse and returns 422 invalid_query
with a machine-readable body instead of gn3's 500s:
{"error": {"type": "invalid_query", "message": "...",
"param": "query",
"detail": {"xapian_error_type": "QueryParserError"}}}
A client can read type/param/message, fix the query, and retry
— the committed tests/search_selfcorrection.rs pins that loop.
Known quirks
locus: nulldoes not say whether the marker is absent or the database was unreachable; carrying that reason is planned response enrichment.- With
sort, the forwarder ranks gn3's first 1000 matches; the native engine retrieves the full match set before ranking, so sorted results can differ in the tail. - There is no runtime indicator of which engine is serving; the config is the source of truth (a startup log line would fix this).
- The gn3 forward engine and the native engine differ in what a field
is:
tissueand sex are free text in gn3's index (the README's/searchnote about matching them indescriptionis about that path), while the native parser has atissueprefix. Same query, different reach. Phrase behaviour above is the native engine's; gn3's own index may carry positions.
