about summary refs log tree commit diff

Search engines: proxy and native

Everything below is about how to select and run a search engine, not how either engine is built. Two engines implement the same Engine seam (src/search/mod.rs): a forwarder that calls gn3's /api/search (the default), and a native engine that searches a xapian index directly from Rust. The default is unchanged — taking this changes no behavior; the native engine is opt-in.

One config key selects the engine

SEARCH_ENGINE (config field search_engine, src/config.rs):

  • unset or forward: the gn3 forwarder — the default, and the parity oracle while the native engine is being verified
  • native: the native engine

Selecting the engine is a config change; flipping back needs no rebuild or redeploy. The native engine also needs XAPIAN_DB_PATH pointing at a xapian index. On production that is /var/lib/xapian; locally, an index built with the existing indexer:

gen3/scripts/index-genenetwork create-xapian-index <index-dir> <sql-uri> <sparql-uri>

Index building stays with the existing indexer -- gn3's scripts/index-genenetwork; only searching moved into gn4.

Boolean prefix values are case-insensitive

The index stores every boolean term in lowercase (XDSbxdpublish, Awilliams, XSmouse, XCx, XYbdnf, XImrna) and holds no mixed-case variants at all, so the native engine lowercases the value before forming the term: dataset:BXDPublish and dataset:bxdpublish are the same query, as are author:Williams/author:williams and chr:X/chr:x. gn3 only does this for group; for dataset and the rest an uppercase value silently matched nothing (dataset:BXDPublish returned 0 hits against an index holding 15 060 BXDPublish traits).

The native engine reads the index, not the database

Search itself never touches MariaDB — the engine opens the index and parses the query (the grammar is gn3's: operators, boolean and probabilistic prefixes, SI position ranges, Hs/Mm locations; see genenetwork3/api/search.py). The database is used only to enrich each hit with locus, the peak marker (PublishXRef), and a database failure leaves locus null rather than failing the search. An instance with no usable SQL_URI serves complete search results with null loci.

What the index covers, and whose coordinates it carries

The index is a projection of MariaDB, rebuilt by src/indexer/'s two crawls (rows.rs): genes (ProbeSet expression rows) and phenotypes (published traits). The only filter is confidentiality < 1 AND public > 0 -- there is no species filter, no dataset whitelist and no marker cutoff -- so coverage is "every public dataset in the database" and changes only when the database does.

Measured on the deployed index (gn4-index, 97 GB):

documents
gene (XTgene) 44 842 506
phenotype (XTphenotype) 22 601

Fourteen species carry data, i.e. exactly the Species rows that have public datasets: arabidopsis, barley, bat, drosophila, human, killifish, medaka, monkey, mouse, nematode, rat, soybean, tomato, turquoise killifish. (The Species table has a poplar row with no public data, which is why it has no XS term.) Mouse, human and rat account for almost all of it -- 27.7 M, 15.2 M and 1.4 M gene documents.

Positions are GN's current build per species, not a modern assembly. The gene crawl takes ProbeSet.Chr/Mb (probe position) and Geno.Chr/Mb (mapped locus), which are the current columns -- not the _2016 and _mm8 ones the schema also carries. What that amounts to per species is in genenetwork_datatypes.md: hg19 for human, mm10-era for mouse, rn6-era for rat. A position from a search hit is therefore "GN's current coordinates for that species" and should be quoted as such; a locus whose Locus has no Geno row for its species has no locus position at all (the join is a LEFT JOIN keyed on the species), and gene documents carry the probe's position separately from the locus's.

What the Xapian query language actually does here

The native engine is Xapian, and q is parsed by Xapian's own QueryParser (build_query_parser in src/search/native.rs), so the semantics are Xapian's rather than SQL's or a web search box's. The consequences that surprise callers, all measured against the production index:

  • Juxtaposition is OR. alcohol dependence is alcohol OR dependence; AND has to be written. Counts for the phenotype index: alcohol ~1 200, alcohol OR dependence ~1 660, alcohol AND dependence 45.
  • Free text is stemmed, English, STEM_ALL_Z, case-insensitive. insulin, insulins and INSULIN are the same query.
  • Quoted phrases match nothing, as do NEAR and ADJ: the stemmed free-text terms carry no positions in this index, so phrase queries parse and then match zero documents. "central nervous system" is 0 while central AND nervous AND system is not. A hyphenated word is a phrase to Xapian, so insulin-like also matches nothing — write insulin AND like, or just insulin.
  • No wildcards. Wildcard expansion is not enabled, so alcohol* parses as alcohol and alco* matches nothing.
  • A prefix whose value is absent matches nothing, silently: it is not an error, and no spelling suggestion is offered.
  • Boolean (add_custom_boolean_prefix) fields are matched exactly with the value lowercased by the engine — group, dataset, species, author, tissue, symbol, chr, peakchr, position/pos, Hs, Mm. Probabilistic ones behave like free text — description, rif/RIF, wiki/WIKI. Numeric ranges (mean, peak, peakmb, additive, year) take field:lo..hi with either end optional.

Malformed queries: 422 with a typed body

The native engine checks the parse and returns 422 invalid_query with a machine-readable body instead of gn3's 500s:

{"error": {"type": "invalid_query", "message": "...",
           "param": "query",
           "detail": {"xapian_error_type": "QueryParserError"}}}

A client can read type/param/message, fix the query, and retry — the committed tests/search_selfcorrection.rs pins that loop.

Known quirks

  • locus: null does not say whether the marker is absent or the database was unreachable; carrying that reason is planned response enrichment.
  • With sort, the forwarder ranks gn3's first 1000 matches; the native engine retrieves the full match set before ranking, so sorted results can differ in the tail.
  • There is no runtime indicator of which engine is serving; the config is the source of truth (a startup log line would fix this).
  • The gn3 forward engine and the native engine differ in what a field is: tissue and sex are free text in gn3's index (the README's /search note about matching them in description is about that path), while the native parser has a tissue prefix. Same query, different reach. Phrase behaviour above is the native engine's; gn3's own index may carry positions.