# Search engines: proxy and native Everything below is about **how to select and run a search engine**, not how either engine is built. Two engines implement the same `Engine` seam (`src/search/mod.rs`): a forwarder that calls gn3's `/api/search` (the default), and a native engine that searches a xapian index directly from Rust. The default is unchanged — taking this changes no behavior; the native engine is opt-in. ## One config key selects the engine `SEARCH_ENGINE` (config field `search_engine`, `src/config.rs`): - unset or `forward`: the gn3 forwarder — the default, and the parity oracle while the native engine is being verified - `native`: the native engine Selecting the engine is a config change; flipping back needs no rebuild or redeploy. The native engine also needs `XAPIAN_DB_PATH` pointing at a xapian index. On production that is `/var/lib/xapian`; locally, an index built with the existing indexer: gen3/scripts/index-genenetwork create-xapian-index Index building stays with the existing indexer -- gn3's `scripts/index-genenetwork`; only searching moved into gn4. ## Boolean prefix values are case-insensitive The index stores every boolean term in lowercase (`XDSbxdpublish`, `Awilliams`, `XSmouse`, `XCx`, `XYbdnf`, `XImrna`) and holds no mixed-case variants at all, so the native engine lowercases the value before forming the term: `dataset:BXDPublish` and `dataset:bxdpublish` are the same query, as are `author:Williams`/`author:williams` and `chr:X`/`chr:x`. gn3 only does this for `group`; for `dataset` and the rest an uppercase value silently matched nothing (`dataset:BXDPublish` returned 0 hits against an index holding 15 060 `BXDPublish` traits). ## The native engine reads the index, not the database Search itself never touches MariaDB — the engine opens the index and parses the query (the grammar is gn3's: operators, boolean and probabilistic prefixes, SI position ranges, `Hs`/`Mm` locations; see `genenetwork3/api/search.py`). The database is used only to enrich each hit with `locus`, the peak marker (`PublishXRef`), and a database failure leaves `locus` null rather than failing the search. An instance with no usable `SQL_URI` serves complete search results with null loci. ## What the index covers, and whose coordinates it carries The index is a **projection of MariaDB**, rebuilt by `src/indexer/`'s two crawls (`rows.rs`): genes (ProbeSet expression rows) and phenotypes (published traits). The only filter is `confidentiality < 1 AND public > 0` -- there is no species filter, no dataset whitelist and no marker cutoff -- so coverage is "every public dataset in the database" and changes only when the database does. Measured on the deployed index (`gn4-index`, 97 GB): | | documents | |---|---| | gene (`XTgene`) | 44 842 506 | | phenotype (`XTphenotype`) | 22 601 | Fourteen species carry data, i.e. exactly the `Species` rows that have public datasets: arabidopsis, barley, bat, drosophila, human, killifish, medaka, monkey, mouse, nematode, rat, soybean, tomato, turquoise killifish. (The `Species` table has a `poplar` row with no public data, which is why it has no `XS` term.) Mouse, human and rat account for almost all of it -- 27.7 M, 15.2 M and 1.4 M gene documents. **Positions are GN's current build per species, not a modern assembly.** The gene crawl takes `ProbeSet.Chr`/`Mb` (probe position) and `Geno.Chr`/`Mb` (mapped locus), which are the *current* columns -- not the `_2016` and `_mm8` ones the schema also carries. What that amounts to per species is in [genenetwork_datatypes.md](genenetwork_datatypes.md#reference-genomes-and-coordinate-generations): hg19 for human, mm10-era for mouse, rn6-era for rat. A position from a search hit is therefore "GN's current coordinates for that species" and should be quoted as such; a locus whose `Locus` has no `Geno` row for its species has no locus position at all (the join is a `LEFT JOIN` keyed on the species), and gene documents carry the probe's position separately from the locus's. ## What the Xapian query language actually does here The native engine is Xapian, and `q` is parsed by Xapian's own `QueryParser` (`build_query_parser` in `src/search/native.rs`), so the semantics are Xapian's rather than SQL's or a web search box's. The consequences that surprise callers, all measured against the production index: - **Juxtaposition is OR.** `alcohol dependence` is `alcohol OR dependence`; `AND` has to be written. Counts for the phenotype index: `alcohol` ~1 200, `alcohol OR dependence` ~1 660, `alcohol AND dependence` 45. - **Free text is stemmed, English, `STEM_ALL_Z`, case-insensitive.** `insulin`, `insulins` and `INSULIN` are the same query. - **Quoted phrases match nothing**, as do `NEAR` and `ADJ`: the stemmed free-text terms carry no positions in this index, so phrase queries parse and then match zero documents. `"central nervous system"` is 0 while `central AND nervous AND system` is not. A hyphenated word is a phrase to Xapian, so `insulin-like` also matches nothing — write `insulin AND like`, or just `insulin`. - **No wildcards.** Wildcard expansion is not enabled, so `alcohol*` parses as `alcohol` and `alco*` matches nothing. - **A prefix whose value is absent matches nothing**, silently: it is not an error, and no spelling suggestion is offered. - Boolean (`add_custom_boolean_prefix`) fields are matched exactly with the value lowercased by the engine — `group`, `dataset`, `species`, `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position`/`pos`, `Hs`, `Mm`. Probabilistic ones behave like free text — `description`, `rif`/`RIF`, `wiki`/`WIKI`. Numeric ranges (`mean`, `peak`, `peakmb`, `additive`, `year`) take `field:lo..hi` with either end optional. ## Malformed queries: 422 with a typed body The native engine checks the parse and returns 422 `invalid_query` with a machine-readable body instead of gn3's 500s: ```json {"error": {"type": "invalid_query", "message": "...", "param": "query", "detail": {"xapian_error_type": "QueryParserError"}}} ``` A client can read `type`/`param`/`message`, fix the query, and retry — the committed `tests/search_selfcorrection.rs` pins that loop. ## Known quirks - `locus: null` does not say whether the marker is absent or the database was unreachable; carrying that reason is planned response enrichment. - With `sort`, the forwarder ranks gn3's first 1000 matches; the native engine retrieves the full match set before ranking, so sorted results can differ in the tail. - There is no runtime indicator of which engine is serving; the config is the source of truth (a startup log line would fix this). - The gn3 forward engine and the native engine differ in what a field is: `tissue` and sex are free text in gn3's index (the README's `/search` note about matching them in `description` is about that path), while the native parser has a `tissue` prefix. Same query, different reach. Phrase behaviour above is the native engine's; gn3's own index may carry positions.