about summary refs log tree commit diff
path: root/docs/search-engines.md
blob: 69c784f8b8e4a063a1b46802ef7342d908c79d0d (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
# Search engines: proxy and native

Everything below is about **how to select and run a search engine**,
not how either engine is built. Two engines implement the same
`Engine` seam (`src/search/mod.rs`): a forwarder that calls gn3's
`/api/search` (the default), and a native engine that searches a
xapian index directly from Rust. The default is unchanged — taking
this changes no behavior; the native engine is opt-in.

## One config key selects the engine

`SEARCH_ENGINE` (config field `search_engine`, `src/config.rs`):

- unset or `forward`: the gn3 forwarder — the default, and the
  parity oracle while the native engine is being verified
- `native`: the native engine

Selecting the engine is a config change; flipping back needs no
rebuild or redeploy. The native engine also needs `XAPIAN_DB_PATH`
pointing at a xapian index. On production that is `/var/lib/xapian`;
locally, an index built with the existing indexer:

    gen3/scripts/index-genenetwork create-xapian-index <index-dir> <sql-uri> <sparql-uri>

Index building stays with the existing indexer -- gn3's
`scripts/index-genenetwork`; only searching moved into gn4.

## Boolean prefix values are case-insensitive

The index stores every boolean term in lowercase (`XDSbxdpublish`,
`Awilliams`, `XSmouse`, `XCx`, `XYbdnf`, `XImrna`) and holds no
mixed-case variants at all, so the native engine lowercases the value
before forming the term: `dataset:BXDPublish` and `dataset:bxdpublish`
are the same query, as are `author:Williams`/`author:williams` and
`chr:X`/`chr:x`. gn3 only does this for `group`; for `dataset` and the
rest an uppercase value silently matched nothing (`dataset:BXDPublish`
returned 0 hits against an index holding 15 060 `BXDPublish` traits).

## The native engine reads the index, not the database

Search itself never touches MariaDB — the engine opens the index
and parses the query (the grammar is gn3's: operators, boolean and
probabilistic prefixes, SI position ranges, `Hs`/`Mm` locations;
see `genenetwork3/api/search.py`). The database is used only to
enrich each hit with `locus`, the peak marker (`PublishXRef`), and
a database failure leaves `locus` null rather than failing the
search. An instance with no usable `SQL_URI` serves complete search
results with null loci.

## What the index covers, and whose coordinates it carries

The index is a **projection of MariaDB**, rebuilt by `src/indexer/`'s two
crawls (`rows.rs`): genes (ProbeSet expression rows) and phenotypes
(published traits). The only filter is `confidentiality < 1 AND public > 0`
-- there is no species filter, no dataset whitelist and no marker cutoff --
so coverage is "every public dataset in the database" and changes only when
the database does.

Measured on the deployed index (`gn4-index`, 97 GB):

| | documents |
|---|---|
| gene (`XTgene`) | 44 842 506 |
| phenotype (`XTphenotype`) | 22 601 |

Fourteen species carry data, i.e. exactly the `Species` rows that have public
datasets: arabidopsis, barley, bat, drosophila, human, killifish, medaka,
monkey, mouse, nematode, rat, soybean, tomato, turquoise killifish. (The
`Species` table has a `poplar` row with no public data, which is why it has no
`XS` term.) Mouse, human and rat account for almost all of it -- 27.7 M,
15.2 M and 1.4 M gene documents.

**Positions are GN's current build per species, not a modern assembly.** The
gene crawl takes `ProbeSet.Chr`/`Mb` (probe position) and `Geno.Chr`/`Mb`
(mapped locus), which are the *current* columns -- not the `_2016` and
`_mm8` ones the schema also carries. What that amounts to per species is in
[genenetwork_datatypes.md](genenetwork_datatypes.md#reference-genomes-and-coordinate-generations):
hg19 for human, mm10-era for mouse, rn6-era for rat. A position from a search
hit is therefore "GN's current coordinates for that species" and should be
quoted as such; a locus whose `Locus` has no `Geno` row for its species has no
locus position at all (the join is a `LEFT JOIN` keyed on the species), and
gene documents carry the probe's position separately from the locus's.

## What the Xapian query language actually does here

The native engine is Xapian, and `q` is parsed by Xapian's own
`QueryParser` (`build_query_parser` in `src/search/native.rs`), so the
semantics are Xapian's rather than SQL's or a web search box's. The
consequences that surprise callers, all measured against the production
index:

- **Juxtaposition is OR.** `alcohol dependence` is
  `alcohol OR dependence`; `AND` has to be written. Counts for the
  phenotype index: `alcohol` ~1 200, `alcohol OR dependence` ~1 660,
  `alcohol AND dependence` 45.
- **Free text is stemmed, English, `STEM_ALL_Z`, case-insensitive.**
  `insulin`, `insulins` and `INSULIN` are the same query.
- **Quoted phrases match nothing**, as do `NEAR` and `ADJ`: the stemmed
  free-text terms carry no positions in this index, so phrase queries
  parse and then match zero documents. `"central nervous system"` is 0
  while `central AND nervous AND system` is not. A hyphenated word is a
  phrase to Xapian, so `insulin-like` also matches nothing — write
  `insulin AND like`, or just `insulin`.
- **No wildcards.** Wildcard expansion is not enabled, so `alcohol*`
  parses as `alcohol` and `alco*` matches nothing.
- **A prefix whose value is absent matches nothing**, silently: it is
  not an error, and no spelling suggestion is offered.
- Boolean (`add_custom_boolean_prefix`) fields are matched exactly with
  the value lowercased by the engine — `group`, `dataset`, `species`,
  `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position`/`pos`,
  `Hs`, `Mm`. Probabilistic ones behave like free text — `description`,
  `rif`/`RIF`, `wiki`/`WIKI`. Numeric ranges (`mean`, `peak`, `peakmb`,
  `additive`, `year`) take `field:lo..hi` with either end optional.

## Malformed queries: 422 with a typed body

The native engine checks the parse and returns 422 `invalid_query`
with a machine-readable body instead of gn3's 500s:

```json
{"error": {"type": "invalid_query", "message": "...",
           "param": "query",
           "detail": {"xapian_error_type": "QueryParserError"}}}
```

A client can read `type`/`param`/`message`, fix the query, and retry
— the committed `tests/search_selfcorrection.rs` pins that loop.

## Known quirks

- `locus: null` does not say whether the marker is absent or the
  database was unreachable; carrying that reason is planned
  response enrichment.
- With `sort`, the forwarder ranks gn3's first 1000 matches; the
  native engine retrieves the full match set before ranking, so
  sorted results can differ in the tail.
- There is no runtime indicator of which engine is serving; the
  config is the source of truth (a startup log line would fix this).
- The gn3 forward engine and the native engine differ in what a field
  is: `tissue` and sex are free text in gn3's index (the README's
  `/search` note about matching them in `description` is about that
  path), while the native parser has a `tissue` prefix. Same query,
  different reach. Phrase behaviour above is the native engine's; gn3's
  own index may carry positions.