diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-07 14:46:13 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-07 14:46:13 +0000 |
| commit | ac20f209da4740ae8a79b7a1482c540fc4846fa2 (patch) | |
| tree | 7cc72e3c8cb4a53af6e99062afa6e76644ca012b /docs | |
| parent | 84838f6e8f42a2a794641bc766f273fbd152e7fd (diff) | |
| download | genenetwork4-ac20f209da4740ae8a79b7a1482c540fc4846fa2.tar.gz | |
docs: search is Xapian, and what that means for a query
The AI instructions described the field prefixes but not the semantics, so a reader would reasonably assume AND between words, working quoted phrases and wildcard prefixes -- none of which is true here. Measuring against the production index: juxtaposition is OR (`alcohol` 1 200 vs `alcohol AND dependence` 45), free text is stemmed and case-insensitive, quoted phrases and NEAR/ADJ match *nothing* because the stemmed terms carry no positions (a hyphenated word is a phrase, so `insulin-like` also matches nothing), and wildcards are not enabled (`alcohol*` is just `alcohol`). * docs/www/ai-instruction.md (3. Search): rewritten around the Xapian QueryParser semantics and a table of exact / stemmed / numeric-range fields. * docs/search-engines.md: the same, for whoever maintains the engine, plus the gn3-forward-vs-native difference in what counts as a field.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/search-engines.md | 37 | ||||
| -rw-r--r-- | docs/www/ai-instruction.md | 47 |
2 files changed, 76 insertions, 8 deletions
diff --git a/docs/search-engines.md b/docs/search-engines.md index c2393a5..0a43cf2 100644 --- a/docs/search-engines.md +++ b/docs/search-engines.md @@ -47,6 +47,37 @@ a database failure leaves `locus` null rather than failing the search. An instance with no usable `SQL_URI` serves complete search results with null loci. +## What the Xapian query language actually does here + +The native engine is Xapian, and `q` is parsed by Xapian's own +`QueryParser` (`build_query_parser` in `src/search/native.rs`), so the +semantics are Xapian's rather than SQL's or a web search box's. The +consequences that surprise callers, all measured against the production +index: + +- **Juxtaposition is OR.** `alcohol dependence` is + `alcohol OR dependence`; `AND` has to be written. Counts for the + phenotype index: `alcohol` ~1 200, `alcohol OR dependence` ~1 660, + `alcohol AND dependence` 45. +- **Free text is stemmed, English, `STEM_ALL_Z`, case-insensitive.** + `insulin`, `insulins` and `INSULIN` are the same query. +- **Quoted phrases match nothing**, as do `NEAR` and `ADJ`: the stemmed + free-text terms carry no positions in this index, so phrase queries + parse and then match zero documents. `"central nervous system"` is 0 + while `central AND nervous AND system` is not. A hyphenated word is a + phrase to Xapian, so `insulin-like` also matches nothing — write + `insulin AND like`, or just `insulin`. +- **No wildcards.** Wildcard expansion is not enabled, so `alcohol*` + parses as `alcohol` and `alco*` matches nothing. +- **A prefix whose value is absent matches nothing**, silently: it is + not an error, and no spelling suggestion is offered. +- Boolean (`add_custom_boolean_prefix`) fields are matched exactly with + the value lowercased by the engine — `group`, `dataset`, `species`, + `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position`/`pos`, + `Hs`, `Mm`. Probabilistic ones behave like free text — `description`, + `rif`/`RIF`, `wiki`/`WIKI`. Numeric ranges (`mean`, `peak`, `peakmb`, + `additive`, `year`) take `field:lo..hi` with either end optional. + ## Malformed queries: 422 with a typed body The native engine checks the parse and returns 422 `invalid_query` @@ -71,3 +102,9 @@ A client can read `type`/`param`/`message`, fix the query, and retry sorted results can differ in the tail. - There is no runtime indicator of which engine is serving; the config is the source of truth (a startup log line would fix this). +- The gn3 forward engine and the native engine differ in what a field + is: `tissue` and sex are free text in gn3's index (the README's + `/search` note about matching them in `description` is about that + path), while the native parser has a `tissue` prefix. Same query, + different reach. Phrase behaviour above is the native engine's; gn3's + own index may carry positions. diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md index 404731b..2e3a59d 100644 --- a/docs/www/ai-instruction.md +++ b/docs/www/ai-instruction.md @@ -130,17 +130,48 @@ Paths are relative to `https://vesuvio.pu.ac.ke/gn4`. | `/publications/{pubmed_id}/traits` | traits a paper reports, across groups. `{pubmed_id}` is the **PubMed** id (`2760876`), which a trait record carries as `pubmed_id` — not the internal `publication_id` (`116`) beside it, which will 404 there | | `/accessions/{accession}` | resolve a GN accession (`GN600`) to its dataset | | `/mapping-methods` | QTL methods offered | -| `/search?q={query}` | full-text search over published phenotypes | +| `/search?q={query}` | Xapian full-text search over published phenotypes | ### Search -`q` takes the Xapian query grammar: free text, `AND`/`OR`/`NOT`, -`"quoted phrases"`, and field prefixes — `group:BXD`, -`dataset:BXDPublish`, `author:smith`, `species:mouse`, `symbol:insr`, -`rif:insulin`, `chr:1`, `peakchr:1`, plus numeric ranges on -`mean`, `peak`, `peakmb`, `additive` and `year` written -`field:lo..hi` (either end may be omitted). Boolean prefix values are -case-insensitive. `type` selects `phenotype` (default) or `gene`. +`/search` is served by a **Xapian** index, queried with Xapian's own +`QueryParser` in the same configuration GeneNetwork3 used. So the syntax +is Xapian's, not Google's or SQL's, and a few of its behaviours will +surprise you: + +* **Juxtaposed terms are OR** — `alcohol dependence` means + `alcohol OR dependence`, not AND. Write `AND` when you mean it: on + this index `alcohol` matches ~1 200 traits, `alcohol OR dependence` + ~1 660, `alcohol AND dependence` 45. +* **Free text is English-stemmed and case-insensitive** — `insulin`, + `insulins` and `INSULIN` all match the same documents. You do not need + to guess singular/plural forms. +* **Quoted phrases match nothing on this index.** `"central nervous + system"` returns 0 even though that text is everywhere in the + descriptions, because the free-text terms are indexed stemmed and + carry no positions in this index. `NEAR` and `ADJ` are in the same boat, and a + hyphenated word (`insulin-like`) is parsed as a phrase, so it matches + nothing either. Use `term AND term` instead of a phrase. +* **No wildcards.** `alcohol*` behaves exactly like `alcohol`, and + `alco*` matches nothing — there is no prefix matching. +* Operators are `AND`, `OR`, `NOT`, `XOR` (uppercase) and parentheses. + A dangling operator or an unbalanced quote is a `422 invalid_query` + with Xapian's own classification in `detail`; fix the query, do not + retry. + +Fields are written `field:value`: + +| kind | fields | behaviour | +|---|---|---| +| exact (boolean) | `group`, `dataset`, `species`, `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position` (alias `pos`), `Hs`, `Mm` | match the stored term exactly; values are lowercased, so case does not matter (`dataset:BXDPublish` works). `Hs`/`Mm` are species-location shorthands | +| stemmed text | `description`, `rif` (alias `RIF`), `wiki` (alias `WIKI`) | weighted, stemmed like free text | +| numeric ranges | `mean`, `peak`, `peakmb`, `additive`, `year` | `field:lo..hi`; either end may be omitted (`mean:10..`, `mean:..15`) | + +A prefix whose value is not in the index simply matches nothing — it is +not an error. `tissue:foo` returning 0 means "no such value", not "bad +syntax". Check a hit's own fields to see the values that exist. + +`type` selects `phenotype` (default) or `gene`. `limit` defaults to 20 and caps at **100**; `page` pages through hits. There is **no total** in the response — to count, page until a short |
