about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-07 14:46:13 +0000
committerPjotr <wrk@bosa.lan>2026-11-07 14:46:13 +0000
commitac20f209da4740ae8a79b7a1482c540fc4846fa2 (patch)
tree7cc72e3c8cb4a53af6e99062afa6e76644ca012b /docs
parent84838f6e8f42a2a794641bc766f273fbd152e7fd (diff)
downloadgenenetwork4-ac20f209da4740ae8a79b7a1482c540fc4846fa2.tar.gz
docs: search is Xapian, and what that means for a query
The AI instructions described the field prefixes but not the semantics,
so a reader would reasonably assume AND between words, working quoted
phrases and wildcard prefixes -- none of which is true here. Measuring
against the production index: juxtaposition is OR (`alcohol` 1 200 vs
`alcohol AND dependence` 45), free text is stemmed and
case-insensitive, quoted phrases and NEAR/ADJ match *nothing* because
the stemmed terms carry no positions (a hyphenated word is a phrase, so
`insulin-like` also matches nothing), and wildcards are not enabled
(`alcohol*` is just `alcohol`).

* docs/www/ai-instruction.md (3. Search): rewritten around the Xapian
QueryParser semantics and a table of exact / stemmed / numeric-range
fields.
* docs/search-engines.md: the same, for whoever maintains the engine,
plus the gn3-forward-vs-native difference in what counts as a field.
Diffstat (limited to 'docs')
-rw-r--r--docs/search-engines.md37
-rw-r--r--docs/www/ai-instruction.md47
2 files changed, 76 insertions, 8 deletions
diff --git a/docs/search-engines.md b/docs/search-engines.md
index c2393a5..0a43cf2 100644
--- a/docs/search-engines.md
+++ b/docs/search-engines.md
@@ -47,6 +47,37 @@ a database failure leaves `locus` null rather than failing the
 search. An instance with no usable `SQL_URI` serves complete search
 results with null loci.
 
+## What the Xapian query language actually does here
+
+The native engine is Xapian, and `q` is parsed by Xapian's own
+`QueryParser` (`build_query_parser` in `src/search/native.rs`), so the
+semantics are Xapian's rather than SQL's or a web search box's. The
+consequences that surprise callers, all measured against the production
+index:
+
+- **Juxtaposition is OR.** `alcohol dependence` is
+  `alcohol OR dependence`; `AND` has to be written. Counts for the
+  phenotype index: `alcohol` ~1 200, `alcohol OR dependence` ~1 660,
+  `alcohol AND dependence` 45.
+- **Free text is stemmed, English, `STEM_ALL_Z`, case-insensitive.**
+  `insulin`, `insulins` and `INSULIN` are the same query.
+- **Quoted phrases match nothing**, as do `NEAR` and `ADJ`: the stemmed
+  free-text terms carry no positions in this index, so phrase queries
+  parse and then match zero documents. `"central nervous system"` is 0
+  while `central AND nervous AND system` is not. A hyphenated word is a
+  phrase to Xapian, so `insulin-like` also matches nothing — write
+  `insulin AND like`, or just `insulin`.
+- **No wildcards.** Wildcard expansion is not enabled, so `alcohol*`
+  parses as `alcohol` and `alco*` matches nothing.
+- **A prefix whose value is absent matches nothing**, silently: it is
+  not an error, and no spelling suggestion is offered.
+- Boolean (`add_custom_boolean_prefix`) fields are matched exactly with
+  the value lowercased by the engine — `group`, `dataset`, `species`,
+  `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position`/`pos`,
+  `Hs`, `Mm`. Probabilistic ones behave like free text — `description`,
+  `rif`/`RIF`, `wiki`/`WIKI`. Numeric ranges (`mean`, `peak`, `peakmb`,
+  `additive`, `year`) take `field:lo..hi` with either end optional.
+
 ## Malformed queries: 422 with a typed body
 
 The native engine checks the parse and returns 422 `invalid_query`
@@ -71,3 +102,9 @@ A client can read `type`/`param`/`message`, fix the query, and retry
   sorted results can differ in the tail.
 - There is no runtime indicator of which engine is serving; the
   config is the source of truth (a startup log line would fix this).
+- The gn3 forward engine and the native engine differ in what a field
+  is: `tissue` and sex are free text in gn3's index (the README's
+  `/search` note about matching them in `description` is about that
+  path), while the native parser has a `tissue` prefix. Same query,
+  different reach. Phrase behaviour above is the native engine's; gn3's
+  own index may carry positions.
diff --git a/docs/www/ai-instruction.md b/docs/www/ai-instruction.md
index 404731b..2e3a59d 100644
--- a/docs/www/ai-instruction.md
+++ b/docs/www/ai-instruction.md
@@ -130,17 +130,48 @@ Paths are relative to `https://vesuvio.pu.ac.ke/gn4`.
 | `/publications/{pubmed_id}/traits` | traits a paper reports, across groups. `{pubmed_id}` is the **PubMed** id (`2760876`), which a trait record carries as `pubmed_id` — not the internal `publication_id` (`116`) beside it, which will 404 there |
 | `/accessions/{accession}` | resolve a GN accession (`GN600`) to its dataset |
 | `/mapping-methods` | QTL methods offered |
-| `/search?q={query}` | full-text search over published phenotypes |
+| `/search?q={query}` | Xapian full-text search over published phenotypes |
 
 ### Search
 
-`q` takes the Xapian query grammar: free text, `AND`/`OR`/`NOT`,
-`"quoted phrases"`, and field prefixes — `group:BXD`,
-`dataset:BXDPublish`, `author:smith`, `species:mouse`, `symbol:insr`,
-`rif:insulin`, `chr:1`, `peakchr:1`, plus numeric ranges on
-`mean`, `peak`, `peakmb`, `additive` and `year` written
-`field:lo..hi` (either end may be omitted). Boolean prefix values are
-case-insensitive. `type` selects `phenotype` (default) or `gene`.
+`/search` is served by a **Xapian** index, queried with Xapian's own
+`QueryParser` in the same configuration GeneNetwork3 used. So the syntax
+is Xapian's, not Google's or SQL's, and a few of its behaviours will
+surprise you:
+
+* **Juxtaposed terms are OR** — `alcohol dependence` means
+  `alcohol OR dependence`, not AND. Write `AND` when you mean it: on
+  this index `alcohol` matches ~1 200 traits, `alcohol OR dependence`
+  ~1 660, `alcohol AND dependence` 45.
+* **Free text is English-stemmed and case-insensitive** — `insulin`,
+  `insulins` and `INSULIN` all match the same documents. You do not need
+  to guess singular/plural forms.
+* **Quoted phrases match nothing on this index.** `"central nervous
+  system"` returns 0 even though that text is everywhere in the
+  descriptions, because the free-text terms are indexed stemmed and
+  carry no positions in this index. `NEAR` and `ADJ` are in the same boat, and a
+  hyphenated word (`insulin-like`) is parsed as a phrase, so it matches
+  nothing either. Use `term AND term` instead of a phrase.
+* **No wildcards.** `alcohol*` behaves exactly like `alcohol`, and
+  `alco*` matches nothing — there is no prefix matching.
+* Operators are `AND`, `OR`, `NOT`, `XOR` (uppercase) and parentheses.
+  A dangling operator or an unbalanced quote is a `422 invalid_query`
+  with Xapian's own classification in `detail`; fix the query, do not
+  retry.
+
+Fields are written `field:value`:
+
+| kind | fields | behaviour |
+|---|---|---|
+| exact (boolean) | `group`, `dataset`, `species`, `author`, `tissue`, `symbol`, `chr`, `peakchr`, `position` (alias `pos`), `Hs`, `Mm` | match the stored term exactly; values are lowercased, so case does not matter (`dataset:BXDPublish` works). `Hs`/`Mm` are species-location shorthands |
+| stemmed text | `description`, `rif` (alias `RIF`), `wiki` (alias `WIKI`) | weighted, stemmed like free text |
+| numeric ranges | `mean`, `peak`, `peakmb`, `additive`, `year` | `field:lo..hi`; either end may be omitted (`mean:10..`, `mean:..15`) |
+
+A prefix whose value is not in the index simply matches nothing — it is
+not an error. `tissue:foo` returning 0 means "no such value", not "bad
+syntax". Check a hit's own fields to see the values that exist.
+
+`type` selects `phenotype` (default) or `gene`.
 
 `limit` defaults to 20 and caps at **100**; `page` pages through hits.
 There is **no total** in the response — to count, page until a short