about summary refs log tree commit diff
path: root/docs/search-engines.md
AgeCommit message (Collapse)Author
9 daysdocs: which reference genome the coordinates are on, and what the index coversPjotr
Two things a reader needs before quoting a position from this API, both measured against db_webqtl rather than inferred from the schema: - No table stores an assembly id. The only labels are prose in Species.MenuName: human hg19, mouse mm10, rat "mRatBN7.2/rn7", fly dm6, nematode ce10, arabidopsis araTha1, medaka Oryl23, bat gsor23, killifish MPIA_NFZ_2, tomato GCF_001406875.1, soybean JGI-PGF v2 2021, barley and poplar unlabelled. - Every position column exists in three generations (Mb, Mb_2016, legacy mouse Mb_mm8; likewise Length/Length_2016/Length_mm8 on Chr_Length) and the labels do not all match the numbers. Human is hg19 in both generations (chr1 249,250,621; SOX2 chr3 181.43, not hg38's 181.71). Mouse Length_2016 chr1 = 197,195,432 is mm9 exactly, while current Length = 197,426,691 matches no standard build -- but mouse probes are mm10-era (Sox2 chr3 34.5). Rat current Length chr1 = 282,763,074 is rn6 exactly, though MenuName advertises rn7. Geno.Mb_2016 exists only for human, mouse, soybean, barley; the mouse-named Mb_mm8 is populated for barley too. docs/search-engines.md gains the coverage side: the index is a projection of MariaDB with no species filter (only confidentiality < 1 AND public > 0), so 14 species -- every Species row with public data -- with 44,842,506 gene and 22,601 phenotype documents, dominated by mouse (27.7M), human (15.2M) and rat (1.4M). The crawl reads the *current* columns, so the index inherits hg19 / mm10-era / rn6-era coordinates, never a modern assembly; and rat positions deserve suspicion (Sox2's 44 probes span chr2 1.2-117.6 Mb).
14 daysdocs: search is Xapian, and what that means for a queryPjotr
The AI instructions described the field prefixes but not the semantics, so a reader would reasonably assume AND between words, working quoted phrases and wildcard prefixes -- none of which is true here. Measuring against the production index: juxtaposition is OR (`alcohol` 1 200 vs `alcohol AND dependence` 45), free text is stemmed and case-insensitive, quoted phrases and NEAR/ADJ match *nothing* because the stemmed terms carry no positions (a hyphenated word is a phrase, so `insulin-like` also matches nothing), and wildcards are not enabled (`alcohol*` is just `alcohol`). * docs/www/ai-instruction.md (3. Search): rewritten around the Xapian QueryParser semantics and a table of exact / stemmed / numeric-range fields. * docs/search-engines.md: the same, for whoever maintains the engine, plus the gn3-forward-vs-native difference in what counts as a field.
2026-11-06search: lowercase boolean prefix valuesPjotr
The index stores every boolean term in lowercase and holds no mixed-case variants at all: XDSbxdpublish (15 060 docs), Awilliams (10 958), XSmouse, XCx, XYbdnf, XImrna all exist, while XDSBXDPublish, AWilliams, XSMouse, XCX, XYBDNF and XIMRNA report "not in database" on both the dev index and a full production one. Matching the value exactly therefore rejected the way a caller naturally writes it -- dataset:BXDPublish returned 0 hits against an index holding 15 060 BXDPublish traits, author:Williams 0 against 10 958 williams authors. Lowercase the value for every boolean prefix, generalising the group-only rule gn3 already has (search.py:204-209). No graded parity shape changes: they all use lowercase values already. Pinned per prefix in boolean_prefixes_parse_to_exact_index_terms, so a registration that regresses to an exact add_boolean_prefix fails there instead of silently returning zero hits.
2026-10-08docs/search-engines.md: the indexer is gn3's scripts/index-genenetworkAlexander_Kabui
-- name the repo so the command is findable from gn4.
2026-10-08docs/search-engines.md: give the indexer command its scriptAlexander_Kabui
prefix (scripts/index-genenetwork, verified against the click CLI); plain 'planned response enrichment' instead of naming a proposal; state the behavior-preserving default up front.
2026-10-08docs/search-engines.md: how to select and run a search engine (theAlexander_Kabui
gn3 proxy by default, the native xapian engine on SEARCH_ENGINE=native + XAPIAN_DB_PATH); README.md (search): point to it. House style: scope line, takeaway headings, code pointers, known quirks (locus null reason, sort tail difference, no runtime indicator).