about summary refs log tree commit diff

gn-search

The GeneNetwork search engine and indexer, extracted from genenetwork4 so the search path can be built, tested and deployed independently.

Status: the engine answers real queries against a real index, and GN4 consumes it as a library.

Crates

crates/schema/    document shape: hit, error vocabulary, index manifest
crates/engine/    reader: query parser, index open, hit decode   (moved from GN4)
crates/indexer/   writer: MySQL crawl, prefix/slot schema, sidecar manifest, delta

The engine is moved, not re-implemented: crates/engine/src/native.rs is GN4's src/search/native.rs with three changes — the index path is a parameter rather than a global, the Engine impl stayed in GN4, and one test calls search_blocking directly.

Use from GN4

gn-search-engine = { git = "https://github.com/Alexanderlacuna/gn-search", rev = "<sha>", features = ["schema"] }

Pinned by revision, as GN4 already pins its xapian bindings. features = ["schema"] supplies the utoipa::ToSchema derive that GN4's Envelope<T: Serialize + ToSchema> requires.

use gn_search_engine::{search, SearchError, SearchHit};

impl Engine for CrateEngine {
    async fn search(&self, q: &str, type_: &str, limit: u64, page: u64)
        -> Result<Vec<SearchHit>, SearchError>
    {
        let (query, search_type, path) =
            (q.to_string(), type_.to_string(), self.index_path.clone());
        tokio::task::spawn_blocking(move || search(&path, &query, &search_type, limit, page))
            .await
            .map_err(|e| SearchError::Internal(format!("search task failed: {e}")))?
    }
}

Engine names

GN4 selects a search engine with one variable. There are three values, and an unrecognised one is refused by name rather than falling through to another engine.

SEARCH_ENGINE=crate      this crate: in-process xapian, opens XAPIAN_DB_PATH
SEARCH_ENGINE=native     the former name for the same thing (the engine used to
                         live in GN4); kept so existing deployments keep working
SEARCH_ENGINE=forward    GN4's own gn3 proxy, an HTTP client — untouched by this
                         crate, and deleted together with gn3
anything else            500 internal_error naming the value it was given:
                         SEARCH_ENGINE="cratee" is not an engine: expected
                         "crate", "native" or "forward"

forward is the current default in GN4, so nothing changes for a deployment until it sets a value. The crate itself never speaks HTTP: it only opens a local index, which is why the proxy is not here.

GN4 retains the route, envelope, lod, locus, sort/order, links, error mapping, authentication and OpenAPI. None of that is below the seam.

Verified against GN4 (guix shell): the library builds, four seam/route tests pass (including the refusal of an unknown engine name), three of GN4's own engine tests pass against a 47,040-document index, and GET /search returns 200 with real hits.

Build and test

The xapian bindings require clang and a pinned xapian, so build in the GeneNetwork guix shell:

guix time-machine -C ~/.config/gn4/channels.scm -- \
  shell -L ~/guix-bioinformatics -D -f ~/project5/genenetwork4/guix.scm -- \
  bash -c 'cd gn-search && cargo test --workspace --features gn-search-engine/fixture'

93 tests pass (lib 40, grammar 26, corpus runner 9, manifest 7, hits 6, errors 4). The corpus runner grades 54 recorded shapes from genenetwork4/test/data/parity-fixtures; anything it cannot grade is reported SKIP with a reason.

Index builds

The indexer writes a glass database from the GeneNetwork MySQL dump. It reads three things from the environment and nothing else:

export SQL_URI='mysql://user:password@host/db_webqtl'          # where the data is
export XAPIAN_DB_PATH=/srv/gn-index/gen-$(date -u +%Y%m%dT%H%M)   # where the index goes
export GN4_RDF_CACHE=/tmp/gn4-rdf-cache.json                   # RIF + wiki dictionaries (optional)

A complete build, then refreshes:

# full: builds into $XAPIAN_DB_PATH, commits in chunks, swaps atomically when done
gn-indexer full --index "$XAPIAN_DB_PATH"

# afterwards, under a lock, in place
gn-indexer delta        --index "$XAPIAN_DB_PATH"   # system-versioning window: only changed datasets
gn-indexer incremental  --index "$XAPIAN_DB_PATH"   # full crawl, per-document hash diff

What a build leaves behind, and how to read it back:

ls "$XAPIAN_DB_PATH"                      # iamglass · postlist.glass · termlist.glass · position.glass · docdata.glass · flintlock
cat "$XAPIAN_DB_PATH".sidecar.json        # mode · doc_count · rif_hash · wiki_hash · per-shard {file, count, content_sha256}
xapian-delve "$XAPIAN_DB_PATH"            # "number of documents = …" — the coarse check

Then point an engine at it and ask a question — the same index serves both:

XAPIAN_DB_PATH="$XAPIAN_DB_PATH" cargo run --features serve --bin gn-search
curl 'http://127.0.0.1:7641/search?query=alcohol&type=phenotype&limit=3'

Changing the schema

The table is edited in one place — crates/schema/src/schema.rs:

add a field in an unused slot      keep SCHEMA_VERSION, extend the table; readers ignore what they do not know
change what an entry MEANS         bump SCHEMA_VERSION, rebuild the index; readers that implement the old
                                   version refuse the new index by name instead of answering from the wrong slot
then                               rebuild: `gn-indexer full --index …` writes the new digest into the sidecar

The schema, briefly

A document's meaning is a set of term prefixes and numeric value slots, defined once in crates/schema/src/schema.rs and compiled by both crates/indexer (which writes them) and crates/engine (which queries them):

prefix   XS species · XG group · XI tissue · XDS dataset · XY symbol · XC chromosome ·
         XPC peak chromosome · A authors · XRF RIF dictionary · XWK wiki
terms    XT<type>  (XTgene, XTphenotype)      Q<doctype>:<name>:<dataset>  (a document's stable identity)
slots    0 mean · 1 peak/LRS · 2 position (Mb) · 3 peak position (Mb) · 4 additive · 5 year · 6 n_samples

Writer and reader must agree on that layout. Two things enforce it:

one table       prefixes, terms and slots are declared once in gn-search-schema, which
                both crates depend on — disagreement is a compile error, not a search bug
one stamp       a build writes schema_version and the digest of that table into the sidecar;
                an engine compares it before opening the database:
                  schema-mismatch: index was written with schema v1 (9f2a…), this engine
                  implements schema v1 (0c77…) — rebuild the index or update the reader

An index built before the stamp is treated as unknown, not as a match, so existing indexes keep serving.

Licence

AGPL-3.0-or-later, as GeneNetwork. The engine and indexer are derived from genenetwork4.