1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
# gn-search
The GeneNetwork search engine and indexer, extracted from
[genenetwork4](https://git.genenetwork.org/public/genenetwork4) so the search
path can be built, tested and deployed independently.
Status: the engine answers real queries against a real index, and GN4 consumes it
as a library.
## Crates
```
crates/schema/ document shape: hit, error vocabulary, index manifest
crates/engine/ reader: query parser, index open, hit decode (moved from GN4)
crates/indexer/ writer: MySQL crawl, prefix/slot schema, sidecar manifest, delta
```
The engine is moved, not re-implemented: `crates/engine/src/native.rs` is GN4's
`src/search/native.rs` with three changes — the index path is a parameter rather
than a global, the `Engine` impl stayed in GN4, and one test calls
`search_blocking` directly.
## Use from GN4
```toml
gn-search-engine = { git = "https://github.com/Alexanderlacuna/gn-search", rev = "<sha>", features = ["schema"] }
```
Pinned by revision, as GN4 already pins its xapian bindings. `features =
["schema"]` supplies the `utoipa::ToSchema` derive that GN4's
`Envelope<T: Serialize + ToSchema>` requires.
```rust
use gn_search_engine::{search, SearchError, SearchHit};
impl Engine for CrateEngine {
async fn search(&self, q: &str, type_: &str, limit: u64, page: u64)
-> Result<Vec<SearchHit>, SearchError>
{
let (query, search_type, path) =
(q.to_string(), type_.to_string(), self.index_path.clone());
tokio::task::spawn_blocking(move || search(&path, &query, &search_type, limit, page))
.await
.map_err(|e| SearchError::Internal(format!("search task failed: {e}")))?
}
}
```
## Engine names
GN4 selects a search engine with one variable. There are three values, and an
unrecognised one is refused by name rather than falling through to another
engine.
```
SEARCH_ENGINE=crate this crate: in-process xapian, opens XAPIAN_DB_PATH
SEARCH_ENGINE=native the former name for the same thing (the engine used to
live in GN4); kept so existing deployments keep working
SEARCH_ENGINE=forward GN4's own gn3 proxy, an HTTP client — untouched by this
crate, and deleted together with gn3
anything else 500 internal_error naming the value it was given:
SEARCH_ENGINE="cratee" is not an engine: expected
"crate", "native" or "forward"
```
`forward` is the current default in GN4, so nothing changes for a deployment
until it sets a value. The crate itself never speaks HTTP: it only opens a local
index, which is why the proxy is not here.
GN4 retains the route, envelope, `lod`, `locus`, sort/order, links, error
mapping, authentication and OpenAPI. None of that is below the seam.
Verified against GN4 (guix shell): the library builds, four seam/route tests
pass (including the refusal of an unknown engine name), three of GN4's own
engine tests pass against a 47,040-document index, and `GET /search` returns 200
with real hits.
## Build and test
The xapian bindings require clang and a pinned xapian, so build in the
GeneNetwork guix shell:
```sh
guix time-machine -C ~/.config/gn4/channels.scm -- \
shell -L ~/guix-bioinformatics -D -f ~/project5/genenetwork4/guix.scm -- \
bash -c 'cd gn-search && cargo test --workspace --features gn-search-engine/fixture'
```
93 tests pass (lib 40, grammar 26, corpus runner 9, manifest 7, hits 6, errors 4).
The corpus runner grades 54 recorded shapes from
`genenetwork4/test/data/parity-fixtures`; anything it cannot grade is reported
`SKIP` with a reason.
## Index builds
The indexer writes a glass database from the GeneNetwork MySQL dump. It reads
three things from the environment and nothing else:
```sh
export SQL_URI='mysql://user:password@host/db_webqtl' # where the data is
export XAPIAN_DB_PATH=/srv/gn-index/gen-$(date -u +%Y%m%dT%H%M) # where the index goes
export GN4_RDF_CACHE=/tmp/gn4-rdf-cache.json # RIF + wiki dictionaries (optional)
```
A complete build, then refreshes:
```sh
# full: builds into $XAPIAN_DB_PATH, commits in chunks, swaps atomically when done
gn-indexer full --index "$XAPIAN_DB_PATH"
# afterwards, under a lock, in place
gn-indexer delta --index "$XAPIAN_DB_PATH" # system-versioning window: only changed datasets
gn-indexer incremental --index "$XAPIAN_DB_PATH" # full crawl, per-document hash diff
```
What a build leaves behind, and how to read it back:
```sh
ls "$XAPIAN_DB_PATH" # iamglass · postlist.glass · termlist.glass · position.glass · docdata.glass · flintlock
cat "$XAPIAN_DB_PATH".sidecar.json # mode · doc_count · rif_hash · wiki_hash · per-shard {file, count, content_sha256}
xapian-delve "$XAPIAN_DB_PATH" # "number of documents = …" — the coarse check
```
Then point an engine at it and ask a question — the same index serves both:
```sh
XAPIAN_DB_PATH="$XAPIAN_DB_PATH" cargo run --features serve --bin gn-search
curl 'http://127.0.0.1:7641/search?query=alcohol&type=phenotype&limit=3'
```
## Changing the schema
The table is edited in one place — `crates/schema/src/schema.rs`:
```
add a field in an unused slot keep SCHEMA_VERSION, extend the table; readers ignore what they do not know
change what an entry MEANS bump SCHEMA_VERSION, rebuild the index; readers that implement the old
version refuse the new index by name instead of answering from the wrong slot
then rebuild: `gn-indexer full --index …` writes the new digest into the sidecar
```
## The schema, briefly
A document's meaning is a set of term prefixes and numeric value slots, defined
once in `crates/schema/src/schema.rs` and compiled by both `crates/indexer` (which
writes them) and `crates/engine` (which queries them):
```
prefix XS species · XG group · XI tissue · XDS dataset · XY symbol · XC chromosome ·
XPC peak chromosome · A authors · XRF RIF dictionary · XWK wiki
terms XT<type> (XTgene, XTphenotype) Q<doctype>:<name>:<dataset> (a document's stable identity)
slots 0 mean · 1 peak/LRS · 2 position (Mb) · 3 peak position (Mb) · 4 additive · 5 year · 6 n_samples
```
Writer and reader must agree on that layout. Two things enforce it:
```
one table prefixes, terms and slots are declared once in gn-search-schema, which
both crates depend on — disagreement is a compile error, not a search bug
one stamp a build writes schema_version and the digest of that table into the sidecar;
an engine compares it before opening the database:
schema-mismatch: index was written with schema v1 (9f2a…), this engine
implements schema v1 (0c77…) — rebuild the index or update the reader
```
An index built before the stamp is treated as *unknown*, not as a match, so
existing indexes keep serving.
## Licence
AGPL-3.0-or-later, as GeneNetwork. The engine and indexer are derived from
genenetwork4.
|