about summary refs log tree commit diff
path: root/src/indexer
AgeCommit message (Collapse)Author
2026-10-14indexer delta: fall back to FULL on a case-only dataset renameAlexander_Kabui
* src/indexer/delta.rs (scan): enumerate dataset labels under COLLATE utf8mb4_bin so a case-only rename's old and new labels both enter scope; route to FULL when a case-variant label pair is in the affected scope. * src/indexer/run.rs (run_delta): seed the duplicate-check map with manifest labels not in scope; route to FULL when a manifest carries an unknown versioning flavour. * scripts/gn-indexer-lag.sh: honour the table-prefix seam only with the explicit test opt-in; treat a literal NULL MAX as no change; guard an empty lag value. * scripts/gn-indexer-run.sh: --index=PATH-aware argument scan.
2026-10-14indexer: add versioning-based DELTA modeAlexander_Kabui
* src/indexer/delta.rs: change detection over MariaDB system versioning (ROW_START/ROW_END): coverage marker, versioning probe, purge check, per-table candidate queries and identity-term fan-out. * src/indexer/run.rs (run_delta): reuse the streaming crawl and shard diff, scoped to the affected datasets. * src/indexer/sidecar.rs: manifest v3 with versioning_marker/versioning_flavour. * src/bin/gn-indexer.rs: add the delta mode. * scripts/gn-indexer-run.sh, scripts/gn-indexer-lag.sh: flock wrapper and lag SLA.
2026-10-14* src/indexer/run.rs (test hooks): require GN4_INDEXER_TEST_HOOKS.Alexander_Kabui
GN4_INDEXER_HALT_AFTER_COMMITS parks the process and GN4_INDEXER_COMMIT_CHUNK reshapes the commit cadence; either set by accident in a production environment could hang or change a run. Make both inert unless GN4_INDEXER_TEST_HOOKS=1.
2026-10-14* src/indexer/sidecar.rs (shard filename): use a 64-bit digest.Alexander_Kabui
shard_filename truncated the label's SHA-256 to 32 bits, so two labels sharing a sanitized stem and a 32-bit prefix could merge into one shard (last writer wins). Use 16 hex chars and assert length and distinctness.
2026-10-14* src/indexer/run.rs (duplicates): catch cross-dataset collisions.Alexander_Kabui
Duplicate-identity detection kept only the current dataset's set, so two labels differing only by case lowercased to the same <dataset> and collided across shards unseen; the manifest doc_count would then double-count and every later run would re-FULL forever. Keep an O(#datasets) lowercased-label map and flag a second distinct raw label that lowers to a seen key.
2026-10-14* src/indexer/rows.rs (crawl order): match the merge's byte order.Alexander_Kabui
The crawl ordered by dataset with the Name column's utf8mb4_general_ci collation while merge_streams compares labels by Rust byte order, so a label could arrive in two non-contiguous runs and the second begin_dataset would drop the first run's rows. Pin COLLATE utf8mb4_bin on both ORDER BY clauses so the SQL order matches the merge.
2026-10-14* src/indexer/sidecar.rs (sharding): one sidecar shard per dataset.Alexander_Kabui
A monolithic sidecar rewrote and re-read the whole identity map each run; sharding it by dataset under <index>.sidecar.json plus <index>.sidecar.d/ lets a run load and rewrite only the shards it touches and keeps resident memory at O(largest dataset). * src/indexer/run.rs: crawl the SQL sources one dataset at a time through a sink and commit every 5000 documents instead of buffering the whole crawl; a bounded-channel variant was tried and reverted (several times slower). * src/indexer/rows.rs: order and group the crawl streams by dataset. * src/bin/gn-indexer.rs: follow the sharded sink wiring.
2026-10-14* src/indexer/run.rs (write path): stream documents, not buffer all.Alexander_Kabui
Building and writing each document as the crawl yields it is leaner than collecting the whole set in a Vec, so live documents stay O(1) and peak memory stops scaling with the corpus. * src/indexer/livemem.rs: the live-document counter that proves it. * src/indexer/build.rs, src/indexer/mod.rs, src/bin/gn-indexer.rs: adapt to the streamed build and write signatures. * Cargo.lock: repin xapian-rs 97fd343 -> 70c5eac (the write-API set).
2026-10-12indexer: rustfmt the module re-exports and lib registration.Alexander_Kabui
2026-10-12* src/bin/gn-indexer.rs: new gn-indexer binary. FULL bootstraps a freshAlexander_Kabui
index; INCREMENTAL crawls, hashes each document's complete source, replaces only changed documents by identity term, deletes vanished identity terms, and writes nothing when nothing changed. * src/indexer/schema.rs: the identity/type/prefix/slot definitions shared by the writer and the native reader. * src/indexer/rows.rs: the gene and phenotype crawl. Both queries carry confidentiality < 1 AND public > 0; the Python phenotype query has no WHERE clause (the metadata leak this closes by construction). * src/indexer/build.rs: document construction, mirroring index_genes / index_phenotypes call for call (TermGenerator configuration, prefix order, value slots, popped-before-storage fields, identity term). * src/indexer/pyjson.rs: byte-exact reproduction of Python json.dumps (separators, ensure_ascii escaping, shortest round-trip float repr with CPython's half-to-even tie-breaking). * src/indexer/cache.rs: the RDF RIF/wiki dictionaries, reproducing build_rdf_cache(remove_common_words=True) from the authoritative cache file; each dictionary is one monolithic hashed input. * src/indexer/sha256.rs: dependency-free SHA-256 for the source and cache hashes. * src/indexer/sidecar.rs, src/indexer/run.rs: per-document source hashes in a sidecar next to the index (atomic rename strictly after the index commit), FULL and INCREMENTAL orchestration, identity-term uniqueness assertion, and the startup key-set mismatch fallback. * src/lib.rs: register the indexer module. * tests/data/pyfloat.txt: reference-docdata float-token fixture for the serializer test (generated read-only from the reference index).