| Age | Commit message (Collapse) | Author |
|
|
|
* src/config.rs (DbMode, Config.db_mode, parse_db_mode): add
GN4_DB_MODE=dev|production in the layered config; dev runs with no
configured DB; fail-closed parse (a typo refuses).
* src/db.rs (DbError, pool, probe), src/error.rs (ApiError::from_db):
mint a typed 503 db_unavailable at the db::pool() choke point,
distinguishing Unavailable from Query.
* src/main.rs: loud no-DB startup line; production refuses to start
(exit non-zero) before axum::serve.
* src/routes/{accessions,datasets,groups,mapping_methods,publications,species,traits}.rs:
route every DB error through ApiError::from_db.
* src/bin/gn-indexer.rs: refuse to build an index when the DB is unreachable.
* tests/no_db.rs: pin no-DB degradation (search 200 partial, DB routes 503).
His ask (DM 2026-10-12): "Make it an option to switch it off for
development machines? I mean mariadb." The no-DB hardening is ours
(review-driven). Default GN4_DB_MODE left unflipped (=dev); the flip needs
owner sign-off.
|
|
* src/indexer/delta.rs: change detection over MariaDB system versioning
(ROW_START/ROW_END): coverage marker, versioning probe, purge check,
per-table candidate queries and identity-term fan-out.
* src/indexer/run.rs (run_delta): reuse the streaming crawl and shard diff,
scoped to the affected datasets.
* src/indexer/sidecar.rs: manifest v3 with versioning_marker/versioning_flavour.
* src/bin/gn-indexer.rs: add the delta mode.
* scripts/gn-indexer-run.sh, scripts/gn-indexer-lag.sh: flock wrapper and lag SLA.
|
|
A monolithic sidecar rewrote and re-read the whole identity map each run;
sharding it by dataset under <index>.sidecar.json plus <index>.sidecar.d/
lets a run load and rewrite only the shards it touches and keeps resident
memory at O(largest dataset).
* src/indexer/run.rs: crawl the SQL sources one dataset at a time through a
sink and commit every 5000 documents instead of buffering the whole crawl;
a bounded-channel variant was tried and reverted (several times slower).
* src/indexer/rows.rs: order and group the crawl streams by dataset.
* src/bin/gn-indexer.rs: follow the sharded sink wiring.
|
|
Building and writing each document as the crawl yields it is leaner than
collecting the whole set in a Vec, so live documents stay O(1) and peak
memory stops scaling with the corpus.
* src/indexer/livemem.rs: the live-document counter that proves it.
* src/indexer/build.rs, src/indexer/mod.rs, src/bin/gn-indexer.rs: adapt
to the streamed build and write signatures.
* Cargo.lock: repin xapian-rs 97fd343 -> 70c5eac (the write-API set).
|
|
index; INCREMENTAL crawls, hashes each document's complete source,
replaces only changed documents by identity term, deletes vanished
identity terms, and writes nothing when nothing changed.
* src/indexer/schema.rs: the identity/type/prefix/slot definitions
shared by the writer and the native reader.
* src/indexer/rows.rs: the gene and phenotype crawl. Both queries
carry confidentiality < 1 AND public > 0; the Python phenotype query
has no WHERE clause (the metadata leak this closes by construction).
* src/indexer/build.rs: document construction, mirroring index_genes /
index_phenotypes call for call (TermGenerator configuration, prefix
order, value slots, popped-before-storage fields, identity term).
* src/indexer/pyjson.rs: byte-exact reproduction of Python json.dumps
(separators, ensure_ascii escaping, shortest round-trip float repr
with CPython's half-to-even tie-breaking).
* src/indexer/cache.rs: the RDF RIF/wiki dictionaries, reproducing
build_rdf_cache(remove_common_words=True) from the authoritative
cache file; each dictionary is one monolithic hashed input.
* src/indexer/sha256.rs: dependency-free SHA-256 for the source and
cache hashes.
* src/indexer/sidecar.rs, src/indexer/run.rs: per-document source
hashes in a sidecar next to the index (atomic rename strictly after
the index commit), FULL and INCREMENTAL orchestration, identity-term
uniqueness assertion, and the startup key-set mismatch fallback.
* src/lib.rs: register the indexer module.
* tests/data/pyfloat.txt: reference-docdata float-token fixture for the
serializer test (generated read-only from the reference index).
|