diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-12 13:52:12 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-12 13:52:12 +0000 |
| commit | 3fab7adf0ec147dceb150d861fdbd5f5b3693384 (patch) | |
| tree | 9e2f0b50a6d639eb821a9295bdfb4723f9a01535 /doc | |
| parent | 5cdd45571c550708f691855891c7919fe076ab05 (diff) | |
| download | ai-memory-3fab7adf0ec147dceb150d861fdbd5f5b3693384.tar.gz | |
Add a topic index: Xapian over the store, driven through Guile
SQLite stays the truth; this is a derived index that lets a reader enter by concept rather than by symbol. Two commands, one prefix vocabulary: ai-memory --index DIR update-index [--rebuild] ai-memory --index DIR search 'topic:pve repo:genenetwork4' Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:, date:. Free text is stemmed; default operator AND, --or to widen. Topic terms come from the session id split on hyphens, so a session named "gemma-pve" answers topic:pve and topic:gemma -- naming a session after a concept is how an entry becomes findable by it. Why Guile and a subprocess: libxapian is C++-only, so a foreign call from Odin means a hand-written C shim with its own exception boundary; guile-xapian already ships one, tested and maintained. So scripts/index.scm owns every Xapian call (426 lines: schema constants, record builders, updater, query parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines of process plumbing. The boundary is a store path in and JSON out, which also means the search reply needs no parser on this side. Incremental in the way that matters: documents are deleted by their unique term and re-added, and Xapian applies that as changed postings only. The whole body is re-tokenised each run (~1 s at a few hundred documents) which buys not keeping any "what did I index last time" state that could go stale. Two discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by term silently appends (the string lands on the docid overload), so delete then add; and Database-get-document by term returns nothing, so no digest compare was attempted. The package's check phase now exercises it end to end -- throwaway store, one entry, update-index, then `search topic:index | grep -q '"count":1'` -- because a check that only links proves nothing about the four Guile dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for guile-xapian and guile-sqlite3. doc/SKILL.md gains the usage, the prefix list and the reason for the subprocess.
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/SKILL.md | 38 |
1 files changed, 38 insertions, 0 deletions
diff --git a/doc/SKILL.md b/doc/SKILL.md index fb17189..570bde7 100644 --- a/doc/SKILL.md +++ b/doc/SKILL.md @@ -33,6 +33,10 @@ guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib" ``` +`guix build -f guix.scm` also runs the package's check phase, which builds a +throwaway store, records an entry, indexes it and queries it -- that is what +proves the Guile side is wired up. + The store defaults to `<repo>/ai-memory.sqlite`; override with `--db PATH` or `$AI_MEMORY_DB`. `--repo PATH` / `-C PATH` selects the repository, and may be given before or after the command. @@ -68,6 +72,40 @@ ai-memory -C REPO attach HEAD --session fix-parser `record` and `attach` import the commit on demand, so they work immediately after committing even if `sync` has not run yet. +## The topic index (optional, derived) + +A second, *derived* store: a Xapian index over the SQLite one, so a reader can +enter by **concept** ("topic:pve") or by repository and kind ("repo:genenetwork4 +type:commit") instead of by symbol name. It is disposable -- delete the +directory and `update-index --rebuild` -- and SQLite stays the truth. + +```sh +ai-memory --index DIR update-index # bring it up to date +ai-memory --index DIR update-index --rebuild # after a schema change +ai-memory --index DIR search 'topic:pve' +ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4' +``` + +(The directory defaults to `<store>.d/index`, override with `--index` or +`$AI_MEMORY_INDEX`. `search` without `--index` keeps using SQLite FTS.) + +**Prefixes**: `topic:`, `repo:`, `type:` (`reasoning`|`commit`), `session:`, +`commit:` (7-character hash), `date:` (`yyyymmdd`). Everything else is free +text, stemmed. The default operator is AND; `--or` widens it. + +Topic terms come from the **session id**, split on hyphens: a session called +`gemma-pve` answers `topic:gemma` and `topic:pve`. So the cheapest way to make +an entry findable by concept is to name the session after the concept. + +Why a subprocess and a Guile script: `libxapian` is C++-only, so a foreign call +from Odin needs a hand-written C shim. `guile-xapian` already has one, tested +and maintained (`scripts/index.scm` owns every Xapian call; ai-memory shells +out to it). The boundary is a store path in, JSON out. + +The JSON a search returns is the record as indexed -- `key`, `repo`, `session`, +`kind`, `date`, `commit`, `title`, `body`, `terms` -- plus `docid` and +`percent`. `count` is how many came back, `matches` how many the index found. + ## Rules - **Never write to `commits`, `symbols` or `touches` by hand.** They are derived; |
