about summary refs log tree commit diff
path: root/doc
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-12 13:52:12 +0000
committerPjotr <wrk@bosa.lan>2026-11-12 13:52:12 +0000
commit3fab7adf0ec147dceb150d861fdbd5f5b3693384 (patch)
tree9e2f0b50a6d639eb821a9295bdfb4723f9a01535 /doc
parent5cdd45571c550708f691855891c7919fe076ab05 (diff)
downloadai-memory-3fab7adf0ec147dceb150d861fdbd5f5b3693384.tar.gz
Add a topic index: Xapian over the store, driven through Guile
SQLite stays the truth; this is a derived index that lets a reader enter by
concept rather than by symbol. Two commands, one prefix vocabulary:

  ai-memory --index DIR update-index [--rebuild]
  ai-memory --index DIR search 'topic:pve repo:genenetwork4'

Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:,
date:. Free text is stemmed; default operator AND, --or to widen. Topic
terms come from the session id split on hyphens, so a session named
"gemma-pve" answers topic:pve and topic:gemma -- naming a session after a
concept is how an entry becomes findable by it.

Why Guile and a subprocess: libxapian is C++-only, so a foreign call from
Odin means a hand-written C shim with its own exception boundary; guile-xapian
already ships one, tested and maintained. So scripts/index.scm owns every
Xapian call (426 lines: schema constants, record builders, updater, query
parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines
of process plumbing. The boundary is a store path in and JSON out, which also
means the search reply needs no parser on this side.

Incremental in the way that matters: documents are deleted by their unique
term and re-added, and Xapian applies that as changed postings only. The whole
body is re-tokenised each run (~1 s at a few hundred documents) which buys not
keeping any "what did I index last time" state that could go stale. Two
discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by
term silently appends (the string lands on the docid overload), so delete then
add; and Database-get-document by term returns nothing, so no digest compare
was attempted.

The package's check phase now exercises it end to end -- throwaway store,
one entry, update-index, then `search topic:index | grep -q '"count":1'` --
because a check that only links proves nothing about the four Guile
dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where
index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for
guile-xapian and guile-sqlite3.

doc/SKILL.md gains the usage, the prefix list and the reason for the
subprocess.
Diffstat (limited to 'doc')
-rw-r--r--doc/SKILL.md38
1 files changed, 38 insertions, 0 deletions
diff --git a/doc/SKILL.md b/doc/SKILL.md
index fb17189..570bde7 100644
--- a/doc/SKILL.md
+++ b/doc/SKILL.md
@@ -33,6 +33,10 @@ guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style
   "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib"
 ```
 
+`guix build -f guix.scm` also runs the package's check phase, which builds a
+throwaway store, records an entry, indexes it and queries it -- that is what
+proves the Guile side is wired up.
+
 The store defaults to `<repo>/ai-memory.sqlite`; override with `--db PATH` or
 `$AI_MEMORY_DB`.  `--repo PATH` / `-C PATH` selects the repository, and may be
 given before or after the command.
@@ -68,6 +72,40 @@ ai-memory -C REPO attach HEAD --session fix-parser
 `record` and `attach` import the commit on demand, so they work immediately after
 committing even if `sync` has not run yet.
 
+## The topic index (optional, derived)
+
+A second, *derived* store: a Xapian index over the SQLite one, so a reader can
+enter by **concept** ("topic:pve") or by repository and kind ("repo:genenetwork4
+type:commit") instead of by symbol name. It is disposable -- delete the
+directory and `update-index --rebuild` -- and SQLite stays the truth.
+
+```sh
+ai-memory --index DIR update-index            # bring it up to date
+ai-memory --index DIR update-index --rebuild  # after a schema change
+ai-memory --index DIR search 'topic:pve'
+ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4'
+```
+
+(The directory defaults to `<store>.d/index`, override with `--index` or
+`$AI_MEMORY_INDEX`. `search` without `--index` keeps using SQLite FTS.)
+
+**Prefixes**: `topic:`, `repo:`, `type:` (`reasoning`|`commit`), `session:`,
+`commit:` (7-character hash), `date:` (`yyyymmdd`). Everything else is free
+text, stemmed. The default operator is AND; `--or` widens it.
+
+Topic terms come from the **session id**, split on hyphens: a session called
+`gemma-pve` answers `topic:gemma` and `topic:pve`. So the cheapest way to make
+an entry findable by concept is to name the session after the concept.
+
+Why a subprocess and a Guile script: `libxapian` is C++-only, so a foreign call
+from Odin needs a hand-written C shim. `guile-xapian` already has one, tested
+and maintained (`scripts/index.scm` owns every Xapian call; ai-memory shells
+out to it). The boundary is a store path in, JSON out.
+
+The JSON a search returns is the record as indexed -- `key`, `repo`, `session`,
+`kind`, `date`, `commit`, `title`, `body`, `terms` -- plus `docid` and
+`percent`. `count` is how many came back, `matches` how many the index found.
+
 ## Rules
 
 - **Never write to `commits`, `symbols` or `touches` by hand.**  They are derived;