about summary refs log tree commit diff
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-12 13:52:12 +0000
committerPjotr <wrk@bosa.lan>2026-11-12 13:52:12 +0000
commit3fab7adf0ec147dceb150d861fdbd5f5b3693384 (patch)
tree9e2f0b50a6d639eb821a9295bdfb4723f9a01535
parent5cdd45571c550708f691855891c7919fe076ab05 (diff)
downloadai-memory-3fab7adf0ec147dceb150d861fdbd5f5b3693384.tar.gz
Add a topic index: Xapian over the store, driven through Guile
SQLite stays the truth; this is a derived index that lets a reader enter by
concept rather than by symbol. Two commands, one prefix vocabulary:

  ai-memory --index DIR update-index [--rebuild]
  ai-memory --index DIR search 'topic:pve repo:genenetwork4'

Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:,
date:. Free text is stemmed; default operator AND, --or to widen. Topic
terms come from the session id split on hyphens, so a session named
"gemma-pve" answers topic:pve and topic:gemma -- naming a session after a
concept is how an entry becomes findable by it.

Why Guile and a subprocess: libxapian is C++-only, so a foreign call from
Odin means a hand-written C shim with its own exception boundary; guile-xapian
already ships one, tested and maintained. So scripts/index.scm owns every
Xapian call (426 lines: schema constants, record builders, updater, query
parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines
of process plumbing. The boundary is a store path in and JSON out, which also
means the search reply needs no parser on this side.

Incremental in the way that matters: documents are deleted by their unique
term and re-added, and Xapian applies that as changed postings only. The whole
body is re-tokenised each run (~1 s at a few hundred documents) which buys not
keeping any "what did I index last time" state that could go stale. Two
discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by
term silently appends (the string lands on the docid overload), so delete then
add; and Database-get-document by term returns nothing, so no digest compare
was attempted.

The package's check phase now exercises it end to end -- throwaway store,
one entry, update-index, then `search topic:index | grep -q '"count":1'` --
because a check that only links proves nothing about the four Guile
dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where
index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for
guile-xapian and guile-sqlite3.

doc/SKILL.md gains the usage, the prefix list and the reason for the
subprocess.
-rw-r--r--doc/SKILL.md38
-rw-r--r--guix.scm54
-rw-r--r--index.odin145
-rw-r--r--main.odin44
-rw-r--r--scripts/index.scm426
5 files changed, 701 insertions, 6 deletions
diff --git a/doc/SKILL.md b/doc/SKILL.md
index fb17189..570bde7 100644
--- a/doc/SKILL.md
+++ b/doc/SKILL.md
@@ -33,6 +33,10 @@ guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style
   "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib"
 ```
 
+`guix build -f guix.scm` also runs the package's check phase, which builds a
+throwaway store, records an entry, indexes it and queries it -- that is what
+proves the Guile side is wired up.
+
 The store defaults to `<repo>/ai-memory.sqlite`; override with `--db PATH` or
 `$AI_MEMORY_DB`.  `--repo PATH` / `-C PATH` selects the repository, and may be
 given before or after the command.
@@ -68,6 +72,40 @@ ai-memory -C REPO attach HEAD --session fix-parser
 `record` and `attach` import the commit on demand, so they work immediately after
 committing even if `sync` has not run yet.
 
+## The topic index (optional, derived)
+
+A second, *derived* store: a Xapian index over the SQLite one, so a reader can
+enter by **concept** ("topic:pve") or by repository and kind ("repo:genenetwork4
+type:commit") instead of by symbol name. It is disposable -- delete the
+directory and `update-index --rebuild` -- and SQLite stays the truth.
+
+```sh
+ai-memory --index DIR update-index            # bring it up to date
+ai-memory --index DIR update-index --rebuild  # after a schema change
+ai-memory --index DIR search 'topic:pve'
+ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4'
+```
+
+(The directory defaults to `<store>.d/index`, override with `--index` or
+`$AI_MEMORY_INDEX`. `search` without `--index` keeps using SQLite FTS.)
+
+**Prefixes**: `topic:`, `repo:`, `type:` (`reasoning`|`commit`), `session:`,
+`commit:` (7-character hash), `date:` (`yyyymmdd`). Everything else is free
+text, stemmed. The default operator is AND; `--or` widens it.
+
+Topic terms come from the **session id**, split on hyphens: a session called
+`gemma-pve` answers `topic:gemma` and `topic:pve`. So the cheapest way to make
+an entry findable by concept is to name the session after the concept.
+
+Why a subprocess and a Guile script: `libxapian` is C++-only, so a foreign call
+from Odin needs a hand-written C shim. `guile-xapian` already has one, tested
+and maintained (`scripts/index.scm` owns every Xapian call; ai-memory shells
+out to it). The boundary is a store path in, JSON out.
+
+The JSON a search returns is the record as indexed -- `key`, `repo`, `session`,
+`kind`, `date`, `commit`, `title`, `body`, `terms` -- plus `docid` and
+`percent`. `count` is how many came back, `matches` how many the index found.
+
 ## Rules
 
 - **Never write to `commits`, `symbols` or `touches` by hand.**  They are derived;
diff --git a/guix.scm b/guix.scm
index 62fcf75..72ab494 100644
--- a/guix.scm
+++ b/guix.scm
@@ -38,6 +38,8 @@
   #:use-module (guix packages)
   #:use-module (guix utils)
   #:use-module (gnu packages bash)
+  #:use-module (gnu packages guile)
+  #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages llvm)
   #:use-module (gnu packages sqlite))
 
@@ -215,7 +217,10 @@ to link the system SQLite3 instead of a bundled copy.")
     ;; store, which are machine state rather than source.
     (source (local-file %source-dir #:recursive? #t #:select? source-file?))
     (build-system gnu-build-system)
-    (inputs (list sqlite))                  ;libsqlite3.so, linked at run time
+    ;; libsqlite3.so is linked at run time; the topic index (scripts/index.scm)
+    ;; is a Guile program, because libxapian is C++-only and guile-xapian
+    ;; already has the C++ shim that a foreign call would otherwise need.
+    (inputs (list sqlite guile-3.0 guile-xapian guile-sqlite3))
     (native-inputs (list odin odin-sqlite3))
     (arguments
      (list
@@ -246,11 +251,54 @@ to link the system SQLite3 instead of a bundled copy.")
                   (apply invoke "odin" "build" "."
                          "-out:bin/ai-memory-check" flags)
                   ;; Links and runs; fail loudly if libsqlite3 is missing.
-                  (invoke "./bin/ai-memory-check"))))
+                  (invoke "./bin/ai-memory-check")
+                  ;; The topic index end to end: a store with one entry, the
+                  ;; Guile indexer over it, and a prefixed query -- which is
+                  ;; what proves the script's dependencies are complete.
+                  (mkdir-p "tmp")
+                  (setenv "AI_MEMORY_DB"
+                          (string-append (getcwd) "/tmp/store.sqlite"))
+                  (setenv "AI_MEMORY_INDEX_SCRIPT"
+                          (string-append (getcwd) "/scripts/index.scm"))
+                  (setenv "GUILE" (string-append #$(this-package-input "guile")
+                                                 "/bin/guile"))
+                  ;; guile-xapian needs guile-lib (snippets) and the sqlite
+                  ;; binding; both come from the package inputs, so the check
+                  ;; fails here rather than in a user's first `update-index'.
+                  (setenv "GUILE_LOAD_PATH"
+                          (string-append #$(this-package-input "guile-xapian")
+                                         "/share/guile/site/3.0:"
+                                         #$(this-package-input "guile-sqlite3")
+                                         "/share/guile/site/3.0"))
+                  (invoke "./bin/ai-memory" "init")
+                  ;; A session id is what gives the entry its topic: terms
+                  ;; ("topic-index" -> topic:topic and topic:index), so the
+                  ;; query below is a real concept query, not a text match.
+                  (invoke "./bin/ai-memory" "record" "--kind" "conclusion"
+                          "--session" "topic-index"
+                          "--body" "the index answers topic: queries")
+                  (invoke "./bin/ai-memory" "--index" "tmp/index" "update-index")
+                  ;; The reply is JSON on stdout: pipe it into grep rather
+                  ;; than capturing it (with-output-to-string captures the
+                  ;; Scheme port, not the child's stdout).
+                  (invoke "sh" "-c"
+                          (string-append "./bin/ai-memory --index tmp/index"
+                                         " search topic:index"
+                                         " | grep -q '\"count\":1'")))))
             (replace 'install
               (lambda _
                 (install-file "bin/ai-memory"
-                              (string-append #$output "/bin"))))))))
+                              (string-append #$output "/bin"))
+                ;; The indexer travels with the binary: index.odin looks for
+                ;; ../share/ai-memory/index.scm beside it.
+                (install-file "scripts/index.scm"
+                              (string-append #$output "/share/ai-memory"))
+                (wrap-program (string-append #$output "/bin/ai-memory")
+                  `("GUILE" = (,(string-append #$(this-package-input "guile") "/bin/guile")))
+                  `("GUILE_LOAD_PATH" = (,(string-append #$(this-package-input "guile-xapian")
+                                                         "/share/guile/site/3.0:"
+                                                         #$(this-package-input "guile-sqlite3")
+                                                         "/share/guile/site/3.0"))))))))))
     (home-page "https://github.com/ai-memory/ai-memory")
     (synopsis "Memory for AI assistants")
     (description
diff --git a/index.odin b/index.odin
new file mode 100644
index 0000000..eb6e3f4
--- /dev/null
+++ b/index.odin
@@ -0,0 +1,145 @@
+package main
+
+// The topic index: a derived Xapian index over the store, which lets a
+// reader enter by concept ("topic:pve") or by repository and type rather than
+// by symbol name.
+//
+// SQLite stays the source of truth. This is a subprocess because libxapian is
+// C++-only: a call into it from Odin means a hand-written C shim, while
+// guile-xapian already has one, tested. So the boundary is a script path in
+// and JSON out -- see scripts/index.scm, which owns every Xapian call.
+//
+//	ai-memory --index DIR update-index [--rebuild]
+//	ai-memory --index DIR search 'topic:pve repo:genenetwork4'
+//
+// Where the pieces are:
+//   --index PATH                or $AI_MEMORY_INDEX       the Xapian directory
+//   --index-script PATH         or $AI_MEMORY_INDEX_SCRIPT  scripts/index.scm
+//   $GUILE                      or "guile"                the interpreter
+
+import "core:fmt"
+import "core:os"
+import "core:strings"
+
+// Set by the global options.
+INDEX_PATH: string
+INDEX_SCRIPT: string
+
+// The script defaults to $AI_MEMORY_INDEX_SCRIPT, then to an installed copy
+// beside the binary (share/ai-memory), then to the working tree's own copy --
+// so a developer running ./bin/ai-memory from the repository needs no setup.
+index_script :: proc() -> string {
+	if len(INDEX_SCRIPT) > 0 {
+		return INDEX_SCRIPT
+	}
+	if env := os.get_env("AI_MEMORY_INDEX_SCRIPT", context.temp_allocator); len(env) > 0 {
+		return env
+	}
+	if dir, err := os.get_executable_directory(context.temp_allocator); err == nil {
+		installed := strings.concatenate({dir, "/../share/ai-memory/index.scm"}, context.temp_allocator)
+		if os.exists(installed) {
+			return installed
+		}
+	}
+	return "scripts/index.scm"
+}
+
+guile_command :: proc() -> string {
+	if env := os.get_env("GUILE", context.temp_allocator); len(env) > 0 {
+		return env
+	}
+	return "guile"
+}
+
+// Run the index script and hand back its streams. `--no-auto-compile' keeps
+// the first call from writing a cache into the user's home directory, which
+// matters because the script runs from a read-only store path when installed.
+run_index_script :: proc(args: []string) -> (stdout: []byte, stderr: []byte, ok: bool) {
+	script := index_script()
+	if !os.exists(script) {
+		fmt.eprintfln("ai-memory: no index script at %q (set $AI_MEMORY_INDEX_SCRIPT)", script)
+		return nil, nil, false
+	}
+	command := make([dynamic]string, 0, len(args) + 5, context.temp_allocator)
+	append(&command, guile_command(), "--no-auto-compile", "-s", script)
+	for arg in args {
+		append(&command, arg)
+	}
+
+	state, out, err_out, perr := os.process_exec({command = command[:]}, context.temp_allocator)
+	if perr != nil {
+		fmt.eprintfln("ai-memory: could not run %q: %v", guile_command(), perr)
+		return nil, nil, false
+	}
+	if !state.success {
+		fmt.eprintfln("ai-memory: the index script failed")
+		if len(err_out) > 0 {
+			fmt.eprintfln("%s", string(err_out))
+		}
+		return out, err_out, false
+	}
+	return out, err_out, true
+}
+
+// `--store' is the SQLite path the store already resolved: the index is over
+// *this* store, so the caller never names it twice.
+cmd_update_index :: proc(rebuild: bool) -> bool {
+	db := db_path()
+	if !os.exists(db) {
+		fmt.eprintfln("ai-memory: no store at %s -- run `init' first", db)
+		return false
+	}
+	args := make([dynamic]string, 0, 8, context.temp_allocator)
+	append(&args, "update", "--store", db, "--index", index_path())
+	if rebuild {
+		append(&args, "--rebuild")
+	}
+	_, stderr, ok := run_index_script(args[:])
+	if ok && len(stderr) > 0 {
+		// The script's progress note is informational: pass it through.
+		fmt.eprintfln("%s", string(stderr))
+	}
+	return ok
+}
+
+// Search through the index instead of the store's own FTS. The script's JSON
+// is printed verbatim: it is the reply format, and re-encoding it here would
+// only add a parser that can disagree with the writer.
+cmd_search_index :: proc(words: []string, limit: int) -> bool {
+	args := make([dynamic]string, 0, 8, context.temp_allocator)
+	append(&args, "search", "--index", index_path())
+	if limit > 0 {
+		append(&args, "--limit", fmt.aprintf("%d", limit, allocator = context.temp_allocator))
+	}
+	for word in words {
+		append(&args, word)
+	}
+	stdout, _, ok := run_index_script(args[:])
+	if ok {
+		fmt.printfln("%s", strings.trim_space(string(stdout)))
+	}
+	return ok
+}
+
+// The index directory: --index, else $AI_MEMORY_INDEX, else <store>.d/index
+// so that a store and its derived index sit together and both are gitignored.
+// Whether the caller asked for the index: an explicit --index, or the
+// environment variable that stands in for it. (The default directory is
+// *not* a request: without one of these, `search' keeps its SQLite
+// behaviour, so nothing changes for a store that has no index.)
+index_requested :: proc() -> bool {
+	if len(INDEX_PATH) > 0 {
+		return true
+	}
+	return len(os.get_env("AI_MEMORY_INDEX", context.temp_allocator)) > 0
+}
+
+index_path :: proc() -> string {
+	if len(INDEX_PATH) > 0 {
+		return INDEX_PATH
+	}
+	if env := os.get_env("AI_MEMORY_INDEX", context.temp_allocator); len(env) > 0 {
+		return env
+	}
+	return strings.concatenate({db_path(), ".d/index"}, context.temp_allocator)
+}
diff --git a/main.odin b/main.odin
index b51b239..6079bd5 100644
--- a/main.odin
+++ b/main.odin
@@ -23,10 +23,22 @@ usage: ai-memory <command> [options]
   log [--limit N]                  recent commits and how much reasoning they carry
   status                           row counts
 
+topic index (derived from the store; see scripts/index.scm):
+
+  update-index [--rebuild]         bring the Xapian index up to date
+  search [--limit N] QUERY...      with --index: search it, with prefixes
+                                   topic: repo: type: session: commit: date:
+                                   (without --index, search stays SQLite FTS)
+
 global options (before or after the command):
 
   --repo PATH, -C PATH             work on the git repository at PATH
   --db PATH                        use PATH for the store instead of the default
+  --index DIR                      the topic index directory
+                                   (default: <store>.d/index; $AI_MEMORY_INDEX)
+  --index-script PATH              the Guile indexer (default: installed
+                                   share/ai-memory/index.scm, else
+                                   scripts/index.scm; $AI_MEMORY_INDEX_SCRIPT)
 
 The store defaults to <repo>/ai-memory.sqlite; set $AI_MEMORY_DB to override.`
 
@@ -54,6 +66,10 @@ main :: proc() {
 			name = "--repo"
 		case a == "--db":
 			name = "--db"
+		case a == "--index":
+			name = "--index"
+		case a == "--index-script":
+			name = "--index-script"
 		case strings.has_prefix(a, "--repo="):
 			name = "--repo"
 			value = a[len("--repo="):]
@@ -62,6 +78,14 @@ main :: proc() {
 			name = "--db"
 			value = a[len("--db="):]
 			has_value = true
+		case strings.has_prefix(a, "--index="):
+			name = "--index"
+			value = a[len("--index="):]
+			has_value = true
+		case strings.has_prefix(a, "--index-script="):
+			name = "--index-script"
+			value = a[len("--index-script="):]
+			has_value = true
 		}
 
 		if len(name) == 0 {
@@ -80,9 +104,14 @@ main :: proc() {
 			i += 1
 		}
 
-		if name == "--db" {
+		switch name {
+		case "--db":
 			DB_PATH = value
-		} else {
+		case "--index":
+			INDEX_PATH = value
+		case "--index-script":
+			INDEX_SCRIPT = value
+		case:
 			REPO_DIR = value
 		}
 	}
@@ -115,8 +144,17 @@ main :: proc() {
 		ok = cmd_attach(args[1:])
 	case "why":
 		ok = cmd_why(args[1:])
+	case "update-index":
+		ok = cmd_update_index(flag_bool(args[1:], "--rebuild"))
 	case "search":
-		ok = cmd_search(args[1:])
+		// The index answers concept queries; the store's own FTS answers
+		// text. Asking for --index is how a caller chooses.
+		if index_requested() {
+			ok = cmd_search_index(args[1:],
+			                      parse_int(flag_or(args[1:], "--limit", "0"), 0))
+		} else {
+			ok = cmd_search(args[1:])
+		}
 	case "log":
 		ok = cmd_log(args[1:])
 	case "status":
diff --git a/scripts/index.scm b/scripts/index.scm
new file mode 100644
index 0000000..ac83850
--- /dev/null
+++ b/scripts/index.scm
@@ -0,0 +1,426 @@
+;;; index.scm --- a topic index over the ai-memory store, in Xapian
+;;; Copyright © 2026 Pjotr Prins <wrk@bosa.lan>
+;;;
+;;; ai-memory's SQLite store is the truth; this is a derived, disposable
+;;; Xapian index over it, in the notmuch style: one document per unit, keyed
+;;; by a unique term, fielded terms for routing, free text stemmed. Drop the
+;;; directory and `update --rebuild' to regenerate it.
+;;;
+;;; Why a separate program rather than FFI inside ai-memory: libxapian is
+;;; C++-only, so a foreign call means a hand-written C shim. guile-xapian
+;;; already has one -- tested, maintained -- and this is the API it exposes.
+;;; ai-memory shells out to this script; the boundary is a store path in and
+;;; JSON out, and SQLite stays the source of truth.
+;;;
+;;; Usage
+;;;   index.scm update --store STORE.sqlite --index DIR [--rebuild] [--quiet]
+;;;   index.scm search --index DIR [--limit N] [--or] QUERY...
+;;;   index.scm count  --index DIR
+;;;
+;;; Documents
+;;;   reasoning:<id>   one per reasoning entry (session, kind, commit)
+;;;   commit:<hash>    one per commit (subject + body, author date)
+;;;
+;;; Terms (boolean, case-folded on both sides, so prefix values are compared
+;;; exactly rather than stemmed)
+;;;   K<key>       identity: the replace key, never queried by a user
+;;;   T<type>      reasoning | commit
+;;;   R<repo>      repository name
+;;;   S<session>   session id
+;;;   O<topic>     one per word of the session id, so `topic:pve' is an entry
+;;;                point into the pve story
+;;;   C<hash>      the commit an entry is attached to (7 characters)
+;;;   D<yyyymmdd>  author date
+;;;
+;;;; Value slots
+;;;   0  date (yyyymmdd; lexicographic order is chronological)
+;;;   1  digest of the indexed text, for the incremental compare
+;;;   2  the document's JSON record -- so a hit is one value read and needs
+;;;      no second query
+;;;
+;;; `update' is incremental: each row's text is digested and compared with
+;;; slot 1 of the document already in the index; only new or changed rows are
+;;; replaced. Deletions are not propagated -- the store is append-only, so a
+;;; row that disappears is a `--rebuild'.
+
+(use-modules (ice-9 format)
+             (ice-9 match)
+             (ice-9 getopt-long)
+             (rnrs arithmetic bitwise)
+             (srfi srfi-1)
+             (srfi srfi-13)
+             (sqlite3)
+             (xapian wrap))
+
+;;; ---------------------------------------------------------------------
+;;; Schema. One place, so the indexer and the query parser cannot drift.
+
+(define SCHEMA-VERSION 1)
+
+(define SLOT-DATE 0)
+(define SLOT-DIGEST 1)
+(define SLOT-DOCDATA 2)
+
+;; User-facing prefix -> boolean term prefix.
+(define PREFIXES
+  '(("type:"    . "T")
+    ("repo:"    . "R")
+    ("session:" . "S")
+    ("topic:"   . "O")
+    ("commit:"  . "C")
+    ("date:"    . "D")))
+
+(define (term prefix value)
+  (string-append prefix (string-downcase value)))
+
+;;; ---------------------------------------------------------------------
+;;; Helpers
+
+(define CURRENT-OPTIONS (make-parameter (list)))
+
+(define (quiet?)
+  (option-ref (CURRENT-OPTIONS) 'quiet #f))
+
+(define (die fmt . args)
+  (format (current-error-port) "index.scm: ~a\n" (apply format #f fmt args))
+  (exit 1))
+
+(define (note fmt . args)
+  (unless (quiet?)
+    (apply format (current-error-port) fmt args)
+    (newline (current-error-port))))
+
+;; FNV-1a over the text: the only property needed is "changes when the text
+;; changes", and this avoids a crypto dependency for it.
+(define (digest str)
+  (let loop ((i 0) (h 2166136261))
+    (if (>= i (string-length str))
+        (number->string h)
+        (loop (+ i 1)
+              (modulo (* (bitwise-xor h (char->integer (string-ref str i)))
+                         16777619)
+                      4294967296)))))
+
+(define (yyyymmdd epoch)
+  (let* ((t (gmtime epoch))
+         (y (+ 1900 (tm:year t)))
+         (m (+ 1 (tm:mon t)))
+         (d (tm:mday t)))
+    (+ (* y 10000) (* m 100) d)))
+
+;; The store knows its repository only as a directory; the basename is the
+;; name a reader uses ("genenetwork4"), so that is the R term.
+(define (repo-name store)
+  (basename (dirname (canonicalize-path store))))
+
+(define (non-empty-string? value)
+  (and (string? value) (not (string-null? value))))
+
+;;; ---------------------------------------------------------------------
+;;; Database handles
+
+(define (open-writable index rebuild)
+  (unless rebuild
+    ;; Xapian creates its files inside DIR, but not DIR.
+    (unless (zero? (system* "mkdir" "-p" index))
+      (die "could not create ~a" index)))
+  (new-WritableDatabase index
+                        (if rebuild (DB-CREATE-OR-OVERWRITE) (DB-CREATE-OR-OPEN))))
+
+(define (open-readable index)
+  (unless (file-exists? index)
+    (die "no index at ~a: run `update' first" index))
+  (new-Database index))
+
+
+;;; ---------------------------------------------------------------------
+;;; JSON, emitted rather than parsed
+;;;
+;;; The index stores a JSON record per document (slot 2) and hands it back
+;;; verbatim, so nothing here ever parses JSON -- which is why this is a
+;;; writer of thirty lines instead of a library dependency. Objects are
+;;; alists of dotted pairs; values are strings, numbers, or lists of
+;;; strings; nothing else is stored.
+
+(define (json-escape text)
+  (let ((out (open-output-string)))
+    (string-for-each
+     (lambda (ch)
+       (case ch
+         ((#\") (display "\\\"" out))
+         ((#\\) (display "\\\\" out))
+         ((#\newline) (display "\\n" out))
+         ((#\tab) (display "\\t" out))
+         ((#\return) (display "\\r" out))
+         (else
+          (if (< (char->integer ch) 32)
+              (format out "\\u~4,'0x" (char->integer ch))
+              (write-char ch out)))))
+     text)
+    (get-output-string out)))
+
+(define (json-value value)
+  (cond ((string? value) (string-append "\"" (json-escape value) "\""))
+        ((number? value) (number->string value))
+        ((boolean? value) (if value "true" "false"))
+        ((list? value) (string-append "["
+                                      (string-join (map json-value value) ",")
+                                      "]"))
+        ((null? value) "null")
+        (else (json-value (format #f "~a" value)))))
+
+(define (json-object record)
+  (string-append "{"
+                 (string-join (map (lambda (pair)
+                                     (string-append (json-value (symbol->string (car pair)))
+                                                    ":"
+                                                    (json-value (second pair))))
+                                   record)
+                              ",")
+                 "}"))
+
+;;; ---------------------------------------------------------------------
+;;; Records -> documents
+;;;
+;;; A record is an alist with symbol keys, so it is both what we index and
+;;; what slot 2 carries: the caller renders exactly what the indexer stored.
+;;; guile-json wants object entries as two-element lists (key value).
+
+(define (field key value) (list key value))
+
+;; An entry is (key value), so the cdr is a one-element list -- assq-ref would
+;; hand back the wrapping list rather than the value.
+(define (lookup record key)
+  (let ((entry (assq key record)))
+    (and entry (second entry))))
+
+(define (reasoning-record repo row)
+  (match row
+    ((id session step kind body commit_hash created_at subject)
+     (let* ((key (format #f "reasoning:~a" id))
+            (date (yyyymmdd created_at))
+            (title (if (string? subject) (string-trim-both subject) ""))
+            (topics (if (non-empty-string? session)
+                        (map (lambda (word) (term "O" word))
+                             (string-split session #\-))
+                        (list)))
+            (terms (append (list (term "T" "reasoning")
+                                 (term "R" repo)
+                                 (term "D" (number->string date)))
+                           (if (non-empty-string? session)
+                               (cons (term "S" session) topics)
+                               (list))
+                           (if (non-empty-string? commit_hash)
+                               (list (term "C" (substring commit_hash 0 7)))
+                               (list)))))
+       (list key
+             (list (field 'type "reasoning")
+                   (field 'key key)
+                   (field 'repo repo)
+                   (field 'session (or session ""))
+                   (field 'kind (or kind ""))
+                   (field 'step (or step 0))
+                   (field 'date date)
+                   (field 'commit (or commit_hash ""))
+                   (field 'title title)
+                   (field 'terms terms)
+                   (field 'body (or body ""))
+                   (field 'text (string-append title " " (or body "")))))))
+    (else (die "unexpected reasoning row: ~s" row))))
+
+(define (commit-record repo row)
+  (match row
+    ((hash subject body author_ts)
+     (let* ((key (format #f "commit:~a" hash))
+            (date (yyyymmdd author_ts))
+            (subject (or subject "")))
+       (list key
+             (list (field 'type "commit")
+                   (field 'key key)
+                   (field 'repo repo)
+                   (field 'date date)
+                   (field 'commit hash)
+                   (field 'title subject)
+                   (field 'terms (list (term "T" "commit")
+                                       (term "R" repo)
+                                       (term "D" (number->string date))
+                                       (term "C" (substring hash 0 7))))
+                   (field 'body (or body ""))
+                   (field 'text (string-append subject " " (or body "")))))))
+    (else (die "unexpected commit row: ~s" row))))
+
+(define (records repo reasoning commits)
+  (append (map (lambda (row) (reasoning-record repo row)) reasoning)
+          (map (lambda (row) (commit-record repo row)) commits)))
+
+;;; ---------------------------------------------------------------------
+;;; Writing
+
+(define (build-document stemmer record)
+  (let ((doc (new-Document))
+        (text (lookup record 'text))
+        (date (lookup record 'date)))
+    (Document-set-data doc (json-object record))
+    ;; The identity term first: it is what the next run deletes by, so a
+    ;; document without it can never be replaced -- only duplicated.
+    (Document-add-boolean-term doc (string-append "K" (lookup record 'key)))
+    (for-each (lambda (t) (Document-add-boolean-term doc t))
+              (lookup record 'terms))
+    (Document-add-value doc SLOT-DATE (number->string date))
+    (Document-add-value doc SLOT-DIGEST (digest text))
+    (Document-add-value doc SLOT-DOCDATA (json-object record))
+    (let ((tg (new-TermGenerator)))
+      (TermGenerator-set-stemmer tg stemmer)
+      (TermGenerator-set-document tg doc)
+      (TermGenerator-index-text tg text))
+    doc))
+
+;; Every document is re-created: deleted by its unique term, then added.
+;; Xapian applies that as a delete plus an insert of the changed postings, so
+;; the on-disk work is incremental; this program merely re-tokenises every
+;; body (~1 s at a few thousand documents), which buys not having to remember
+;; what was indexed last time -- state that can go stale.
+(define (index-record! db stemmer key record)
+  ;; delete-then-add rather than replace-document: `delete_document' by
+  ;; unique term works through this binding, while `replace_document' with a
+  ;; term silently appends (it lands on the docid overload), which duplicates
+  ;; the whole store on the second run. Deleting a term that is not there is
+  ;; a no-op, so this is also the first-index case.
+  (let ((term (string-append "K" key)))
+    (false-if-exception (WritableDatabase-delete-document db term))
+    (WritableDatabase-add-document db (build-document stemmer record))))
+
+;;; ---------------------------------------------------------------------
+;;; Reading the store
+
+;; guile-sqlite3 gives results through a prepared statement: sqlite-map
+;; runs a proc per row and returns them as a list.
+(define (query db sql)
+  (let ((statement (sqlite-prepare db sql)))
+    (let ((rows (sqlite-map vector->list statement)))
+      (sqlite-finalize statement)
+      rows)))
+
+(define (store-rows store)
+  (let ((db (sqlite-open store SQLITE_OPEN_READONLY)))
+    (let ((reasoning (query db "SELECT r.id, r.session_id, r.step, r.kind, r.body,
+                                      COALESCE(r.commit_hash, ''), r.created_at,
+                                      COALESCE(c.subject, '')
+                               FROM reasoning r
+                               LEFT JOIN commits c ON c.hash = r.commit_hash
+                               ORDER BY r.id"))
+          (commits (query db "SELECT hash, subject, COALESCE(body, ''), author_ts
+                              FROM commits ORDER BY author_ts")))
+      (sqlite-close db)
+      (values reasoning commits))))
+
+;;; ---------------------------------------------------------------------
+;;; Commands
+
+(define (cmd-update opts)
+  (let ((store (option-ref opts 'store #f))
+        (index (option-ref opts 'index #f))
+        (rebuild (option-ref opts 'rebuild #f)))
+    (unless store (die "update needs --store STORE.sqlite"))
+    (unless index (die "update needs --index DIR"))
+    (unless (file-exists? store) (die "no store at ~a" store))
+    (call-with-values (lambda () (store-rows store))
+      (lambda (reasoning commits)
+        (let* ((repo (repo-name store))
+               (records (records repo reasoning commits))
+               (db (open-writable index rebuild))
+               (stemmer (new-Stem "english"))
+               (written 0))
+          (for-each (lambda (entry)
+                      (index-record! db stemmer (first entry) (second entry))
+                      (set! written (+ written 1)))
+                    records)
+          (WritableDatabase-commit db)
+          (delete-WritableDatabase db)
+          (note "~a: ~a documents (~a reasoning, ~a commits)"
+                (if rebuild "rebuilt" "updated") written
+                (length reasoning) (length commits)))))))
+
+(define (make-query-parser db default-op)
+  (let ((qp (new-QueryParser)))
+    (QueryParser-set-stemmer qp (new-Stem "english"))
+    (QueryParser-set-database qp db)
+    (QueryParser-set-default-op qp default-op)
+    (for-each (lambda (pair)
+                (QueryParser-add-boolean-prefix qp (car pair) (cdr pair)))
+              PREFIXES)
+    qp))
+
+(define (cmd-search opts words)
+  (let ((index (option-ref opts 'index #f))
+        (limit (string->number (option-ref opts 'limit "10")))
+        (query-string (string-join words " ")))
+    (unless index (die "search needs --index DIR"))
+    (when (string-null? (string-trim-both query-string))
+      (die "search needs a query"))
+    (let* ((db (open-readable index))
+           (qp (make-query-parser db (if (option-ref opts 'or #f)
+                                         (Query-OP-OR)
+                                         (Query-OP-AND))))
+           (enq (new-Enquire db)))
+      (Enquire-set-query enq (QueryParser-parse-query qp query-string))
+      (Enquire-set-sort-by-value enq SLOT-DATE #t)
+      (let* ((mset (Enquire-get-mset enq 0 limit))
+             (n (MSet-size mset))
+             (hits (let loop ((i 0) (acc (list)))
+                     (if (>= i n)
+                         acc
+                         (let* ((item (MSet-get-hit mset i))
+                                (doc (MSetIterator-get-document item))
+                                (record (Document-get-value doc SLOT-DOCDATA)))
+                           (loop (+ i 1)
+                                 (cons (string-append
+                                        "{\"docid\":" (number->string (MSetIterator-get-docid item))
+                                        ",\"percent\":" (number->string (MSetIterator-get-percent item))
+                                        ",\"record\":" record "}")
+                                       acc)))))))
+        (display
+         (string-append
+          "{\"query\":" (json-value query-string)
+          ",\"count\":" (number->string (length hits))
+          ",\"matches\":" (number->string (MSet-get-matches-estimated mset))
+          ",\"hits\":["
+          (string-join (reverse hits) ",")
+          "]}"))
+        (newline))
+      (delete-Database db))))
+
+(define (cmd-count opts)
+  (let ((db (open-readable (option-ref opts 'index #f))))
+    (format #t "~a\n" (Database-get-doccount db))
+    (delete-Database db)))
+
+(define (usage)
+  (format #t "usage: index.scm update --store STORE.sqlite --index DIR [--rebuild]~@
+              \x20      index.scm search --index DIR [--limit N] [--or] QUERY...~@
+              \x20      index.scm count  --index DIR~%"))
+
+(define OPTION-SPEC
+  '((store (value #t))
+    (index (value #t))
+    (limit (value #t))
+    (rebuild)
+    (or)
+    (quiet)
+    (help (single-char #\h))))
+
+(define (main args)
+  (let* ((opts (getopt-long args OPTION-SPEC))
+         (rest (option-ref opts '() '())))
+    (parameterize ((CURRENT-OPTIONS opts))
+      (match rest
+        (("update" . _) (cmd-update opts))
+        (("search" . words) (cmd-search opts words))
+        (("count" . _) (cmd-count opts))
+        (("schema" . _) (format #t "~a\n" SCHEMA-VERSION))
+        (("--help" . _) (usage))
+        (((? (lambda (a) (member a '("-h" "--help"))) _) . _) (usage))
+        ((cmd . _) (die "unknown command: ~a" cmd))
+        (() (usage))))))
+
+(main (command-line))