diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-12 13:52:12 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-12 13:52:12 +0000 |
| commit | 3fab7adf0ec147dceb150d861fdbd5f5b3693384 (patch) | |
| tree | 9e2f0b50a6d639eb821a9295bdfb4723f9a01535 | |
| parent | 5cdd45571c550708f691855891c7919fe076ab05 (diff) | |
| download | ai-memory-3fab7adf0ec147dceb150d861fdbd5f5b3693384.tar.gz | |
Add a topic index: Xapian over the store, driven through Guile
SQLite stays the truth; this is a derived index that lets a reader enter by concept rather than by symbol. Two commands, one prefix vocabulary: ai-memory --index DIR update-index [--rebuild] ai-memory --index DIR search 'topic:pve repo:genenetwork4' Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:, date:. Free text is stemmed; default operator AND, --or to widen. Topic terms come from the session id split on hyphens, so a session named "gemma-pve" answers topic:pve and topic:gemma -- naming a session after a concept is how an entry becomes findable by it. Why Guile and a subprocess: libxapian is C++-only, so a foreign call from Odin means a hand-written C shim with its own exception boundary; guile-xapian already ships one, tested and maintained. So scripts/index.scm owns every Xapian call (426 lines: schema constants, record builders, updater, query parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines of process plumbing. The boundary is a store path in and JSON out, which also means the search reply needs no parser on this side. Incremental in the way that matters: documents are deleted by their unique term and re-added, and Xapian applies that as changed postings only. The whole body is re-tokenised each run (~1 s at a few hundred documents) which buys not keeping any "what did I index last time" state that could go stale. Two discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by term silently appends (the string lands on the docid overload), so delete then add; and Database-get-document by term returns nothing, so no digest compare was attempted. The package's check phase now exercises it end to end -- throwaway store, one entry, update-index, then `search topic:index | grep -q '"count":1'` -- because a check that only links proves nothing about the four Guile dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for guile-xapian and guile-sqlite3. doc/SKILL.md gains the usage, the prefix list and the reason for the subprocess.
| -rw-r--r-- | doc/SKILL.md | 38 | ||||
| -rw-r--r-- | guix.scm | 54 | ||||
| -rw-r--r-- | index.odin | 145 | ||||
| -rw-r--r-- | main.odin | 44 | ||||
| -rw-r--r-- | scripts/index.scm | 426 |
5 files changed, 701 insertions, 6 deletions
diff --git a/doc/SKILL.md b/doc/SKILL.md index fb17189..570bde7 100644 --- a/doc/SKILL.md +++ b/doc/SKILL.md @@ -33,6 +33,10 @@ guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib" ``` +`guix build -f guix.scm` also runs the package's check phase, which builds a +throwaway store, records an entry, indexes it and queries it -- that is what +proves the Guile side is wired up. + The store defaults to `<repo>/ai-memory.sqlite`; override with `--db PATH` or `$AI_MEMORY_DB`. `--repo PATH` / `-C PATH` selects the repository, and may be given before or after the command. @@ -68,6 +72,40 @@ ai-memory -C REPO attach HEAD --session fix-parser `record` and `attach` import the commit on demand, so they work immediately after committing even if `sync` has not run yet. +## The topic index (optional, derived) + +A second, *derived* store: a Xapian index over the SQLite one, so a reader can +enter by **concept** ("topic:pve") or by repository and kind ("repo:genenetwork4 +type:commit") instead of by symbol name. It is disposable -- delete the +directory and `update-index --rebuild` -- and SQLite stays the truth. + +```sh +ai-memory --index DIR update-index # bring it up to date +ai-memory --index DIR update-index --rebuild # after a schema change +ai-memory --index DIR search 'topic:pve' +ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4' +``` + +(The directory defaults to `<store>.d/index`, override with `--index` or +`$AI_MEMORY_INDEX`. `search` without `--index` keeps using SQLite FTS.) + +**Prefixes**: `topic:`, `repo:`, `type:` (`reasoning`|`commit`), `session:`, +`commit:` (7-character hash), `date:` (`yyyymmdd`). Everything else is free +text, stemmed. The default operator is AND; `--or` widens it. + +Topic terms come from the **session id**, split on hyphens: a session called +`gemma-pve` answers `topic:gemma` and `topic:pve`. So the cheapest way to make +an entry findable by concept is to name the session after the concept. + +Why a subprocess and a Guile script: `libxapian` is C++-only, so a foreign call +from Odin needs a hand-written C shim. `guile-xapian` already has one, tested +and maintained (`scripts/index.scm` owns every Xapian call; ai-memory shells +out to it). The boundary is a store path in, JSON out. + +The JSON a search returns is the record as indexed -- `key`, `repo`, `session`, +`kind`, `date`, `commit`, `title`, `body`, `terms` -- plus `docid` and +`percent`. `count` is how many came back, `matches` how many the index found. + ## Rules - **Never write to `commits`, `symbols` or `touches` by hand.** They are derived; diff --git a/guix.scm b/guix.scm index 62fcf75..72ab494 100644 --- a/guix.scm +++ b/guix.scm @@ -38,6 +38,8 @@ #:use-module (guix packages) #:use-module (guix utils) #:use-module (gnu packages bash) + #:use-module (gnu packages guile) + #:use-module (gnu packages guile-xyz) #:use-module (gnu packages llvm) #:use-module (gnu packages sqlite)) @@ -215,7 +217,10 @@ to link the system SQLite3 instead of a bundled copy.") ;; store, which are machine state rather than source. (source (local-file %source-dir #:recursive? #t #:select? source-file?)) (build-system gnu-build-system) - (inputs (list sqlite)) ;libsqlite3.so, linked at run time + ;; libsqlite3.so is linked at run time; the topic index (scripts/index.scm) + ;; is a Guile program, because libxapian is C++-only and guile-xapian + ;; already has the C++ shim that a foreign call would otherwise need. + (inputs (list sqlite guile-3.0 guile-xapian guile-sqlite3)) (native-inputs (list odin odin-sqlite3)) (arguments (list @@ -246,11 +251,54 @@ to link the system SQLite3 instead of a bundled copy.") (apply invoke "odin" "build" "." "-out:bin/ai-memory-check" flags) ;; Links and runs; fail loudly if libsqlite3 is missing. - (invoke "./bin/ai-memory-check")))) + (invoke "./bin/ai-memory-check") + ;; The topic index end to end: a store with one entry, the + ;; Guile indexer over it, and a prefixed query -- which is + ;; what proves the script's dependencies are complete. + (mkdir-p "tmp") + (setenv "AI_MEMORY_DB" + (string-append (getcwd) "/tmp/store.sqlite")) + (setenv "AI_MEMORY_INDEX_SCRIPT" + (string-append (getcwd) "/scripts/index.scm")) + (setenv "GUILE" (string-append #$(this-package-input "guile") + "/bin/guile")) + ;; guile-xapian needs guile-lib (snippets) and the sqlite + ;; binding; both come from the package inputs, so the check + ;; fails here rather than in a user's first `update-index'. + (setenv "GUILE_LOAD_PATH" + (string-append #$(this-package-input "guile-xapian") + "/share/guile/site/3.0:" + #$(this-package-input "guile-sqlite3") + "/share/guile/site/3.0")) + (invoke "./bin/ai-memory" "init") + ;; A session id is what gives the entry its topic: terms + ;; ("topic-index" -> topic:topic and topic:index), so the + ;; query below is a real concept query, not a text match. + (invoke "./bin/ai-memory" "record" "--kind" "conclusion" + "--session" "topic-index" + "--body" "the index answers topic: queries") + (invoke "./bin/ai-memory" "--index" "tmp/index" "update-index") + ;; The reply is JSON on stdout: pipe it into grep rather + ;; than capturing it (with-output-to-string captures the + ;; Scheme port, not the child's stdout). + (invoke "sh" "-c" + (string-append "./bin/ai-memory --index tmp/index" + " search topic:index" + " | grep -q '\"count\":1'"))))) (replace 'install (lambda _ (install-file "bin/ai-memory" - (string-append #$output "/bin")))))))) + (string-append #$output "/bin")) + ;; The indexer travels with the binary: index.odin looks for + ;; ../share/ai-memory/index.scm beside it. + (install-file "scripts/index.scm" + (string-append #$output "/share/ai-memory")) + (wrap-program (string-append #$output "/bin/ai-memory") + `("GUILE" = (,(string-append #$(this-package-input "guile") "/bin/guile"))) + `("GUILE_LOAD_PATH" = (,(string-append #$(this-package-input "guile-xapian") + "/share/guile/site/3.0:" + #$(this-package-input "guile-sqlite3") + "/share/guile/site/3.0")))))))))) (home-page "https://github.com/ai-memory/ai-memory") (synopsis "Memory for AI assistants") (description diff --git a/index.odin b/index.odin new file mode 100644 index 0000000..eb6e3f4 --- /dev/null +++ b/index.odin @@ -0,0 +1,145 @@ +package main + +// The topic index: a derived Xapian index over the store, which lets a +// reader enter by concept ("topic:pve") or by repository and type rather than +// by symbol name. +// +// SQLite stays the source of truth. This is a subprocess because libxapian is +// C++-only: a call into it from Odin means a hand-written C shim, while +// guile-xapian already has one, tested. So the boundary is a script path in +// and JSON out -- see scripts/index.scm, which owns every Xapian call. +// +// ai-memory --index DIR update-index [--rebuild] +// ai-memory --index DIR search 'topic:pve repo:genenetwork4' +// +// Where the pieces are: +// --index PATH or $AI_MEMORY_INDEX the Xapian directory +// --index-script PATH or $AI_MEMORY_INDEX_SCRIPT scripts/index.scm +// $GUILE or "guile" the interpreter + +import "core:fmt" +import "core:os" +import "core:strings" + +// Set by the global options. +INDEX_PATH: string +INDEX_SCRIPT: string + +// The script defaults to $AI_MEMORY_INDEX_SCRIPT, then to an installed copy +// beside the binary (share/ai-memory), then to the working tree's own copy -- +// so a developer running ./bin/ai-memory from the repository needs no setup. +index_script :: proc() -> string { + if len(INDEX_SCRIPT) > 0 { + return INDEX_SCRIPT + } + if env := os.get_env("AI_MEMORY_INDEX_SCRIPT", context.temp_allocator); len(env) > 0 { + return env + } + if dir, err := os.get_executable_directory(context.temp_allocator); err == nil { + installed := strings.concatenate({dir, "/../share/ai-memory/index.scm"}, context.temp_allocator) + if os.exists(installed) { + return installed + } + } + return "scripts/index.scm" +} + +guile_command :: proc() -> string { + if env := os.get_env("GUILE", context.temp_allocator); len(env) > 0 { + return env + } + return "guile" +} + +// Run the index script and hand back its streams. `--no-auto-compile' keeps +// the first call from writing a cache into the user's home directory, which +// matters because the script runs from a read-only store path when installed. +run_index_script :: proc(args: []string) -> (stdout: []byte, stderr: []byte, ok: bool) { + script := index_script() + if !os.exists(script) { + fmt.eprintfln("ai-memory: no index script at %q (set $AI_MEMORY_INDEX_SCRIPT)", script) + return nil, nil, false + } + command := make([dynamic]string, 0, len(args) + 5, context.temp_allocator) + append(&command, guile_command(), "--no-auto-compile", "-s", script) + for arg in args { + append(&command, arg) + } + + state, out, err_out, perr := os.process_exec({command = command[:]}, context.temp_allocator) + if perr != nil { + fmt.eprintfln("ai-memory: could not run %q: %v", guile_command(), perr) + return nil, nil, false + } + if !state.success { + fmt.eprintfln("ai-memory: the index script failed") + if len(err_out) > 0 { + fmt.eprintfln("%s", string(err_out)) + } + return out, err_out, false + } + return out, err_out, true +} + +// `--store' is the SQLite path the store already resolved: the index is over +// *this* store, so the caller never names it twice. +cmd_update_index :: proc(rebuild: bool) -> bool { + db := db_path() + if !os.exists(db) { + fmt.eprintfln("ai-memory: no store at %s -- run `init' first", db) + return false + } + args := make([dynamic]string, 0, 8, context.temp_allocator) + append(&args, "update", "--store", db, "--index", index_path()) + if rebuild { + append(&args, "--rebuild") + } + _, stderr, ok := run_index_script(args[:]) + if ok && len(stderr) > 0 { + // The script's progress note is informational: pass it through. + fmt.eprintfln("%s", string(stderr)) + } + return ok +} + +// Search through the index instead of the store's own FTS. The script's JSON +// is printed verbatim: it is the reply format, and re-encoding it here would +// only add a parser that can disagree with the writer. +cmd_search_index :: proc(words: []string, limit: int) -> bool { + args := make([dynamic]string, 0, 8, context.temp_allocator) + append(&args, "search", "--index", index_path()) + if limit > 0 { + append(&args, "--limit", fmt.aprintf("%d", limit, allocator = context.temp_allocator)) + } + for word in words { + append(&args, word) + } + stdout, _, ok := run_index_script(args[:]) + if ok { + fmt.printfln("%s", strings.trim_space(string(stdout))) + } + return ok +} + +// The index directory: --index, else $AI_MEMORY_INDEX, else <store>.d/index +// so that a store and its derived index sit together and both are gitignored. +// Whether the caller asked for the index: an explicit --index, or the +// environment variable that stands in for it. (The default directory is +// *not* a request: without one of these, `search' keeps its SQLite +// behaviour, so nothing changes for a store that has no index.) +index_requested :: proc() -> bool { + if len(INDEX_PATH) > 0 { + return true + } + return len(os.get_env("AI_MEMORY_INDEX", context.temp_allocator)) > 0 +} + +index_path :: proc() -> string { + if len(INDEX_PATH) > 0 { + return INDEX_PATH + } + if env := os.get_env("AI_MEMORY_INDEX", context.temp_allocator); len(env) > 0 { + return env + } + return strings.concatenate({db_path(), ".d/index"}, context.temp_allocator) +} diff --git a/main.odin b/main.odin index b51b239..6079bd5 100644 --- a/main.odin +++ b/main.odin @@ -23,10 +23,22 @@ usage: ai-memory <command> [options] log [--limit N] recent commits and how much reasoning they carry status row counts +topic index (derived from the store; see scripts/index.scm): + + update-index [--rebuild] bring the Xapian index up to date + search [--limit N] QUERY... with --index: search it, with prefixes + topic: repo: type: session: commit: date: + (without --index, search stays SQLite FTS) + global options (before or after the command): --repo PATH, -C PATH work on the git repository at PATH --db PATH use PATH for the store instead of the default + --index DIR the topic index directory + (default: <store>.d/index; $AI_MEMORY_INDEX) + --index-script PATH the Guile indexer (default: installed + share/ai-memory/index.scm, else + scripts/index.scm; $AI_MEMORY_INDEX_SCRIPT) The store defaults to <repo>/ai-memory.sqlite; set $AI_MEMORY_DB to override.` @@ -54,6 +66,10 @@ main :: proc() { name = "--repo" case a == "--db": name = "--db" + case a == "--index": + name = "--index" + case a == "--index-script": + name = "--index-script" case strings.has_prefix(a, "--repo="): name = "--repo" value = a[len("--repo="):] @@ -62,6 +78,14 @@ main :: proc() { name = "--db" value = a[len("--db="):] has_value = true + case strings.has_prefix(a, "--index="): + name = "--index" + value = a[len("--index="):] + has_value = true + case strings.has_prefix(a, "--index-script="): + name = "--index-script" + value = a[len("--index-script="):] + has_value = true } if len(name) == 0 { @@ -80,9 +104,14 @@ main :: proc() { i += 1 } - if name == "--db" { + switch name { + case "--db": DB_PATH = value - } else { + case "--index": + INDEX_PATH = value + case "--index-script": + INDEX_SCRIPT = value + case: REPO_DIR = value } } @@ -115,8 +144,17 @@ main :: proc() { ok = cmd_attach(args[1:]) case "why": ok = cmd_why(args[1:]) + case "update-index": + ok = cmd_update_index(flag_bool(args[1:], "--rebuild")) case "search": - ok = cmd_search(args[1:]) + // The index answers concept queries; the store's own FTS answers + // text. Asking for --index is how a caller chooses. + if index_requested() { + ok = cmd_search_index(args[1:], + parse_int(flag_or(args[1:], "--limit", "0"), 0)) + } else { + ok = cmd_search(args[1:]) + } case "log": ok = cmd_log(args[1:]) case "status": diff --git a/scripts/index.scm b/scripts/index.scm new file mode 100644 index 0000000..ac83850 --- /dev/null +++ b/scripts/index.scm @@ -0,0 +1,426 @@ +;;; index.scm --- a topic index over the ai-memory store, in Xapian +;;; Copyright © 2026 Pjotr Prins <wrk@bosa.lan> +;;; +;;; ai-memory's SQLite store is the truth; this is a derived, disposable +;;; Xapian index over it, in the notmuch style: one document per unit, keyed +;;; by a unique term, fielded terms for routing, free text stemmed. Drop the +;;; directory and `update --rebuild' to regenerate it. +;;; +;;; Why a separate program rather than FFI inside ai-memory: libxapian is +;;; C++-only, so a foreign call means a hand-written C shim. guile-xapian +;;; already has one -- tested, maintained -- and this is the API it exposes. +;;; ai-memory shells out to this script; the boundary is a store path in and +;;; JSON out, and SQLite stays the source of truth. +;;; +;;; Usage +;;; index.scm update --store STORE.sqlite --index DIR [--rebuild] [--quiet] +;;; index.scm search --index DIR [--limit N] [--or] QUERY... +;;; index.scm count --index DIR +;;; +;;; Documents +;;; reasoning:<id> one per reasoning entry (session, kind, commit) +;;; commit:<hash> one per commit (subject + body, author date) +;;; +;;; Terms (boolean, case-folded on both sides, so prefix values are compared +;;; exactly rather than stemmed) +;;; K<key> identity: the replace key, never queried by a user +;;; T<type> reasoning | commit +;;; R<repo> repository name +;;; S<session> session id +;;; O<topic> one per word of the session id, so `topic:pve' is an entry +;;; point into the pve story +;;; C<hash> the commit an entry is attached to (7 characters) +;;; D<yyyymmdd> author date +;;; +;;;; Value slots +;;; 0 date (yyyymmdd; lexicographic order is chronological) +;;; 1 digest of the indexed text, for the incremental compare +;;; 2 the document's JSON record -- so a hit is one value read and needs +;;; no second query +;;; +;;; `update' is incremental: each row's text is digested and compared with +;;; slot 1 of the document already in the index; only new or changed rows are +;;; replaced. Deletions are not propagated -- the store is append-only, so a +;;; row that disappears is a `--rebuild'. + +(use-modules (ice-9 format) + (ice-9 match) + (ice-9 getopt-long) + (rnrs arithmetic bitwise) + (srfi srfi-1) + (srfi srfi-13) + (sqlite3) + (xapian wrap)) + +;;; --------------------------------------------------------------------- +;;; Schema. One place, so the indexer and the query parser cannot drift. + +(define SCHEMA-VERSION 1) + +(define SLOT-DATE 0) +(define SLOT-DIGEST 1) +(define SLOT-DOCDATA 2) + +;; User-facing prefix -> boolean term prefix. +(define PREFIXES + '(("type:" . "T") + ("repo:" . "R") + ("session:" . "S") + ("topic:" . "O") + ("commit:" . "C") + ("date:" . "D"))) + +(define (term prefix value) + (string-append prefix (string-downcase value))) + +;;; --------------------------------------------------------------------- +;;; Helpers + +(define CURRENT-OPTIONS (make-parameter (list))) + +(define (quiet?) + (option-ref (CURRENT-OPTIONS) 'quiet #f)) + +(define (die fmt . args) + (format (current-error-port) "index.scm: ~a\n" (apply format #f fmt args)) + (exit 1)) + +(define (note fmt . args) + (unless (quiet?) + (apply format (current-error-port) fmt args) + (newline (current-error-port)))) + +;; FNV-1a over the text: the only property needed is "changes when the text +;; changes", and this avoids a crypto dependency for it. +(define (digest str) + (let loop ((i 0) (h 2166136261)) + (if (>= i (string-length str)) + (number->string h) + (loop (+ i 1) + (modulo (* (bitwise-xor h (char->integer (string-ref str i))) + 16777619) + 4294967296))))) + +(define (yyyymmdd epoch) + (let* ((t (gmtime epoch)) + (y (+ 1900 (tm:year t))) + (m (+ 1 (tm:mon t))) + (d (tm:mday t))) + (+ (* y 10000) (* m 100) d))) + +;; The store knows its repository only as a directory; the basename is the +;; name a reader uses ("genenetwork4"), so that is the R term. +(define (repo-name store) + (basename (dirname (canonicalize-path store)))) + +(define (non-empty-string? value) + (and (string? value) (not (string-null? value)))) + +;;; --------------------------------------------------------------------- +;;; Database handles + +(define (open-writable index rebuild) + (unless rebuild + ;; Xapian creates its files inside DIR, but not DIR. + (unless (zero? (system* "mkdir" "-p" index)) + (die "could not create ~a" index))) + (new-WritableDatabase index + (if rebuild (DB-CREATE-OR-OVERWRITE) (DB-CREATE-OR-OPEN)))) + +(define (open-readable index) + (unless (file-exists? index) + (die "no index at ~a: run `update' first" index)) + (new-Database index)) + + +;;; --------------------------------------------------------------------- +;;; JSON, emitted rather than parsed +;;; +;;; The index stores a JSON record per document (slot 2) and hands it back +;;; verbatim, so nothing here ever parses JSON -- which is why this is a +;;; writer of thirty lines instead of a library dependency. Objects are +;;; alists of dotted pairs; values are strings, numbers, or lists of +;;; strings; nothing else is stored. + +(define (json-escape text) + (let ((out (open-output-string))) + (string-for-each + (lambda (ch) + (case ch + ((#\") (display "\\\"" out)) + ((#\\) (display "\\\\" out)) + ((#\newline) (display "\\n" out)) + ((#\tab) (display "\\t" out)) + ((#\return) (display "\\r" out)) + (else + (if (< (char->integer ch) 32) + (format out "\\u~4,'0x" (char->integer ch)) + (write-char ch out))))) + text) + (get-output-string out))) + +(define (json-value value) + (cond ((string? value) (string-append "\"" (json-escape value) "\"")) + ((number? value) (number->string value)) + ((boolean? value) (if value "true" "false")) + ((list? value) (string-append "[" + (string-join (map json-value value) ",") + "]")) + ((null? value) "null") + (else (json-value (format #f "~a" value))))) + +(define (json-object record) + (string-append "{" + (string-join (map (lambda (pair) + (string-append (json-value (symbol->string (car pair))) + ":" + (json-value (second pair)))) + record) + ",") + "}")) + +;;; --------------------------------------------------------------------- +;;; Records -> documents +;;; +;;; A record is an alist with symbol keys, so it is both what we index and +;;; what slot 2 carries: the caller renders exactly what the indexer stored. +;;; guile-json wants object entries as two-element lists (key value). + +(define (field key value) (list key value)) + +;; An entry is (key value), so the cdr is a one-element list -- assq-ref would +;; hand back the wrapping list rather than the value. +(define (lookup record key) + (let ((entry (assq key record))) + (and entry (second entry)))) + +(define (reasoning-record repo row) + (match row + ((id session step kind body commit_hash created_at subject) + (let* ((key (format #f "reasoning:~a" id)) + (date (yyyymmdd created_at)) + (title (if (string? subject) (string-trim-both subject) "")) + (topics (if (non-empty-string? session) + (map (lambda (word) (term "O" word)) + (string-split session #\-)) + (list))) + (terms (append (list (term "T" "reasoning") + (term "R" repo) + (term "D" (number->string date))) + (if (non-empty-string? session) + (cons (term "S" session) topics) + (list)) + (if (non-empty-string? commit_hash) + (list (term "C" (substring commit_hash 0 7))) + (list))))) + (list key + (list (field 'type "reasoning") + (field 'key key) + (field 'repo repo) + (field 'session (or session "")) + (field 'kind (or kind "")) + (field 'step (or step 0)) + (field 'date date) + (field 'commit (or commit_hash "")) + (field 'title title) + (field 'terms terms) + (field 'body (or body "")) + (field 'text (string-append title " " (or body ""))))))) + (else (die "unexpected reasoning row: ~s" row)))) + +(define (commit-record repo row) + (match row + ((hash subject body author_ts) + (let* ((key (format #f "commit:~a" hash)) + (date (yyyymmdd author_ts)) + (subject (or subject ""))) + (list key + (list (field 'type "commit") + (field 'key key) + (field 'repo repo) + (field 'date date) + (field 'commit hash) + (field 'title subject) + (field 'terms (list (term "T" "commit") + (term "R" repo) + (term "D" (number->string date)) + (term "C" (substring hash 0 7)))) + (field 'body (or body "")) + (field 'text (string-append subject " " (or body ""))))))) + (else (die "unexpected commit row: ~s" row)))) + +(define (records repo reasoning commits) + (append (map (lambda (row) (reasoning-record repo row)) reasoning) + (map (lambda (row) (commit-record repo row)) commits))) + +;;; --------------------------------------------------------------------- +;;; Writing + +(define (build-document stemmer record) + (let ((doc (new-Document)) + (text (lookup record 'text)) + (date (lookup record 'date))) + (Document-set-data doc (json-object record)) + ;; The identity term first: it is what the next run deletes by, so a + ;; document without it can never be replaced -- only duplicated. + (Document-add-boolean-term doc (string-append "K" (lookup record 'key))) + (for-each (lambda (t) (Document-add-boolean-term doc t)) + (lookup record 'terms)) + (Document-add-value doc SLOT-DATE (number->string date)) + (Document-add-value doc SLOT-DIGEST (digest text)) + (Document-add-value doc SLOT-DOCDATA (json-object record)) + (let ((tg (new-TermGenerator))) + (TermGenerator-set-stemmer tg stemmer) + (TermGenerator-set-document tg doc) + (TermGenerator-index-text tg text)) + doc)) + +;; Every document is re-created: deleted by its unique term, then added. +;; Xapian applies that as a delete plus an insert of the changed postings, so +;; the on-disk work is incremental; this program merely re-tokenises every +;; body (~1 s at a few thousand documents), which buys not having to remember +;; what was indexed last time -- state that can go stale. +(define (index-record! db stemmer key record) + ;; delete-then-add rather than replace-document: `delete_document' by + ;; unique term works through this binding, while `replace_document' with a + ;; term silently appends (it lands on the docid overload), which duplicates + ;; the whole store on the second run. Deleting a term that is not there is + ;; a no-op, so this is also the first-index case. + (let ((term (string-append "K" key))) + (false-if-exception (WritableDatabase-delete-document db term)) + (WritableDatabase-add-document db (build-document stemmer record)))) + +;;; --------------------------------------------------------------------- +;;; Reading the store + +;; guile-sqlite3 gives results through a prepared statement: sqlite-map +;; runs a proc per row and returns them as a list. +(define (query db sql) + (let ((statement (sqlite-prepare db sql))) + (let ((rows (sqlite-map vector->list statement))) + (sqlite-finalize statement) + rows))) + +(define (store-rows store) + (let ((db (sqlite-open store SQLITE_OPEN_READONLY))) + (let ((reasoning (query db "SELECT r.id, r.session_id, r.step, r.kind, r.body, + COALESCE(r.commit_hash, ''), r.created_at, + COALESCE(c.subject, '') + FROM reasoning r + LEFT JOIN commits c ON c.hash = r.commit_hash + ORDER BY r.id")) + (commits (query db "SELECT hash, subject, COALESCE(body, ''), author_ts + FROM commits ORDER BY author_ts"))) + (sqlite-close db) + (values reasoning commits)))) + +;;; --------------------------------------------------------------------- +;;; Commands + +(define (cmd-update opts) + (let ((store (option-ref opts 'store #f)) + (index (option-ref opts 'index #f)) + (rebuild (option-ref opts 'rebuild #f))) + (unless store (die "update needs --store STORE.sqlite")) + (unless index (die "update needs --index DIR")) + (unless (file-exists? store) (die "no store at ~a" store)) + (call-with-values (lambda () (store-rows store)) + (lambda (reasoning commits) + (let* ((repo (repo-name store)) + (records (records repo reasoning commits)) + (db (open-writable index rebuild)) + (stemmer (new-Stem "english")) + (written 0)) + (for-each (lambda (entry) + (index-record! db stemmer (first entry) (second entry)) + (set! written (+ written 1))) + records) + (WritableDatabase-commit db) + (delete-WritableDatabase db) + (note "~a: ~a documents (~a reasoning, ~a commits)" + (if rebuild "rebuilt" "updated") written + (length reasoning) (length commits))))))) + +(define (make-query-parser db default-op) + (let ((qp (new-QueryParser))) + (QueryParser-set-stemmer qp (new-Stem "english")) + (QueryParser-set-database qp db) + (QueryParser-set-default-op qp default-op) + (for-each (lambda (pair) + (QueryParser-add-boolean-prefix qp (car pair) (cdr pair))) + PREFIXES) + qp)) + +(define (cmd-search opts words) + (let ((index (option-ref opts 'index #f)) + (limit (string->number (option-ref opts 'limit "10"))) + (query-string (string-join words " "))) + (unless index (die "search needs --index DIR")) + (when (string-null? (string-trim-both query-string)) + (die "search needs a query")) + (let* ((db (open-readable index)) + (qp (make-query-parser db (if (option-ref opts 'or #f) + (Query-OP-OR) + (Query-OP-AND)))) + (enq (new-Enquire db))) + (Enquire-set-query enq (QueryParser-parse-query qp query-string)) + (Enquire-set-sort-by-value enq SLOT-DATE #t) + (let* ((mset (Enquire-get-mset enq 0 limit)) + (n (MSet-size mset)) + (hits (let loop ((i 0) (acc (list))) + (if (>= i n) + acc + (let* ((item (MSet-get-hit mset i)) + (doc (MSetIterator-get-document item)) + (record (Document-get-value doc SLOT-DOCDATA))) + (loop (+ i 1) + (cons (string-append + "{\"docid\":" (number->string (MSetIterator-get-docid item)) + ",\"percent\":" (number->string (MSetIterator-get-percent item)) + ",\"record\":" record "}") + acc))))))) + (display + (string-append + "{\"query\":" (json-value query-string) + ",\"count\":" (number->string (length hits)) + ",\"matches\":" (number->string (MSet-get-matches-estimated mset)) + ",\"hits\":[" + (string-join (reverse hits) ",") + "]}")) + (newline)) + (delete-Database db)))) + +(define (cmd-count opts) + (let ((db (open-readable (option-ref opts 'index #f)))) + (format #t "~a\n" (Database-get-doccount db)) + (delete-Database db))) + +(define (usage) + (format #t "usage: index.scm update --store STORE.sqlite --index DIR [--rebuild]~@ + \x20 index.scm search --index DIR [--limit N] [--or] QUERY...~@ + \x20 index.scm count --index DIR~%")) + +(define OPTION-SPEC + '((store (value #t)) + (index (value #t)) + (limit (value #t)) + (rebuild) + (or) + (quiet) + (help (single-char #\h)))) + +(define (main args) + (let* ((opts (getopt-long args OPTION-SPEC)) + (rest (option-ref opts '() '()))) + (parameterize ((CURRENT-OPTIONS opts)) + (match rest + (("update" . _) (cmd-update opts)) + (("search" . words) (cmd-search opts words)) + (("count" . _) (cmd-count opts)) + (("schema" . _) (format #t "~a\n" SCHEMA-VERSION)) + (("--help" . _) (usage)) + (((? (lambda (a) (member a '("-h" "--help"))) _) . _) (usage)) + ((cmd . _) (die "unknown command: ~a" cmd)) + (() (usage)))))) + +(main (command-line)) |
