| Age | Commit message (Collapse) | Author |
|
A reasoning document was titled with the subject of the commit it is attached
to (COALESCE(c.subject,'') through a LEFT JOIN), which is wrong twice over: an
entry recorded with --commit none -- every entry before its commit exists --
was indexed with an empty title, and its gist, the assessed one-line claim
that `list` and `why` show, was not indexed as text at all; an attached entry
merely repeated the title its commit document already carries. The title is
now the entry's own gist, from reasoning.gist, and the join is gone.
`update-index --rebuild` also died with "mkdir failed (No such file or
directory)" whenever the index directory was absent -- the state of every
store that has never been indexed, and the state README.md tells a reader to
produce by deleting the index. Only the non-rebuild path created the
directory; `mkdir -p` now runs before the database handle either way.
The check phase records a gist deliberately different from the body and
asserts both that it answers a search and that it is the document's title, so
a regression fails the build rather than the reader.
.gitignore covers *.sqlite.d/, the default index directory, which neither
*.sqlite nor *.sqlite-* matched -- the README already claimed the index was
ignored.
doc/SKILL.md says what `title` means in a search reply: the gist for a
reasoning entry, the subject for a commit.
|
|
topic:pve answered with nine reasoning entries and no commits, which is
backwards for anyone following the story: the pve thread is carried by the
commits (cd3e65b, 64e63bd, 3a42b5f, 8e11ec0) as much as by the notes.
Commit documents now take the sessions of the reasoning attached to them
(one GROUP_CONCAT query over reasoning, joined in memory) and gain the same
S<session> and O<topic> terms, so the same query returns 13 documents
instead of 9.
Also documents the honest limit of session-derived topics in doc/SKILL.md:
a session named gemma-wrapper-diagnostics about the same work does not
answer topic:pve, while a plain-text pve search still finds it. topic: is
the curated answer, free text is the recall answer -- name a session after
the concept you want it found by.
|
|
SQLite stays the truth; this is a derived index that lets a reader enter by
concept rather than by symbol. Two commands, one prefix vocabulary:
ai-memory --index DIR update-index [--rebuild]
ai-memory --index DIR search 'topic:pve repo:genenetwork4'
Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:,
date:. Free text is stemmed; default operator AND, --or to widen. Topic
terms come from the session id split on hyphens, so a session named
"gemma-pve" answers topic:pve and topic:gemma -- naming a session after a
concept is how an entry becomes findable by it.
Why Guile and a subprocess: libxapian is C++-only, so a foreign call from
Odin means a hand-written C shim with its own exception boundary; guile-xapian
already ships one, tested and maintained. So scripts/index.scm owns every
Xapian call (426 lines: schema constants, record builders, updater, query
parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines
of process plumbing. The boundary is a store path in and JSON out, which also
means the search reply needs no parser on this side.
Incremental in the way that matters: documents are deleted by their unique
term and re-added, and Xapian applies that as changed postings only. The whole
body is re-tokenised each run (~1 s at a few hundred documents) which buys not
keeping any "what did I index last time" state that could go stale. Two
discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by
term silently appends (the string lands on the docid overload), so delete then
add; and Database-get-document by term returns nothing, so no digest compare
was attempted.
The package's check phase now exercises it end to end -- throwaway store,
one entry, update-index, then `search topic:index | grep -q '"count":1'` --
because a check that only links proves nothing about the four Guile
dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where
index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for
guile-xapian and guile-sqlite3.
doc/SKILL.md gains the usage, the prefix list and the reason for the
subprocess.
|