| Age | Commit message (Collapse) | Author |
|
A commit that only adds a file recorded nothing: git has no before-side for
the new file, so its hunks carry no context ("@@ -0,0 +1,207 @@"), and Python
and Ruby had no declaration scanner to fall back on. Adding
bin/geno2mdb.py to gemma-wrapper left one commit row and two reasoning
entries with no symbol at all, so `why` could not reach them while `search`
and `list` could.
Both languages are now scanned line by line, which is the right tool here:
in Python and Ruby a declaration is the first token on its line, so the
keyword decides -- `def foo(bar):` declares, `foo(bar)` calls -- the
distinction no C++ line scanner can make.
Python def / async def -> fn, class -> type, and at column 0 only
NAME = value -> const, NAME: Type -> var (an indented assignment
is a local inside a def)
Ruby def / def self.NAME (with ? ! = suffixes) -> fn, class / module ->
type, and a column-0 capitalised NAME = value -> const
An added file needs no special case: every one of its lines is a '+' line,
which git_touched already walks, so the same path that records an added line
in an existing file now records a new file's declarations. Names strip .py
and .rb like .odin and .rs (script.helper); the C family keeps its extension
because a header declaration and the source definition are different rows.
The lookup's last resort was `s.name like '%name'` -- a suffix match -- so
`why geno2mdb` could never find a file's own symbols even though SKILL.md
documents a LIKE '%...%' scan. It is a substring match now, so the file base
works, and the doc is true again.
The check phase commits script.py and script.rb into its throwaway
repository and asserts both land as `add` rows and that `why script`
resolves; measured on gemma-wrapper, sync adds 344 touches over 319 commits
and `why read_snps` reaches the reasoning attached to the commit that
introduced it.
|
|
wrap-program exports unconditionally, so `export AI_MEMORY_CTAGS=...` in the
wrapper overwrote any value the user set: an installed ai-memory always used
the store's ctags, and the documented override did nothing. Found by testing
the degradation path -- `AI_MEMORY_CTAGS=/nonexistent/ctags` still produced
`add` rows, which is only possible if the override was ignored.
The default is now CTAGS_DEFAULT, a #config constant that guix.scm fills with
-define:CTAGS_PATH=...; $AI_MEMORY_CTAGS is consulted first, and a plain
`odin build` leaves the literal `ctags` for PATH. The check phase builds with
the same define, so it tests what users get.
|
|
C and C++ were the one language family where the store could only say "a hunk
landed inside this function". A diff line cannot be read for declarations
there: inside a body `foo(bar);` calls something, inside a class body the same
bytes declare a constructor, and pangemma indents both the same way (of 248538
changed lines in its history, 14991 are indented `name(` lines, and the most
frequent are calls). So the file is handed to universal-ctags and the tags of a
commit's blob are diffed against its parent's: present only in the new blob is
an add, only in the old one is a remove, and a name whose tag survives both is
left to hunk context, which is the honest answer for "the change was inside
this".
The cost is bounded by a cache keyed by git object id (new table `blobs`): a
blob's tags can never change, so each file version is parsed once per store.
On the pangemma history (688 commits, 1163 distinct C/C++ blobs) the first full
pass takes 26 s and touches go from 1404 context-only rows to 6888 add, 5125
remove, 1505 context; the second pass takes 11 s and runs no ctags at all.
ctags is optional at run time. Without it the store behaves exactly as before
and `sync` says so once. The Guix package bakes its absolute path into
$AI_MEMORY_CTAGS through wrap-program, and the check phase now builds a
two-commit git repository with one C++ file to prove it end to end: the
function added in the second commit is `add`, the one whose body changed is
`context`.
Details found by running it on a real history, all in this commit:
- lambdas are tagged by default under names ctags derives from a hash of the
file (`__anon426fc91f0102`), so every touched file reported seven functions
added and seven removed per commit; `--extras=-{anonymous}` turns them off.
- `--kinds-C++=+p` is needed or prototypes are invisible and a declaration in a
header would never be seen.
- include guards (`#define CALC_H`, `#define __CALC_H__`) are filtered: they are
bookkeeping, while other macros are kept as `const`.
- merge_decl keyed a commit's declarations by bare identifier, so two files
declaring the same name in one commit collided and one lost its attribution.
It is keyed by (file, name) now -- a bug that applied to Odin and Rust too.
|
|
A reasoning document was titled with the subject of the commit it is attached
to (COALESCE(c.subject,'') through a LEFT JOIN), which is wrong twice over: an
entry recorded with --commit none -- every entry before its commit exists --
was indexed with an empty title, and its gist, the assessed one-line claim
that `list` and `why` show, was not indexed as text at all; an attached entry
merely repeated the title its commit document already carries. The title is
now the entry's own gist, from reasoning.gist, and the join is gone.
`update-index --rebuild` also died with "mkdir failed (No such file or
directory)" whenever the index directory was absent -- the state of every
store that has never been indexed, and the state README.md tells a reader to
produce by deleting the index. Only the non-rebuild path created the
directory; `mkdir -p` now runs before the database handle either way.
The check phase records a gist deliberately different from the body and
asserts both that it answers a search and that it is the document's title, so
a regression fails the build rather than the reader.
.gitignore covers *.sqlite.d/, the default index directory, which neither
*.sqlite nor *.sqlite-* matched -- the README already claimed the index was
ignored.
doc/SKILL.md says what `title` means in a search reply: the gist for a
reasoning entry, the subject for a commit.
|
|
Two failures found by using the store from more than one binary at once.
A stale ai-memory opened a version 5 store and left it stamped version 3.
SQLite cannot drop a column, so the data survived (gist_source was still
there), but the stamp lied -- and the next newer binary then re-ran data
steps it believed were pending. The migration chain ends at the version the
*binary* knows and wrote that number unconditionally, which is what did it.
So: schema_version() reads the version out of the SCHEMA the binary embeds,
so the number lives in schema.sql alone and the two cannot disagree; the
stamp only ever moves forward (`if version > started_at`); and a store whose
version is greater than this binary's is refused before anything is touched:
ai-memory: this store is schema v9; this binary knows v5
ai-memory: it was written by a newer ai-memory -- upgrade this binary
(scripts/install.sh). Nothing was changed.
The second failure was the migration order: a store at version 3 that a
newer binary had already written gists into died on every command with
"no such column: gist_source", because the version 3 -> 4 backfill referenced
a column the version 4 -> 5 step adds. The gist columns are now ensured
(guarded by the table existing, so a fresh store still gets them from SCHEMA)
before any backfill runs, and the skew hint fires on any "no such column" or
"no such table", not only on an empty error message.
Verified: a v9 store is refused with the stamp untouched; a store the stale
binary knocked back to v3 upgrades to v5 with its assessed gists intact; the
gist immutability test still passes; and all six stores on this box are now at
v5 after the new binary opened them. schema_version() needed care in Odin:
SCHEMA is a compile-time constant, so it is copied to a string before being
walked by index.
|
|
The first version worked exactly once. guix build --root cannot replace a
root that already exists -- it answers
guix build: error: failed to create GC root '.../root': File exists
-- so the second run, which is the normal case, died before building. The
script now removes the old symlink first and refuses to touch anything that is
not a symlink, since a directory at that path is someone else's mistake.
Verified by running it three times in a row against a tree that changed between
runs (adding this file moves the store path, which is what makes the root worth
having).
|
|
guix build -f guix.scm prints a path that changes with every rebuild of the
tree, so the practical way to use it has been to copy a store hash into a
script -- and a copied hash outlives the source. That is not hypothetical:
a store on this box was created by a binary that predated the gist column,
and the mismatch surfaced as an empty 'sql error (Error):' at write time.
scripts/install.sh builds from this checkout, GC-roots the result
(AI_MEMORY_ROOT, default ~/.local/state/ai-memory/root) so a guix gc cannot
take it away, links AI_MEMORY_LINK (default ~/.local/bin/ai-memory) so nobody
has to remember a hash, and then runs the binary once to prove it starts.
Two builds in a row give the same store path, which is what makes the root
stable between runs and not merely between rebuilds of the same day.
README and doc/SKILL.md now lead with it, and keep the bare guix build as the
unpinned alternative.
|
|
A gist is a judgement about what an entry claims, so it must not drift once
made. The store did not hold that invariant: db_backfill_gist ran
update reasoning set gist = ?1 where id = ?2
for every row, so any migration that re-derived gists would have replaced
every assessed --gist with the body's first sentence. It only escaped
notice because the backfill happens to run once, on the version 3 -> 4
upgrade.
reasoning.gist_source now says how each gist came about -- 'assessed' when
the caller said what the entry claims (--gist), 'derived' when it is the
body's first sentence -- and the backfill is guarded:
where gist is null or gist_source = 'derived'
so a derived gist may be recomputed when the derivation improves, and an
assessed one is frozen for the life of the entry. Version 5 of the schema
adds the column; a store that predates it cannot say which of its gists were
assessed, and the two mistakes are not equal -- labelling an assessed gist
derived lets a later migration overwrite a judgement, while labelling a
derived one assessed merely freezes a first sentence -- so existing gists
become 'assessed' and only rows with no gist are left to the backfill.
test/gist_immutable.sh pins it: record one assessed and one derived gist,
set user_version back to 3 so the next open re-runs the migration, and assert
both are byte-identical afterwards. Against the old backfill the assessed
one changes and the test fails.
Measured on the production store (139 entries, version 4): the migration to 5
kept every gist and labelled all 139 assessed, no gist was blanked, and
listings (log, why, search) are unchanged. Two new entries came out as
expected: ('Derived one', derived) and ('Assessed one', assessed).
|
|
`ai-memory record --commit 1ae2f5d` (a short hash) answered
imported 1ae2f5d (qtl2: permutations for the LOD thresholds)
ai-memory: sql error (Error): SQL logic error
and left the store half-updated: the commit row was imported under its full
hash, the reasoning row was not written at all, and the message named neither
the cause nor the fix. Feed the same command the full forty characters and it
worked, which is the sort of difference that reads as flakiness.
Three changes, in the order they matter:
* resolve the ref first. record's --commit and attach's positional argument
both go through resolve_commit_ref, which asks git (`rev-parse --verify`) and
keeps the full hash; if git does not know the ref the command refuses before
opening a transaction, says which ref and which repository, and suggests
--commit none plus a later attach. The store keys commits by the full hash,
so a short ref written to reasoning.commit_hash names a row that cannot exist.
* one transaction around the import and the insert (begin immediate ... commit,
rollback on any failure), so a failed record leaves a store that looks exactly
as it did. Verified against a store whose schema predates the binary: no new
commit row survives the failure.
* an actionable message: an empty sqlite3 errmsg on SQLITE_ERROR now says the
store's schema is older than the binary and to run init, instead of "sql
error (Error):" with nothing after it.
Measured, fresh store: --commit 1ae2f5d records against
1ae2f5d712e18c016fdb7bd04da99eab4da6915b; --commit deadbeef refuses and writes
nothing; --commit HEAD~1 resolves; attach 1ae2f5d links an unattached entry.
|
|
list, why, search and log printed either nothing or a full ISO timestamp, and
a stamp that wide pushes the hash and the gist apart. One helper, fmt_date,
renders UTC dd/mm and adds the year only when it is not the current one; an
unattached entry is dated by when it was written, since it has no commit date.
|
|
A store can carry a newer column under an older version number: the
genenetwork4 store reported user_version = 3 while already having
reasoning.gist, so the v3->v4 ALTER failed with "duplicate column name:
gist" and every command on that store died in open_db -- search included.
Each step now adds its column only if it is missing, inspects pragma
table_info, and writes the version at the end. Backfills still run only
when a column was actually added.
|
|
`list`, `why` and `search` were truncating bodies at 96 characters, which
keeps whatever happens to come first -- usually the least informative clause
of the entry ("Only the range or every chromosome?...").
reasoning.gist (schema v4) holds the claim on one line, written at record
time via --gist. Listings show it; --full adds the body beneath. Existing
entries are backfilled from their first sentence, which is a floor, not an
equivalent: it says what the entry is about. Recording without --gist still
works and says so on stderr.
|
|
why SYMBOL and search both need a starting point -- a symbol name or a word
you already know. `list` is the inventory: short hash, entry id, kind, and
the body flattened to a line. Unattached entries sort first and read
"uncommitted"; a superseded entry reads "kind->#N". --session, --kind and
--commit narrow it.
|
|
The project had no README: the only prose was doc/SKILL.md, which is the
working protocol for an assistant already using the tool, not an introduction
to it. README.md now says what the store is for (a session's context is finite
and the conversation is gone when it ends; the diff says what, not why), what
it stores (three derived tables, one authored), how to build it, and the
vocabulary it needs: why / search / record / attach / sync / log / status,
with the topic index and its prefixes, the rules the tool relies on
(append-only reasoning, --supersedes, --basis) and the two limits worth
knowing up front -- the store is per repository and local, so a fact spanning
repos needs recording in each and anything that must outlive the machine
belongs in commits and docs too.
The examples are real output from this deployment's own stores, not invented:
the why entry shows the [fn file.function, context] annotation and the
reasoning under it, log shows the per-commit reasoning counts, status the row
counts. The record examples were run against a scratch store before being
written down.
LICENSE is BSD 3-Clause (Copyright (c) 2026 Pjotr Prins), which also settles
the TODO in guix.scm: the package now declares license:bsd-3 instead of the
placeholder expat. The package installs README.md and doc/SKILL.md beside the
binary under share/doc/ai-memory/; LICENSE travels in
share/doc/ai-memory-<version>/ from the standard install-license-files phase,
so it is not installed twice.
|
|
A session's context is finite and the conversation that produced a piece of
work is gone when it ends. Say that out loud in the skill doc, because it
changes what recording is for: not bookkeeping, but the next session's only
entry point.
New section 'Session continuity', with the measured numbers from the
GEMMA/ravanan work in genenetwork4: finding the runbook costs 16 ms through
SQLite FTS (~100 ms through the topic index, 1.3 s to build the index), and
re-running what the entry says cost ~20 s of machine time (container up ->
endpoint verified). Four rules follow from that: record RUNBOOK entries with
the commands, not just the conclusions; name the session after the concept you
want to find it by; record the gotchas (three of the four requirements in that
session were wrong at first); and point at the long-form docs instead of
copying them.
Also states two limits plainly: the store is per repository, so a fact
spanning repos needs recording in each (or one shared index with a repo:
filter), and it is local gitignored SQLite -- it survives a cleared session,
not a lost machine, so anything that must outlive the box belongs in commits
and docs too.
Session protocol gains step 6: write for a reader who has none of this
conversation.
|
|
topic:pve answered with nine reasoning entries and no commits, which is
backwards for anyone following the story: the pve thread is carried by the
commits (cd3e65b, 64e63bd, 3a42b5f, 8e11ec0) as much as by the notes.
Commit documents now take the sessions of the reasoning attached to them
(one GROUP_CONCAT query over reasoning, joined in memory) and gain the same
S<session> and O<topic> terms, so the same query returns 13 documents
instead of 9.
Also documents the honest limit of session-derived topics in doc/SKILL.md:
a session named gemma-wrapper-diagnostics about the same work does not
answer topic:pve, while a plain-text pve search still finds it. topic: is
the curated answer, free text is the recall answer -- name a session after
the concept you want it found by.
|
|
SQLite stays the truth; this is a derived index that lets a reader enter by
concept rather than by symbol. Two commands, one prefix vocabulary:
ai-memory --index DIR update-index [--rebuild]
ai-memory --index DIR search 'topic:pve repo:genenetwork4'
Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:,
date:. Free text is stemmed; default operator AND, --or to widen. Topic
terms come from the session id split on hyphens, so a session named
"gemma-pve" answers topic:pve and topic:gemma -- naming a session after a
concept is how an entry becomes findable by it.
Why Guile and a subprocess: libxapian is C++-only, so a foreign call from
Odin means a hand-written C shim with its own exception boundary; guile-xapian
already ships one, tested and maintained. So scripts/index.scm owns every
Xapian call (426 lines: schema constants, record builders, updater, query
parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines
of process plumbing. The boundary is a store path in and JSON out, which also
means the search reply needs no parser on this side.
Incremental in the way that matters: documents are deleted by their unique
term and re-added, and Xapian applies that as changed postings only. The whole
body is re-tokenised each run (~1 s at a few hundred documents) which buys not
keeping any "what did I index last time" state that could go stale. Two
discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by
term silently appends (the string lands on the docid overload), so delete then
add; and Database-get-document by term returns nothing, so no digest compare
was attempted.
The package's check phase now exercises it end to end -- throwaway store,
one entry, update-index, then `search topic:index | grep -q '"count":1'` --
because a check that only links proves nothing about the four Guile
dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where
index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for
guile-xapian and guile-sqlite3.
doc/SKILL.md gains the usage, the prefix list and the reason for the
subprocess.
|
|
`insert or ignore' returns success for a touch that already exists, so the
old counter reported every re-sync as if it had imported the whole history
again. Count the table rows before and after instead.
|
|
.gitignore now ignores *.sqlite and *.sqlite-*, so a store written by --db or
$AI_MEMORY_DB is never committed under a different name.
doc/SKILL.md gains a "Rewriting history" section: amend/rebase/squash leave
touches and reasoning attached to hashes that no longer exist, sync only ever
adds, and the foreign-key cascade that removes the stale commit's touches is
off unless pragma foreign_keys=on is set in the sqlite3 CLI.
|
|
commits, symbols and touches are derived from `git log`/`git show` by
`sync`; only reasoning is authored, and it is append-only. Each entry
records a basis (how it was established) and may supersede an earlier
entry instead of rewriting it.
Symbols come from two signals: declarations added or removed on diff
lines (Odin and Rust), and the enclosing item named in git's own hunk
headers, which needs no parser and works for any language with a
userdiff driver.
Links SQLite through the odin-sqlite3 bindings; guix.scm builds both the
compiler and the tool. doc/SKILL.md is the operating guide.
|