about summary refs log tree commit diff

name: gn-memory description: Keep a repository's ai-memory store current — import git history, record reasoning while working, and answer "why does this code look like this?" (or "how did we run that?") after a session's context has been cleared.


gn-memory

ai-memory is a per-repository store that links what changed (derived from git) to why it changed (written by you). Three tables:

table source contents
commits derived hash, author timestamp, subject, body
symbols + touches derived every declaration a commit added, removed, or merely enclosed (hunk context)
reasoning authored append-only thoughts, conclusions and questions; each records its basis and may supersede an earlier entry

Query flow: symbol → commits that touched it → reasoning behind those commits.

Getting the binary

scripts/install.sh              # build, GC-root, link ~/.local/bin/ai-memory

That is the one to use: a bare `guix build' prints a path that changes with every rebuild, so anything you write down goes stale, and a stale binary against a newer store is the schema skew that fails at write time rather than at start-up.

guix build -f guix.scm          # build (odin + sqlite are in guix.scm)
$(guix build -f guix.scm)/bin/ai-memory --help

Or build locally into bin/ (gitignored) with the flags listed at the top of guix.scm:

guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style \
  -collection:sqlite3=$GUIX_ENVIRONMENT/share/odin-sqlite3 \
  -define:SQLITE3_SYSTEM_LIB=true -define:SQLITE3_DYNAMIC_LIB=true \
  "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib"

guix build -f guix.scm also runs the package's check phase, which builds a throwaway store, records an entry, indexes it, queries it, and then makes a two-commit git repository holding one C++ file to prove the ctags half (an added function is add, a changed body is context).

universal-ctags is what turns C and C++ files into symbols; the Guix package bakes its absolute path into $AI_MEMORY_CTAGS, so nothing depends on PATH. If the binary is missing the store still works -- C and C++ simply keep hunk context and get no add/remove -- and sync says so once on stderr.

The store defaults to <repo>/ai-memory.sqlite; override with --db PATH or $AI_MEMORY_DB. --repo PATH / -C PATH selects the repository, and may be given before or after the command.

Session protocol

  1. Once per repo: ai-memory -C REPO init (idempotent, creates/upgrades the schema).
  2. After every commit: ai-memory -C REPO sync. It is cheap and idempotent — re-running it only refreshes derived rows. Use import instead of sync only to pull in a whole history the first time (import has no limit).
  3. Before changing code: ask the store, do not guess. ai-memory -C REPO why SYMBOL and ai-memory -C REPO search TEXT.
  4. While working: record as you go, not at the end. --kind thought for what you are weighing, --kind conclusion for what you settled on, and --kind question when you need an answer from the user.
  5. Before the commit exists there is nothing to attach to: use --commit none, then after committing run ai-memory -C REPO attach HEAD --session S.
  6. Write for a reader who has none of this conversation. The next session starts with an empty context window; see Session continuity for what that means for operational work in particular.
  7. Correcting yourself is not a normal record. --supersedes ID changes what why and search answer from then on, so it needs the user's consent first — see the Rules below.
ai-memory -C REPO record --session fix-parser --step 1 \
  --kind thought --basis "read at git.odin:120-141" \
  --gist "Hunk headers name the enclosing fn; a new fn still needs the scan" \
  --body "Hunk headers name the enclosing fn; a new fn needs the declarative scan." --commit none
ai-memory -C REPO record --session fix-parser --step 2 \
  --kind conclusion --basis "measured on genenetwork4, 284 commits in 2.5 s" \
  --gist "Keep both signals; add ctags only if add/remove proves noisy" \
  --body "Keep both signals; add ctags only if add/remove proves noisy."
ai-memory -C REPO attach HEAD --session fix-parser

record and attach import the commit on demand, so they work immediately after committing even if sync has not run yet.

Session continuity: recovering after the context is cleared

A model's context is finite -- a million tokens if you are lucky, and the conversation that produced a piece of work is gone when it ends. The store is the part that outlives it: git-derived history for what happened, authored reasoning for why, and a topic index over both for finding it by concept. Treat recording as part of the work, not bookkeeping after it, because the next session starts with none of this conversation.

What recovery looks like in practice, measured on the GEMMA/ravanan work in genenetwork4 (2026-11-12):

step cost
search "slurm container" (SQLite FTS, no index needed) 16 ms
the same through the topic index ~100 ms (1.3 s to build the index once)
re-running what the entry says (container up → endpoint verified) ~20 s of machine time

The first two lines are the point: rediscovering what to type costs milliseconds, where reconstructing it from the diff, the docs and a guess would cost the session.

So, what to record so that a cold reader arrives at a working command:

  • A RUNBOOK entry for anything operational, with the commands themselves — not just what you concluded. An entry that says "the container has a stale JWT and needs HOME writable" is useful; an entry that also holds the four commands, the port, the fingerprint and the fastest test is recovery. Write it as if the reader has none of your context, because they will not.
  • Name the session after the concept you want to find it by (topic: terms come from it), and attach reasoning to its commit so why SYMBOL and topic: both reach it.
  • Record the gotchas, not just the happy path. In that session three of the four things that had to be right were wrong at first (a ravanan build without the endpoint fix, the job's HOME not writable in the container, ccwl missing from PATH). Those are the sentences that save the next hour.
  • Point at the long-form docs (docs/…, a session note in another repo) rather than copying them: the store is for the shortest path to the right document, not a second copy of it.

Two honest limits. The store is per repository, so a fact that spans repos (the pve story in genenetwork4, gemma-wrapper and guix-bioinformatics) needs recording in each — or one shared index with a repo: filter. And the store is local, gitignored SQLite: it survives a cleared session, not a lost machine. Anything that must outlive the box belongs in commits and docs as well — which is also why every entry here is paired with a commit message that says the same thing.

The topic index (optional, derived)

A second, derived store: a Xapian index over the SQLite one, so a reader can enter by concept ("topic:pve") or by repository and kind ("repo:genenetwork4 type:commit") instead of by symbol name. It is disposable -- delete the directory and update-index --rebuild -- and SQLite stays the truth.

ai-memory --index DIR update-index            # bring it up to date
ai-memory --index DIR update-index --rebuild  # after a schema change
ai-memory --index DIR search 'topic:pve'
ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4'

(The directory defaults to <store>.d/index, override with --index or $AI_MEMORY_INDEX. search without --index keeps using SQLite FTS.)

Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit: (7-character hash), date: (yyyymmdd). Everything else is free text, stemmed. The default operator is AND; --or widens it.

Topic terms come from the session id, split on hyphens: a session called gemma-pve answers topic:gemma and topic:pve. So the cheapest way to make an entry findable by concept is to name the session after the concept -- a session called gemma-wrapper-diagnostics about the same work does not answer topic:pve, while a plain-text pve search still finds it. That is the difference between the two: topic: is the curated answer, free text is the recall answer.

A commit inherits the topics of the reasoning attached to it, so topic:pve returns both the reasoning and the commits that carry it.

Why a subprocess and a Guile script: libxapian is C++-only, so a foreign call from Odin needs a hand-written C shim. guile-xapian already has one, tested and maintained (scripts/index.scm owns every Xapian call; ai-memory shells out to it). The boundary is a store path in, JSON out.

The JSON a search returns is the record as indexed -- key, repo, session, kind, date, commit, title, body, terms -- plus docid and percent. count is how many came back, matches how many the index found.

For a reasoning entry title is its gist (the assessed one-line claim), not the subject of the commit it is attached to: that subject already titles the commit document, and an entry recorded with --commit none has no commit to borrow from. The gist is indexed as text too, so a claim written only in the gist is findable; a commit document's title is its subject, as before.

Rules

  • Never write to commits, symbols or touches by hand. They are derived; sync owns them and hand edits will be silently overwritten or duplicated.
  • reasoning is append-only. Do not edit or delete an entry to change your mind: record a correction with --supersedes ID. The old entry stays visible as #12 [superseded by #15], why and search answer with the correction, and --all shows the whole chain. attach is the only other update the tool makes (setting a NULL commit_hash).
  • Ask the user before superseding. --supersedes rewrites history as far as every later reader is concerned: why and search stop answering with the old entry, and that entry collapses to a one-line stub. The row survives, but the record's answer does not. So: say which entry id you propose to replace, what it claimed, what the replacement says, and on what basis — then wait for a yes. Do not bundle it into a batch of records, and never supersede an entry written by someone else without their consent. If the user declines, either record the correction as a new standing entry without --supersedes (both claims stay visible and the contradiction is on the record) or do not record it at all.
  • Every entry says how it was established (--basis): read at src/lib.rs:412-470, from commit message, measured on the gn4 instance, or plainly inferred. Inference and observation look identical in prose, and that is exactly what makes a wrong entry unrecognisable later.
  • Every entry carries a gist (--gist), and the gist is a claim, not a subject line: "GEMMA does accept -lmm 9 -pve; boundary se is -nan, so store None" beats "About pve reporting". Write it while recording, because that is the only moment the context is in hand — list, why and search show the gist and hide the body behind --full. Without --gist the tool falls back to the body's first sentence and says so on stderr; that fallback says what the entry is about, which is why prose-heavy entries read as truncated fragments.
  • One session per task, monotonically increasing --step. (session_id, step, kind) is unique, so re-running a step is a no-op rather than a duplicate.
  • Make bodies self-contained. Say what you considered, what you chose, what you rejected and why. A future reader has the diff but not the conversation.
  • Name what you renamed. Symbols are <file-base>.<identifier> (e.g. gemma.compute_gemma); an old name keeps its history, so mention the new name in the body to keep search useful.
  • Never commit the store or the build output. /bin/ and ai-memory.sqlite* are in .gitignore; commit only source. Importing history is read-only — -C never touches the working tree.

Rewriting history

git commit --amend, rebase and squash change commit hashes. Every touches and reasoning row attached to the old hash is then stale, and sync will not clean it up — it only adds. The repair, in this order:

  1. detach the reasoning for the old hash: update reasoning set commit_hash = null where commit_hash = '<old>'
  2. ai-memory -C REPO sync — imports the new hash
  3. ai-memory -C REPO attach <new|HEAD> --session S — re-links the entries
  4. delete the old commit row: pragma foreign_keys=on; delete from commits where hash = '<old>'

Step 4 needs the pragma: the sqlite3 CLI leaves foreign keys off, so the delete runs without cascading and the old commit's touches stay behind as orphans. Nothing complains — status just reports double the touches, and why points at a hash that is no longer in git log. Check afterwards with pragma foreign_key_check and by counting rows whose commit_hash has no matching commits row.

This is the one place where hand-written SQL is correct: the tool repairs nothing. Amending before recording is cheaper than repairing afterwards.

Interpretation

  • [fn gemma.compute_gemma, context] means a hunk landed inside that function, not that the function was declared. add/remove/mod are declaration changes.
  • Declarations come from three places: a line scanner for .odin and .rs, and for C and C++ (.c .h .cc .cpp .cxx .hpp .hh .hxx .ipp .tpp .inl) universal-ctags over the file itself, diffed between the commit's blob and its parent's -- which is why a C++ add means the declaration is new, not that a line mentioning it moved. Everything else still gets hunk-context symbols ([fn name, context]) thanks to git's userdiff drivers, so "which function was I in" works for Python, Go, … too.
  • The ctags pass is cached per git blob (table blobs), keyed by object id: a file version is parsed once per store, and re-syncing an imported history runs no ctags at all. Dropping the table only costs time.
  • C and C++ keep their extension in the symbol name (lmm.cpp.mdb_analyze), so a declaration in a header and the definition in the source are two symbols that why compute_maf shows side by side. Directories are still stripped, so a file that moved between directories reads as the same symbol (that collision predates ctags and applies to every language).
  • Kinds are the same five as everywhere else: fn for functions, prototypes and methods; type for class, struct, union, enum, typedef; const for macros except include guards; var for file-scope variables. Class members, enumerators, lambdas and namespaces are deliberately not tags here -- they churn without changing what a hunk was about.
  • why SYMBOL tries three things in order, all but the last indexed: the qualified name (gemma.compute_gemma), then the bare identifier via symbols.basename (compute_gemma), then a LIKE '%…%' scan for partial names.
  • log prints how many reasoning entries each commit carries — commits with (0) are the ones worth annotating later.
  • list is the inventory view: one line per entry, short-hash dd/mm #id kind gist. Unattached entries come first (labelled uncommitted, dated by when they were written), then the newest commit's, and a superseded entry reads kind->#N. Dates are UTC and carry the year only when it is not the current one, so a listing stays narrow without becoming ambiguous after a year. --session, --kind and --commit narrow it; skim list before opening entries with why.
  • why SYMBOL and search TEXT print the gist per entry; --full adds the body indented beneath it, and --all admits superseded entries. Gist-first is the default because a listing of twenty bodies is not read.

Troubleshooting

  • no commits to import — the repo has no history yet, or --repo points elsewhere.
  • why finds nothing — run sync first; or the symbol lives in a file type we do not parse for declarations.
  • Touch counts looking double, or a commit in the output of log that is not in git log — a hash was rewritten (amend/rebase). See Rewriting history; the stale rows do not repair themselves.
  • search finds nothing after manual DB edits — FTS is trigger-maintained; rebuild with insert into reasoning_fts(reasoning_fts) values('rebuild').
  • Direct inspection is fine (read-only):
sqlite3 ai-memory.sqlite \
  "select c.hash, c.subject, r.kind, r.body
     from symbols s join touches t on t.symbol_id = s.id
       join commits c on c.hash = t.commit_hash
       left join reasoning r on r.commit_hash = c.hash
    where s.name like '%compute_gemma'
    order by c.author_ts desc;"
  • commit, symbol and touch are SQL keywords; the tables are plural.