--- name: gn-memory description: Keep a repository's ai-memory store current — import git history, record reasoning while working, and answer "why does this code look like this?" (or "how did we run that?") after a session's context has been cleared. --- # gn-memory `ai-memory` is a per-repository store that links **what changed** (derived from git) to **why it changed** (written by you). Three tables: | table | source | contents | |---|---|---| | `commits` | derived | hash, author timestamp, subject, body | | `symbols` + `touches` | derived | every declaration a commit added, removed, or merely *enclosed* (hunk context) | | `reasoning` | **authored** | append-only thoughts, conclusions and questions; each records its `basis` and may `supersede` an earlier entry | Query flow: symbol → commits that touched it → reasoning behind those commits. ## Getting the binary ```sh scripts/install.sh # build, GC-root, link ~/.local/bin/ai-memory ``` That is the one to use: a bare `guix build' prints a path that changes with every rebuild, so anything you write down goes stale, and a stale binary against a newer store is the schema skew that fails at write time rather than at start-up. ```sh guix build -f guix.scm # build (odin + sqlite are in guix.scm) $(guix build -f guix.scm)/bin/ai-memory --help ``` Or build locally into `bin/` (gitignored) with the flags listed at the top of `guix.scm`: ```sh guix shell -D -f guix.scm -- odin build . -out:bin/ai-memory -vet -strict-style \ -collection:sqlite3=$GUIX_ENVIRONMENT/share/odin-sqlite3 \ -define:SQLITE3_SYSTEM_LIB=true -define:SQLITE3_DYNAMIC_LIB=true \ "-extra-linker-flags:-L$GUIX_ENVIRONMENT/lib -Wl,-rpath,$GUIX_ENVIRONMENT/lib" ``` `guix build -f guix.scm` also runs the package's check phase, which builds a throwaway store, records an entry, indexes it, queries it, and then makes a two-commit git repository holding one C++ file to prove the ctags half (an added function is `add`, a changed body is `context`). `universal-ctags` is what turns C and C++ files into symbols; the Guix package bakes its absolute path into `$AI_MEMORY_CTAGS`, so nothing depends on PATH. If the binary is missing the store still works -- C and C++ simply keep hunk context and get no `add`/`remove` -- and `sync` says so once on stderr. The store defaults to `/ai-memory.sqlite`; override with `--db PATH` or `$AI_MEMORY_DB`. `--repo PATH` / `-C PATH` selects the repository, and may be given before or after the command. ## Session protocol 1. **Once per repo**: `ai-memory -C REPO init` (idempotent, creates/upgrades the schema). 2. **After every commit**: `ai-memory -C REPO sync`. It is cheap and idempotent — re-running it only refreshes derived rows. Use `import` instead of `sync` only to pull in a *whole* history the first time (`import` has no limit). 3. **Before changing code**: ask the store, do not guess. `ai-memory -C REPO why SYMBOL` and `ai-memory -C REPO search TEXT`. 4. **While working**: record as you go, not at the end. `--kind thought` for what you are weighing, `--kind conclusion` for what you settled on, and `--kind question` when you need an answer from the user. 5. **Before the commit exists** there is nothing to attach to: use `--commit none`, then after committing run `ai-memory -C REPO attach HEAD --session S`. 6. **Write for a reader who has none of this conversation.** The next session starts with an empty context window; see [Session continuity](#session-continuity-recovering-after-the-context-is-cleared) for what that means for operational work in particular. 6. **Correcting yourself is not a normal record.** `--supersedes ID` changes what `why` and `search` answer from then on, so it needs the user's consent first — see the Rules below. ```sh ai-memory -C REPO record --session fix-parser --step 1 \ --kind thought --basis "read at git.odin:120-141" \ --gist "Hunk headers name the enclosing fn; a new fn still needs the scan" \ --body "Hunk headers name the enclosing fn; a new fn needs the declarative scan." --commit none ai-memory -C REPO record --session fix-parser --step 2 \ --kind conclusion --basis "measured on genenetwork4, 284 commits in 2.5 s" \ --gist "Keep both signals; add ctags only if add/remove proves noisy" \ --body "Keep both signals; add ctags only if add/remove proves noisy." ai-memory -C REPO attach HEAD --session fix-parser ``` `record` and `attach` import the commit on demand, so they work immediately after committing even if `sync` has not run yet. ## Session continuity: recovering after the context is cleared A model's context is finite -- a million tokens if you are lucky, and the conversation that produced a piece of work is gone when it ends. The store is the part that outlives it: git-derived history for *what* happened, authored reasoning for *why*, and a topic index over both for finding it by concept. Treat recording as part of the work, not bookkeeping after it, because the next session starts with none of this conversation. What recovery looks like in practice, measured on the GEMMA/ravanan work in `genenetwork4` (2026-11-12): | step | cost | |---|---| | `search "slurm container"` (SQLite FTS, no index needed) | **16 ms** | | the same through the topic index | ~100 ms (1.3 s to build the index once) | | re-running what the entry says (container up → endpoint verified) | ~20 s of machine time | The first two lines are the point: rediscovering *what to type* costs milliseconds, where reconstructing it from the diff, the docs and a guess would cost the session. So, what to record so that a cold reader arrives at a working command: - **A RUNBOOK entry for anything operational**, with the commands themselves — not just what you concluded. An entry that says "the container has a stale JWT and needs `HOME` writable" is useful; an entry that also holds the four commands, the port, the fingerprint and the fastest test is *recovery*. Write it as if the reader has none of your context, because they will not. - **Name the session after the concept** you want to find it by (`topic:` terms come from it), and **attach reasoning to its commit** so `why SYMBOL` and `topic:` both reach it. - **Record the gotchas, not just the happy path.** In that session three of the four things that had to be right were wrong at first (a ravanan build without the endpoint fix, the job's `HOME` not writable in the container, `ccwl` missing from `PATH`). Those are the sentences that save the next hour. - **Point at the long-form docs** (`docs/…`, a session note in another repo) rather than copying them: the store is for the shortest path to the right document, not a second copy of it. Two honest limits. The store is **per repository**, so a fact that spans repos (the `pve` story in `genenetwork4`, `gemma-wrapper` and `guix-bioinformatics`) needs recording in each — or one shared index with a `repo:` filter. And the store is **local, gitignored SQLite**: it survives a cleared session, not a lost machine. Anything that must outlive the box belongs in commits and docs as well — which is also why every entry here is paired with a commit message that says the same thing. ## The topic index (optional, derived) A second, *derived* store: a Xapian index over the SQLite one, so a reader can enter by **concept** ("topic:pve") or by repository and kind ("repo:genenetwork4 type:commit") instead of by symbol name. It is disposable -- delete the directory and `update-index --rebuild` -- and SQLite stays the truth. ```sh ai-memory --index DIR update-index # bring it up to date ai-memory --index DIR update-index --rebuild # after a schema change ai-memory --index DIR search 'topic:pve' ai-memory --index DIR search --limit 5 'type:commit ravanan repo:genenetwork4' ``` (The directory defaults to `.d/index`, override with `--index` or `$AI_MEMORY_INDEX`. `search` without `--index` keeps using SQLite FTS.) **Prefixes**: `topic:`, `repo:`, `type:` (`reasoning`|`commit`), `session:`, `commit:` (7-character hash), `date:` (`yyyymmdd`). Everything else is free text, stemmed. The default operator is AND; `--or` widens it. Topic terms come from the **session id**, split on hyphens: a session called `gemma-pve` answers `topic:gemma` and `topic:pve`. So the cheapest way to make an entry findable by concept is to **name the session after the concept** -- a session called `gemma-wrapper-diagnostics` about the same work does not answer `topic:pve`, while a plain-text `pve` search still finds it. That is the difference between the two: `topic:` is the curated answer, free text is the recall answer. A **commit inherits the topics of the reasoning attached to it**, so `topic:pve` returns both the reasoning and the commits that carry it. Why a subprocess and a Guile script: `libxapian` is C++-only, so a foreign call from Odin needs a hand-written C shim. `guile-xapian` already has one, tested and maintained (`scripts/index.scm` owns every Xapian call; ai-memory shells out to it). The boundary is a store path in, JSON out. The JSON a search returns is the record as indexed -- `key`, `repo`, `session`, `kind`, `date`, `commit`, `title`, `body`, `terms` -- plus `docid` and `percent`. `count` is how many came back, `matches` how many the index found. For a reasoning entry `title` is its **gist** (the assessed one-line claim), not the subject of the commit it is attached to: that subject already titles the commit document, and an entry recorded with `--commit none` has no commit to borrow from. The gist is indexed as text too, so a claim written only in the gist is findable; a commit document's `title` is its subject, as before. ## Rules - **Never write to `commits`, `symbols` or `touches` by hand.** They are derived; `sync` owns them and hand edits will be silently overwritten or duplicated. - **`reasoning` is append-only.** Do not edit or delete an entry to change your mind: record a correction with `--supersedes ID`. The old entry stays visible as `#12 [superseded by #15]`, `why` and `search` answer with the correction, and `--all` shows the whole chain. `attach` is the only other update the tool makes (setting a NULL `commit_hash`). - **Ask the user before superseding.** `--supersedes` rewrites history as far as every later reader is concerned: `why` and `search` stop answering with the old entry, and that entry collapses to a one-line stub. The row survives, but the record's answer does not. So: say which entry id you propose to replace, what it claimed, what the replacement says, and on what basis — then wait for a yes. Do not bundle it into a batch of records, and never supersede an entry written by someone else without their consent. If the user declines, either record the correction as a new standing entry without `--supersedes` (both claims stay visible and the contradiction is on the record) or do not record it at all. - **Every entry says how it was established** (`--basis`): `read at src/lib.rs:412-470`, `from commit message`, `measured on the gn4 instance`, or plainly `inferred`. Inference and observation look identical in prose, and that is exactly what makes a wrong entry unrecognisable later. - **Every entry carries a gist** (`--gist`), and the gist is a *claim*, not a subject line: "GEMMA does accept -lmm 9 -pve; boundary se is -nan, so store None" beats "About pve reporting". Write it while recording, because that is the only moment the context is in hand — `list`, `why` and `search` show the gist and hide the body behind `--full`. Without `--gist` the tool falls back to the body's first sentence and says so on stderr; that fallback says what the entry is *about*, which is why prose-heavy entries read as truncated fragments. - **One session per task**, monotonically increasing `--step`. `(session_id, step, kind)` is unique, so re-running a step is a no-op rather than a duplicate. - **Make bodies self-contained.** Say what you considered, what you chose, what you rejected and why. A future reader has the diff but not the conversation. - **Name what you renamed.** Symbols are `.` (e.g. `gemma.compute_gemma`); an old name keeps its history, so mention the new name in the body to keep `search` useful. - **Never commit the store or the build output.** `/bin/` and `ai-memory.sqlite*` are in `.gitignore`; commit only source. Importing history is read-only — `-C` never touches the working tree. ## Rewriting history `git commit --amend`, `rebase` and `squash` change commit hashes. Every `touches` and `reasoning` row attached to the old hash is then stale, and `sync` will **not** clean it up — it only adds. The repair, in this order: 1. detach the reasoning for the old hash: `update reasoning set commit_hash = null where commit_hash = ''` 2. `ai-memory -C REPO sync` — imports the new hash 3. `ai-memory -C REPO attach --session S` — re-links the entries 4. delete the old commit row: `pragma foreign_keys=on; delete from commits where hash = ''` Step 4 needs the pragma: the `sqlite3` CLI leaves foreign keys **off**, so the delete runs without cascading and the old commit's touches stay behind as orphans. Nothing complains — `status` just reports double the touches, and `why` points at a hash that is no longer in `git log`. Check afterwards with `pragma foreign_key_check` and by counting rows whose `commit_hash` has no matching `commits` row. This is the one place where hand-written SQL is correct: the tool repairs nothing. Amending *before* recording is cheaper than repairing afterwards. ## Interpretation - `[fn gemma.compute_gemma, context]` means a hunk landed *inside* that function, not that the function was declared. `add`/`remove`/`mod` are declaration changes. - Declarations come from three places: a line scanner for `.odin` and `.rs`, and for C and C++ (`.c .h .cc .cpp .cxx .hpp .hh .hxx .ipp .tpp .inl`) `universal-ctags` over the file itself, diffed between the commit's blob and its parent's -- which is why a C++ `add` means the declaration is new, not that a line mentioning it moved. Everything else still gets hunk-context symbols (`[fn name, context]`) thanks to git's userdiff drivers, so "which function was I in" works for Python, Go, … too. - The ctags pass is cached per git blob (table `blobs`), keyed by object id: a file version is parsed once per store, and re-syncing an imported history runs no ctags at all. Dropping the table only costs time. - C and C++ keep their extension in the symbol name (`lmm.cpp.mdb_analyze`), so a declaration in a header and the definition in the source are two symbols that `why compute_maf` shows side by side. Directories are still stripped, so a file that moved between directories reads as the same symbol (that collision predates ctags and applies to every language). - Kinds are the same five as everywhere else: `fn` for functions, prototypes and methods; `type` for class, struct, union, enum, typedef; `const` for macros except include guards; `var` for file-scope variables. Class members, enumerators, lambdas and namespaces are deliberately not tags here -- they churn without changing what a hunk was about. - `why SYMBOL` tries three things in order, all but the last indexed: the qualified name (`gemma.compute_gemma`), then the bare identifier via `symbols.basename` (`compute_gemma`), then a `LIKE '%…%'` scan for partial names. - `log` prints how many reasoning entries each commit carries — commits with `(0)` are the ones worth annotating later. - `list` is the inventory view: one line per entry, `short-hash dd/mm #id kind gist`. Unattached entries come first (labelled `uncommitted`, dated by when they were written), then the newest commit's, and a superseded entry reads `kind->#N`. Dates are UTC and carry the year only when it is not the current one, so a listing stays narrow without becoming ambiguous after a year. `--session`, `--kind` and `--commit` narrow it; skim `list` before opening entries with `why`. - `why SYMBOL` and `search TEXT` print the gist per entry; `--full` adds the body indented beneath it, and `--all` admits superseded entries. Gist-first is the default because a listing of twenty bodies is not read. ## Troubleshooting - `no commits to import` — the repo has no history yet, or `--repo` points elsewhere. - `why` finds nothing — run `sync` first; or the symbol lives in a file type we do not parse for declarations. - Touch counts looking double, or a commit in the output of `log` that is not in `git log` — a hash was rewritten (amend/rebase). See *Rewriting history*; the stale rows do not repair themselves. - `search` finds nothing after manual DB edits — FTS is trigger-maintained; rebuild with `insert into reasoning_fts(reasoning_fts) values('rebuild')`. - Direct inspection is fine (read-only): ```sh sqlite3 ai-memory.sqlite \ "select c.hash, c.subject, r.kind, r.body from symbols s join touches t on t.symbol_id = s.id join commits c on c.hash = t.commit_hash left join reasoning r on r.commit_hash = c.hash where s.name like '%compute_gemma' order by c.author_ts desc;" ``` - `commit`, `symbol` and `touch` are SQL keywords; the tables are plural.