about summary refs log tree commit diff
path: root/doc
AgeCommit message (Collapse)Author
32 hourssymbols: C and C++ declarations from ctags, diffed between blobsPjotr Prins
C and C++ were the one language family where the store could only say "a hunk landed inside this function". A diff line cannot be read for declarations there: inside a body `foo(bar);` calls something, inside a class body the same bytes declare a constructor, and pangemma indents both the same way (of 248538 changed lines in its history, 14991 are indented `name(` lines, and the most frequent are calls). So the file is handed to universal-ctags and the tags of a commit's blob are diffed against its parent's: present only in the new blob is an add, only in the old one is a remove, and a name whose tag survives both is left to hunk context, which is the honest answer for "the change was inside this". The cost is bounded by a cache keyed by git object id (new table `blobs`): a blob's tags can never change, so each file version is parsed once per store. On the pangemma history (688 commits, 1163 distinct C/C++ blobs) the first full pass takes 26 s and touches go from 1404 context-only rows to 6888 add, 5125 remove, 1505 context; the second pass takes 11 s and runs no ctags at all. ctags is optional at run time. Without it the store behaves exactly as before and `sync` says so once. The Guix package bakes its absolute path into $AI_MEMORY_CTAGS through wrap-program, and the check phase now builds a two-commit git repository with one C++ file to prove it end to end: the function added in the second commit is `add`, the one whose body changed is `context`. Details found by running it on a real history, all in this commit: - lambdas are tagged by default under names ctags derives from a hash of the file (`__anon426fc91f0102`), so every touched file reported seven functions added and seven removed per commit; `--extras=-{anonymous}` turns them off. - `--kinds-C++=+p` is needed or prototypes are invisible and a declaration in a header would never be seen. - include guards (`#define CALC_H`, `#define __CALC_H__`) are filtered: they are bookkeeping, while other macros are kept as `const`. - merge_decl keyed a commit's declarations by bare identifier, so two files declaring the same name in one commit collided and one lost its attribution. It is keyed by (file, name) now -- a bug that applied to Odin and Rust too.
33 hoursindex: title a reasoning doc with its gist, and make --rebuild stand alonePjotr Prins
A reasoning document was titled with the subject of the commit it is attached to (COALESCE(c.subject,'') through a LEFT JOIN), which is wrong twice over: an entry recorded with --commit none -- every entry before its commit exists -- was indexed with an empty title, and its gist, the assessed one-line claim that `list` and `why` show, was not indexed as text at all; an attached entry merely repeated the title its commit document already carries. The title is now the entry's own gist, from reasoning.gist, and the join is gone. `update-index --rebuild` also died with "mkdir failed (No such file or directory)" whenever the index directory was absent -- the state of every store that has never been indexed, and the state README.md tells a reader to produce by deleting the index. Only the non-rebuild path created the directory; `mkdir -p` now runs before the database handle either way. The check phase records a gist deliberately different from the body and asserts both that it answers a search and that it is the document's title, so a regression fails the build rather than the reader. .gitignore covers *.sqlite.d/, the default index directory, which neither *.sqlite nor *.sqlite-* matched -- the README already claimed the index was ignored. doc/SKILL.md says what `title` means in a search reply: the gist for a reasoning entry, the subject for a commit.
39 hoursinstall.sh: build, root and link, so no store path is ever written downPjotr
guix build -f guix.scm prints a path that changes with every rebuild of the tree, so the practical way to use it has been to copy a store hash into a script -- and a copied hash outlives the source. That is not hypothetical: a store on this box was created by a binary that predated the gist column, and the mismatch surfaced as an empty 'sql error (Error):' at write time. scripts/install.sh builds from this checkout, GC-roots the result (AI_MEMORY_ROOT, default ~/.local/state/ai-memory/root) so a guix gc cannot take it away, links AI_MEMORY_LINK (default ~/.local/bin/ai-memory) so nobody has to remember a hash, and then runs the binary once to prove it starts. Two builds in a row give the same store path, which is what makes the root stable between runs and not merely between rebuilds of the same day. README and doc/SKILL.md now lead with it, and keep the bare guix build as the unpinned alternative.
4 daysdates: dd/mm next to every hash in gist listingsPjotr Prins
list, why, search and log printed either nothing or a full ISO timestamp, and a stamp that wide pushes the hash and the gist apart. One helper, fmt_date, renders UTC dd/mm and adds the year only when it is not the current one; an unattached entry is dated by when it was written, since it has no commit date.
4 daysgists: a one-line claim per entry, shown by defaultPjotr Prins
`list`, `why` and `search` were truncating bodies at 96 characters, which keeps whatever happens to come first -- usually the least informative clause of the entry ("Only the range or every chromosome?..."). reasoning.gist (schema v4) holds the claim on one line, written at record time via --gist. Listings show it; --full adds the body beneath. Existing entries are backfilled from their first sentence, which is a floor, not an equivalent: it says what the entry is about. Recording without --gist still works and says so on stderr.
4 dayslist: one line per reasoning entry, short hash firstPjotr Prins
why SYMBOL and search both need a starting point -- a symbol name or a word you already know. `list` is the inventory: short hash, entry id, kind, and the body flattened to a line. Unattached entries sort first and read "uncommitted"; a superseded entry reads "kind->#N". --session, --kind and --commit narrow it.
5 daysdocs: the store is how a cleared session recoversPjotr
A session's context is finite and the conversation that produced a piece of work is gone when it ends. Say that out loud in the skill doc, because it changes what recording is for: not bookkeeping, but the next session's only entry point. New section 'Session continuity', with the measured numbers from the GEMMA/ravanan work in genenetwork4: finding the runbook costs 16 ms through SQLite FTS (~100 ms through the topic index, 1.3 s to build the index), and re-running what the entry says cost ~20 s of machine time (container up -> endpoint verified). Four rules follow from that: record RUNBOOK entries with the commands, not just the conclusions; name the session after the concept you want to find it by; record the gotchas (three of the four requirements in that session were wrong at first); and point at the long-form docs instead of copying them. Also states two limits plainly: the store is per repository, so a fact spanning repos needs recording in each (or one shared index with a repo: filter), and it is local gitignored SQLite -- it survives a cleared session, not a lost machine, so anything that must outlive the box belongs in commits and docs too. Session protocol gains step 6: write for a reader who has none of this conversation.
5 daysindex: a commit inherits the topics of the reasoning attached to itPjotr
topic:pve answered with nine reasoning entries and no commits, which is backwards for anyone following the story: the pve thread is carried by the commits (cd3e65b, 64e63bd, 3a42b5f, 8e11ec0) as much as by the notes. Commit documents now take the sessions of the reasoning attached to them (one GROUP_CONCAT query over reasoning, joined in memory) and gain the same S<session> and O<topic> terms, so the same query returns 13 documents instead of 9. Also documents the honest limit of session-derived topics in doc/SKILL.md: a session named gemma-wrapper-diagnostics about the same work does not answer topic:pve, while a plain-text pve search still finds it. topic: is the curated answer, free text is the recall answer -- name a session after the concept you want it found by.
5 daysAdd a topic index: Xapian over the store, driven through GuilePjotr
SQLite stays the truth; this is a derived index that lets a reader enter by concept rather than by symbol. Two commands, one prefix vocabulary: ai-memory --index DIR update-index [--rebuild] ai-memory --index DIR search 'topic:pve repo:genenetwork4' Prefixes: topic:, repo:, type: (reasoning|commit), session:, commit:, date:. Free text is stemmed; default operator AND, --or to widen. Topic terms come from the session id split on hyphens, so a session named "gemma-pve" answers topic:pve and topic:gemma -- naming a session after a concept is how an entry becomes findable by it. Why Guile and a subprocess: libxapian is C++-only, so a foreign call from Odin means a hand-written C shim with its own exception boundary; guile-xapian already ships one, tested and maintained. So scripts/index.scm owns every Xapian call (426 lines: schema constants, record builders, updater, query parser, JSON writer) and ai-memory shells out to it -- index.odin is 145 lines of process plumbing. The boundary is a store path in and JSON out, which also means the search reply needs no parser on this side. Incremental in the way that matters: documents are deleted by their unique term and re-added, and Xapian applies that as changed postings only. The whole body is re-tokenised each run (~1 s at a few hundred documents) which buys not keeping any "what did I index last time" state that could go stale. Two discoveries shaped it, both from guile-xapian 0.4.0's SWIG surface: replace by term silently appends (the string lands on the docid overload), so delete then add; and Database-get-document by term returns nothing, so no digest compare was attempted. The package's check phase now exercises it end to end -- throwaway store, one entry, update-index, then `search topic:index | grep -q '"count":1'` -- because a check that only links proves nothing about the four Guile dependencies. guix.scm installs scripts/index.scm to share/ai-memory (where index.odin looks) and wraps the binary with GUILE and GUILE_LOAD_PATH for guile-xapian and guile-sqlite3. doc/SKILL.md gains the usage, the prefix list and the reason for the subprocess.
6 daysignore any SQLite store; document repairing a rewritten commit hashPjotr Prins
.gitignore now ignores *.sqlite and *.sqlite-*, so a store written by --db or $AI_MEMORY_DB is never committed under a different name. doc/SKILL.md gains a "Rewriting history" section: amend/rebase/squash leave touches and reasoning attached to hashes that no longer exist, sync only ever adds, and the foreign-key cascade that removes the stale commit's touches is off unless pragma foreign_keys=on is set in the sqlite3 CLI.
6 daysinitial ai-memory: derive commits and symbols from git, author the whyPjotr Prins
commits, symbols and touches are derived from `git log`/`git show` by `sync`; only reasoning is authored, and it is append-only. Each entry records a basis (how it was established) and may supersede an earlier entry instead of rewriting it. Symbols come from two signals: declarations added or removed on diff lines (Odin and Rust), and the enclosing item named in git's own hunk headers, which needs no parser and works for any language with a userdiff driver. Links SQLite through the odin-sqlite3 bindings; guix.scm builds both the compiler and the tool. doc/SKILL.md is the operating guide.