about summary refs log tree commit diff
path: root/db.odin
AgeCommit message (Collapse)Author
40 hourssymbols: C and C++ declarations from ctags, diffed between blobsPjotr Prins
C and C++ were the one language family where the store could only say "a hunk landed inside this function". A diff line cannot be read for declarations there: inside a body `foo(bar);` calls something, inside a class body the same bytes declare a constructor, and pangemma indents both the same way (of 248538 changed lines in its history, 14991 are indented `name(` lines, and the most frequent are calls). So the file is handed to universal-ctags and the tags of a commit's blob are diffed against its parent's: present only in the new blob is an add, only in the old one is a remove, and a name whose tag survives both is left to hunk context, which is the honest answer for "the change was inside this". The cost is bounded by a cache keyed by git object id (new table `blobs`): a blob's tags can never change, so each file version is parsed once per store. On the pangemma history (688 commits, 1163 distinct C/C++ blobs) the first full pass takes 26 s and touches go from 1404 context-only rows to 6888 add, 5125 remove, 1505 context; the second pass takes 11 s and runs no ctags at all. ctags is optional at run time. Without it the store behaves exactly as before and `sync` says so once. The Guix package bakes its absolute path into $AI_MEMORY_CTAGS through wrap-program, and the check phase now builds a two-commit git repository with one C++ file to prove it end to end: the function added in the second commit is `add`, the one whose body changed is `context`. Details found by running it on a real history, all in this commit: - lambdas are tagged by default under names ctags derives from a hash of the file (`__anon426fc91f0102`), so every touched file reported seven functions added and seven removed per commit; `--extras=-{anonymous}` turns them off. - `--kinds-C++=+p` is needed or prototypes are invisible and a declaration in a header would never be seen. - include guards (`#define CALC_H`, `#define __CALC_H__`) are filtered: they are bookkeeping, while other macros are kept as `const`. - merge_decl keyed a commit's declarations by bare identifier, so two files declaring the same name in one commit collided and one lost its attribution. It is keyed by (file, name) now -- a bug that applied to Odin and Rust too.
40 hoursdb: never lower the schema stamp, and refuse a store a newer binary wrotePjotr
Two failures found by using the store from more than one binary at once. A stale ai-memory opened a version 5 store and left it stamped version 3. SQLite cannot drop a column, so the data survived (gist_source was still there), but the stamp lied -- and the next newer binary then re-ran data steps it believed were pending. The migration chain ends at the version the *binary* knows and wrote that number unconditionally, which is what did it. So: schema_version() reads the version out of the SCHEMA the binary embeds, so the number lives in schema.sql alone and the two cannot disagree; the stamp only ever moves forward (`if version > started_at`); and a store whose version is greater than this binary's is refused before anything is touched: ai-memory: this store is schema v9; this binary knows v5 ai-memory: it was written by a newer ai-memory -- upgrade this binary (scripts/install.sh). Nothing was changed. The second failure was the migration order: a store at version 3 that a newer binary had already written gists into died on every command with "no such column: gist_source", because the version 3 -> 4 backfill referenced a column the version 4 -> 5 step adds. The gist columns are now ensured (guarded by the table existing, so a fresh store still gets them from SCHEMA) before any backfill runs, and the skew hint fires on any "no such column" or "no such table", not only on an empty error message. Verified: a v9 store is refused with the stamp untouched; a store the stale binary knocked back to v3 upgrades to v5 with its assessed gists intact; the gist immutability test still passes; and all six stores on this box are now at v5 after the new binary opened them. schema_version() needed care in Odin: SCHEMA is a compile-time constant, so it is copied to a string before being walked by index.
2 daysgists: freeze what was assessed, recompute only what was derivedPjotr
A gist is a judgement about what an entry claims, so it must not drift once made. The store did not hold that invariant: db_backfill_gist ran update reasoning set gist = ?1 where id = ?2 for every row, so any migration that re-derived gists would have replaced every assessed --gist with the body's first sentence. It only escaped notice because the backfill happens to run once, on the version 3 -> 4 upgrade. reasoning.gist_source now says how each gist came about -- 'assessed' when the caller said what the entry claims (--gist), 'derived' when it is the body's first sentence -- and the backfill is guarded: where gist is null or gist_source = 'derived' so a derived gist may be recomputed when the derivation improves, and an assessed one is frozen for the life of the entry. Version 5 of the schema adds the column; a store that predates it cannot say which of its gists were assessed, and the two mistakes are not equal -- labelling an assessed gist derived lets a later migration overwrite a judgement, while labelling a derived one assessed merely freezes a first sentence -- so existing gists become 'assessed' and only rows with no gist are left to the backfill. test/gist_immutable.sh pins it: record one assessed and one derived gist, set user_version back to 3 so the next open re-runs the migration, and assert both are byte-identical afterwards. Against the old backfill the assessed one changes and the test fails. Measured on the production store (139 entries, version 4): the migration to 5 kept every gist and labelled all 139 assessed, no gist was blanked, and listings (log, why, search) are unchanged. Two new entries came out as expected: ('Derived one', derived) and ('Assessed one', assessed).
2 daysrecord/attach: resolve the commit ref before SQL, and write nothing on failurePjotr
`ai-memory record --commit 1ae2f5d` (a short hash) answered imported 1ae2f5d (qtl2: permutations for the LOD thresholds) ai-memory: sql error (Error): SQL logic error and left the store half-updated: the commit row was imported under its full hash, the reasoning row was not written at all, and the message named neither the cause nor the fix. Feed the same command the full forty characters and it worked, which is the sort of difference that reads as flakiness. Three changes, in the order they matter: * resolve the ref first. record's --commit and attach's positional argument both go through resolve_commit_ref, which asks git (`rev-parse --verify`) and keeps the full hash; if git does not know the ref the command refuses before opening a transaction, says which ref and which repository, and suggests --commit none plus a later attach. The store keys commits by the full hash, so a short ref written to reasoning.commit_hash names a row that cannot exist. * one transaction around the import and the insert (begin immediate ... commit, rollback on any failure), so a failed record leaves a store that looks exactly as it did. Verified against a store whose schema predates the binary: no new commit row survives the failure. * an actionable message: an empty sqlite3 errmsg on SQLITE_ERROR now says the store's schema is older than the binary and to run init, instead of "sql error (Error):" with nothing after it. Measured, fresh store: --commit 1ae2f5d records against 1ae2f5d712e18c016fdb7bd04da99eab4da6915b; --commit deadbeef refuses and writes nothing; --commit HEAD~1 resolves; attach 1ae2f5d links an unattached entry.
4 daysmigrate by inspecting the schema, not by trusting user_versionPjotr Prins
A store can carry a newer column under an older version number: the genenetwork4 store reported user_version = 3 while already having reasoning.gist, so the v3->v4 ALTER failed with "duplicate column name: gist" and every command on that store died in open_db -- search included. Each step now adds its column only if it is missing, inspects pragma table_info, and writes the version at the end. Backfills still run only when a column was actually added.
5 daysgists: a one-line claim per entry, shown by defaultPjotr Prins
`list`, `why` and `search` were truncating bodies at 96 characters, which keeps whatever happens to come first -- usually the least informative clause of the entry ("Only the range or every chromosome?..."). reasoning.gist (schema v4) holds the claim on one line, written at record time via --gist. Listings show it; --full adds the body beneath. Existing entries are backfilled from their first sentence, which is a floor, not an equivalent: it says what the entry is about. Recording without --gist still works and says so on stderr.
6 daysinitial ai-memory: derive commits and symbols from git, author the whyPjotr Prins
commits, symbols and touches are derived from `git log`/`git show` by `sync`; only reasoning is authored, and it is append-only. Each entry records a basis (how it was established) and may supersede an earlier entry instead of rewriting it. Symbols come from two signals: declarations added or removed on diff lines (Odin and Rust), and the enclosing item named in git's own hunk headers, which needs no parser and works for any language with a userdiff driver. Links SQLite through the odin-sqlite3 bindings; guix.scm builds both the compiler and the tool. doc/SKILL.md is the operating guide.