path: root/docs/design/ai-kb.org

#+TITLE: Design: AI Knowledge Base (ai-kb)
#+AUTHOR: Craig Jennings
#+DATE: 2026-05-24
#+OPTIONS: toc:nil num:nil

* Status

Ready. Six reviews incorporated (=ai-kb-review.org= through =-review6.org=; all 2026-05-24). Review 6's UX/performance pass is folded in: one safety model for human Emacs edits (after-save runs index+lint+commit), a required =:SUMMARY:=, the generated index made invisible to backlink/orphan logic, first-class browsing commands (dashboard/find/search/show/backlinks/map), a full org-roam *profile* on switch (not just dir/db), conditional sync, lexical query ranking, a =raw/= size/type policy, and performance budgets. The four original blockers (version control + recovery, switch-state safety, startup surface, project-awareness) and review 4's two write-loop caveats (push-failure contract, index regeneration) have decisions. Review 3's operational shape (a repo-resident agent-neutral contract, a minimal CLI, maintenance commands, multi-agent provenance) is adopted. Review 5's implementation-hardening is folded in: the commit gate runs the *full* =ai-kb lint= (not just node org-lint), an explicit org-lint fatal-check list, observable push failures, a testable =ai-kb query= contract, a Step 1a/1b split, and ID-first durable pointers. Cross-agent is *not a near-term goal* (Craig, 2026-05-24): v1 ships the Claude adapter over the neutral contract, and other-agent adapters (Codex/Ollama, MCP) are deferred to [[*vNext][vNext]]. The architecture is decided and Step 1 is buildable; the build-time implementation choices (limits, perf budgets, scoring weights, map, after-save UX + recursion guard) are settled with calibratable defaults in [[*Open decisions][Open decisions]].

In scope: Step 1 (store + contract/CLI + global rule + provisioning) and Step 2 (Emacs browsing layer). Step 3 (migrating =.ai/sessions= and workflows in) and the full LLM-Wiki layer are *deferred to their own specs* — see [[*vNext][vNext]].

* Scope decision: memory store, not (yet) an LLM Wiki

ai-kb v1 is a *global, durable, cross-project memory store* for AI coding agents (Claude Code today; agent-neutral by contract): org-roam nodes holding lessons, principles, Craig's preferences, reusable procedures, and durable observations. It is the concrete first slice of the broader "org-roam as agent memory" vision in [[file:agentic-knowledgebase.org][agentic-knowledgebase.org]].

It is *not* a Karpathy-style LLM Wiki in v1. That pattern — immutable =raw/= sources, compiled =wiki/= synthesis pages, =schema.org=, source hashes, and full ingest/query/lint pipelines — is a larger product whose value is *grounding compiled knowledge in re-checkable sources*. v1 adopts the one piece that pays off immediately: a =raw/= capture for *external* sources (see [[*Grounding external sources][Grounding external sources]]). The rest of that machinery is the documented evolution path (see [[*vNext][vNext]]); v1's structure is chosen so it can grow that way without a rewrite.

* Problem

AI coding agents start every session cold. Continuity today is per-project and flat. There is no home for durable, *general* knowledge that should follow the agent into every repo — engineering lessons, Craig's cross-project preferences, reusable procedures (e.g. "move a local repo to git.cjennings.net with a mirror-to-GitHub hook") — and no link structure relating one piece of knowledge to another. A flat shared lessons file would solve "knowledge that follows the agent" alone; org-roam is chosen over it for *link structure*, *first-class browsing* (node-find, backlink buffer, graph), and a substrate that grows toward the agentic-KB vision. The complexity is earned by those three.

* Memory tiers

Naming the tiers (after Nexus's vocabulary) so every agent routes consistently:

- *T1 — session scratch:* the current chat, spawned-agent handoffs. Ephemeral.
- *T2 — project memory:* per-project =~/.claude/projects/<encoded-cwd>/memory/=, =.ai/notes.org=, active project decisions. Minor or project-specific.
- *T3 — ai-kb:* global, durable, cross-project. Significant and general: lessons, principles, preferences, reusable procedures, durable observations.

T2's =MEMORY.md= shrinks toward an index: for significant items it points at the T3 (ai-kb) node rather than holding the content. ai-kb is *not* a dump for T1/T2 breadcrumbs — the proactive-write bar (below) keeps it T3-only.

* Concept: two layers

- *Store* — a git repository of org files (each a valid org-roam node). The agent reads/writes these directly and never touches the SQLite database; the files are the source of truth.
- *Emacs/org-roam integration* — so Craig can browse with backlinks and the graph. org-roam keys off one global =org-roam-directory= + =org-roam-db-location= per session, so ai-kb cannot be live alongside the personal roam; the integration is a *switch* that installs a full org-roam profile (see [[*The Emacs switch: a full org-roam profile][the switch profile]]).

* Storage, version control, and recovery

ai-kb is its *own git repository* — not in =~/sync/org= (Syncthing has proven unreliable for backup/restore: no history, silent =.sync-conflict= files) and *not* in =~/.emacs.d= (publicly mirrored to GitHub; ai-kb holds personal/work-private knowledge that would leak).

- *Location:* =~/.local/share/ai-kb= (XDG =$XDG_DATA_HOME/ai-kb=).
- *Origin:* a bare repo on =git.cjennings.net= (=git@cjennings.net:ai-kb.git=), *private — no public GitHub mirror*. This is the recovery layer: full history, clone-to-restore.
- *No Syncthing.* git is the sole sync and backup; multi-machine concurrency surfaces as ordinary git merges, not silent conflict files.
- *org-roam scope:* =org-roam-directory= points at the repo root; =raw/= *and the generated index files* (=index*.org=) are *excluded* from the scan (=org-roam-file-exclude-regexp=) so neither raw captures nor the index become noisy roam nodes. The LLM-Wiki vNext would add a compiled =wiki/= layer; v1 keeps compiled nodes flat at root.
- *Generated files are invisible to semantics.* =index*.org= and =raw/= are excluded from the org-roam scan, the graph/map, and curation's backlink/orphan calculations. The index references nodes as *plain text* (=Title (UUID)=), never =[[id:...]]= links — otherwise every node would gain an artificial backlink from the index and orphan detection would be meaningless. The index is a navigation artifact, not a semantic backlink source.

* Write protocol and synchronization

The agent writes nodes from the shell, possibly from several machines or concurrent processes, so the write path is a defined protocol, not a bare =git push=. Encapsulated in the =ai-kb remember= operation (see [[*The agent contract and operations][operations]]):

1. *Before write:* =git fetch=; if behind and clean, =git pull --ff-only=; if diverged or the tree is dirty with unrelated changes, *abort and surface* — don't auto-merge.
2. *Write/edit the node.*
3. *Regenerate the index* from node properties (see [[*Startup surface and retrieval contract][Startup surface]]).
4. *Validate the whole change, not just the node — this is the safety boundary.* Run the full =ai-kb lint= over the change set before committing: =org-lint= fatal checks on *both* the edited node and the regenerated index, index freshness/completeness, duplicate =:ID:=, broken =[[id:...]]= links (excl =raw/=), missing required properties, invalid project slugs, and a credential/secret scan of nodes *and* =raw/=. Any failure aborts the commit. (Gating only on single-node org-lint would let a stale index or a leaked secret through — then the write protocol isn't the boundary the spec claims.)
5. *Commit locally — always.* The local commit is the durable record.
6. *Push — best-effort, non-blocking, never fatal, and observable.* =remember= commits locally and does *not* push; a background =systemd --user= timer (~15 min) pushes when the repo is ahead. A failed push (offline, network blip, gpg-agent SSH key not loaded — observed this session) is *logged to a state file* (under the repo or =$XDG_STATE_HOME/ai-kb=) and ignored, never erroring or hanging the agent. Visibility comes from three surfaces so the KB can't go quietly local-only: the log, =ai-kb doctor= (reports "ahead N", "push failed", or "remote diverged"), and a one-line startup/adapter nudge when there are unpushed commits or a recorded rejection.
7. *On push rejection* (remote moved): do *not* blind-retry. Fetch, record the divergence, leave the local commit intact for resolution.
8. *Same-machine concurrency:* =flock= around =remember= serializes concurrent agents (Claude + Codex + an Emacs save) so they don't race. A v1 file lock; not a daemon.

* Why a separate database

org-roam supports one active =org-roam-directory= / =org-roam-db-location= at a time. ai-kb gets its own directory (the repo above) and its own database (=~/.emacs.d/org-roam-ai.db= — a regenerable cache). The personal roam (=~/sync/org/roam/= + =~/.emacs.d/org-roam.db=, recipes etc.) is never scanned or modified.

* The sync model

The =.org= files are truth; the SQLite db is a cache indexing nodes and =[[id:...]]= links that powers Emacs's backlink buffer, node-find, and graph. Editing in Emacs updates the cache on save (=org-roam-db-autosync-mode=); agent shell writes don't, so =org-roam-db-sync= re-scans. The key consequence: *the agent never needs the db to check links* — they live in the files and are grepped (always current). *Craig's Emacs browsing* needs the cache current, so the switch-to-ai-kb command syncs on entry; the agent may also fire =emacsclient -e '(cj/ai-kb-db-sync)'= for immediacy, but agent correctness never depends on Emacs running.

* Proactive-write rule

The agent writes a node *unprompted* when something is =durable= (true beyond this session) *and* =general= (T3, not tied to the current repo; project-specific knowledge goes to T2). The bar, to keep out noise: genuinely worth recalling or linking later — a principle, a reusable procedure, a preference, a non-obvious lesson — not routine status or anything re-derivable from code or git. New nodes link to related existing ones (grep candidates by title/tag first) and trigger an index regeneration.

*Contradiction guard:* if a write would contradict an existing node that affects agent behavior or a stated preference, the agent does *not* silently overwrite. It marks both =:STATUS: contested=, records the conflict, and asks Craig before changing the canonical node.

* Node format and conventions

#+begin_src org
:PROPERTIES:
:ID:          <uuid, generated with `uuidgen`>
:PROJECTS:    :general:           ; or :deepsat: :emacs: ... (see slug rule below)
:CREATED:     2026-05-24
:UPDATED:     2026-05-24
:CREATED_BY:  claude-code         ; claude-code | codex | ollama | human
:CONFIDENCE:  user-stated         ; user-stated | observed | inferred | external
:VISIBILITY:  personal            ; personal | work-private
:SOURCE:      chat 2026-05-24     ; free-form, or a raw/ path for external sources
:STATUS:      current             ; current | contested | superseded
:SUMMARY:     One sentence, written for retrieval and index display.
:END:
#+title: Concise node title
#+filetags: :principle:emacs:

Body. Link related nodes with [[id:OTHER-UUID][Their title]], optionally prefixed
with a relation label: SUPERSEDES, CONTRADICTS, RELATES_TO, IMPLEMENTS, DERIVED_FROM.
#+end_src

- *Filename:* org-roam convention — =YYYYMMDDHHMMSS-slug.org= (or =slug.org= for stable, frequently-linked nodes). Filenames are for humans and can change during curation, so they are *not* the durable identity.
- *ID:* a real UUID (=uuidgen=) — org-roam won't index a node without one, and the =:ID:= is the *durable identity*. External pointers (a per-project =MEMORY.md= → an ai-kb node) are *ID-first*: =ai-kb: <Title> (<UUID>)=, resolved by ID with title as fallback, so a rename or curation merge doesn't dangle the pointer.
- *Type tags* (=#+filetags:=): =:principle:= =:preference:= =:procedure:= =:observation:= =:reference:=.
- *Project slugs* (=:PROJECTS:=): derived from the project directory basename (so =~/.emacs.d= → =:emacs:=, the DeepSat repo → =:deepsat:=), with =:general:= for cross-cutting nodes. The derivation rule lives in the contract so every agent produces the same slug; new slugs are recorded in the index's project list.
- *Provenance:* =:CREATED_BY:= and =:CONFIDENCE:= let later curation and trust policy distinguish "Craig stated this" from "a model inferred it." =:CONFIDENCE:= here is *provenance* (how the claim was obtained), not a numeric grounding score — the latter is vNext. =:VISIBILITY:= is two-valued in v1 (the full =public|work-private|secret= taxonomy is vNext); secrets are never stored at all (see [[*Security and privacy][Security]]).
- *Summary:* a *required* one-line =:SUMMARY:= property, written for retrieval. =ai-kb index= and =ai-kb query= read it straight from the property, so the index rebuilds fast and locally — no inferring from the first paragraph (inconsistent) and no LLM call (slow, nonlocal).
- *Relation labels:* a small fixed vocabulary used in link context now; full typed-link catalog storage is vNext.

* Grounding external sources

The one LLM-Wiki piece adopted in v1: keep compiled knowledge re-checkable where an external source exists.

- *Node authored from an external source* (web article, fetched doc, transcript, API result): capture under =raw/= and point =:SOURCE:= at that path. *By default store the URL, retrieval date, and the relevant excerpt* — store full external text only when it is user-owned, licensed for the use, or operationally necessary (this is a private KB, but copyright still applies). A later agent can re-ground a suspicious node against the source instead of trusting its own prior summary.
- *Node authored from the conversation or direct observation*: only the free-form =:SOURCE:= pointer; no raw capture (the source is not an external artifact).
- =raw/= is append-only in spirit and excluded from org-roam's scan.
- *Size/type policy:* a small org stub (URL + retrieval date + bounded excerpt) is the default capture, under a maximum excerpt size; a larger full source file goes under =raw/files/= only when explicitly requested. The credential scan runs over *text* only — binary files are skipped (by type or byte-sniff). =ai-kb doctor= reports the raw-directory size and =curate --dry-run= flags unusually large raw files and raw captures with no compiled node, so bloat stays visible.

* Startup surface and retrieval contract

Passive grep-on-demand gets under-used; loading the whole KB wastes context. Two tiers:

- *L1 — always loaded:* the global rule adapter (=claude-rules/ai-kb.md= for Claude), tiny. It carries the path, routing rule, link-grep recipes, and: *when a task may involve durable knowledge, read the index first.* Include concrete example triggers so it actually fires — e.g. "before choosing a formatter/test/lint convention," "before a multi-step procedure you've likely done before (repo setup, release, deploy)," "when Craig states a preference," "when you hit a non-obvious gotcha worth keeping."
- *L2 — on demand:* =index.org= at the repo root, read at session start only when L1's condition applies.
- *Full nodes* read only when the index points at them or Craig asks.

=index.org= is *generated output*, never hand-maintained — that is what keeps it from drifting from the nodes. A regeneration script greps node properties (=#+title:=, =:ID:=, type tag, =:PROJECTS:=, =:UPDATED:=, =:STATUS:=) and rebuilds the file with a "generated, do not edit" marker. It runs in provisioning, in the curation pass, on demand, and as step 3 of every =remember=. =lint --index= checks: every listed id resolves, every =current= node is listed, contested/superseded sections are accurate, the size budget holds (split into =index-procedures.org= etc. when exceeded).

#+begin_example
,* Procedures
| Title | ID | Summary | Projects | Updated |
,* Preferences
| Title | ID | Summary | Projects | Status |
,* Contested / needs review
| Title | Issue | Last touched |
#+end_example

* Checking links (agent recipes)

No database needed; grep the files (excluding =raw/=):

- *Backlinks to a node* — =rg -l "id:<UUID>" ~/.local/share/ai-kb --glob '*.org' --glob '!raw/**'=.
- *Forward links from a node* — grep that node's file for =id:= links.
- *Find a node to link to* — grep titles/tags.

* Node validity (org-lint)

The agent writes raw org from the shell, bypassing Emacs's structural editing, so malformed org (broken drawer, bad property, broken timestamp) can slip in and make =org-roam-db-sync= choke or mis-index. Distinct from the semantic link/credential checks; both run.

- *On write:* =org-lint= via =emacs --batch=, gating on a concrete *fatal-check* list (not the vague "error-level," so a future org-lint behavior change can't silently weaken or over-tighten the gate, and tests target the list directly):
  - malformed property drawer
  - missing or duplicate =:ID:=
  - an invalid required-property line
  - missing =#+title:=
  - structurally invalid org that prevents parse/index
  Style warnings stay non-fatal. A node hitting a fatal check is not committed.
- *In curation:* an =org-lint= sweep over all nodes catches drift or bad Emacs-side hand-edits.

Cheap (sub-second batch on one small file); the safety net that makes "the agent writes raw org" trustworthy. Reuses/extends the project's =scripts/lint-org.el=. This node-level gate is run as part of the full =ai-kb lint= the write protocol invokes before commit (step 4 above) — it never stands alone as the only check.

* The agent contract and operations

The access layer is an *agent-neutral contract*, not a Claude-only prompt snippet. Cross-agent use is not a near-term goal (deferred to vNext), but making the contract repo-resident and neutral *in shape* costs nothing now, future-proofs that path, and — more importantly for v1 — the CLI earns its place on Claude-only grounds: it is the clean, atomic, testable home for the write protocol, index regeneration, and lint, far better than scattering them across prose rules.

- *Canonical contract:* lives *in the repo* (=~/.local/share/ai-kb/AGENT_CONTRACT.org=) — the source of truth for the node format, routing rule, write protocol, and operations. It travels with the store.
- *Adapters* point at it: =claude-rules/ai-kb.md= (symlinked into =~/.claude/rules/= by rulesets =make install=) is the Claude adapter. Other agents get their own thin adapter when wanted (deferred — see [[*Open decisions][Open decisions]]).
- *Operations* — a small =ai-kb= CLI (shell, calling Emacs for org-lint/index work) is the canonical surface, so humans and every agent share one contract. For performance: prefer =emacsclient= when a daemon is up (=emacs --batch= fallback), and run lint + index in a *single* Emacs invocation per =remember= rather than one startup per check. The full-lint gate stays on for v1; if timing crosses the perf budget at scale, split lint into cheap always-on checks (the edited node + index) and a slower full-sweep, but don't pre-optimize.
  - =ai-kb doctor= — repo present, remote reachable + private, branch state, org-roam db buildable, required tools installed (incl. =graphviz= if the map needs it), adapter linked, no obvious secrets, raw-directory size.
  - =ai-kb status= — fast, non-diagnostic state for the dashboard/nudge: branch ahead, last push failure, node count, last index time, curation-due. (May be a =doctor --status= mode.)
  - =ai-kb show <id-or-title>= — resolve an ID-first pointer and print the node (path + body); the testable primitive the Emacs =show-node= wraps.
  - =ai-kb backlinks <id>= — list nodes linking to =<id>=, excluding =raw/= and the generated index.
  - =ai-kb index= — regenerate =index.org= from node properties.
  - =ai-kb query <context>= — read the index, return relevant nodes. It is the surface adapters call before spending context on full nodes, so it has a *testable contract* even though v1 retrieval is plain lexical: default output is plain text (one node per line), with =--json= for tests and tools; fields are title, ID, summary, projects, status, updated, path, and the *match reason* (matched-title / tag / summary / body); it searches index rows + title/tags/properties/body; ranking is a simple lexical score (title > tag/project/status > summary > body) with most-recently-updated as the *tie-breaker* — recency alone would bury old stable preferences and procedures, which are exactly what the store exists to preserve; a default max-result count; =raw/= paths appear only as source references, never as primary results; exit codes distinguish no-match, invalid/missing KB, and a lint/index failure.
  - =ai-kb remember= — the write protocol above (fetch/ff, write, regenerate index, full lint gate, commit; push is the timer's job; under =flock=).
  - =ai-kb lint= — org-lint fatal checks, duplicate ids, broken id-links (excl =raw/=), missing required properties, bad project slugs, stale/incomplete index, credential scan of nodes and =raw/=. This is what =remember= runs before commit and what curation runs as a sweep.
  - =ai-kb curate --dry-run= — report duplicates, orphans, contested/superseded nodes, raw captures with no compiled node, nodes untouched past a horizon.
  - =ai-kb sync= — =org-roam-db-sync= against ai-kb (Emacs-side helper).
- *Admin split:* destructive operations — merge nodes, delete a node or raw capture, rewrite backlinks, mark superseded — are *human-confirmed only*, never automatic.
- *Capability levels* (named so adapters know their lane): =file-only= (read/grep/template-write), =cli= (call =ai-kb=), =mcp= and =semantic= are vNext. Claude v1 uses =cli= with the rule adapter; until the CLI exists, =file-only= following the contract template is the bootstrap path.

* The Emacs switch: a full org-roam profile

Switching is *not* a two-variable rebind. The personal org-roam surface has many globals and hooks that would misroute into ai-kb: =org-roam-directory=, =org-roam-db-location=, =org-roam-dailies-directory= (personal journals), capture templates, tag/topic/recipe find wrappers that reference personal template paths, the agenda/refile finalize hook (=cj/org-roam-add-node-to-agenda-files-finalize-hook=) that can add captured nodes to personal agenda files, and the completed-task→daily hook (=cj/org-roam-copy-todo-to-today=). The switch therefore installs an *ai-kb profile*, restored exactly on exit:

- directory + db location + the file-exclude regexp (=raw/= + =index*.org=)
- dailies disabled (or pointed nowhere personal)
- ai-kb-only capture templates
- topic/project/recipe find wrappers disabled or rebound to the ai-kb profile
- agenda/refile finalize hook + completed-task→daily hook neutralized so nothing from ai-kb lands in personal agenda files or journals
- *Abnormal exit:* if Emacs is killed while switched, the config re-asserts the personal profile at startup, so a crash can't leave personal hooks rescoped into ai-kb.

Tests assert *profile-level* behavior — not just dir/db restore, but that the completed-task and agenda hooks don't fire into ai-kb while switched, and that personal templates/dailies are untouched.

* Human edits must use the same safety model

One safety boundary for *both* agent and human writes. =ai-kb remember= linting + indexing + committing must not be bypassed when Craig edits a node in Emacs and saves. The v1 mechanism: an =ai-kb= minor mode on buffers under the store with an =after-save-hook= that runs the same post-save sequence under =flock= — regenerate index, full =ai-kb lint=, commit, update push state. A lint failure on save surfaces the problem rather than silently committing a broken node. (Read-only-with-an-edit-command is the fallback if the after-save approach proves fiddly; either way there is exactly one write path.)

* Emacs browsing surface

The spec promises first-class browsing, so Step 2 names the commands rather than leaving Craig to remember low-level org-roam + git details. All operate within the ai-kb profile and exclude =raw/= + generated index:

- =cj/ai-kb-dashboard= — a status buffer (or =index.org= with a banner): active KB, node count, unpushed commits, push-failure state, curation-due, last index time, last sync time. Wraps =ai-kb status=/=doctor=.
- =cj/ai-kb-find-node= — =org-roam-node-find= in the profile.
- =cj/ai-kb-search= — =ai-kb query= or =consult-ripgrep= scoped to the store.
- =cj/ai-kb-show-node= — resolve an ID-first pointer (=ai-kb: Title (UUID)=) and open the node.
- =cj/ai-kb-backlinks= — backlinks excluding =raw/= and the generated index.
- =cj/ai-kb-map= — a graph/map via built-in =org-roam-graph= or a small DOT export from =[[id:...]]= links, excluding =raw/= + index, filterable by project/tag/status. =graphviz= is checked by =ai-kb doctor= if this command needs it. Richer interactive graph (=org-roam-ui=) is vNext.

* Sync only when stale

=org-roam-db-sync= on every switch becomes a visible pause as the store grows, and agent correctness never depends on the db. So =ai-kb sync= (and the switch's entry sync) runs *only when needed* — db missing, or db older than the newest node/index — or when forced with a prefix arg, showing a "syncing…/done" status. Consider running it asynchronously from the dashboard/switch with a pending/running/done indicator.

* Maintenance and curation

The proactive-write bar controls intake; nothing controls rot, and the system creates memories unprompted, so a minimal maintenance loop is v1 (read-only commands; destructive execution human-gated):

- =ai-kb doctor= / =ai-kb lint= — health and validity (above).
- =ai-kb curate --dry-run= surfaces four buckets — duplicates to merge, stale/superseded nodes, orphans (no back- or forward-links), over-broad nodes to split. Craig decides; the agent executes *human-confirmed* merges/splits, repointing =[[id:]]= backlinks (grep + rewrite) and re-linting. A =:LAST_CURATED:= stamp rotates the pass through least-recently-touched nodes.
- *Trigger (node-count):* curation is "due" when roughly N nodes have been added or gone untouched since the last =:LAST_CURATED:=; =ai-kb doctor= and the session-startup surface emit a one-line nudge when due (mirroring the existing task-review habit). The interactive pass itself is a global workflow at =~/code/rulesets/.ai/workflows/ai-kb-curate.org=, available from every project's session.
- *Pointer integrity:* external pointers are ID-first, so a rename is safe; but a merge or supersede *changes the canonical ID*, so before merging/deleting, grep for inbound references (other nodes' =[[id:]]= and per-project =MEMORY.md= =ai-kb: ... (UUID)= pointers) and repoint the old ID to the new.

* Security and privacy

ai-kb lives in a *private* repo (cjennings.net only, no public mirror), removing the main leak surface. v1 rule: *private but not a secret store* — no credentials/tokens/keys in nodes or =raw/=; =ai-kb lint= scans both for common credential patterns before commit and *fails* on a hit (secrets move to a secure reference, not ai-kb). =:VISIBILITY:= is two-valued (=personal= / =work-private=) in v1; the full =public|work-private|secret= taxonomy and a public/private split are vNext, for when sharing or publishing a subset is real.

* Provisioning

The pieces span the rulesets repo, this repo, and the ai-kb repo. =make ai-kb-init= (wrapping =scripts/setup-ai-kb.sh=) is idempotent. *One-time server bootstrap* (distinct from the per-machine clone, and not doable by the local script): =sudo git init --bare /var/git/ai-kb.git && chown= on cjennings.net, plus the github-mirror hook left *off* for this repo.

Per-machine, ordered:
1. Clone =git@cjennings.net:ai-kb.git= to =~/.local/share/ai-kb= (or =git init= + add remote on the very first machine).
2. =make ai-kb-init=: seed =index.org= + a README/index node with a generated =:ID:=; install the =ai-kb= CLI onto =PATH=; =ai-kb index=; install and =systemctl --user enable --now= the =ai-kb-push.timer= + =.service= units (the debounced background push); best-effort =ai-kb sync= if an Emacs server is up.
3. =cd ~/code/rulesets && make install= — symlinks the =claude-rules/ai-kb.md= adapter into =~/.claude/rules/= (and syncs the =ai-kb-curate.org= workflow into projects via the usual startup rsync).
4. =ai-kb doctor= to confirm the machine is wired correctly (repo, remote, CLI on PATH, push timer active, adapter linked, db buildable, no secrets).

* Build plan

Step 1 splits into two slices by dependency — =remember= needs =index= + =lint=, and the adapter needs =remember=, so the safe write path (1a) lands first and the read/maintenance/timer pieces (1b) follow. Same scope, cleaner sequencing.

*** Step 1a — the safe write path (minimum usable)

- The =ai-kb= git repo (bare on cjennings.net + clone at the XDG path), seed =index.org=, =AGENT_CONTRACT.org=.
- =ai-kb index= (regenerate from properties incl. =:SUMMARY:=), =ai-kb lint= (full check set: org-lint fatal gate + required-property check incl. =:SUMMARY:= + credential scan), =ai-kb remember= (write protocol: fetch/ff, write, regen index, full-lint gate, commit, =flock=; lint+index in one Emacs invocation), =ai-kb doctor= / =ai-kb status= (health + push-state + raw-size report).
- =claude-rules/ai-kb.md= adapter (points at the contract; routing + proactive + contradiction rules + concrete L1 triggers + "use =ai-kb remember=, never bypass =ai-kb lint="); =make install= links it.
- =scripts/setup-ai-kb.sh= + =make ai-kb-init=; the one-time server bootstrap documented.

After 1a the agent can remember, lint, and check health — the safe write path exists.

*** Step 1b — retrieval, maintenance, push

- =ai-kb query= (the testable retrieval contract: lexical score + recency tie-break + match reason) plus the =ai-kb show= / =ai-kb backlinks= inspection helpers.
- =ai-kb curate --dry-run= (incl. large/orphan =raw/= reporting) and =ai-kb sync= (only-when-stale).
- =ai-kb-push.timer= + =ai-kb-push.service= =systemd --user= units (debounced background push) installed and enabled by =setup-ai-kb.sh=, plus the push-failure log + =doctor=/startup surfacing.
- =~/code/rulesets/.ai/workflows/ai-kb-curate.org= — the human-gated curation workflow, surfaced when the node-count trigger makes it due.

** Step 2 — Emacs browsing layer

In =org-roam-config.el=:
- The *ai-kb org-roam profile* (=cj/org-roam-switch-to-ai-kb= / =…-to-personal=): dir + db + exclude regexp (=raw/= + =index*.org=), dailies/templates/find-wrappers/agenda+completed-task hooks all rescoped or neutralized, restored exactly on exit, re-asserted at startup after an abnormal exit.
- *Edit safety:* an =ai-kb= minor mode whose =after-save-hook= runs index + full lint + commit + push-state under =flock=, so human edits use the one safety model.
- *Conditional sync* =cj/ai-kb-db-sync=: only when the db is missing/stale or forced, with a status indicator.
- *Browsing surface:* =cj/ai-kb-dashboard=, =-find-node=, =-search=, =-show-node=, =-backlinks=, =-map= (built-in =org-roam-graph= or DOT export, excl =raw/=+index).
- =C-c n= keybindings (e.g. =C-c n a= switch / =C-c n A= back / a small transient for the browsing commands), which-key labels; profile-level + edit-path ERT tests + =/review-code=.

** Step 3 and the LLM-Wiki layer — deferred

Separate specs. See [[*vNext][vNext]].

* Test strategy

- *CLI / write path:* a write with the remote unreachable still commits locally and does *not* error the agent (push deferred); =flock= serializes concurrent =remember=; each fatal org-lint check (malformed drawer, missing/dup =:ID:=, invalid required property, missing =#+title:=, unparseable org) rejects the commit while a style warning does not; and — the safety boundary — =remember= aborts the commit when the full =ai-kb lint= fails (stale index, broken link, leaked secret in =raw/=), not only on node org-lint.
- *Index:* regeneration from a fixture KB produces the expected entries; a node added out-of-band appears only after regeneration (proves no drift); =lint --index= flags a missing/stale entry.
- *Lint gates:* a node missing =:SUMMARY:= (or any required property) fails =ai-kb lint=; the credential scan rejects a secret in a node or =raw/= text file and skips binaries.
- *query contract:* =ai-kb query --json= returns the specified fields (incl. match reason), exit codes, and =raw/= only as source refs on a fixture KB; a title match outranks a body-only match, with recency only breaking ties (an old preference is not buried under a newer body-only hit).
- *Index is not a backlink source:* a node referenced only by =index.org= still reports as an orphan in =curate=; the index contains no =[[id:...]]= links.
- *Push observability:* a simulated push failure is recorded to the state file and surfaced by =ai-kb doctor= / =ai-kb status= ("ahead"/"push failed").
- *Link recipes* (fixture KB): backlink-by-grep (excluding =raw/= + index) and forward-link-by-grep return correct sets.
- *Step 2 profile:* switch installs the ai-kb profile and switch-back restores personal *exactly* — completed-task hook, agenda/refile finalize hook, dailies, and capture templates all untouched by ai-kb while switched; a save in an ai-kb buffer runs the index+lint+commit sequence (and a bad save surfaces the lint failure rather than committing); startup re-asserts personal state after a simulated abnormal exit.
- *Performance* (=:perf= tag): fixture KBs at 100 and 1,000 nodes; assert =index=, =query=, =lint=, and =remember= stay under a stated time budget (catches an accidental per-check Emacs startup or an O(n^2) scan early).
- *Provisioning* (bats): =setup-ai-kb.sh= idempotent; seeds a node with a valid =:ID:= and =:SUMMARY:=; =doctor= passes on a freshly-provisioned repo.

* Scaling path (planned, not built)

- v1: =rg= over org files + a generated =index.org=.
- v1.5: =ai-kb query= grows richer ranking over title/tags/properties/body.
- vNext: a local BM25/vector tool (=qmd= or similar) over the nodes, preserving links; no embeddings in v1.

* Review dispositions

Everything not listed was accepted as written and woven in. Listed: modified, rejected, or owner-deferred recommendations, with reasons.

- *Review 2 core reframe → MODIFIED (scope).* v1 is the memory store, not a full LLM Wiki; per Review 2's own off-ramp and Craig's stated intent.
- *Review 2 #1 (raw/wiki/schema) → PARTIALLY ADOPTED.* =raw/= for external sources only; compiled =wiki/= + =schema.org= stay vNext (most memories have no external source).
- *Review 2 #2 (ingest/query/lint) → MODIFIED.* query = index + =rg= (now =ai-kb query=); lint = =ai-kb lint= + curation; org-lint is the write-time validity gate; the heavy ingest pipeline → vNext.
- *Review 2 #3 (provenance + hashes + numeric confidence) → MODIFIED to provenance-lite,* reconciled with Review 3: adopted =:CREATED_BY:/:CONFIDENCE:(provenance)/:VISIBILITY:/:SOURCE:/:STATUS:=; dropped source *hashes* and numeric confidence scoring (raw-corpus grounding machinery → vNext).
- *Review 2 #8 (exclude =raw/= from scan) → ADOPTED.*
- *Review 2 #10 (full visibility taxonomy + lint) → MODIFIED:* v1 = private repo + no-secrets rule + credential lint; four-level taxonomy → vNext.
- *Review 3 #1 (agent-neutral contract + CLI) → ADOPTED (contract + CLI); cross-agent ADAPTERS deferred to vNext (Craig, 2026-05-24).* The contract lives in the repo and a minimal CLI is the operation surface — justified on Claude-only correctness (atomic safe writes), with neutrality as cheap future-proofing. Codex/Ollama adapters + MCP wait until cross-agent is actually adopted.
- *Review 3 capability levels =mcp=/=semantic= → DEFERRED.* vNext.
- *Review 4 #1 (push-failure contract) → ADOPTED,* and strengthened to debounced best-effort push (commit always; push never blocks/fails the agent) — directly informed by the gpg-agent SSH failure observed this session.
- *Review 4 #2 (index regeneration) → ADOPTED:* generated by =ai-kb index=, never hand-maintained.
- *Storage location → Option 1 (emacs home) REJECTED* (public mirror leaks); *XDG dedicated private repo ADOPTED;* Syncthing dropped.
- *Curation full workflow → kept v1-minimal:* read-only =curate --dry-run= ships v1; the interactive merge/split flow is human-gated.
- *Review 5 (all six) → ACCEPTED.* #1 (the only blocker): =remember= runs the *full* =ai-kb lint= — index freshness, dup IDs, broken links, secret scan — before commit, not just node org-lint. #2: an explicit org-lint fatal-check list (tests target it). #3: push failures are observable (state-file log + =doctor= + startup nudge). #4: =ai-kb query= gets a testable contract (text/=--json=, fixed fields, ordering, exit codes). #5: Step 1 split into 1a (safe write path) / 1b (query/curate/sync/timer/workflow). #6: durable pointers are ID-first (=ai-kb: <Title> (<UUID>)=), not filename-first. Nothing rejected — all six were sound hardening.
- *Review 6 (all ten + enhancements) → ACCEPTED.* The UX/performance pass, all sound. #1 (the key gap): human Emacs edits use the *same* safety model as agent writes — an ai-kb minor mode whose after-save-hook runs index + full lint + commit under =flock=, so there's one write path, not two. #2: the generated =index.org= is invisible to backlink/orphan logic (excluded from the scan; its references are plain =Title (UUID)= text, not =id:= links). #3: a required =:SUMMARY:= property, so the index/query rebuild from properties without inferring or calling an LLM. #4: =ai-kb query= ranks lexically (title > tag/project/status > summary > body) with recency only as a tie-break, and returns a match reason. #5: performance budgets (100/1,000-node fixtures) + lint+index in one Emacs invocation + =emacsclient=-preferred-with-batch-fallback; the full-lint gate stays, with a cheap/full split held in reserve. #6: switch installs a full org-roam *profile* (dailies, templates, find wrappers, agenda/refile + completed-task hooks all rescoped), not a two-variable swap. #7/#8: a first-class browsing surface (=dashboard/find-node/search/show-node/backlinks/map=), map via built-in =org-roam-graph= or DOT export with =graphviz= in =doctor=. #9: a =raw/= size/type policy (bounded excerpt default, =raw/files/= for large, text-only secret scan, size reporting in =doctor=/=curate=). #10: sync only when stale. Enhancements: =ai-kb show=/=backlinks=/=status= CLI helpers and the generated-files-ignored rule, all folded in.

* Agreed decisions

- Building from the rulesets session is sanctioned cross-project work (Craig, 2026-05-24).
- ai-kb is intentionally global; the one sanctioned exception to =cross-project.md=.
- Scope: memory store v1; LLM Wiki deferred.
- Storage: dedicated private git repo, XDG path, no Syncthing.
- Write path: commit always, push best-effort/non-blocking/debounced; safe-fetch before; =flock=.
- Operations are an agent-neutral contract fronted by a minimal =ai-kb= CLI; destructive ops human-only.
- Cross-agent is not a near-term goal (Craig, 2026-05-24): v1 ships the Claude adapter; other-agent adapters + MCP are deferred to vNext. The contract stays neutral in shape so they are additive later.
- *Resolved 2026-05-24:* store path = =~/.local/share/ai-kb= (XDG); CLI = a shell wrapper calling =emacs --batch= for the org-lint/sync steps; push = a background =systemd --user= timer (~15 min, push only if ahead), commits always local; curation = node-count trigger surfaced by =ai-kb doctor= + startup, workflow at =~/code/rulesets/.ai/workflows/ai-kb-curate.org=.

* Open decisions

Architecture is decided. These implementation choices are now settled with build-time defaults (2026-05-24); the numeric ones in the first two are starting points to calibrate against the real repo and machine, not invariants.

- [X] *Concrete limits.* Raw excerpt soft cap ~2,000 words (≈16 KB); anything larger is captured as a small pointer-stub plus the full file under =raw/files/=, and only on explicit request. =curate --dry-run= flags any =raw/= file over 256 KB as "unusually large." Curation nudge fires at 150 nodes, then re-fires every +50, tracked by =:LAST_CURATED:= rotation.
- [X] *Performance budgets* (=:perf= fixtures; one =emacsclient= round-trip assumed, batch fallback ≈ +1s; calibrate, don't treat as invariants): =index= 100 < 0.5s / 1,000 < 3s; =query= 100 < 0.2s / 1,000 < 1s; =lint= 100 < 1s / 1,000 < 6s; =remember= (write + index + full lint, remote mocked) 100 < 1.5s / 1,000 < 8s; =sync= 100 < 2s / 1,000 < 15s. A miss is a *signal* (an accidental per-check Emacs startup, an O(n²) scan), surfaced for investigation, not an automatic build failure.
- [X] *Lexical scoring weights.* A node's score is the sum of the weight of each field that matches, counted once per field: title 100, tag/project/status 50 each, summary 20, body 5. No term-frequency weighting in v1 — a field either matches or it doesn't. Recency tie-break: when scores are equal, the higher =:UPDATED:= wins.
- [X] *Map implementation.* Built-in =org-roam-graph= first — the profile's =org-roam-file-exclude-regexp= already keeps =raw/= and =index*.org= out of the db, so the graph inherits the right scope for free, and it is the least code. A custom DOT export is the fallback only if project/tag/status *filtering* proves necessary (=org-roam-graph= can't filter), which is a small additive step on top.
- [X] *After-save failure UX.* The save always writes to disk and the buffer stays fully editable — never read-only, never blocked. The pipeline runs after the write; on lint failure it *does not commit*, writes the findings to a =*ai-kb-lint*= buffer (popped to, not focus-stealing), and the uncommitted-failing state shows in the modeline + dashboard. Craig fixes and re-saves; a clean save commits. A briefly saved-but-uncommitted file is the intended state, not a trap.
- [X] *After-save recursion guard.* Two layers. (a) The =ai-kb= minor mode's activation predicate excludes =index*.org= and =raw/=, so generated and captured files never carry the hook. (b) The pipeline binds a re-entrancy flag (=cj/ai-kb--in-pipeline=) that the after-save-hook checks and early-returns on, so programmatic =index.org= regeneration and the commit-time write can't retrigger it. Index regeneration also prefers =write-region= over =save-buffer= to avoid the hook entirely.

* vNext

Each is valuable but out of v1 scope; the v1 bar is token-efficient, safe, and fully recoverable. Reasons given so a future reader need not re-litigate.

- *Step 3 — migrate =.ai/sessions/= + workflows into ai-kb.* Its own spec. *Why not v1:* moving working systems is a migration with its own tradeoffs; prove ai-kb first.
- *Other-agent adapters (Codex, Ollama) + MCP server.* *Why not v1:* cross-agent is not a near-term goal (Craig, 2026-05-24). The contract is already repo-resident and neutral in shape, so adapters are purely additive when a second agent is actually adopted. Ollama specifically will need a host wrapper (run =ai-kb query= before the turn; =remember --confirm= human-gated) since a model runtime won't curate on its own.
- *Compiled =wiki/= layer + =schema.org=, source hashes, numeric confidence.* *Why not v1:* pay off only with a substantial external-source corpus to compile and drift-check; v1 captures sources selectively under =raw/= already.
- *Formal ingest pipeline.* *Why not v1:* the external-corpus workflow that triggers the wiki layer; premature without it.
- *Semantic / embedding retrieval, =qmd=.* *Why not v1:* find-by-meaning pays off above ~hundreds of nodes; =rg= + index is faster and dependency-free below that.
- *Event-sourced JSONL catalog + SQLite projection; typed-link graph traversal + content-addressed spans.* *Why not v1:* Nexus-scale infrastructure; org-roam's db + grep cover v1, and clean links/properties now let this be built later without rewriting nodes.
- *Plan library / operator-DAG execution* (AgenticScholar, Plan*RAG). *Why not v1:* the near-term lesson is only "don't hide retrieval procedure in prose" — met by the =ai-kb query= contract; multi-hop planning waits.
- *=log.org= op-log.* *Why not v1:* git history already records every write; add later if the git log is too coarse.
- *Full =:VISIBILITY:= taxonomy + public/private split.* *Why not v1:* the private repo + no-secrets rule cover the floor.
- *Full agentic-knowledgebase vision* (project hubs; person/decision/thread/meeting/problem/runbook types; =cj/agent-*= commands). *Why not v1:* a much larger product; ai-kb is its first slice.
- *Live dual-roam browsing* (no switch). *Why not v1:* org-roam supports one active db per session today.

* Relationship to existing mechanisms

- *Per-project claude memory (T2)* — stays the session-recall layer; shrinks to an index pointing into ai-kb (T3) for significant items.
- *.ai/notes.org and .ai/sessions/* — unchanged in v1 (migration is deferred Step 3).
- *Personal org-roam (recipes, etc.)* — never touched; reached by switching.
- *agentic-knowledgebase.org* — the broader vision; ai-kb is its first concrete slice.