#+TITLE: Docs Lifecycle — Spec #+AUTHOR: Craig Jennings & Claude #+DATE: 2026-07-01 #+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED * DRAFT Docs lifecycle :PROPERTIES: :ID: 80b0787b-4a60-4c82-8a16-b383d3e3c8f2 :END: - 2026-07-01 Wed @ 22:13:00 -0400 — drafted from the five decisions settled 2026-06-28 (todo.org "Spec storage location + lifecycle-status convention"). * Metadata | Status | draft | |----------+------------------------------------------------------------------| | Owner | Craig Jennings | |----------+------------------------------------------------------------------| | Reviewer | Craig Jennings | |----------+------------------------------------------------------------------| | Date | 2026-07-01 | |----------+------------------------------------------------------------------| | Related | [[file:../design/2026-06-15-spec-storage-lifecycle-proposal.org][source proposal]]; todo.org "Spec storage location + | | | lifecycle-status convention" | |----------+------------------------------------------------------------------| * Summary Formal specs and working notes currently share one directory per project, and a spec's lifecycle state (drafted, in progress, shipped, dead) is invisible without opening the file. This spec adopts two coupled conventions — a location split (=docs/specs/= for formal specs, =docs/design/= for notes) and an authoritative in-file status carried by an org TODO keyword on a top-level status heading — plus =org-id= links for rename-safety, a general =docs-lifecycle= rule capturing the shape, and a one-time confirmed retrofit that sorts every project's existing pile. * Problem / Context .emacs.d triaged ~28 design docs and had to run a four-agent sweep reading every spec against the code to reconstruct which had shipped (6 implemented, 8 in progress, 12 not started, 1 superseded). Nothing in the filename, location, or file records the state, so the answer to "what's open?" degrades into "open every file and infer." rulesets has the same shape: 41 files in =docs/design/= of which only 3 carry a formal spec spine, plus two =-spec.org= files misfiled at the =docs/= root. The cost compounds with every doc added, and every project inherits the problem through the shared spec-create workflow. Two forces beyond triage cost: - *Links are load-bearing.* =todo.org= tasks, session archives, and sibling docs link specs by =file:= path. Any convention that renames or moves files on every status change (the filename-suffix approach) breaks those links repeatedly across a cross-linked, template-synced doc set. - *The convention is worthless if legacy docs stay misfiled* (Craig, 2026-06-28). Template sync distributes rules and workflows but cannot perform a one-time per-project migration, so the design must include a reach mechanism that gets each project's existing pile sorted once. * Goals and Non-Goals ** Goals - A directory listing answers "which docs are specs, and what state is each in" without opening files. - Status transitions cost one keyword edit — no rename, no link surgery. - Cross-doc spec links survive moves and renames. - The shape is captured once as a general rule (=docs-lifecycle=) so future artifact collections (brainstorm piles, recording queues) can reuse it. - Every existing project's =docs/design/= pile gets sorted exactly once, with human confirmation on each classification. ** Non-Goals - No automation of status flips — the keyword is edited by whoever changes the state (spec-create, spec-review, spec-response, or a human), not by a watcher. - No retroactive rewriting of session archives or git history that reference old paths; only live inbound links (=todo.org=, =notes.org=, docs) are updated by the retrofit. - No new tracking database or index file — the files are the index. ** Scope tiers - v1: the location split, the status-heading convention, the org-id link standard, the =docs-lifecycle= rule, spec-create/spec-review/spec-response updates, the retrofit helper + startup nudge, and the rulesets pilot. - Out of scope: applying the lifecycle shape to non-doc collections (the rule documents the pattern; adopting it elsewhere is per-collection work). - vNext: an org-agenda custom view over =docs/specs/*.org= keyed on the status keywords (nice-to-have once the keywords exist; log to todo.org). * Design ** The location split - =docs/specs/= — formal specs only. A *spec* is a doc proposing a buildable change that carries a =Decisions= section and =Implementation phases= (the spec-create spine). Filenames keep the existing =YYYY-MM-DD--spec.org= shape — the =-spec.org= suffix stays because spec-review's Phase 0 precondition keys on it; only the *status* suffixes from the original proposal are dropped. - =docs/design/= — everything else: brainstorms, inventories, proposals, research notes, frozen source material. Review findings live inside the spec they review (current spec-review behavior), so standalone review files are legacy notes and stay in =docs/design/=. ** The status heading (the authoritative record) Each spec's first element after the file header is a single top-level *status heading* carrying the org TODO keyword: #+begin_example ,#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED ,* DOING :PROPERTIES: :ID: :END: - #+end_example - *The keyword is authoritative.* The Metadata table's =Status= field mirrors it in lowercase for readers already in the table; on disagreement the heading wins. - *Vocabulary:* =DRAFT= (being written) → =READY= (review passed, buildable) → =DOING= (implementation in progress) → =IMPLEMENTED= / =SUPERSEDED= / =CANCELLED= (terminal). This supersedes the old =TODO | DONE SUPERSEDED CANCELLED= header line; the retrofit updates it. - *Glanceability without opening files:* one grep gives the full board — #+begin_src sh rg -H '^\* (DRAFT|READY|DOING|IMPLEMENTED|SUPERSEDED|CANCELLED) ' docs/specs/ #+end_src and because the keyword sits on a real org heading, an org-agenda view over =docs/specs/= works for free (the vNext item). - *The heading body is the dated status history* — one line per transition (=YYYY-MM-DD Day @ HH:MM:SS -ZZZZ — =), the record a filename could never carry. - Why a dedicated status heading rather than restructuring each spec under one top-level heading: demoting every section in every existing spec is a large, link-hostile rewrite; a prepended heading is additive, retrofittable by script, and leaves the familiar flat section layout untouched. ** Rename-safe links The status heading carries an =:ID:= UUID. Cross-doc references to a spec use =[[id:]]= rather than =file:= paths, so the one move the retrofit performs (and any future move) can't orphan them. =file:= links remain fine for intra-doc anchors and for notes that never move. The KB's existing id-resolution recipe applies: =rg ':ID:[[:space:]]+' docs/=. ** The =docs-lifecycle= rule (the generalization) A new =claude-rules/docs-lifecycle.md= captures the reusable shape, with spec-create as the first instance: 1. Separate formal artifacts from working notes by location. 2. Lifecycle state lives *in* the artifact, on a scannable, greppable carrier (an org keyword heading), with a dated history. 3. Links use rename-safe identifiers. 4. A growing collection earns this treatment when "which of these are live?" starts requiring a file-by-file read. ** The retrofit (reach mechanism for existing piles) A synced helper, =.ai/scripts/spec-sort=, run once per project: 1. *Classify* each =docs/design/*.org= (and stray =docs/*-spec.org=) by heuristic: a doc with a =Decisions= section plus =Implementation phases= or a =Metadata= table is a spec candidate; everything else is a note. The heuristic *proposes*; a human confirms every move (classification is a judgment call — Craig, 2026-06-28). 2. *Move* confirmed specs to =docs/specs/=, prepend the status heading (keyword proposed from the doc's current Status field or review history; confirmed by the human), assign an =:ID:=, and update the =#+TODO:= header line. 3. *Relink* inbound =file:= references (=todo.org=, =notes.org=, =.ai/=, other docs) to the new path — and to =id:= form where the target is a spec. 4. *Stamp* =:LAST_SPEC_SORT: YYYY-MM-DD= in =notes.org= Workflow State. A startup nudge (one line in Phase C) fires in projects that have a =docs/design/= but no =:LAST_SPEC_SORT:= marker, so each project runs the sort exactly once without template sync having to perform it. * Alternatives Considered ** Filename status suffix (=-spec-doing.org=, =-spec-implemented.org=) - Good, because the state is visible in a bare =ls= with no tooling. - Bad, because every transition renames a file in a cross-linked, template-synced doc set — each rename is link surgery or a broken link, and the churn lands in git history and inbound =todo.org= links. - Neutral, because the ls-visibility it buys is matched by the one-line =rg= over status headings. - Rejected 2026-06-28 (Craig chose org-keyword over his earlier filename-suffix lean). ** Status field in the Metadata table only (no keyword) - Good, because the field already exists and needs no new structure. - Bad, because a table cell is neither org-agenda-scannable nor reliably greppable across format drift, and it carries no dated history. - Neutral, because the field stays anyway — as the in-table mirror. ** Relink-helper instead of org-id (keep =file:= links, fix them on every move) - Good, because readers see plain paths. - Bad, because it makes every future move a tooling event, and one missed run silently breaks links — the failure mode is invisible until someone clicks. - Neutral, because the retrofit needs relink logic once regardless; org-id just makes it a one-time need. * Decisions [5/5] All five were settled with Craig on 2026-06-28 (recorded in todo.org; migrated here per that note). ** DONE Location split — adopt - Context: specs and notes share one directory; telling them apart requires opening files. - Decision: =docs/specs/= for formal specs (Decisions + phases spine); =docs/design/= for notes. Documented in spec-create and the docs-lifecycle rule. - Consequences: easier — a listing answers "what's formal"; harder — one-time migration and link updates (the retrofit). ** DONE Status mechanism — org keyword authoritative, no filename suffix - Context: filename suffix vs org keyword; suffix wins =ls= visibility, keyword wins link stability and zero-rename transitions. - Decision: the org TODO keyword on the spec's top status heading is authoritative, mirrored by the Metadata =Status= field. No status suffixes in filenames. - Consequences: easier — a transition is one keyword edit and links never break; harder — glanceability needs the one-line =rg= (or the vNext agenda view) instead of bare =ls=. ** DONE Link safety — org-id for cross-doc spec links - Context: both the migration move and any future rename break =file:= links. - Decision: specs carry =:ID:= UUIDs on the status heading; cross-doc references use =[[id:...]]=. - Consequences: easier — moves are free; harder — following a link outside org needs the =rg ':ID:'= lookup. ** DONE Generalize as a =docs-lifecycle= rule - Context: the shape (in-artifact lifecycle state, formal-vs-notes split, rename-safe links) recurs for any processed-document collection. - Decision: capture it in =claude-rules/docs-lifecycle.md= with spec-create as the first instance. - Consequences: easier — the next collection reuses a decided pattern; harder — the rule must stay honest as the spec instance evolves. ** DONE Retrofit existing files across ALL projects - Context: template sync distributes conventions but cannot perform a per-project one-time migration; legacy piles would stay misfiled forever. - Decision: ship a confirmed classify-move-relink helper (=spec-sort=) plus a startup nudge gated on =:LAST_SPEC_SORT:=; the helper proposes, a human confirms. Pilot on rulesets first. - Consequences: easier — every project converges without manual archaeology; harder — the helper needs real relink logic and tests, and classification stays a judgment call. * Implementation phases ** Phase 1 — Rule + template updates Write =claude-rules/docs-lifecycle.md=. Update spec-create (emit into =docs/specs/=, new =#+TODO:= vocabulary, status heading with =:ID:= in the template, transition mechanics), spec-review (precondition path =docs/specs/=; flipping =DRAFT= → =READY= on a passing review updates the heading + history line), and spec-response (status flips it owns). Tree stays working: new specs land in the new shape; old specs are untouched until Phase 3. ** Phase 2 — The =spec-sort= helper Build =.ai/scripts/spec-sort= (classify → confirm → move + prepend status heading + assign =:ID:= → relink inbound references → stamp =:LAST_SPEC_SORT:=), with bats coverage for classification, moving, relinking, idempotence, and the confirm gate. Tree stays working: the script is callable but nothing invokes it yet. ** Phase 3 — Pilot on rulesets Run =spec-sort= against rulesets' own =docs/= (41 design files, 3 spec-spine candidates, 2 stray root specs). Fix what the pilot surfaces before any other project runs it. Tree stays working: moves are confirmed one by one, links updated in the same pass. ** Phase 4 — Startup nudge + broadcast Add the Phase C nudge (fires when =docs/design/= exists and =:LAST_SPEC_SORT:= is absent). Send .emacs.d a note that the convention is live and its ~28-doc pile is ready to sort. Tree stays working: the nudge is one read-only line per session until acted on. * Acceptance criteria - [ ] =rg '^\* (DRAFT|READY|DOING|IMPLEMENTED|SUPERSEDED|CANCELLED) ' docs/specs/= lists every rulesets spec with its state, and the answer matches reality. - [ ] A status transition on a spec changes exactly one keyword plus one history line — no rename, no link edits. - [ ] Every doc remaining in rulesets =docs/design/= is a note (no Decisions + phases spine); both stray =docs/= root specs are re-homed. - [ ] All inbound =todo.org= / =notes.org= / doc links to moved specs resolve after the pilot. - [ ] spec-create emits new specs into =docs/specs/= in the new shape; spec-review refuses a spec outside it. - [ ] A project with an unsorted =docs/design/= gets the startup nudge; one confirmed =spec-sort= run clears it via =:LAST_SPEC_SORT:=. * Readiness dimensions - Data model & ownership: the spec file owns its state; the Metadata mirror is display-only. No external index to drift. - Errors, empty states & failure: =spec-sort= on a project with no =docs/= is a silent no-op; an ambiguous classification is surfaced, never auto-moved; a relink pass that finds zero inbound links is normal. - Security & privacy: N/A because the docs are already in-repo; no new exposure surface. - Observability: the status grep is the dashboard; =spec-sort= prints every proposed move and every rewritten link. - Performance & scale: N/A because collections are tens of files; everything is one-shot or grep-speed. - Reuse & lost opportunities: reuses org TODO keywords, org-id, the existing scheme-header pattern of declared vocabularies, and spec-review's in-file findings convention. - Architecture fit & weak points: weak point is the classification heuristic — mitigated by the confirm gate. The status heading is additive, so old readers of spec files see one extra heading and nothing breaks. - Config surface: none new — one marker line (=:LAST_SPEC_SORT:=) in the existing Workflow State section. - Documentation plan: the docs-lifecycle rule is the documentation; spec-create's template is the worked example. - Dev tooling: =spec-sort= ships with bats tests under the existing glob-discovered suite. - Rollout, compatibility & rollback: additive per project, one project at a time, rulesets first. Rollback of a sort is =git revert= of the pilot commit (moves + relinks are one commit). - External APIs & deps: N/A — plain files, =rg=, =uuidgen=. * Risks, Rabbit Holes, and Drawbacks - *Relink misses an inbound link shape* (org radio links, bare paths in scripts). Dodge: the pilot greps for the old path after moving and fails loudly on any residue. - *Heuristic over-classifies notes as specs.* Dodge: the confirm gate is mandatory; the helper never moves unconfirmed. - *Keyword vocabulary drift* between this spec, the rule, and spec-create's template. Dodge: the rule names the vocabulary once and the others link it. * Testing / Verification / Rollout bats for =spec-sort= (classification, move, relink, idempotence, confirm gate, marker stamp). The pilot run on rulesets is the live verification; the post-move residue grep is the acceptance check. Rollout is per-project via the startup nudge, each run human-confirmed. * References / Appendix - Source proposal: [[file:../design/2026-06-15-spec-storage-lifecycle-proposal.org]] (.emacs.d handoff, 2026-06-15). - Decisions record: todo.org "Spec storage location + lifecycle-status convention" (settled 2026-06-28). - This file is the convention's first resident: it lives in =docs/specs/=, carries the status heading + =:ID:=, and drops the status filename suffix. * Review and iteration history ** 2026-07-01 Wed @ 22:13:00 -0400 — Claude — author - What: initial draft, written from the five pre-ratified decisions. - Why: the queued-specs half of the 2026-06-30 session goal; decisions were settled 2026-06-28 and needed migration into a buildable spec. - Artifacts: todo.org task "Spec storage location + lifecycle-status convention"; source proposal above.