From 83ac3201023f8736c234da27a0642f21786adcfc Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 22 Apr 2026 17:00:39 -0500 Subject: docs: add design docs for coverage and dev-setup-project Two new design docs for pending todo.org tickets. docs/design/coverage.org describes diff-aware coverage reporting with pluggable backends. Primary use case is pre-commit feedback on in-flight changes. LCOV is the shared output format across languages. docs/design/dev-setup-project.org describes an interactive helper that detects a project's shape and writes per-subdirectory .dir-locals.el files for the F4/F6/F7 dev block, with optional starter Makefile generation. Three-tier detection: existing Makefile, existing package.json or pyproject.toml scripts, or fall-back generation. Both tickets in todo.org reference their design docs via org file: links. --- docs/design/coverage.org | 160 ++++++++++++++++++++++++++++++++++++++ docs/design/dev-setup-project.org | 158 +++++++++++++++++++++++++++++++++++++ 2 files changed, 318 insertions(+) create mode 100644 docs/design/coverage.org create mode 100644 docs/design/dev-setup-project.org (limited to 'docs/design') diff --git a/docs/design/coverage.org b/docs/design/coverage.org new file mode 100644 index 00000000..a913a2bc --- /dev/null +++ b/docs/design/coverage.org @@ -0,0 +1,160 @@ +#+TITLE: Design: Coverage Reporting +#+AUTHOR: Craig Jennings +#+DATE: 2026-04-22 + +* Status + +Draft. Not yet implemented. + +* Problem + +Before committing or opening a PR, there's no quick way to answer "are the lines I just changed actually covered by tests?" Line-level coverage for the *whole* project is also missing, and there's no artifact to track coverage over time. + +The primary user-facing need is the first one: point-in-time feedback on in-flight changes, triggered from Emacs. The other two (whole-project report, long-term artifact) fall out naturally once the primary path exists. + +The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language. + +* Non-Goals + +- Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns. +- Mutation testing or any signal other than line coverage. +- CI integration beyond emitting an LCOV artifact. No coveralls, no GitHub Actions wiring. +- Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.). + +* Approaches Considered + +** Recommended: diff-aware report with pluggable backends + +Core engine reads an LCOV file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce LCOV in their own way and register themselves with the core. + +*Pros:* Directly serves the primary use case. LCOV is a universal format, so new languages plug in without touching the core. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation. + +*Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines). + +** Rejected: non-interactive pre-commit hook + +Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes. + +** Rejected: coverage as a =review-code= skill criterion + +Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward. + +** Rejected: mutation testing instead of line coverage + +Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation. + +* Design + +** Architecture + +Three files: + +- =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic. +- =modules/coverage-elisp.el= — the initial backend. Registers itself on load. +- (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering. + +=init.el= requires the core and the active backends. + +*** Backend protocol + +Each backend is a plist registered into =cj/coverage-backends=: + +#+begin_src emacs-lisp +(:name 'elisp + :detect (lambda () ...) ; non-nil if current project matches + :run (lambda (cb) ...) ; kick off coverage build; invoke CB with LCOV path + :lcov-path (lambda () ...)) ; where the LCOV lives (for re-reading without running) +#+end_src + +Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance. + +*** Pure helpers + +- =cj/--coverage-parse-lcov FILE= → hash-table ={file → covered-line-set}=. +- =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers. +- =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked. + +All three are pure, fully ERT-tested. + +** Data Flow + +1. User invokes =cj/coverage-report= (bound to =F7=). +2. Core resolves the backend for the current project. +3. =completing-read= prompts for scope: + - "Working tree — all uncommitted changes" + - "Staged — about to commit" + - "Branch vs parent" (uses =cj/coverage-base-branch= → =@{upstream}= → =main= in order) + - "Branch vs main" (explicit) +4. Freshness check: if =lcov.info= is missing, or older than the newest changed file, prompt "Run coverage now?" Yes runs the backend's =:run= asynchronously via =compile=; no reads the stale file anyway. +5. Parse LCOV, compute changed lines, intersect. +6. Display a report buffer in a mode derived from =compilation-mode=. + +** Persistence + +- =.coverage/lcov.info= at the project root, gitignored. Overwritten on each run. +- No long-term storage. Historical tracking is explicitly out of scope for v1. + +** Error Handling + +*Pre-flight:* +- No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=. +- =.dir-locals.el= names an unknown backend → error listing registered backends. +- Not in a git repository → error; don't swallow git's stderr. +- "Branch vs main" scope on a repo with no common ancestor (orphan branch, shallow clone missing the fork point) → "no merge base with main" error, suggest "Working tree" or "Staged" scope. + +*During the coverage run:* +- Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data. +- Run completes but no LCOV produced → error naming the expected path. + +*Post-flight classification:* three buckets, not two. +- *Covered* — changed line in LCOV's covered-line set. +- *Uncovered* — changed line in a tracked file but not covered. +- *Not tracked* — changed file isn't in LCOV at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code." + +*Happy-path degenerates:* +- Zero changed lines in scope → "No changes in this scope; nothing to report." +- All changed lines covered → "N of N changed lines covered. " + +** Keybindings + +*Global:* +- =F7= → =cj/coverage-report= (prompts scope, shows report). +- =C-u F7= → force re-run regardless of LCOV freshness. + +*In the report buffer* (compilation-mode derived, most inherited for free): +- =RET= → jump to source under point. +- =n= / =p= → next / previous uncovered line. +- =g= → refresh (re-run + redisplay). +- =q= → bury buffer. + +*Globally available via compilation-mode integration:* +- =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer. +- =C-x `= → visit next uncovered line without leaving the current buffer. + +The =F4=–=F7= developer block (compile+run, debug, test, coverage) gets its full rework in a separate todo ticket. The coverage work binds =F7= now because it's its final position. + +** Testing + +*Pure helpers, fully tested* (Normal / Boundary / Error for each): +- =cj/--coverage-parse-lcov= — handcrafted LCOV fragments in temp files; empty, headers-only, spaces/unicode in filenames, malformed lines, missing =end_of_record=. +- =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case. +- =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs. + +*Backend registry, structurally tested:* +- =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins. + +*Not tested:* +- =cj/coverage-report= interactive command — one smoke test with a prepared LCOV and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display. +- The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design. + +* Open Questions + +- [ ] Which tests should a coverage run actually execute? All of them (simple, slow for 265 files), or only the test files whose target modules changed (fast, but dependent-test discovery in Elisp is non-trivial)? Deferred until implementation. +- [ ] Default behavior when LCOV is stale but not missing: prompt, or auto-rerun? Current design prompts. Revisit after first use. +- [ ] Whether =cj/coverage-base-branch= should be a single value or a list of candidates (useful if you routinely stack PRs more than one level deep). Single value for v1. + +* Next Steps + +1. Replace the existing =[#C] Integrate undercover.el for test coverage= entry in =todo.org= with a sharper implementation ticket referencing this design. +2. Begin implementation, starting with the pure helpers (TDD) and the elisp backend, then the =cj/coverage-report= command, then the =make coverage= Makefile target. +3. Open questions above → individual =arch-decide= ADRs if they turn out to be load-bearing; otherwise resolve inline during implementation. diff --git a/docs/design/dev-setup-project.org b/docs/design/dev-setup-project.org new file mode 100644 index 00000000..280b015b --- /dev/null +++ b/docs/design/dev-setup-project.org @@ -0,0 +1,158 @@ +#+TITLE: Design: cj/dev-setup-project +#+AUTHOR: Craig Jennings +#+DATE: 2026-04-22 + +* Status + +Draft. Not yet implemented. + +* Problem + +Adopting the F4 / F6 / F7 dev-block keybindings (compile+run, test, coverage) on a new project means configuring projectile's per-project compile/run/test commands plus the coverage backend. That's a few minutes of ceremony per project, and the polyglot Docker case (backend + frontend in subdirectories) needs per-subproject configuration that projectile's cache doesn't handle cleanly. + +=cj/dev-setup-project= is the interactive helper that removes that ceremony. It detects the project shape, proposes the right =.dir-locals.el= content for each subproject, optionally generates a starter Makefile when none exists, and writes everything in one reviewed step. + +* Non-Goals + +- Running the detected commands. The helper only writes configuration; you invoke F4 / F6 / F7 afterwards. +- Managing Dockerfile changes, compose file edits, or container orchestration. Those stay hand-owned. +- Replacing projectile's cache for simple single-language projects. If you're fine with projectile's prompt-and-cache, don't run the helper. +- Supporting every possible project shape. The helper targets the shapes the user actually uses: pure Elisp, pure Go, pure Python, pure Node/TS, Docker Compose polyglot. + +* Approaches Considered + +** Recommended: detect + review buffer + user commits + +Interactive command opens a review buffer pre-populated with proposals. User edits inline. On =C-c C-c=, helper writes the files. + +Detection is three-tier: existing Makefile, existing package.json / pyproject.toml scripts, or fall back to generating a starter Makefile. Re-runs use the same buffer with status banners (UNCHANGED, WILL UPDATE, WILL CREATE) so nothing changes silently. + +*Pros:* Zero silent surprises. User sees exactly what's going to change. Reuses the same UX for initial setup and re-runs. + +*Cons:* More code than a "just write the files" approach. Review buffer mode is a small but non-trivial piece of UX. + +** Rejected: silent auto-detect and commit + +Helper inspects project, writes =.dir-locals.el= immediately with best-guess conventions, prints a summary. Zero friction on the easy cases. Wrong results on edge cases go unnoticed until you hit F4/F6 and they misfire. Not worth the friction savings. + +** Rejected: wizard (prompt each question in sequence) + +Helper asks "Test command: [default: make test] > " and so on. Explicit and safe, but slow and the series of minibuffer prompts is a worse fit than a single editable review buffer. + +** Rejected: hybrid (silent for obvious cases, wizard for polyglot) + +Two code paths to maintain. The review-buffer approach is already fast for obvious cases (one =C-c C-c= to accept the proposal) and correct for polyglot cases. No need for a second path. + +* Design + +** Detection + +Three tiers, checked in order. + +*** Tier 1: existing Makefile + +Parse Makefile for =.PHONY:= declarations and bare =^target:= lines. Collect the target names. + +Best-guess role mapping: +- *compile* role: prefer =build=, =compile=, =install= +- *run* role: prefer =run=, =start=, =dev=, =serve= +- *test* role: prefer =test=, =tests=, =check= + +If multiple targets match (e.g., both =test= and =check=), pick the first match and list the others in the review buffer as "other available targets." + +*** Tier 2: existing package.json scripts or pyproject.toml sections + +- =package.json= with a =scripts= block: parse the block, same best-guess mapping (=dev= → run, =build= → compile, =test= → test). Command prefix is =npm run=. +- =pyproject.toml= with =[tool.pytest]= or =[project.scripts]=: for v1, skip this — fall back to =pytest= as the test command if =pytest= is on PATH. More sophisticated parsing can come later. + +*** Tier 3: no build file found + +Propose a starter Makefile in the review buffer. User edits or declines. + +The starter Makefile adapts to the detected project type: + +- Elisp: =make compile=, =make test= wrapping =emacs --batch= invocations. +- Go: =go build=, =go run=, =go test ./...=. +- Python (non-Docker): =pip install -r requirements.txt=, =python -m =, =pytest=. +- Node/TS (non-Docker): =npm install=, =npm run dev=, =npm test=. +- Docker Compose polyglot: =docker compose build=, calls to user-named external run script (prompted), =docker compose exec = for tests per service. + +** Review Buffer + +Custom major mode derived from =emacs-lisp-mode= with two local bindings: + +- =C-c C-c= — parse the buffer, validate all blocks, write files, show summary. +- =C-c C-k= — abort, write nothing. + +Block syntax: =;; ==== ====[ ]== banner lines delimit each file's proposed content. Status banner is one of: + +- (unset, initial setup) — file will be created +- =[UNCHANGED]= — current file matches proposal; skipped unless user edits +- =[WILL UPDATE]= — current file differs; shown with both current and proposed for the user to pick +- =[WILL CREATE]= — file doesn't exist yet; will be created + +=;; ==== .gitignore (append if missing) ===== is a special banner — lines under it are appended to =.gitignore= if not already present. + +=;; ==== Makefile ====[...]= is a special banner — only honored if no Makefile exists at the target path. On re-run with an existing Makefile, this banner is suppressed entirely. + +** Escape Hatch + +A =.dir-locals.el= containing =;;; cj/dev-setup-project: ignore= as the first line is skipped on re-run. Lets the user diverge intentionally without every re-run reverting. + +** Write Step + +On =C-c C-c=: + +1. Parse all blocks. Validate each is well-formed elisp (or well-formed Makefile / gitignore entries). +2. If any block is malformed, show an error in the review buffer and do not write. +3. For each WILL UPDATE / WILL CREATE block: write the file. +4. For the gitignore block: append each line only if not present (idempotent). +5. Clear projectile's per-project command cache for this project (so new commands take effect on next F4/F6). +6. Print a summary: ="Wrote backend/.dir-locals.el, frontend/.dir-locals.el, appended 2 lines to .gitignore."= + +** Coverage Backend Forward References + +The helper writes =(cj/coverage-backend . python)=, =(cj/coverage-backend . typescript)=, etc. even when those backends don't exist yet (MVP coverage ships Elisp only). The binding silently does nothing until the backend lands; after that, it activates automatically. Simpler than leaving empty and coming back. + +** Example Flows + +*** Fresh setup on orchestration_dashboard_mvp (Tier 3, no Makefile) + +Review buffer proposes a Makefile (calling the user's existing =reset-dashboard.sh= as the =run= target) plus backend/ and frontend/ =.dir-locals.el= files plus gitignore updates. User edits the Makefile's =run= target to match their actual script path. =C-c C-c=. Four files written. + +*** Fresh setup on .emacs.d (Tier 1, rich Makefile) + +Review buffer shows target-to-role mapping derived from the existing 14-target Makefile (=make compile= → compile role, =make test= → test role; =run= role left nil since this is a config project). Single file written: =.dir-locals.el= at project root. + +*** Re-run after adding a new compose service + +The helper detects a new =worker= service in docker-compose.yml with a =./worker/= build context. Existing backend/ and frontend/ files show =[UNCHANGED]=. New =worker/.dir-locals.el= block shows =[WILL CREATE]=. =C-c C-c=. One file written. + +*** Re-run after renaming a Makefile target + +Makefile's =test-frontend= was renamed to =test-frontend-unit=. The helper detects the mismatch and shows frontend/.dir-locals.el as =[WILL UPDATE]= with current and proposed visible. User either accepts (the test command updates) or edits the buffer to keep =test-frontend=. Nothing silent. + +* Testing + +Pure helpers, fully tested per the project's Normal / Boundary / Error discipline: + +- =cj/--dev-setup-parse-makefile-targets FILE= — handcrafted Makefiles. Normal: two-target file with .PHONY. Boundary: tabs vs spaces, continuation lines, pattern-rule targets (skip them). Error: file missing, non-Makefile content. +- =cj/--dev-setup-parse-package-json-scripts FILE= — synthetic package.json fixtures. Normal: valid scripts block. Boundary: no scripts block, empty scripts. Error: malformed JSON. +- =cj/--dev-setup-detect-project-shape ROOT= — temp directories with combinations of marker files. Assert returned shape plist. Normal: each single-language case. Boundary: docker-compose polyglot with one subproject, with two subprojects, with a service that uses an external image (no subproject). Error: empty directory returns 'unknown. +- =cj/--dev-setup-map-targets-to-roles TARGETS= — input list of target names, output role mapping. Normal: well-named project (build/run/test). Boundary: unusual names (start instead of run; check instead of test). Error: empty input returns empty mapping. +- =cj/--dev-setup-review-buffer-parse CONTENTS= — the buffer-format parser. Normal: well-formed buffer with multiple blocks. Boundary: single block, block with empty body. Error: missing banner, malformed elisp inside a dir-locals block. + +Not tested (by design): +- The interactive command =cj/dev-setup-project= itself — one smoke test that runs against a prepared temp project and asserts the expected files exist after =C-c C-c=. +- The review-buffer major mode's keybindings. + +* Open Questions + +- [ ] Whether to also detect Cargo.toml (Rust), pom.xml (Java/Maven), etc. v1 targets Elisp, Go, Python, TS/JS. Rust/Java defer. +- [ ] Whether =cj/dev-setup-project= should also offer to add a =make coverage= target when generating a Makefile. Probably yes — it's the natural partner to the coverage work. +- [ ] Whether to support a project-wide config override file (=.cj-dev-setup.el= at project root) that pins choices regardless of what detection finds. Defer unless the detection-only path proves annoying. + +* Next Steps + +1. Implement after the F-key rework ticket ships. +2. Open questions above → resolve inline or via =arch-decide= if they turn out to be load-bearing. -- cgit v1.2.3