#+TITLE: Design: Coverage Reporting #+AUTHOR: Craig Jennings #+DATE: 2026-04-22 * Status Draft. Not yet implemented. * Problem Before committing or opening a PR, there's no quick way to answer "are the lines I just changed actually covered by tests?" Line-level coverage for the *whole* project is also missing, and there's no artifact to track coverage over time. The primary user-facing need is the first one: point-in-time feedback on in-flight changes, triggered from Emacs. The other two (whole-project report, long-term artifact) fall out naturally once the primary path exists. The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language. * Non-Goals - Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns. - Mutation testing or any signal other than line coverage. - CI integration beyond emitting a simplecov JSON artifact. No coveralls, no GitHub Actions wiring. - Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.). * Approaches Considered ** Recommended: diff-aware report with pluggable backends Core engine reads a simplecov JSON file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce simplecov in their own way and register themselves with the core. *Pros:* Directly serves the primary use case. Simplecov is broadly supported across language coverage tools, and Undercover's ~:merge-report t~ option works for simplecov (but not for LCOV), which is essential for the per-file coverage-run strategy. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation. *Note on format:* An earlier draft of this design used LCOV. That was changed to simplecov after discovering that Undercover's LCOV writer does not implement report-merging — per-file coverage runs would require custom merge logic or an external ~lcov~ tool. Simplecov's native merge-report support made it the cleaner fit without changing anything about the pluggable backend story. *Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines). ** Rejected: non-interactive pre-commit hook Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes. ** Rejected: coverage as a =review-code= skill criterion Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward. ** Rejected: mutation testing instead of line coverage Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation. * Design ** Architecture Three files: - =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic. - =modules/coverage-elisp.el= — the initial backend. Registers itself on load. - (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering. =init.el= requires the core and the active backends. *** Backend protocol Each backend is a plist registered into =cj/coverage-backends=: #+begin_src emacs-lisp (:name 'elisp :detect (lambda () ...) ; non-nil if current project matches :run (lambda (cb) ...) ; kick off coverage build; invoke CB with report path :report-path (lambda () ...)) ; where the simplecov JSON lives (for re-reading without running) #+end_src Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance. *** Pure helpers - =cj/--coverage-parse-simplecov FILE= → hash-table ={file → covered-line-set}=. - =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers. - =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked. All three are pure, fully ERT-tested. ** Data Flow 1. User invokes =cj/coverage-report= (bound to =F7=). 2. Core resolves the backend for the current project. 3. =completing-read= prompts for scope: - "Working tree — all uncommitted changes" - "Staged — about to commit" - "Branch vs parent" (uses =cj/coverage-base-branch= → =@{upstream}= → =main= in order) - "Branch vs main" (explicit) 4. Freshness check: if =simplecov.json= is missing, or older than the newest changed file, prompt "Run coverage now?" Yes runs the backend's =:run= asynchronously via =compile=; no reads the stale file anyway. 5. Parse simplecov, compute changed lines, intersect. 6. Display a report buffer in a mode derived from =compilation-mode=. ** Persistence - =.coverage/simplecov.json= at the project root, gitignored. Overwritten on each run. - No long-term storage. Historical tracking is explicitly out of scope for v1. ** Error Handling *Pre-flight:* - No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=. - =.dir-locals.el= names an unknown backend → error listing registered backends. - Not in a git repository → error; don't swallow git's stderr. - "Branch vs main" scope on a repo with no common ancestor (orphan branch, shallow clone missing the fork point) → "no merge base with main" error, suggest "Working tree" or "Staged" scope. *During the coverage run:* - Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data. - Run completes but no simplecov.json produced → error naming the expected path. *Post-flight classification:* three buckets, not two. - *Covered* — changed line in the simplecov covered-line set. - *Uncovered* — changed line in a tracked file but not covered. - *Not tracked* — changed file isn't in the simplecov data at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code." *Happy-path degenerates:* - Zero changed lines in scope → "No changes in this scope; nothing to report." - All changed lines covered → "N of N changed lines covered. " ** Keybindings *Global:* - =F7= → =cj/coverage-report= (prompts scope, shows report). - =C-u F7= → force re-run regardless of report freshness. *In the report buffer* (compilation-mode derived, most inherited for free): - =RET= → jump to source under point. - =n= / =p= → next / previous uncovered line. - =g= → refresh (re-run + redisplay). - =q= → bury buffer. *Globally available via compilation-mode integration:* - =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer. - =C-x `= → visit next uncovered line without leaving the current buffer. The =F4=–=F7= developer block (compile+run, debug, test, coverage) gets its full rework in a separate todo ticket. The coverage work binds =F7= now because it's its final position. ** Testing *Pure helpers, fully tested* (Normal / Boundary / Error for each): - =cj/--coverage-parse-simplecov= — handcrafted simplecov JSON in temp files; empty object, all-null coverage arrays, spaces in filenames, multiple test-name keys unioned, malformed JSON. - =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case. - =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs. *Backend registry, structurally tested:* - =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins. *Not tested:* - =cj/coverage-report= interactive command — one smoke test with a prepared simplecov report and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display. - The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design. * Open Questions - [ ] Which tests should a coverage run actually execute? All of them (simple, slow for 265 files), or only the test files whose target modules changed (fast, but dependent-test discovery in Elisp is non-trivial)? Deferred until implementation. - [ ] Default behavior when the simplecov report is stale but not missing: prompt, or auto-rerun? Current design prompts. Revisit after first use. - [ ] Whether =cj/coverage-base-branch= should be a single value or a list of candidates (useful if you routinely stack PRs more than one level deep). Single value for v1. * Next Steps 1. Replace the existing =[#C] Integrate undercover.el for test coverage= entry in =todo.org= with a sharper implementation ticket referencing this design. 2. Begin implementation, starting with the pure helpers (TDD) and the elisp backend, then the =cj/coverage-report= command, then the =make coverage= Makefile target. 3. Open questions above → individual =arch-decide= ADRs if they turn out to be load-bearing; otherwise resolve inline during implementation.