#+TITLE: Design: Coverage Reporting #+AUTHOR: Craig Jennings #+DATE: 2026-04-22 * Status Implemented for Elisp. The shipped path is local-first: =make coverage= produces =.coverage/simplecov.json= with Undercover, and =cj/coverage-report= reads that artifact to show either diff-aware coverage or a whole-project summary from Emacs. Python, TypeScript, and Go backends remain future work. * Problem Before this work, there was no quick way to answer "are the lines I just changed actually covered by tests?" Line-level coverage for the *whole* project was also missing, and there was no local artifact to inspect. The primary user-facing need is the first one: point-in-time feedback on in-flight changes, triggered from Emacs. The implemented system also supports a whole-project summary and writes a local SimpleCov JSON artifact. The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language. * Non-Goals - Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns. - Mutation testing or any signal other than line coverage. - CI integration beyond emitting a simplecov JSON artifact. No coveralls, no GitHub Actions wiring. - Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.). * Approaches Considered ** Recommended: diff-aware report with pluggable backends Core engine reads a simplecov JSON file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce simplecov in their own way and register themselves with the core. *Pros:* Directly serves the primary use case. Simplecov is broadly supported across language coverage tools, and Undercover's ~:merge-report t~ option works for simplecov (but not for LCOV), which is essential for the per-file coverage-run strategy. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation. *Note on format:* An earlier draft of this design used LCOV. That was changed to simplecov after discovering that Undercover's LCOV writer does not implement report-merging — per-file coverage runs would require custom merge logic or an external ~lcov~ tool. Simplecov's native merge-report support made it the cleaner fit without changing anything about the pluggable backend story. *Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines). ** Rejected: non-interactive pre-commit hook Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes. ** Rejected: coverage as a =review-code= skill criterion Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward. ** Rejected: mutation testing instead of line coverage Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation. * Design ** Architecture Three files: - =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic. - =modules/coverage-elisp.el= — the initial backend. Registers itself on load. - (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering. =init.el= requires the core and the active backends. *** Elisp coverage producer For the Elisp backend, =make coverage= is the only supported producer of the coverage artifact. It removes stale compiled files for instrumented sources, then runs each unit test file in its own batch Emacs process. Before loading the test file, the Makefile loads =tests/run-coverage-file.el=, which initializes packages and configures Undercover: #+begin_src emacs-lisp (undercover "modules/*.el" "gptel-tools/*.el" (:report-format 'simplecov) (:report-file ".coverage/simplecov.json") (:merge-report t) (:send-report nil)) #+end_src Undercover is therefore the instrumentation layer: it instruments =modules/*.el= and =gptel-tools/*.el=, records Edebug stop-point hits while tests execute, and writes the line hit arrays. SimpleCov is the local interchange format consumed by the rest of this design. The split-per-test-file Makefile strategy depends on =:merge-report t=; Undercover can merge SimpleCov reports across separate Emacs processes, while its LCOV writer cannot merge reports. This is the concrete reason the artifact is =.coverage/simplecov.json= rather than =coverage.lcov=. The Makefile excludes tests that are incompatible with instrumented source loading, such as byte-compilation checks. If =.coverage/simplecov.json= is not created, the coverage run is considered failed; downstream report commands should not infer partial coverage from a missing artifact. *** Backend protocol Each backend is a plist registered into =cj/coverage-backends=: #+begin_src emacs-lisp (:name 'elisp :detect (lambda () ...) ; non-nil if current project matches :run (lambda (cb) ...) ; kick off coverage build; invoke CB with report path :report-path (lambda () ...)) ; where the simplecov JSON lives (for re-reading without running) #+end_src Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance. *** Pure helpers - =cj/--coverage-parse-simplecov FILE= → hash-table ={file → covered-line-set}=. - =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers. - =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked. These helpers are pure and covered by focused ERT tests. ** Data Flow 1. User invokes =cj/coverage-report= (bound to =F7=). 2. Core resolves the backend for the current project. 3. =completing-read= prompts for scope: - "Working tree — all uncommitted changes" - "Staged — about to commit" - "Branch vs parent" (uses =@{upstream}= unless a caller passes an explicit base to the helper) - "Branch vs main" (explicit) - "Whole project — all executable lines" 4. If =simplecov.json= is missing, prompt to run coverage. A prefix argument (=C-u F7=) forces a fresh run. Otherwise the existing report is used as-is. 5. Parse simplecov, compute changed lines or all executable lines, intersect. 6. Display a report buffer in a mode derived from =compilation-mode=. ** Persistence - =.coverage/simplecov.json= at the project root, gitignored. Overwritten on each run. - No long-term storage. Historical tracking is explicitly out of scope for v1. ** Error Handling *Pre-flight:* - No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=. - =.dir-locals.el= names an unknown backend → error listing registered backends. - Not in a git repository → error; don't swallow git's stderr. - Branch comparison on a repo with no common ancestor (orphan branch, shallow clone missing the fork point, or missing upstream) reports the underlying git failure. *During the coverage run:* - Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data. - Run completes but no simplecov.json produced → error naming the expected path. *Post-flight classification:* three buckets, not two. - *Covered* — changed line in the simplecov covered-line set. - *Uncovered* — changed line in a tracked file but not covered. - *Not tracked* — changed file isn't in the simplecov data at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code." *Happy-path degenerates:* - Zero changed lines in scope → "No changes in this scope; nothing to report." - All changed lines covered → "N of N changed lines covered. " ** Keybindings *Global:* - =F7= → =cj/coverage-report= (prompts scope, shows report). - =C-u F7= → force re-run regardless of report freshness. *In the report buffer* (compilation-mode derived, most inherited for free): - =RET= → jump to source under point. - =n= / =p= → next / previous uncovered line. - =q= → bury buffer. *Globally available via compilation-mode integration:* - =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer. - =C-x `= → visit next uncovered line without leaving the current buffer. The =F4=–=F7= developer block currently uses =F6= for project-aware test dispatch and =F7= for coverage. ** Testing *Pure helpers, fully tested* (Normal / Boundary / Error for each): - =cj/--coverage-parse-simplecov= — handcrafted simplecov JSON in temp files; empty object, all-null coverage arrays, spaces in filenames, multiple test-name keys unioned, malformed JSON. - =cj/--coverage-simplecov-executable-lines= — whole-project executable-line set, including zero-hit executable lines. - =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case. - =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs. - =cj/--coverage-format-report= and =cj/--coverage-format-summary= — report text and whole-project per-file summary. *Backend registry, structurally tested:* - =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins. *Not tested:* - =cj/coverage-report= interactive command — one smoke test with a prepared simplecov report and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display. - The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design. * Current Limitations - =make coverage= runs all unit test files except known instrumentation conflicts. It does not try to select only tests related to changed modules. - Existing reports are not checked for staleness. Use =C-u F7= or =make coverage= when a fresh report matters. - Only the Elisp backend is implemented. - There is no CI coverage publishing. The generated =.coverage/simplecov.json= file is local and gitignored.