#+TITLE: Design: Coverage Reporting
#+AUTHOR: Craig Jennings
#+DATE: 2026-04-22

* Status

Implemented for Elisp.

The shipped path is local-first: =make coverage= produces
=.coverage/simplecov.json= with Undercover, and =cj/coverage-report= reads that
artifact to show either diff-aware coverage or a whole-project summary from
Emacs.  Python, TypeScript, and Go backends remain future work.

* Problem

Before this work, there was no quick way to answer "are the lines I just
changed actually covered by tests?" Line-level coverage for the *whole*
project was also missing, and there was no local artifact to inspect.

The primary user-facing need is the first one: point-in-time feedback on
in-flight changes, triggered from Emacs.  The implemented system also supports a
whole-project summary and writes a local SimpleCov JSON artifact.

The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language.

* Non-Goals

- Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns.
- Mutation testing or any signal other than line coverage.
- CI integration beyond emitting a simplecov JSON artifact. No coveralls, no GitHub Actions wiring.
- Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.).

* Approaches Considered

** Recommended: diff-aware report with pluggable backends

Core engine reads a simplecov JSON file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce simplecov in their own way and register themselves with the core.

*Pros:* Directly serves the primary use case. Simplecov is broadly supported across language coverage tools, and Undercover's ~:merge-report t~ option works for simplecov (but not for LCOV), which is essential for the per-file coverage-run strategy. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation.

*Note on format:* An earlier draft of this design used LCOV. That was changed to simplecov after discovering that Undercover's LCOV writer does not implement report-merging — per-file coverage runs would require custom merge logic or an external ~lcov~ tool. Simplecov's native merge-report support made it the cleaner fit without changing anything about the pluggable backend story.

*Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines).

** Rejected: non-interactive pre-commit hook

Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes.

** Rejected: coverage as a =review-code= skill criterion

Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward.

** Rejected: mutation testing instead of line coverage

Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation.

* Design

** Architecture

Three files:

- =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic.
- =modules/coverage-elisp.el= — the initial backend. Registers itself on load.
- (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering.

=init.el= requires the core and the active backends.

*** Backend protocol

Each backend is a plist registered into =cj/coverage-backends=:

#+begin_src emacs-lisp
(:name         'elisp
 :detect       (lambda () ...)   ; non-nil if current project matches
 :run          (lambda (cb) ...) ; kick off coverage build; invoke CB with report path
 :report-path  (lambda () ...))  ; where the simplecov JSON lives (for re-reading without running)
#+end_src

Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance.

*** Pure helpers

- =cj/--coverage-parse-simplecov FILE= → hash-table ={file → covered-line-set}=.
- =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers.
- =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked.

These helpers are pure and covered by focused ERT tests.

** Data Flow

1. User invokes =cj/coverage-report= (bound to =F7=).
2. Core resolves the backend for the current project.
3. =completing-read= prompts for scope:
   - "Working tree — all uncommitted changes"
   - "Staged — about to commit"
   - "Branch vs parent" (uses =@{upstream}= unless a caller passes an explicit base to the helper)
   - "Branch vs main" (explicit)
   - "Whole project — all executable lines"
4. If =simplecov.json= is missing, prompt to run coverage.  A prefix argument
   (=C-u F7=) forces a fresh run.  Otherwise the existing report is used as-is.
5. Parse simplecov, compute changed lines or all executable lines, intersect.
6. Display a report buffer in a mode derived from =compilation-mode=.

** Persistence

- =.coverage/simplecov.json= at the project root, gitignored. Overwritten on each run.
- No long-term storage. Historical tracking is explicitly out of scope for v1.

** Error Handling

*Pre-flight:*
- No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=.
- =.dir-locals.el= names an unknown backend → error listing registered backends.
- Not in a git repository → error; don't swallow git's stderr.
- Branch comparison on a repo with no common ancestor (orphan branch, shallow
  clone missing the fork point, or missing upstream) reports the underlying git
  failure.

*During the coverage run:*
- Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data.
- Run completes but no simplecov.json produced → error naming the expected path.

*Post-flight classification:* three buckets, not two.
- *Covered* — changed line in the simplecov covered-line set.
- *Uncovered* — changed line in a tracked file but not covered.
- *Not tracked* — changed file isn't in the simplecov data at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code."

*Happy-path degenerates:*
- Zero changed lines in scope → "No changes in this scope; nothing to report."
- All changed lines covered → "N of N changed lines covered. "

** Keybindings

*Global:*
- =F7= → =cj/coverage-report= (prompts scope, shows report).
- =C-u F7= → force re-run regardless of report freshness.

*In the report buffer* (compilation-mode derived, most inherited for free):
- =RET= → jump to source under point.
- =n= / =p= → next / previous uncovered line.
- =q= → bury buffer.

*Globally available via compilation-mode integration:*
- =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer.
- =C-x `= → visit next uncovered line without leaving the current buffer.

The =F4=–=F7= developer block currently uses =F6= for project-aware test
dispatch and =F7= for coverage.

** Testing

*Pure helpers, fully tested* (Normal / Boundary / Error for each):
- =cj/--coverage-parse-simplecov= — handcrafted simplecov JSON in temp files; empty object, all-null coverage arrays, spaces in filenames, multiple test-name keys unioned, malformed JSON.
- =cj/--coverage-simplecov-executable-lines= — whole-project executable-line set, including zero-hit executable lines.
- =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case.
- =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs.
- =cj/--coverage-format-report= and =cj/--coverage-format-summary= — report text and whole-project per-file summary.

*Backend registry, structurally tested:*
- =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins.

*Not tested:*
- =cj/coverage-report= interactive command — one smoke test with a prepared simplecov report and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display.
- The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design.

* Current Limitations

- =make coverage= runs all unit test files except known instrumentation
  conflicts.  It does not try to select only tests related to changed modules.
- Existing reports are not checked for staleness.  Use =C-u F7= or
  =make coverage= when a fresh report matters.
- Only the Elisp backend is implemented.
- There is no CI coverage publishing.  The generated
  =.coverage/simplecov.json= file is local and gitignored.