aboutsummaryrefslogtreecommitdiff
path: root/docs/design/coverage.org
diff options
context:
space:
mode:
Diffstat (limited to 'docs/design/coverage.org')
-rw-r--r--docs/design/coverage.org160
1 files changed, 160 insertions, 0 deletions
diff --git a/docs/design/coverage.org b/docs/design/coverage.org
new file mode 100644
index 00000000..a913a2bc
--- /dev/null
+++ b/docs/design/coverage.org
@@ -0,0 +1,160 @@
+#+TITLE: Design: Coverage Reporting
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-04-22
+
+* Status
+
+Draft. Not yet implemented.
+
+* Problem
+
+Before committing or opening a PR, there's no quick way to answer "are the lines I just changed actually covered by tests?" Line-level coverage for the *whole* project is also missing, and there's no artifact to track coverage over time.
+
+The primary user-facing need is the first one: point-in-time feedback on in-flight changes, triggered from Emacs. The other two (whole-project report, long-term artifact) fall out naturally once the primary path exists.
+
+The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language.
+
+* Non-Goals
+
+- Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns.
+- Mutation testing or any signal other than line coverage.
+- CI integration beyond emitting an LCOV artifact. No coveralls, no GitHub Actions wiring.
+- Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.).
+
+* Approaches Considered
+
+** Recommended: diff-aware report with pluggable backends
+
+Core engine reads an LCOV file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce LCOV in their own way and register themselves with the core.
+
+*Pros:* Directly serves the primary use case. LCOV is a universal format, so new languages plug in without touching the core. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation.
+
+*Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines).
+
+** Rejected: non-interactive pre-commit hook
+
+Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes.
+
+** Rejected: coverage as a =review-code= skill criterion
+
+Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward.
+
+** Rejected: mutation testing instead of line coverage
+
+Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation.
+
+* Design
+
+** Architecture
+
+Three files:
+
+- =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic.
+- =modules/coverage-elisp.el= — the initial backend. Registers itself on load.
+- (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering.
+
+=init.el= requires the core and the active backends.
+
+*** Backend protocol
+
+Each backend is a plist registered into =cj/coverage-backends=:
+
+#+begin_src emacs-lisp
+(:name 'elisp
+ :detect (lambda () ...) ; non-nil if current project matches
+ :run (lambda (cb) ...) ; kick off coverage build; invoke CB with LCOV path
+ :lcov-path (lambda () ...)) ; where the LCOV lives (for re-reading without running)
+#+end_src
+
+Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance.
+
+*** Pure helpers
+
+- =cj/--coverage-parse-lcov FILE= → hash-table ={file → covered-line-set}=.
+- =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers.
+- =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked.
+
+All three are pure, fully ERT-tested.
+
+** Data Flow
+
+1. User invokes =cj/coverage-report= (bound to =F7=).
+2. Core resolves the backend for the current project.
+3. =completing-read= prompts for scope:
+ - "Working tree — all uncommitted changes"
+ - "Staged — about to commit"
+ - "Branch vs parent" (uses =cj/coverage-base-branch= → =@{upstream}= → =main= in order)
+ - "Branch vs main" (explicit)
+4. Freshness check: if =lcov.info= is missing, or older than the newest changed file, prompt "Run coverage now?" Yes runs the backend's =:run= asynchronously via =compile=; no reads the stale file anyway.
+5. Parse LCOV, compute changed lines, intersect.
+6. Display a report buffer in a mode derived from =compilation-mode=.
+
+** Persistence
+
+- =.coverage/lcov.info= at the project root, gitignored. Overwritten on each run.
+- No long-term storage. Historical tracking is explicitly out of scope for v1.
+
+** Error Handling
+
+*Pre-flight:*
+- No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=.
+- =.dir-locals.el= names an unknown backend → error listing registered backends.
+- Not in a git repository → error; don't swallow git's stderr.
+- "Branch vs main" scope on a repo with no common ancestor (orphan branch, shallow clone missing the fork point) → "no merge base with main" error, suggest "Working tree" or "Staged" scope.
+
+*During the coverage run:*
+- Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data.
+- Run completes but no LCOV produced → error naming the expected path.
+
+*Post-flight classification:* three buckets, not two.
+- *Covered* — changed line in LCOV's covered-line set.
+- *Uncovered* — changed line in a tracked file but not covered.
+- *Not tracked* — changed file isn't in LCOV at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code."
+
+*Happy-path degenerates:*
+- Zero changed lines in scope → "No changes in this scope; nothing to report."
+- All changed lines covered → "N of N changed lines covered. "
+
+** Keybindings
+
+*Global:*
+- =F7= → =cj/coverage-report= (prompts scope, shows report).
+- =C-u F7= → force re-run regardless of LCOV freshness.
+
+*In the report buffer* (compilation-mode derived, most inherited for free):
+- =RET= → jump to source under point.
+- =n= / =p= → next / previous uncovered line.
+- =g= → refresh (re-run + redisplay).
+- =q= → bury buffer.
+
+*Globally available via compilation-mode integration:*
+- =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer.
+- =C-x `= → visit next uncovered line without leaving the current buffer.
+
+The =F4=–=F7= developer block (compile+run, debug, test, coverage) gets its full rework in a separate todo ticket. The coverage work binds =F7= now because it's its final position.
+
+** Testing
+
+*Pure helpers, fully tested* (Normal / Boundary / Error for each):
+- =cj/--coverage-parse-lcov= — handcrafted LCOV fragments in temp files; empty, headers-only, spaces/unicode in filenames, malformed lines, missing =end_of_record=.
+- =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case.
+- =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs.
+
+*Backend registry, structurally tested:*
+- =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins.
+
+*Not tested:*
+- =cj/coverage-report= interactive command — one smoke test with a prepared LCOV and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display.
+- The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design.
+
+* Open Questions
+
+- [ ] Which tests should a coverage run actually execute? All of them (simple, slow for 265 files), or only the test files whose target modules changed (fast, but dependent-test discovery in Elisp is non-trivial)? Deferred until implementation.
+- [ ] Default behavior when LCOV is stale but not missing: prompt, or auto-rerun? Current design prompts. Revisit after first use.
+- [ ] Whether =cj/coverage-base-branch= should be a single value or a list of candidates (useful if you routinely stack PRs more than one level deep). Single value for v1.
+
+* Next Steps
+
+1. Replace the existing =[#C] Integrate undercover.el for test coverage= entry in =todo.org= with a sharper implementation ticket referencing this design.
+2. Begin implementation, starting with the pure helpers (TDD) and the elisp backend, then the =cj/coverage-report= command, then the =make coverage= Makefile target.
+3. Open questions above → individual =arch-decide= ADRs if they turn out to be load-bearing; otherwise resolve inline during implementation.