diff options
| author | Craig Jennings <c@cjennings.net> | 2026-06-15 10:24:40 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-06-15 10:24:40 -0500 |
| commit | 45e0f6e896b2c34de25d5c3aa18474c79d6a1e72 (patch) | |
| tree | 3f4d822aa5da53f4e6bbebbdc7fb400a2b212189 /docs/specs/coverage-spec-implemented.org | |
| parent | a5c9f48220cd52770f10f7627922b9fc8e2204cc (diff) | |
| download | dotemacs-45e0f6e896b2c34de25d5c3aa18474c79d6a1e72.tar.gz dotemacs-45e0f6e896b2c34de25d5c3aa18474c79d6a1e72.zip | |
docs: move specs to docs/specs/ with lifecycle-status filenames
Separate the 27 formal specs from working notes. Specs move to docs/specs/, notes stay in docs/design/. Each spec carries its lifecycle in the filename (-spec, -spec-doing, -spec-implemented, -spec-superseded) plus an authoritative ID and STATUS property drawer. The status came from checking each spec against the code, not the doc's own field: 6 implemented, 8 in progress, 12 not started, 1 superseded.
Inbound links become org-id links so future status renames don't break them; code-comment paths repoint to docs/specs/. Working notes, inventories, reviews, and brainstorms stay in docs/design/.
Diffstat (limited to 'docs/specs/coverage-spec-implemented.org')
| -rw-r--r-- | docs/specs/coverage-spec-implemented.org | 210 |
1 files changed, 210 insertions, 0 deletions
diff --git a/docs/specs/coverage-spec-implemented.org b/docs/specs/coverage-spec-implemented.org new file mode 100644 index 000000000..65734fb3d --- /dev/null +++ b/docs/specs/coverage-spec-implemented.org @@ -0,0 +1,210 @@ +:PROPERTIES: +:ID: 7d7f4486-fad7-4f0a-bd9a-775bd4cd8f7e +:STATUS: implemented +:END: +#+TITLE: Design: Coverage Reporting +#+AUTHOR: Craig Jennings +#+DATE: 2026-04-22 + +* Status + +Implemented for Elisp. + +The shipped path is local-first: =make coverage= produces +=.coverage/simplecov.json= with Undercover, and =cj/coverage-report= reads that +artifact to show either diff-aware coverage or a whole-project summary from +Emacs. Python, TypeScript, and Go backends remain future work. + +* Problem + +Before this work, there was no quick way to answer "are the lines I just +changed actually covered by tests?" Line-level coverage for the *whole* +project was also missing, and there was no local artifact to inspect. + +The primary user-facing need is the first one: point-in-time feedback on +in-flight changes, triggered from Emacs. The implemented system also supports a +whole-project summary and writes a local SimpleCov JSON artifact. + +The tooling should be pluggable so the same workflow covers Elisp today and Python, TypeScript, and Go later — without rebuilding the UI for each language. + +* Non-Goals + +- Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns. +- Mutation testing or any signal other than line coverage. +- CI integration beyond emitting a simplecov JSON artifact. No coveralls, no GitHub Actions wiring. +- Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.). + +* Approaches Considered + +** Recommended: diff-aware report with pluggable backends + +Core engine reads a simplecov JSON file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce simplecov in their own way and register themselves with the core. + +*Pros:* Directly serves the primary use case. Simplecov is broadly supported across language coverage tools, and Undercover's ~:merge-report t~ option works for simplecov (but not for LCOV), which is essential for the per-file coverage-run strategy. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation. + +*Note on format:* An earlier draft of this design used LCOV. That was changed to simplecov after discovering that Undercover's LCOV writer does not implement report-merging — per-file coverage runs would require custom merge logic or an external ~lcov~ tool. Simplecov's native merge-report support made it the cleaner fit without changing anything about the pluggable backend story. + +*Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines). + +** Rejected: non-interactive pre-commit hook + +Would run coverage on every commit and report uncovered-changed-lines to stderr. Literal fit for the use case but adds a long delay to every commit and offers no way to inspect non-staged scopes. + +** Rejected: coverage as a =review-code= skill criterion + +Would fold coverage into the existing pre-commit review skill. Clean in principle, but couples =review-code= to Emacs-specific tooling and makes ad-hoc inspection (outside a review) awkward. + +** Rejected: mutation testing instead of line coverage + +Stronger signal than coverage but minutes-to-hours runtime on the current 265-file suite, and no polished Elisp tool exists. Different conversation. + +* Design + +** Architecture + +Three files: + +- =modules/coverage-core.el= — engine + backend registry + user-facing command. Language-agnostic. +- =modules/coverage-elisp.el= — the initial backend. Registers itself on load. +- (Future) =modules/coverage-python.el=, =coverage-typescript.el=, =coverage-go.el= — each ~30 lines, self-registering. + +=init.el= requires the core and the active backends. + +*** Elisp coverage producer + +For the Elisp backend, =make coverage= is the only supported producer of the +coverage artifact. It removes stale compiled files for instrumented sources, +then runs each unit test file in its own batch Emacs process. Before loading +the test file, the Makefile loads =tests/run-coverage-file.el=, which +initializes packages and configures Undercover: + +#+begin_src emacs-lisp +(undercover "modules/*.el" + "gptel-tools/*.el" + (:report-format 'simplecov) + (:report-file ".coverage/simplecov.json") + (:merge-report t) + (:send-report nil)) +#+end_src + +Undercover is therefore the instrumentation layer: it instruments +=modules/*.el= and =gptel-tools/*.el=, records Edebug stop-point hits while +tests execute, and writes the line hit arrays. SimpleCov is the local +interchange format consumed by the rest of this design. The split-per-test-file +Makefile strategy depends on =:merge-report t=; Undercover can merge SimpleCov +reports across separate Emacs processes, while its LCOV writer cannot merge +reports. This is the concrete reason the artifact is +=.coverage/simplecov.json= rather than =coverage.lcov=. + +The Makefile excludes tests that are incompatible with instrumented source +loading, such as byte-compilation checks. If =.coverage/simplecov.json= is not +created, the coverage run is considered failed; downstream report commands +should not infer partial coverage from a missing artifact. + +*** Backend protocol + +Each backend is a plist registered into =cj/coverage-backends=: + +#+begin_src emacs-lisp +(:name 'elisp + :detect (lambda () ...) ; non-nil if current project matches + :run (lambda (cb) ...) ; kick off coverage build; invoke CB with report path + :report-path (lambda () ...)) ; where the simplecov JSON lives (for re-reading without running) +#+end_src + +Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance. + +*** Pure helpers + +- =cj/--coverage-parse-simplecov FILE= → hash-table ={file → covered-line-set}=. +- =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers. +- =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked. + +These helpers are pure and covered by focused ERT tests. + +** Data Flow + +1. User invokes =cj/coverage-report= (bound to =F7=). +2. Core resolves the backend for the current project. +3. =completing-read= prompts for scope: + - "Working tree — all uncommitted changes" + - "Staged — about to commit" + - "Branch vs parent" (uses =@{upstream}= unless a caller passes an explicit base to the helper) + - "Branch vs main" (explicit) + - "Whole project — all executable lines" +4. If =simplecov.json= is missing, prompt to run coverage. A prefix argument + (=C-u F7=) forces a fresh run. Otherwise the existing report is used as-is. +5. Parse simplecov, compute changed lines or all executable lines, intersect. +6. Display a report buffer in a mode derived from =compilation-mode=. + +** Persistence + +- =.coverage/simplecov.json= at the project root, gitignored. Overwritten on each run. +- No long-term storage. Historical tracking is explicitly out of scope for v1. + +** Error Handling + +*Pre-flight:* +- No backend matches → =user-error= with instructions to register a backend or set =.dir-locals.el=. +- =.dir-locals.el= names an unknown backend → error listing registered backends. +- Not in a git repository → error; don't swallow git's stderr. +- Branch comparison on a repo with no common ancestor (orphan branch, shallow + clone missing the fork point, or missing upstream) reports the underlying git + failure. + +*During the coverage run:* +- Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data. +- Run completes but no simplecov.json produced → error naming the expected path. + +*Post-flight classification:* three buckets, not two. +- *Covered* — changed line in the simplecov covered-line set. +- *Uncovered* — changed line in a tracked file but not covered. +- *Not tracked* — changed file isn't in the simplecov data at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code." + +*Happy-path degenerates:* +- Zero changed lines in scope → "No changes in this scope; nothing to report." +- All changed lines covered → "N of N changed lines covered. " + +** Keybindings + +*Global:* +- =F7= → =cj/coverage-report= (prompts scope, shows report). +- =C-u F7= → force re-run regardless of report freshness. + +*In the report buffer* (compilation-mode derived, most inherited for free): +- =RET= → jump to source under point. +- =n= / =p= → next / previous uncovered line. +- =q= → bury buffer. + +*Globally available via compilation-mode integration:* +- =M-g n= / =M-g p= → =next-error= / =previous-error= on the last compilation buffer. +- =C-x `= → visit next uncovered line without leaving the current buffer. + +The =F4=–=F7= developer block currently uses =F6= for project-aware test +dispatch and =F7= for coverage. + +** Testing + +*Pure helpers, fully tested* (Normal / Boundary / Error for each): +- =cj/--coverage-parse-simplecov= — handcrafted simplecov JSON in temp files; empty object, all-null coverage arrays, spaces in filenames, multiple test-name keys unioned, malformed JSON. +- =cj/--coverage-simplecov-executable-lines= — whole-project executable-line set, including zero-hit executable lines. +- =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case. +- =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs. +- =cj/--coverage-format-report= and =cj/--coverage-format-summary= — report text and whole-project per-file summary. + +*Backend registry, structurally tested:* +- =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins. + +*Not tested:* +- =cj/coverage-report= interactive command — one smoke test with a prepared simplecov report and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display. +- The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design. + +* Current Limitations + +- =make coverage= runs all unit test files except known instrumentation + conflicts. It does not try to select only tests related to changed modules. +- Existing reports are not checked for staleness. Use =C-u F7= or + =make coverage= when a fresh report matters. +- Only the Elisp backend is implemented. +- There is no CI coverage publishing. The generated + =.coverage/simplecov.json= file is local and gitignored. |
