aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-04-22 19:58:00 -0500
committerCraig Jennings <c@cjennings.net>2026-04-22 19:58:00 -0500
commita97266c0e89ef8560824789063512d2613849fc9 (patch)
treeba320ed3d2dfedebade1fa79ece01baaa4750bd3 /docs
parentecca6c5809aa2945d593baae10308c0dcfe6ec17 (diff)
downloaddotemacs-a97266c0e89ef8560824789063512d2613849fc9.tar.gz
dotemacs-a97266c0e89ef8560824789063512d2613849fc9.zip
feat(coverage): wire make coverage target + simplecov pipeline
Completes the coverage v1 pipeline by adding the Makefile target, the undercover driver script, the exclusion list, and the .gitignore entry. Uses simplecov JSON rather than LCOV as the collection format. The LCOV vs simplecov choice: Undercover's :merge-report t option only supports simplecov. Since the pipeline runs tests per-file (matching test-unit's isolation pattern) and accumulates coverage across runs, merge-report is required. LCOV is better-supported by external coverage viewers, but for a primarily interactive workflow the on-disk format is an internal detail. Other moves in this commit: - Renamed cj/--coverage-parse-lcov to cj/--coverage-parse-simplecov and rewrote its tests for the JSON schema. Same signature, same semantics (file to set of covered lines), different parser. - Renamed the backend protocol's :lcov-path key to :report-path, format-neutral and matching the renamed cj/--coverage-elisp-report-path function. - The coverage target deletes modules/*.elc before running so undercover can instrument the .el sources. Without this, byte-compiled versions shadow the instrumentation and only a handful of pre-loaded modules end up with coverage data. - Excluded tests/test-all-comp-errors.el from make coverage runs. That test byte-compiles every module, which fails under undercover's instrumentation. Excluded only from coverage. Normal make test still runs it. - Updated docs/design/coverage.org to reflect the simplecov pivot with a historical note on why we moved off LCOV. Verified end-to-end: make coverage produces .coverage/simplecov.json with 2717 of 4559 executable lines hit across 44 tracked modules.
Diffstat (limited to 'docs')
-rw-r--r--docs/design/coverage.org38
1 files changed, 20 insertions, 18 deletions
diff --git a/docs/design/coverage.org b/docs/design/coverage.org
index a913a2bc..1a9452bf 100644
--- a/docs/design/coverage.org
+++ b/docs/design/coverage.org
@@ -18,16 +18,18 @@ The tooling should be pluggable so the same workflow covers Elisp today and Pyth
- Continuous in-buffer overlays (fringe marks, line highlights). Parked over performance concerns.
- Mutation testing or any signal other than line coverage.
-- CI integration beyond emitting an LCOV artifact. No coveralls, no GitHub Actions wiring.
+- CI integration beyond emitting a simplecov JSON artifact. No coveralls, no GitHub Actions wiring.
- Shadowing or replacing existing test-running commands (=make test=, =make test-file=, etc.).
* Approaches Considered
** Recommended: diff-aware report with pluggable backends
-Core engine reads an LCOV file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce LCOV in their own way and register themselves with the core.
+Core engine reads a simplecov JSON file, shells to ~git diff~ at a selectable scope, intersects, and displays the result in a compilation-mode-derived buffer. Language-specific "backends" each produce simplecov in their own way and register themselves with the core.
-*Pros:* Directly serves the primary use case. LCOV is a universal format, so new languages plug in without touching the core. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation.
+*Pros:* Directly serves the primary use case. Simplecov is broadly supported across language coverage tools, and Undercover's ~:merge-report t~ option works for simplecov (but not for LCOV), which is essential for the per-file coverage-run strategy. Compilation-mode inheritance gives free =next-error= / =previous-error= navigation.
+
+*Note on format:* An earlier draft of this design used LCOV. That was changed to simplecov after discovering that Undercover's LCOV writer does not implement report-merging — per-file coverage runs would require custom merge logic or an external ~lcov~ tool. Simplecov's native merge-report support made it the cleaner fit without changing anything about the pluggable backend story.
*Cons:* More code than a "just run coverage and read the output" approach. Backend registry adds one layer of indirection (small — ~30 lines).
@@ -60,17 +62,17 @@ Three files:
Each backend is a plist registered into =cj/coverage-backends=:
#+begin_src emacs-lisp
-(:name 'elisp
- :detect (lambda () ...) ; non-nil if current project matches
- :run (lambda (cb) ...) ; kick off coverage build; invoke CB with LCOV path
- :lcov-path (lambda () ...)) ; where the LCOV lives (for re-reading without running)
+(:name 'elisp
+ :detect (lambda () ...) ; non-nil if current project matches
+ :run (lambda (cb) ...) ; kick off coverage build; invoke CB with report path
+ :report-path (lambda () ...)) ; where the simplecov JSON lives (for re-reading without running)
#+end_src
Detection precedence: =.dir-locals.el= override (=cj/coverage-backend= set to a backend name), then project-root fingerprints (=go.mod=, =pyproject.toml=, =package.json=, =.el= files + Makefile, etc.). First =:detect= that matches wins. No silent fallback — if nothing matches, the command errors with guidance.
*** Pure helpers
-- =cj/--coverage-parse-lcov FILE= → hash-table ={file → covered-line-set}=.
+- =cj/--coverage-parse-simplecov FILE= → hash-table ={file → covered-line-set}=.
- =cj/--coverage-changed-lines SCOPE BASE= → hash-table ={file → changed-line-set}= by shelling a =git diff --unified=0= for the selected scope and parsing hunk headers.
- =cj/--coverage-intersect COVERED CHANGED= → per-file records with three buckets: covered, uncovered, not-tracked.
@@ -85,13 +87,13 @@ All three are pure, fully ERT-tested.
- "Staged — about to commit"
- "Branch vs parent" (uses =cj/coverage-base-branch= → =@{upstream}= → =main= in order)
- "Branch vs main" (explicit)
-4. Freshness check: if =lcov.info= is missing, or older than the newest changed file, prompt "Run coverage now?" Yes runs the backend's =:run= asynchronously via =compile=; no reads the stale file anyway.
-5. Parse LCOV, compute changed lines, intersect.
+4. Freshness check: if =simplecov.json= is missing, or older than the newest changed file, prompt "Run coverage now?" Yes runs the backend's =:run= asynchronously via =compile=; no reads the stale file anyway.
+5. Parse simplecov, compute changed lines, intersect.
6. Display a report buffer in a mode derived from =compilation-mode=.
** Persistence
-- =.coverage/lcov.info= at the project root, gitignored. Overwritten on each run.
+- =.coverage/simplecov.json= at the project root, gitignored. Overwritten on each run.
- No long-term storage. Historical tracking is explicitly out of scope for v1.
** Error Handling
@@ -104,12 +106,12 @@ All three are pure, fully ERT-tested.
*During the coverage run:*
- Backend =:run= fails (test failure, Make error) → keep the =compile= buffer visible, do *not* proceed to display a report. Partial data is worse than no data.
-- Run completes but no LCOV produced → error naming the expected path.
+- Run completes but no simplecov.json produced → error naming the expected path.
*Post-flight classification:* three buckets, not two.
-- *Covered* — changed line in LCOV's covered-line set.
+- *Covered* — changed line in the simplecov covered-line set.
- *Uncovered* — changed line in a tracked file but not covered.
-- *Not tracked* — changed file isn't in LCOV at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code."
+- *Not tracked* — changed file isn't in the simplecov data at all (test files, READMEs, config). Reported separately — don't conflate "coverage didn't look here" with "tests didn't exercise this code."
*Happy-path degenerates:*
- Zero changed lines in scope → "No changes in this scope; nothing to report."
@@ -119,7 +121,7 @@ All three are pure, fully ERT-tested.
*Global:*
- =F7= → =cj/coverage-report= (prompts scope, shows report).
-- =C-u F7= → force re-run regardless of LCOV freshness.
+- =C-u F7= → force re-run regardless of report freshness.
*In the report buffer* (compilation-mode derived, most inherited for free):
- =RET= → jump to source under point.
@@ -136,7 +138,7 @@ The =F4=–=F7= developer block (compile+run, debug, test, coverage) gets its fu
** Testing
*Pure helpers, fully tested* (Normal / Boundary / Error for each):
-- =cj/--coverage-parse-lcov= — handcrafted LCOV fragments in temp files; empty, headers-only, spaces/unicode in filenames, malformed lines, missing =end_of_record=.
+- =cj/--coverage-parse-simplecov= — handcrafted simplecov JSON in temp files; empty object, all-null coverage arrays, spaces in filenames, multiple test-name keys unioned, malformed JSON.
- =cj/--coverage-changed-lines= — =cl-letf= over =shell-command-to-string= to return canned =git diff= output; single hunk, new-file hunk, deletion-only hunk, binary marker, no-diff case.
- =cj/--coverage-intersect= — pure table-in / table-out; covered ⊇ changed, unknown files, nil/empty inputs.
@@ -144,13 +146,13 @@ The =F4=–=F7= developer block (compile+run, debug, test, coverage) gets its fu
- =cj/coverage-backend-for-project ROOT= — synthetic temp project roots with marker files; assert correct backend. Registration-order test: two backends match, first-registered wins.
*Not tested:*
-- =cj/coverage-report= interactive command — one smoke test with a prepared LCOV and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display.
+- =cj/coverage-report= interactive command — one smoke test with a prepared simplecov report and a stubbed git-diff. No tests for the prompt UI or the compilation-buffer display.
- The elisp backend's =:run= function — shells to =make coverage=; integration-test-shaped, low value, slow. Skipped by design.
* Open Questions
- [ ] Which tests should a coverage run actually execute? All of them (simple, slow for 265 files), or only the test files whose target modules changed (fast, but dependent-test discovery in Elisp is non-trivial)? Deferred until implementation.
-- [ ] Default behavior when LCOV is stale but not missing: prompt, or auto-rerun? Current design prompts. Revisit after first use.
+- [ ] Default behavior when the simplecov report is stale but not missing: prompt, or auto-rerun? Current design prompts. Revisit after first use.
- [ ] Whether =cj/coverage-base-branch= should be a single value or a list of candidates (useful if you routinely stack PRs more than one level deep). Single value for v1.
* Next Steps