docs: Add comprehensive test-reporter specification

Created detailed specification for test reporting system that will:
- Parse ERT batch output and generate test summaries
- Show total tests, passed, failed, duration
- List individual test failures with details
- Enable incremental improvement via TDD

Key Design Decisions:
- Language: Emacs Lisp (dogfooding, no dependencies, better integration)
- Architecture: Collect → Parse → Report
- Integration: Makefile + test-runner.el
- Test isolation: Clear tests on project switch

Implementation Phases:
- Phase 1: Basic stats (tests, passed, failed, duration)
- Phase 2: Failure details (messages, rerun commands)
- Phase 3: Rich details (timing, slowest tests, assertions)

Cross-Project Test Isolation:
- Problem: ERT tests globally registered across all projects
- Solution: Clear tests on project switch + project-aware commands
- Hybrid approach combining automatic and manual control

Integration Points:
- Makefile: Pipe output to test-reporter script
- test-runner.el: Post-run hooks or result wrapping
- ERT: Custom reporter (future enhancement)

Test-Driven Development:
- Parser tested with real ERT output samples
- Fixtures for edge cases
- Incremental improvement as new formats encountered

This is marked as NEXT PRIORITY TASK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 636 insertions, 0 deletions

diff --git a/test-reporter-spec.org b/test-reporter-spec.org
new file mode 100644
index 00000000..9a7ca949
--- /dev/null
+++ b/test-reporter-spec.org
@@ -0,0 +1,636 @@
+#+TITLE: Test Reporter Specification
+#+AUTHOR: Claude & Craig Jennings
+#+DATE: 2025-11-12
+#+FILETAGS: :spec:testing:
+
+* Overview
+
+A test reporting system that analyzes ERT batch test output and generates comprehensive summaries with statistics and failure details.
+
+* Goals
+
+1. Provide clear test statistics after running test suites
+2. Show total tests run, passed, failed, and duration
+3. List all individual test failures with details
+4. Enable incremental improvement through test-driven development
+5. Support both Makefile and interactive Emacs usage
+
+* Requirements
+
+** Must Have (Phase 1)
+- Total test count (not just file count)
+- Pass/fail counts with percentages
+- Total duration in seconds
+- List of failed test names
+- File-based output collection (all ERT output in one file)
+- Basic ERT output parsing
+
+** Should Have (Phase 2)
+- Detailed failure messages (assertion details)
+- Line numbers for failures
+- Test file grouping
+- Rerun command suggestions
+- Per-file statistics
+
+** Could Have (Phase 3)
+- Per-test timing
+- Slowest tests report
+- Historical comparison
+- JSON output for tooling
+- Integration with CI systems
+
+* Design Decisions
+
+** Language Choice: Emacs Lisp vs Python
+
+*** Emacs Lisp (RECOMMENDED)
+
+**** Advantages
+- No external dependencies (pure Emacs)
+- Native ERT understanding (can use ERT's own data structures)
+- Better integration with test-runner.el
+- Dogfooding (testing tool in same language as tests)
+- Can be tested with ERT itself (meta-testing!)
+- Philosophically aligned with Emacs-first workflow
+
+**** Disadvantages
+- String parsing less ergonomic than Python
+- Regex syntax clunkier
+- More verbose file I/O
+
+**** Difficulty: Medium
+
+*** Python Alternative
+
+**** Advantages
+- More ergonomic string parsing
+- Cleaner regex syntax
+- Simpler file I/O
+- Standard library text processing
+
+**** Disadvantages
+- External dependency (requires Python 3)
+- Not "native" to Emacs ecosystem
+- Feels less integrated
+- Separate testing framework needed
+
+**** Difficulty: Easy
+
+*** Decision: Emacs Lisp
+Slightly harder to write but better aligned with ecosystem and enables tighter integration with test-runner.el.
+
+** Architecture
+
+#+BEGIN_SRC
+Makefile/test-runner.el → Collect ERT Output → Parse Output → Generate Report
+                          (/tmp/test-run.txt)   (test-reporter.el)
+#+END_SRC
+
+*** Components
+
+1. *Output Collector* (Makefile or test-runner.el)
+   - Runs ERT tests
+   - Captures all output to single file
+   - Tracks start/end time
+
+2. *Parser* (test-reporter.el)
+   - Reads output file
+   - Parses ERT batch format
+   - Extracts statistics and failures
+
+3. *Reporter* (test-reporter.el)
+   - Formats summary report
+   - Displays to user
+   - Returns appropriate exit code
+
+* ERT Output Format
+
+** Summary Line Format
+#+BEGIN_EXAMPLE
+Ran 15 tests, 14 results as expected, 1 unexpected
+#+END_EXAMPLE
+
+** Test Progress Format
+#+BEGIN_EXAMPLE
+Running 15 tests (2024-11-12 10:30:45-0600, selector `(not (tag :slow))')
+   passed  1/15  test-foo-normal
+   passed  2/15  test-foo-boundary
+   FAILED  3/15  test-foo-error (0.001234 sec) at test-foo.el:42
+#+END_EXAMPLE
+
+** Failure Details Format
+#+BEGIN_EXAMPLE
+   FAILED  3/15  test-foo-error (0.001234 sec) at test-foo.el:42
+       (should (equal result expected))
+       :form (equal nil "expected")
+       :value nil
+       :explanation "Expected non-nil but was nil"
+#+END_EXAMPLE
+
+* Implementation Plan
+
+** Phase 1: Basic Statistics (MVP)
+
+*** Features
+- Parse "Ran X tests, Y expected, Z unexpected" lines
+- Sum across all test files
+- Calculate totals and percentages
+- Display formatted summary
+- Return exit code (0 = all pass, 1 = any failures)
+
+*** Files
+- scripts/test-reporter.el (executable Emacs script)
+- tests/test-test-reporter.el (ERT tests for reporter)
+- Makefile (integration)
+
+*** Sample Output
+#+BEGIN_EXAMPLE
+================================================================================
+TEST SUMMARY
+================================================================================
+Tests:    523 total
+Passed:   520 (99.4%)
+Failed:   3 (0.6%)
+Duration: 45s
+================================================================================
+#+END_EXAMPLE
+
+** Phase 2: Failure Details
+
+*** Features
+- Parse FAILED lines
+- Extract test names and files
+- Capture error messages
+- Group by file
+- Show rerun commands
+
+*** Sample Output
+#+BEGIN_EXAMPLE
+================================================================================
+TEST SUMMARY
+================================================================================
+Tests:    523 total
+Passed:   520 (99.4%)
+Failed:   3 (0.6%)
+Duration: 45s
+
+FAILURES:
+  test-custom-buffer-file.el
+    ✗ test-copy-buffer-empty-buffer
+      Expected non-nil but was nil
+
+  test-org-agenda-build-list.el
+    ✗ test-cache-invalidation
+      Wrong type argument: listp, "string"
+    ✗ test-cache-update
+      Timeout after 5s
+
+RERUN FAILED:
+  make test-file FILE=test-custom-buffer-file.el
+  make test-file FILE=test-org-agenda-build-list.el
+================================================================================
+#+END_EXAMPLE
+
+** Phase 3: Rich Details
+
+*** Features
+- Parse assertion details (:form, :value, :explanation)
+- Extract line numbers
+- Show context around failures
+- Per-test timing
+- Slowest tests report
+
+* Integration Points
+
+** Makefile Integration
+
+*** Current Approach
+#+BEGIN_SRC makefile
+test-unit:
+	@echo "[i] Running unit tests ($(words $(UNIT_TESTS)) files)..."
+	@failed=0; \
+	for test in $(UNIT_TESTS); do \
+		echo "  Testing $$test..."; \
+		$(EMACS_TEST) -l ert -l $$test \
+			--eval "(ert-run-tests-batch-and-exit '(not (tag :slow)))" \
+			|| failed=$$((failed + 1)); \
+	done; \
+	if [ $$failed -eq 0 ]; then \
+		echo "✓ All unit tests passed"; \
+	else \
+		echo "✗ $$failed unit test file(s) failed"; \
+		exit 1; \
+	fi
+#+END_SRC
+
+*** Proposed Approach
+#+BEGIN_SRC makefile
+test-unit:
+	@echo "[i] Running unit tests ($(words $(UNIT_TESTS)) files)..."
+	@output_file="/tmp/emacs-test-run-$$.txt"; \
+	start_time=$$(date +%s); \
+	failed=0; \
+	for test in $(UNIT_TESTS); do \
+		echo "  Testing $$test..."; \
+		$(EMACS_TEST) -l ert -l $$test \
+			--eval "(ert-run-tests-batch-and-exit '(not (tag :slow)))" \
+			2>&1 | tee -a $$output_file || failed=$$((failed + 1)); \
+	done; \
+	end_time=$$(date +%s); \
+	duration=$$((end_time - start_time)); \
+	$(EMACS) --script scripts/test-reporter.el $$output_file $$duration; \
+	rm -f $$output_file; \
+	exit $$failed
+#+END_SRC
+
+** test-runner.el Integration
+
+*** Current Functionality
+- Interactive test running from Emacs
+- Focus/unfocus test patterns
+- Run all tests, focused tests, or specific files
+- Display results in *ert* buffer
+
+*** Integration Opportunities
+
+**** Option A: Post-Run Hook
+#+BEGIN_SRC elisp
+(defcustom cj/test-runner-report-hook nil
+  "Hook run after test execution with results.
+Functions receive (total-tests passed failed duration)."
+  :type 'hook
+  :group 'test-runner)
+
+(defun cj/test-runner--run-with-reporting ()
+  "Run tests and generate report."
+  (let ((start-time (current-time))
+        (output-buffer (generate-new-buffer " *test-output*")))
+    (unwind-protect
+        (progn
+          ;; Redirect ERT output to buffer
+          (let ((standard-output output-buffer))
+            (ert-run-tests-batch selector))
+          ;; Parse output
+          (let ((stats (test-reporter-parse-buffer output-buffer)))
+            (test-reporter-display-summary stats)
+            (run-hook-with-args 'cj/test-runner-report-hook stats)))
+      (kill-buffer output-buffer))))
+#+END_SRC
+
+**** Option B: Wrap ERT Results
+#+BEGIN_SRC elisp
+(defun cj/test-runner-display-summary ()
+  "Display test summary after running tests."
+  (interactive)
+  (when (get-buffer "*ert*")
+    (with-current-buffer "*ert*"
+      (let ((stats (test-reporter-parse-buffer (current-buffer))))
+        (goto-char (point-max))
+        (insert "\n" (test-reporter-format-summary stats))))))
+#+END_SRC
+
+**** Option C: Replace ERT Reporter
+- Implement custom ERT result printer
+- More invasive but cleanest integration
+- See `ert-batch-print` and `ert-batch-backtrace-right-margin`
+
+*** Recommendation
+Start with Option B (wrap results), migrate to Option A (hooks) once stable.
+
+** Cross-Project Test Isolation
+
+*** The Problem
+When switching between Emacs Lisp projects (e.g., ~/.emacs.d → chime.el):
+- ERT tests globally registered in Emacs session
+- ~M-x ert RET t RET~ runs ALL loaded tests from ALL projects
+- Can accidentally run wrong project's tests
+- Current workaround: restart Emacs (loses session state)
+
+*** Root Cause
+#+BEGIN_SRC elisp
+;; ERT stores tests globally
+(defvar ert--test-registry (make-hash-table :test 'equal))
+
+;; Tests registered at load-time
+(ert-deftest my-test () ...)
+;; → Adds to ert--test-registry immediately
+#+END_SRC
+
+*** Proposed Solutions
+
+**** Solution A: Clear Tests on Project Switch
+#+BEGIN_SRC elisp
+(defun cj/ert-clear-all-tests ()
+  "Clear all registered ERT tests."
+  (interactive)
+  (clrhash ert--test-registry)
+  (message "Cleared all ERT tests"))
+
+(defun cj/ert-clear-project-tests (prefix)
+  "Clear ERT tests matching PREFIX (e.g., 'cj/' or 'chime-')."
+  (interactive "sTest prefix to clear: ")
+  (maphash (lambda (name test)
+             (when (string-prefix-p prefix (symbol-name name))
+               (remhash name ert--test-registry)))
+           ert--test-registry)
+  (message "Cleared tests matching '%s'" prefix))
+
+;; Hook into project-switch
+(add-hook 'project-switch-hook #'cj/ert-clear-all-tests)
+#+END_SRC
+
+**** Solution B: Project-Aware Test Selection
+#+BEGIN_SRC elisp
+(defun cj/ert-run-current-project-tests ()
+  "Run only tests for current project."
+  (interactive)
+  (let* ((project-root (project-root (project-current)))
+         (test-prefix (cj/ert--infer-test-prefix project-root)))
+    (ert-run-tests-interactively
+     (lambda (test)
+       (string-prefix-p test-prefix (symbol-name test))))))
+
+(defun cj/ert--infer-test-prefix (project-root)
+  "Infer test prefix from project root."
+  (cond
+   ((string-match-p "\.emacs\.d" project-root) "cj/")
+   ((string-match-p "chime" project-root) "chime-")
+   (t (read-string "Test prefix: "))))
+#+END_SRC
+
+**** Solution C: Namespace-Based Isolation
+#+BEGIN_SRC elisp
+;; Store current project context
+(defvar-local cj/test-project-context nil
+  "Current project context for test filtering.")
+
+(defun cj/ert-set-project-context ()
+  "Set test project context based on current buffer."
+  (setq cj/test-project-context
+        (file-name-nondirectory
+         (directory-file-name (project-root (project-current))))))
+
+;; Filter tests by context
+(defun cj/ert-run-project-tests ()
+  "Run tests for current project context only."
+  (interactive)
+  (cj/ert-set-project-context)
+  (ert-run-tests-interactively
+   `(tag ,(intern cj/test-project-context))))
+
+;; Require tests to declare project
+(ert-deftest chime-my-test ()
+  :tags '(:chime)
+  ...)
+#+END_SRC
+
+*** Recommendation
+*Solution A + B hybrid*:
+1. Clear tests automatically on project switch (Solution A)
+2. Provide explicit "run current project tests" command (Solution B)
+3. Document test naming conventions (cj/ prefix for .emacs.d, etc.)
+4. Add keybinding: ~C-c C-t p~ (run project tests)
+
+This avoids requiring test tags while providing both automatic and manual control.
+
+* Test-Driven Development Approach
+
+** Philosophy
+- Parser gets smarter incrementally
+- Each new ERT output format → new test case → improved parser
+- Regression prevention through comprehensive test suite
+
+** Test Organization
+#+BEGIN_SRC
+tests/test-reporter/
+├── test-test-reporter-basic.el      # Basic parsing tests
+├── test-test-reporter-failures.el   # Failure detail parsing
+├── test-test-reporter-edge-cases.el # Edge cases
+└── fixtures/
+    ├── all-pass.txt                 # Sample ERT output: all pass
+    ├── all-fail.txt                 # Sample ERT output: all fail
+    ├── mixed.txt                    # Sample ERT output: mixed
+    └── malformed.txt                # Sample ERT output: malformed
+#+END_SRC
+
+** Sample Test Cases
+#+BEGIN_SRC elisp
+(ert-deftest test-reporter-parse-summary-line ()
+  "Parse 'Ran X tests, Y expected, Z unexpected' line."
+  (let ((result (test-reporter--parse-summary-line
+                 "Ran 15 tests, 14 results as expected, 1 unexpected")))
+    (should (= 15 (plist-get result :total)))
+    (should (= 14 (plist-get result :passed)))
+    (should (= 1 (plist-get result :failed)))))
+
+(ert-deftest test-reporter-parse-all-pass ()
+  "Parse output with all tests passing."
+  (let ((stats (test-reporter--parse-fixture "all-pass.txt")))
+    (should (= 0 (plist-get stats :failed)))
+    (should (null (plist-get stats :failures)))))
+
+(ert-deftest test-reporter-parse-with-failures ()
+  "Parse output with test failures."
+  (let ((stats (test-reporter--parse-fixture "mixed.txt")))
+    (should (> (plist-get stats :failed) 0))
+    (should (> (length (plist-get stats :failures)) 0))))
+
+(ert-deftest test-reporter-extract-failure-details ()
+  "Extract test name and error message from FAILED line."
+  (let ((failure (test-reporter--parse-failure
+                  "   FAILED  3/15  test-foo-error (0.001234 sec)")))
+    (should (string= "test-foo-error" (plist-get failure :test)))
+    (should (numberp (plist-get failure :index)))
+    (should (numberp (plist-get failure :duration)))))
+#+END_SRC
+
+** Incremental Development Flow
+1. Write test with sample ERT output
+2. Run test (should fail)
+3. Implement parser feature
+4. Run test (should pass)
+5. Encounter new format in real usage
+6. Add new test case with that format
+7. Improve parser
+8. Repeat
+
+* Implementation Details
+
+** File Structure
+
+*** scripts/test-reporter.el
+#+BEGIN_SRC elisp
+#!/usr/bin/env emacs --script
+;;; test-reporter.el --- Parse ERT output and generate test reports -*- lexical-binding: t; -*-
+
+;;; Commentary:
+;; Parses ERT batch test output and generates summary reports.
+;; Usage: emacs --script test-reporter.el OUTPUT_FILE DURATION
+
+;;; Code:
+
+(defun test-reporter--parse-summary-line (line)
+  "Parse 'Ran X tests, Y expected, Z unexpected' line."
+  ...)
+
+(defun test-reporter--parse-failure (line)
+  "Parse FAILED line and extract test details."
+  ...)
+
+(defun test-reporter--parse-output-file (file)
+  "Parse ERT output FILE and return stats plist."
+  ...)
+
+(defun test-reporter--format-summary (stats duration)
+  "Format summary report from STATS and DURATION."
+  ...)
+
+;; Main entry point
+(let ((output-file (nth 0 command-line-args-left))
+      (duration (string-to-number (nth 1 command-line-args-left))))
+  (let ((stats (test-reporter--parse-output-file output-file)))
+    (princ (test-reporter--format-summary stats duration))
+    (kill-emacs (if (> (plist-get stats :failed) 0) 1 0))))
+
+(provide 'test-reporter)
+;;; test-reporter.el ends here
+#+END_SRC
+
+*** tests/test-test-reporter.el
+#+BEGIN_SRC elisp
+;;; test-test-reporter.el --- Tests for test-reporter -*- lexical-binding: t; -*-
+
+(require 'ert)
+(load-file "scripts/test-reporter.el")
+
+(ert-deftest test-reporter-parse-summary-line ()
+  ...)
+
+(provide 'test-test-reporter)
+;;; test-test-reporter.el ends here
+#+END_SRC
+
+** API Design
+
+*** Core Functions
+
+#+BEGIN_SRC elisp
+(defun test-reporter--parse-summary-line (line)
+  "Parse summary line, return (:total N :passed N :failed N) or nil."
+  ...)
+
+(defun test-reporter--parse-failure (line)
+  "Parse failure line, return (:test NAME :index N :duration N) or nil."
+  ...)
+
+(defun test-reporter--parse-output-file (file)
+  "Parse entire output file, return complete stats plist."
+  ...)
+
+(defun test-reporter--format-summary (stats duration)
+  "Format human-readable summary from stats."
+  ...)
+
+(defun test-reporter-parse-buffer (buffer)
+  "Parse test output from BUFFER (for interactive use)."
+  ...)
+
+(defun test-reporter-display-summary (stats)
+  "Display summary in current buffer or separate window."
+  ...)
+#+END_SRC
+
+*** Data Structures
+
+#+BEGIN_SRC elisp
+;; Stats plist format
+(:total 523
+ :passed 520
+ :failed 3
+ :failures ((:test "test-foo"
+             :file "test-foo.el"
+             :line 42
+             :message "Expected non-nil but was nil")
+            ...))
+
+;; Failure plist format
+(:test "test-name"
+ :file "test-file.el"
+ :line 42
+ :index 3
+ :duration 0.001234
+ :message "Error message"
+ :form "(should (equal x y))"
+ :value "nil")
+#+END_SRC
+
+* Success Metrics
+
+** Phase 1 Success Criteria
+- ✓ Total test count displayed (not just file count)
+- ✓ Pass/fail counts with percentages
+- ✓ Duration in whole seconds
+- ✓ Works from Makefile
+- ✓ Exits with correct code (0=pass, 1=fail)
+- ✓ Basic test coverage (>80%)
+
+** Phase 2 Success Criteria
+- ✓ Failed test names listed
+- ✓ Error messages displayed
+- ✓ Grouped by file
+- ✓ Rerun commands suggested
+- ✓ Comprehensive test coverage (>90%)
+
+** Phase 3 Success Criteria
+- ✓ Per-test timing
+- ✓ Slowest tests identified
+- ✓ Rich assertion details
+- ✓ JSON output option
+- ✓ Full test coverage (100%)
+
+* Next Steps
+
+1. Create scripts/test-reporter.el with Phase 1 implementation
+2. Create tests/test-test-reporter.el with basic test cases
+3. Update Makefile to use test-reporter
+4. Test with real test suite
+5. Iterate based on real-world output formats
+6. Expand to Phase 2 features
+7. Integrate with test-runner.el
+
+* Open Questions
+
+1. Should we support both interactive (buffer) and batch (file) modes?
+   - Leaning YES - useful for both Makefile and test-runner.el
+
+2. Should we cache parsed results for re-display?
+   - Leaning NO - parsing is fast enough, keep it simple
+
+3. Should we support custom output formats (JSON, XML)?
+   - Leaning LATER - start with human-readable, add later if needed
+
+4. Should we track historical test results?
+   - Leaning LATER - interesting but out of scope for MVP
+
+5. How to handle test isolation across projects?
+   - Use Solution A+B: Clear on switch + project-aware run command
+
+* References
+
+- ERT documentation: [[info:ert#Top]]
+- modules/test-runner.el (current test infrastructure)
+- Makefile (current test execution)
+- ai-prompts/quality-engineer.org (testing philosophy)
+
+* Meta
+
+This specification was created: 2025-11-12
+This is the NEXT item to work on after current session.
+
+** Remember
+When returning to this task:
+1. Read this spec thoroughly
+2. Start with Phase 1 implementation
+3. Use TDD approach (tests first)
+4. Integrate incrementally
+5. Keep it simple - don't over-engineer