diff options
| author | Craig Jennings <c@cjennings.net> | 2026-05-31 12:31:35 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-05-31 12:31:35 -0500 |
| commit | af478a42b18c4d5e0712c4cb43036126d36c56b5 (patch) | |
| tree | 5ef37e4f4c0e1f749e1d3506dde4147036660854 | |
| parent | ddf48dc7ac780da1aacdff4e03f1d7da255b8f39 (diff) | |
| download | rulesets-af478a42b18c4d5e0712c4cb43036126d36c56b5.tar.gz rulesets-af478a42b18c4d5e0712c4cb43036126d36c56b5.zip | |
feat(python): add coverage-summary to the Python bundle
Second language in the coverage-summary fan-out, after the Elisp pilot. Same kernel: a module no test imports never appears in coverage.py's report, so a line-weighted total skips it silently and the suite looks healthier than it is. This counts every source file on disk that's absent from the report as 0% and weights the project number by file, so untested modules stay visible.
The script at languages/python/claude/scripts/coverage-summary.py parses coverage.py's JSON (files[path].summary.covered_lines / num_statements), resolves report paths against the report's directory since coverage records them relative to where it ran, and recurses the source dir for *.py. Unlike the Elisp version it doesn't print a per-file table, because coverage.py's own coverage report already does. The script adds the missing-file accounting that report lacks. It uses only the standard library, parsing the report rather than importing coverage.
The Python run confirmed the plumbing from the pilot is genuinely generic. install-lang and sync deliver the script and the project-owned coverage-makefile.txt with no Python-specific code. The one gap I had to close: the Python bundle shipped without a gitignore-add.txt, so the .claude/ footprint wasn't ignored and the script would have been committable. Added one mirroring the Elisp footprint plus Python artifacts (__pycache__, .coverage, coverage.json). make test gained a languages/*/tests/test_*.py discovery path alongside the existing Elisp ERT one.
Tests: 12 pytest covering the parser, the file-weighted number, and the missing-file detection including subpackage recursion, plus an install-lang check that the script lands in the gitignored footprint. I proved it against a report matching coverage.py's documented schema and the CLI end to end, but not against a live coverage json run, because coverage.py isn't installed in this repo's env. The first project to adopt it should sanity-check against a real report.
| -rw-r--r-- | Makefile | 5 | ||||
| -rw-r--r-- | languages/python/claude/rules/python-testing.md | 16 | ||||
| -rwxr-xr-x | languages/python/claude/scripts/coverage-summary.py | 157 | ||||
| -rw-r--r-- | languages/python/coverage-makefile.txt | 46 | ||||
| -rw-r--r-- | languages/python/gitignore-add.txt | 13 | ||||
| -rw-r--r-- | languages/python/tests/test_coverage_summary.py | 156 | ||||
| -rw-r--r-- | scripts/tests/install-lang.bats | 11 | ||||
| -rw-r--r-- | todo.org | 13 |
8 files changed, 411 insertions, 6 deletions
@@ -463,6 +463,11 @@ catchup-machine: ## Pull rulesets, refresh install, sync .ai/ across projects, v test: ## Run all test suites (pytest + ERT + bats) @cd .ai/scripts/tests && python3 -m pytest @cd hooks/tests && python3 -m pytest + @set -e; for d in languages/*/tests; do \ + ls "$$d"/test_*.py >/dev/null 2>&1 || continue; \ + echo "pytest: $$d"; \ + ( cd "$$d" && python3 -m pytest -q ); \ + done @set -e; for f in .ai/scripts/tests/test-*.el; do \ [ -e "$$f" ] || continue; \ echo "ert: $$(basename "$$f")"; \ diff --git a/languages/python/claude/rules/python-testing.md b/languages/python/claude/rules/python-testing.md index 4edde35..dedcce4 100644 --- a/languages/python/claude/rules/python-testing.md +++ b/languages/python/claude/rules/python-testing.md @@ -82,6 +82,22 @@ generation in-process. See `testing.md` § Combinatorial Coverage for the general rule and when to skip. +## Measuring Coverage — `make coverage-summary` + +The bundle ships a coverage summary at `.claude/scripts/coverage-summary.py` +and a Makefile fragment (`coverage-makefile.txt`) with `coverage` and +`coverage-summary` targets. After `make coverage` runs the suite under +coverage.py and writes a JSON report, `make coverage-summary` prints a +file-weighted project number and the source files no test imported. + +The number to watch is that missing-file count. A module no test imports never +appears in coverage.py's report, so a line-weighted total skips it silently and +the suite looks healthier than it is. The summary counts every `*.py` under the +source dir that's absent from the report as 0%, so an untested module drags the +project number down where you can see it. It doesn't reimplement the per-file +table — `coverage report` already prints that. Copy the fragment's targets into +your own Makefile to adopt it; the bundle never edits your Makefile. + ## Mocking Guidelines ### Mock these (external boundaries): diff --git a/languages/python/claude/scripts/coverage-summary.py b/languages/python/claude/scripts/coverage-summary.py new file mode 100755 index 0000000..a869fff --- /dev/null +++ b/languages/python/claude/scripts/coverage-summary.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Whole-project coverage summary from a coverage.py JSON report. + +Batch helper for `make coverage-summary`. After `make coverage` writes a +coverage.py JSON report (`coverage json`), this prints a file-weighted project +number and the source files present on disk but absent from the report. + +The value here is the missing-file detection: a module no test imports never +appears in coverage.py's output, so a line-weighted total silently skips it and +the suite looks healthier than it is. This counts such a file as 0% and weights +the project number by file rather than by line, so untested modules stay +visible. coverage.py's own `coverage report` prints the per-file table, so this +deliberately doesn't reimplement that — it adds the accounting `report` lacks. + +Self-contained on the standard library (json only); it parses the report, it +does not import coverage. The coverage.py JSON shape it reads is: + { "files": { "<path>": { "summary": { "covered_lines": N, + "num_statements": M, ... } } } } +Paths are relative to the directory coverage ran in (the project root). + +CLI: + coverage-summary.py REPORT SOURCE_DIR [PROJECT_ROOT] + REPORT path to the coverage.py JSON report + SOURCE_DIR the package/source directory to account for + PROJECT_ROOT defaults to the current directory +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import NoReturn + + +def _die(msg: str) -> NoReturn: + print(f"coverage-summary: {msg}", file=sys.stderr) + raise SystemExit(1) + + +def parse_report(report_path) -> dict: + """Parse a coverage.py JSON report into {abspath: (covered, total)}. + + COVERED is the file's covered_lines, TOTAL its num_statements. coverage.py + records paths relative to the directory it ran in, which is where the report + is written, so relative paths resolve against the report's parent. Exits with + a clear message when the report is missing or malformed. + """ + p = Path(report_path) + if not p.is_file(): + _die(f"coverage report not found: {p}") + base = p.resolve().parent + try: + data = json.loads(p.read_text()) + except (ValueError, OSError) as e: + _die(f"malformed coverage JSON in {p}: {e}") + result: dict = {} + for path, entry in (data.get("files") or {}).items(): + summary = entry.get("summary", {}) if isinstance(entry, dict) else {} + covered = int(summary.get("covered_lines", 0)) + total = int(summary.get("num_statements", 0)) + fp = Path(path) + abspath = fp.resolve() if fp.is_absolute() else (base / fp).resolve() + result[str(abspath)] = (covered, total) + return result + + +def under_dir(table: dict, source_dir, project_root) -> dict: + """Filter TABLE to files under SOURCE_DIR, re-keyed relative to PROJECT_ROOT.""" + source_dir = Path(source_dir).resolve() + project_root = Path(project_root).resolve() + result: dict = {} + for abspath, counts in table.items(): + ap = Path(abspath) + if source_dir == ap or source_dir in ap.parents: + result[str(ap.relative_to(project_root))] = counts + return result + + +def source_files(source_dir, project_root) -> list: + """Return *.py files under SOURCE_DIR (recursive), relative to PROJECT_ROOT. + + Packages nest, so this recurses; __pycache__ is skipped. The caller scopes + coverage by choosing SOURCE_DIR. + """ + source_dir = Path(source_dir).resolve() + project_root = Path(project_root).resolve() + out = [] + for p in source_dir.rglob("*.py"): + if "__pycache__" in p.parts: + continue + out.append(str(p.resolve().relative_to(project_root))) + return sorted(out) + + +def missing(tracked, source_dir, project_root) -> list: + """Source files on disk but absent from TRACKED (the report's keys).""" + tracked_set = set(tracked) + return [f for f in source_files(source_dir, project_root) if f not in tracked_set] + + +def file_pct(covered: int, total: int) -> float: + """COVERED/TOTAL as a percentage. A file with no statements is 100%.""" + return (100.0 * covered / total) if total > 0 else 100.0 + + +def project_pct(report_path, source_dir, project_root) -> float: + """File-weighted project coverage: every tracked file contributes its own + percentage, every source file missing from the report contributes 0%, and + the result is the mean over all files under SOURCE_DIR.""" + tracked = under_dir(parse_report(report_path), source_dir, project_root) + miss = missing(list(tracked), source_dir, project_root) + total_count = len(tracked) + len(miss) + if total_count == 0: + return 0.0 + score = sum(file_pct(c, t) for c, t in tracked.values()) + return score / total_count + + +def summary_text(report_path, source_dir, project_root) -> str: + """Render the project number and the missing-file list (not the per-file + table — `coverage report` already prints that).""" + tracked = under_dir(parse_report(report_path), source_dir, project_root) + miss = missing(list(tracked), source_dir, project_root) + pct = project_pct(report_path, source_dir, project_root) + rel_src = Path(source_dir).resolve().relative_to(Path(project_root).resolve()) + total = len(tracked) + len(miss) + lines = [ + f"Coverage summary for {rel_src}", + "", + f"Project coverage: {pct:.1f}% ({len(tracked)} tracked, {len(miss)} " + f"missing, {total} total; missing files count as 0%)", + "", + f"Not in coverage report: {len(miss)} file{'' if len(miss) == 1 else 's'}", + ] + if miss: + lines.append("These files had no coverage entry; they count as 0% in project coverage.") + lines += [f" {m}" for m in sorted(miss)] + else: + lines.append("Every source file appears in the coverage report.") + lines.append("") + lines.append("(Per-file table: run `coverage report`.)") + return "\n".join(lines) + "\n" + + +def main(argv) -> int: + if not 2 <= len(argv) <= 3: + print("usage: coverage-summary.py REPORT SOURCE_DIR [PROJECT_ROOT]", file=sys.stderr) + return 2 + report, source_dir = argv[0], argv[1] + project_root = argv[2] if len(argv) == 3 else "." + print() + print(summary_text(report, source_dir, project_root), end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/languages/python/coverage-makefile.txt b/languages/python/coverage-makefile.txt new file mode 100644 index 0000000..5764120 --- /dev/null +++ b/languages/python/coverage-makefile.txt @@ -0,0 +1,46 @@ +# Python coverage — Makefile fragment + setup recommendation +# +# This file is owned by the project, not the rulesets bundle. The bundle never +# edits your Makefile. Copy the two targets below into your own Makefile (and +# adjust the variables at the top), then delete this file or keep it as a note. +# +# What you get: +# make coverage runs the test suite under coverage.py, writes a JSON +# report, and prints coverage.py's own per-file table +# make coverage-summary prints a file-weighted project number and — the point +# — every source file on disk that no test imported, +# counted as 0%. +# +# Why the summary matters: a module no test imports never appears in coverage.py's +# output, so a line-weighted total silently skips it. The summary weights by file +# and counts a missing file as 0%, so untested modules stay visible. It does not +# reimplement the per-file table — `coverage report` already prints that. +# +# --------------------------------------------------------------------------- +# Prerequisite: coverage.py +# +# pip install coverage # or pytest-cov, if you prefer the pytest plugin +# +# The summary script itself needs nothing beyond the standard library — it parses +# the JSON report, it does not import coverage. +# --------------------------------------------------------------------------- + +# Variables — adjust to your layout. +PYTHON ?= python3 +SOURCE_DIR ?= src +COVERAGE_FILE ?= coverage.json +# The summary script ships with the bundle under .claude/scripts/ (gitignored). +COVERAGE_SUMMARY ?= .claude/scripts/coverage-summary.py + +coverage: + @$(PYTHON) -m coverage run --source=$(SOURCE_DIR) -m pytest + @$(PYTHON) -m coverage json -o $(COVERAGE_FILE) + @$(PYTHON) -m coverage report + @$(MAKE) coverage-summary + +coverage-summary: + @if [ ! -f $(COVERAGE_FILE) ]; then \ + echo "[!] No coverage file at $(COVERAGE_FILE). Run 'make coverage' first."; \ + exit 1; \ + fi + @$(PYTHON) $(COVERAGE_SUMMARY) $(COVERAGE_FILE) $(SOURCE_DIR) $(CURDIR) diff --git a/languages/python/gitignore-add.txt b/languages/python/gitignore-add.txt new file mode 100644 index 0000000..d380534 --- /dev/null +++ b/languages/python/gitignore-add.txt @@ -0,0 +1,13 @@ +# Claude Code — local tooling, delivered by install/sync, not committed +.claude/ +CLAUDE.md +githooks/ + +# Python bytecode + coverage artifacts (generated) +__pycache__/ +*.pyc +*.pyo +.coverage +coverage.json +htmlcov/ +.pytest_cache/ diff --git a/languages/python/tests/test_coverage_summary.py b/languages/python/tests/test_coverage_summary.py new file mode 100644 index 0000000..b65effd --- /dev/null +++ b/languages/python/tests/test_coverage_summary.py @@ -0,0 +1,156 @@ +"""Tests for the Python bundle coverage-summary kernel. + +The script under test lives at languages/python/claude/scripts/coverage-summary.py +and is loaded by path (hyphenated CLI name, not importable as a module). + +The kernel is the missing-file detection and the file-weighted project number; +coverage.py's own `coverage report` prints the per-file table, so this script +does not reimplement that. Tests build a throwaway package tree plus a faithful +coverage.py JSON report and assert the kernel's numbers against hand-computed +values, with Normal / Boundary / Error coverage per function. +""" +import importlib.util +import json +from pathlib import Path + +import pytest + +# Load the hyphenated CLI script by path. +_SCRIPT = Path(__file__).resolve().parents[1] / "claude" / "scripts" / "coverage-summary.py" +_spec = importlib.util.spec_from_file_location("coverage_summary", _SCRIPT) +cs = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(cs) + + +def write_report(root: Path, files: dict) -> Path: + """Write a coverage.py-shaped JSON report. + + files maps a project-relative path to (covered, total). Mirrors the real + schema: files[path] = {"summary": {"covered_lines", "num_statements", ...}}. + """ + payload = {"meta": {"version": "7.0"}, "files": {}, "totals": {}} + for rel, (cov, tot) in files.items(): + payload["files"][rel] = { + "executed_lines": list(range(1, cov + 1)), + "summary": { + "covered_lines": cov, + "num_statements": tot, + "percent_covered": (100.0 * cov / tot) if tot else 100.0, + "missing_lines": tot - cov, + "excluded_lines": 0, + }, + "missing_lines": list(range(cov + 1, tot + 1)), + "excluded_lines": [], + } + p = root / "coverage.json" + p.write_text(json.dumps(payload)) + return p + + +@pytest.fixture +def project(tmp_path): + """A package tree under src/ plus a place for the report. Returns (root, src).""" + src = tmp_path / "src" + (src / "sub").mkdir(parents=True) + return tmp_path, src + + +# --- parse: covered / total ------------------------------------------------ + +def test_parse_counts_covered_and_statements(project): + root, src = project + report = write_report(root, {"src/a.py": (3, 5)}) + table = cs.parse_report(report) + key = str((root / "src/a.py").resolve()) + assert table[key] == (3, 5) + + +def test_parse_missing_report_raises(project): + root, _ = project + with pytest.raises(SystemExit): + cs.parse_report(root / "nope.json") + + +def test_parse_malformed_json_raises(project): + root, _ = project + bad = root / "coverage.json" + bad.write_text("{not json") + with pytest.raises(SystemExit): + cs.parse_report(bad) + + +# --- file percentage ------------------------------------------------------- + +def test_file_pct_normal(): + assert cs.file_pct(1, 2) == 50.0 + + +def test_file_pct_no_statements_is_100(): + assert cs.file_pct(0, 0) == 100.0 + + +def test_file_pct_fully_covered(): + assert cs.file_pct(4, 4) == 100.0 + + +# --- missing-file detection (the kernel) ----------------------------------- + +def test_missing_finds_ondisk_file_absent_from_report(project): + root, src = project + (src / "tracked.py").write_text("x = 1\n") + (src / "untested.py").write_text("y = 2\n") + report = write_report(root, {"src/tracked.py": (1, 1)}) + table = cs.under_dir(cs.parse_report(report), src, root) + missing = cs.missing(list(table), src, root) + assert missing == ["src/untested.py"] + + +def test_missing_recurses_into_subpackages(project): + root, src = project + (src / "top.py").write_text("a = 1\n") + (src / "sub" / "deep.py").write_text("b = 2\n") + report = write_report(root, {"src/top.py": (1, 1)}) + table = cs.under_dir(cs.parse_report(report), src, root) + missing = cs.missing(list(table), src, root) + assert "src/sub/deep.py" in missing + + +def test_missing_empty_when_all_tracked(project): + root, src = project + (src / "a.py").write_text("x = 1\n") + report = write_report(root, {"src/a.py": (1, 1)}) + table = cs.under_dir(cs.parse_report(report), src, root) + assert cs.missing(list(table), src, root) == [] + + +# --- project number (file-weighted, missing as 0%) ------------------------- + +def test_project_pct_unit_weighted_with_missing_as_zero(project): + root, src = project + for name in ("full.py", "half.py", "untested.py"): + (src / name).write_text("x = 1\n") + report = write_report(root, {"src/full.py": (2, 2), "src/half.py": (1, 2)}) + # full=100, half=50, untested missing=0 -> (100+50+0)/3 = 50.0 + assert cs.project_pct(report, src, root) == 50.0 + + +def test_project_pct_no_missing(project): + root, src = project + for name in ("full.py", "half.py"): + (src / name).write_text("x = 1\n") + report = write_report(root, {"src/full.py": (2, 2), "src/half.py": (1, 2)}) + assert cs.project_pct(report, src, root) == 75.0 + + +# --- end-to-end text ------------------------------------------------------- + +def test_summary_text_reports_missing_and_project_number(project): + root, src = project + (src / "a.py").write_text("x = 1\n") + (src / "orphan.py").write_text("y = 2\n") + report = write_report(root, {"src/a.py": (1, 2)}) + text = cs.summary_text(report, src, root) + assert "orphan.py" in text + assert "0%" in text + # a.py = 50%, orphan missing = 0% -> 25.0% + assert "25.0%" in text diff --git a/scripts/tests/install-lang.bats b/scripts/tests/install-lang.bats index a26c3d5..a523186 100644 --- a/scripts/tests/install-lang.bats +++ b/scripts/tests/install-lang.bats @@ -57,3 +57,14 @@ teardown() { grep -qxF "*.elc" "$PROJECT/.gitignore" grep -qxF "*.eln" "$PROJECT/.gitignore" } + +@test "install-lang python: lands the coverage script in the gitignored footprint" { + run bash "$INSTALL_LANG" python "$PROJECT" + + [ "$status" -eq 0 ] + [ -f "$PROJECT/.claude/scripts/coverage-summary.py" ] + [ -f "$PROJECT/coverage-makefile.txt" ] + # The .claude footprint must be gitignored so the script isn't committed. + grep -qxF ".claude/" "$PROJECT/.gitignore" + grep -qxF "coverage.json" "$PROJECT/.gitignore" +} @@ -1156,18 +1156,19 @@ Reference (dotemacs): =scripts/coverage-summary.el=, =modules/coverage-core.el=, Origin: handoff from the .emacs.d session, 2026-05-25. -** TODO [#C] Fan out coverage-summary to Python, Go, and TypeScript bundles :feature: +** TODO [#C] Fan out coverage-summary to Go and TypeScript bundles :feature: :PROPERTIES: :CREATED: [2026-05-31 Sun] :END: -The Elisp pilot proved the pattern (see the DONE task above). Each remaining bundle needs its own ~40-line parser over that tool's report format, plus a =coverage-makefile.txt= fragment and the prereq harness where one is missing. The bundle plumbing is already generic: =sync-language-bundle.sh= auto-fixes any =claude/scripts/*= and inbox-drops any =coverage-makefile.txt=; =install-lang.sh= seeds the fragment; =make test= discovers =languages/*/tests/test-*.el=. So each language is just: the parser script, its tests, and the fragment. +The Elisp pilot proved the pattern; Python followed (both DONE above). Python confirmed the plumbing is genuinely generic — =sync-language-bundle.sh= auto-fixes any =claude/scripts/*= and inbox-drops any =coverage-makefile.txt=; =install-lang.sh= seeds the fragment; =make test= now discovers both =languages/*/tests/test-*.el= (ERT) and =languages/*/tests/test_*.py= (pytest). So Go and TS are each just: the parser script, its tests, and the fragment. -- Python: =coverage json= per-file JSON, or lean on =coverage report=. Missing-file detection over the package's =*.py= on disk. -- Go: =go test -coverprofile=cover.out=; parse =cover.out= (simple text), or =go tool cover -func=. -- TypeScript/JS: nyc/Istanbul =coverage-final.json= / json-summary. +- Go: =go test -coverprofile=cover.out=; parse =cover.out= (simple text), or =go tool cover -func=. Note Go has no =make test= discovery path yet — add a =go test= runner for =languages/go/tests= when this lands. +- TypeScript/JS: nyc/Istanbul =coverage-final.json= / json-summary. Needs a JS test-discovery path in =make test= too. + +Keep the kernel identical: file-weighted project number, source files absent from the report counted as 0%. Don't reimplement the per-file table where the built-in reporter already prints one — Go's =go tool cover -func= and nyc both do, so those scripts focus on the missing-file list and the project number. -Keep the kernel identical: file-weighted project number, source files absent from the report counted as 0%. Don't reimplement the per-file table where the built-in reporter already prints one — Python and JS both do, so those scripts can focus on the missing-file list and the project number. +Python notes for the next person: the script parses coverage.py's =files[path].summary.{covered_lines,num_statements}= (stable since coverage 5.x), resolves report paths against the report's parent dir (= project root), recurses the source dir for =*.py=, and was proven against a synthetic report matching the documented schema — not yet against a live =coverage json= run (coverage.py wasn't installed in the rulesets env). First real adopter should sanity-check against an actual report. ** TODO [#B] Cross-project pattern catalog :spec:thinking: :PROPERTIES: |
