aboutsummaryrefslogtreecommitdiff
path: root/languages/python
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-05-31 12:31:35 -0500
committerCraig Jennings <c@cjennings.net>2026-05-31 12:31:35 -0500
commitaf478a42b18c4d5e0712c4cb43036126d36c56b5 (patch)
tree5ef37e4f4c0e1f749e1d3506dde4147036660854 /languages/python
parentddf48dc7ac780da1aacdff4e03f1d7da255b8f39 (diff)
downloadrulesets-af478a42b18c4d5e0712c4cb43036126d36c56b5.tar.gz
rulesets-af478a42b18c4d5e0712c4cb43036126d36c56b5.zip
feat(python): add coverage-summary to the Python bundle
Second language in the coverage-summary fan-out, after the Elisp pilot. Same kernel: a module no test imports never appears in coverage.py's report, so a line-weighted total skips it silently and the suite looks healthier than it is. This counts every source file on disk that's absent from the report as 0% and weights the project number by file, so untested modules stay visible. The script at languages/python/claude/scripts/coverage-summary.py parses coverage.py's JSON (files[path].summary.covered_lines / num_statements), resolves report paths against the report's directory since coverage records them relative to where it ran, and recurses the source dir for *.py. Unlike the Elisp version it doesn't print a per-file table, because coverage.py's own coverage report already does. The script adds the missing-file accounting that report lacks. It uses only the standard library, parsing the report rather than importing coverage. The Python run confirmed the plumbing from the pilot is genuinely generic. install-lang and sync deliver the script and the project-owned coverage-makefile.txt with no Python-specific code. The one gap I had to close: the Python bundle shipped without a gitignore-add.txt, so the .claude/ footprint wasn't ignored and the script would have been committable. Added one mirroring the Elisp footprint plus Python artifacts (__pycache__, .coverage, coverage.json). make test gained a languages/*/tests/test_*.py discovery path alongside the existing Elisp ERT one. Tests: 12 pytest covering the parser, the file-weighted number, and the missing-file detection including subpackage recursion, plus an install-lang check that the script lands in the gitignored footprint. I proved it against a report matching coverage.py's documented schema and the CLI end to end, but not against a live coverage json run, because coverage.py isn't installed in this repo's env. The first project to adopt it should sanity-check against a real report.
Diffstat (limited to 'languages/python')
-rw-r--r--languages/python/claude/rules/python-testing.md16
-rwxr-xr-xlanguages/python/claude/scripts/coverage-summary.py157
-rw-r--r--languages/python/coverage-makefile.txt46
-rw-r--r--languages/python/gitignore-add.txt13
-rw-r--r--languages/python/tests/test_coverage_summary.py156
5 files changed, 388 insertions, 0 deletions
diff --git a/languages/python/claude/rules/python-testing.md b/languages/python/claude/rules/python-testing.md
index 4edde35..dedcce4 100644
--- a/languages/python/claude/rules/python-testing.md
+++ b/languages/python/claude/rules/python-testing.md
@@ -82,6 +82,22 @@ generation in-process.
See `testing.md` § Combinatorial Coverage for the general rule and when
to skip.
+## Measuring Coverage — `make coverage-summary`
+
+The bundle ships a coverage summary at `.claude/scripts/coverage-summary.py`
+and a Makefile fragment (`coverage-makefile.txt`) with `coverage` and
+`coverage-summary` targets. After `make coverage` runs the suite under
+coverage.py and writes a JSON report, `make coverage-summary` prints a
+file-weighted project number and the source files no test imported.
+
+The number to watch is that missing-file count. A module no test imports never
+appears in coverage.py's report, so a line-weighted total skips it silently and
+the suite looks healthier than it is. The summary counts every `*.py` under the
+source dir that's absent from the report as 0%, so an untested module drags the
+project number down where you can see it. It doesn't reimplement the per-file
+table — `coverage report` already prints that. Copy the fragment's targets into
+your own Makefile to adopt it; the bundle never edits your Makefile.
+
## Mocking Guidelines
### Mock these (external boundaries):
diff --git a/languages/python/claude/scripts/coverage-summary.py b/languages/python/claude/scripts/coverage-summary.py
new file mode 100755
index 0000000..a869fff
--- /dev/null
+++ b/languages/python/claude/scripts/coverage-summary.py
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Whole-project coverage summary from a coverage.py JSON report.
+
+Batch helper for `make coverage-summary`. After `make coverage` writes a
+coverage.py JSON report (`coverage json`), this prints a file-weighted project
+number and the source files present on disk but absent from the report.
+
+The value here is the missing-file detection: a module no test imports never
+appears in coverage.py's output, so a line-weighted total silently skips it and
+the suite looks healthier than it is. This counts such a file as 0% and weights
+the project number by file rather than by line, so untested modules stay
+visible. coverage.py's own `coverage report` prints the per-file table, so this
+deliberately doesn't reimplement that — it adds the accounting `report` lacks.
+
+Self-contained on the standard library (json only); it parses the report, it
+does not import coverage. The coverage.py JSON shape it reads is:
+ { "files": { "<path>": { "summary": { "covered_lines": N,
+ "num_statements": M, ... } } } }
+Paths are relative to the directory coverage ran in (the project root).
+
+CLI:
+ coverage-summary.py REPORT SOURCE_DIR [PROJECT_ROOT]
+ REPORT path to the coverage.py JSON report
+ SOURCE_DIR the package/source directory to account for
+ PROJECT_ROOT defaults to the current directory
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+from typing import NoReturn
+
+
+def _die(msg: str) -> NoReturn:
+ print(f"coverage-summary: {msg}", file=sys.stderr)
+ raise SystemExit(1)
+
+
+def parse_report(report_path) -> dict:
+ """Parse a coverage.py JSON report into {abspath: (covered, total)}.
+
+ COVERED is the file's covered_lines, TOTAL its num_statements. coverage.py
+ records paths relative to the directory it ran in, which is where the report
+ is written, so relative paths resolve against the report's parent. Exits with
+ a clear message when the report is missing or malformed.
+ """
+ p = Path(report_path)
+ if not p.is_file():
+ _die(f"coverage report not found: {p}")
+ base = p.resolve().parent
+ try:
+ data = json.loads(p.read_text())
+ except (ValueError, OSError) as e:
+ _die(f"malformed coverage JSON in {p}: {e}")
+ result: dict = {}
+ for path, entry in (data.get("files") or {}).items():
+ summary = entry.get("summary", {}) if isinstance(entry, dict) else {}
+ covered = int(summary.get("covered_lines", 0))
+ total = int(summary.get("num_statements", 0))
+ fp = Path(path)
+ abspath = fp.resolve() if fp.is_absolute() else (base / fp).resolve()
+ result[str(abspath)] = (covered, total)
+ return result
+
+
+def under_dir(table: dict, source_dir, project_root) -> dict:
+ """Filter TABLE to files under SOURCE_DIR, re-keyed relative to PROJECT_ROOT."""
+ source_dir = Path(source_dir).resolve()
+ project_root = Path(project_root).resolve()
+ result: dict = {}
+ for abspath, counts in table.items():
+ ap = Path(abspath)
+ if source_dir == ap or source_dir in ap.parents:
+ result[str(ap.relative_to(project_root))] = counts
+ return result
+
+
+def source_files(source_dir, project_root) -> list:
+ """Return *.py files under SOURCE_DIR (recursive), relative to PROJECT_ROOT.
+
+ Packages nest, so this recurses; __pycache__ is skipped. The caller scopes
+ coverage by choosing SOURCE_DIR.
+ """
+ source_dir = Path(source_dir).resolve()
+ project_root = Path(project_root).resolve()
+ out = []
+ for p in source_dir.rglob("*.py"):
+ if "__pycache__" in p.parts:
+ continue
+ out.append(str(p.resolve().relative_to(project_root)))
+ return sorted(out)
+
+
+def missing(tracked, source_dir, project_root) -> list:
+ """Source files on disk but absent from TRACKED (the report's keys)."""
+ tracked_set = set(tracked)
+ return [f for f in source_files(source_dir, project_root) if f not in tracked_set]
+
+
+def file_pct(covered: int, total: int) -> float:
+ """COVERED/TOTAL as a percentage. A file with no statements is 100%."""
+ return (100.0 * covered / total) if total > 0 else 100.0
+
+
+def project_pct(report_path, source_dir, project_root) -> float:
+ """File-weighted project coverage: every tracked file contributes its own
+ percentage, every source file missing from the report contributes 0%, and
+ the result is the mean over all files under SOURCE_DIR."""
+ tracked = under_dir(parse_report(report_path), source_dir, project_root)
+ miss = missing(list(tracked), source_dir, project_root)
+ total_count = len(tracked) + len(miss)
+ if total_count == 0:
+ return 0.0
+ score = sum(file_pct(c, t) for c, t in tracked.values())
+ return score / total_count
+
+
+def summary_text(report_path, source_dir, project_root) -> str:
+ """Render the project number and the missing-file list (not the per-file
+ table — `coverage report` already prints that)."""
+ tracked = under_dir(parse_report(report_path), source_dir, project_root)
+ miss = missing(list(tracked), source_dir, project_root)
+ pct = project_pct(report_path, source_dir, project_root)
+ rel_src = Path(source_dir).resolve().relative_to(Path(project_root).resolve())
+ total = len(tracked) + len(miss)
+ lines = [
+ f"Coverage summary for {rel_src}",
+ "",
+ f"Project coverage: {pct:.1f}% ({len(tracked)} tracked, {len(miss)} "
+ f"missing, {total} total; missing files count as 0%)",
+ "",
+ f"Not in coverage report: {len(miss)} file{'' if len(miss) == 1 else 's'}",
+ ]
+ if miss:
+ lines.append("These files had no coverage entry; they count as 0% in project coverage.")
+ lines += [f" {m}" for m in sorted(miss)]
+ else:
+ lines.append("Every source file appears in the coverage report.")
+ lines.append("")
+ lines.append("(Per-file table: run `coverage report`.)")
+ return "\n".join(lines) + "\n"
+
+
+def main(argv) -> int:
+ if not 2 <= len(argv) <= 3:
+ print("usage: coverage-summary.py REPORT SOURCE_DIR [PROJECT_ROOT]", file=sys.stderr)
+ return 2
+ report, source_dir = argv[0], argv[1]
+ project_root = argv[2] if len(argv) == 3 else "."
+ print()
+ print(summary_text(report, source_dir, project_root), end="")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main(sys.argv[1:]))
diff --git a/languages/python/coverage-makefile.txt b/languages/python/coverage-makefile.txt
new file mode 100644
index 0000000..5764120
--- /dev/null
+++ b/languages/python/coverage-makefile.txt
@@ -0,0 +1,46 @@
+# Python coverage — Makefile fragment + setup recommendation
+#
+# This file is owned by the project, not the rulesets bundle. The bundle never
+# edits your Makefile. Copy the two targets below into your own Makefile (and
+# adjust the variables at the top), then delete this file or keep it as a note.
+#
+# What you get:
+# make coverage runs the test suite under coverage.py, writes a JSON
+# report, and prints coverage.py's own per-file table
+# make coverage-summary prints a file-weighted project number and — the point
+# — every source file on disk that no test imported,
+# counted as 0%.
+#
+# Why the summary matters: a module no test imports never appears in coverage.py's
+# output, so a line-weighted total silently skips it. The summary weights by file
+# and counts a missing file as 0%, so untested modules stay visible. It does not
+# reimplement the per-file table — `coverage report` already prints that.
+#
+# ---------------------------------------------------------------------------
+# Prerequisite: coverage.py
+#
+# pip install coverage # or pytest-cov, if you prefer the pytest plugin
+#
+# The summary script itself needs nothing beyond the standard library — it parses
+# the JSON report, it does not import coverage.
+# ---------------------------------------------------------------------------
+
+# Variables — adjust to your layout.
+PYTHON ?= python3
+SOURCE_DIR ?= src
+COVERAGE_FILE ?= coverage.json
+# The summary script ships with the bundle under .claude/scripts/ (gitignored).
+COVERAGE_SUMMARY ?= .claude/scripts/coverage-summary.py
+
+coverage:
+ @$(PYTHON) -m coverage run --source=$(SOURCE_DIR) -m pytest
+ @$(PYTHON) -m coverage json -o $(COVERAGE_FILE)
+ @$(PYTHON) -m coverage report
+ @$(MAKE) coverage-summary
+
+coverage-summary:
+ @if [ ! -f $(COVERAGE_FILE) ]; then \
+ echo "[!] No coverage file at $(COVERAGE_FILE). Run 'make coverage' first."; \
+ exit 1; \
+ fi
+ @$(PYTHON) $(COVERAGE_SUMMARY) $(COVERAGE_FILE) $(SOURCE_DIR) $(CURDIR)
diff --git a/languages/python/gitignore-add.txt b/languages/python/gitignore-add.txt
new file mode 100644
index 0000000..d380534
--- /dev/null
+++ b/languages/python/gitignore-add.txt
@@ -0,0 +1,13 @@
+# Claude Code — local tooling, delivered by install/sync, not committed
+.claude/
+CLAUDE.md
+githooks/
+
+# Python bytecode + coverage artifacts (generated)
+__pycache__/
+*.pyc
+*.pyo
+.coverage
+coverage.json
+htmlcov/
+.pytest_cache/
diff --git a/languages/python/tests/test_coverage_summary.py b/languages/python/tests/test_coverage_summary.py
new file mode 100644
index 0000000..b65effd
--- /dev/null
+++ b/languages/python/tests/test_coverage_summary.py
@@ -0,0 +1,156 @@
+"""Tests for the Python bundle coverage-summary kernel.
+
+The script under test lives at languages/python/claude/scripts/coverage-summary.py
+and is loaded by path (hyphenated CLI name, not importable as a module).
+
+The kernel is the missing-file detection and the file-weighted project number;
+coverage.py's own `coverage report` prints the per-file table, so this script
+does not reimplement that. Tests build a throwaway package tree plus a faithful
+coverage.py JSON report and assert the kernel's numbers against hand-computed
+values, with Normal / Boundary / Error coverage per function.
+"""
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+
+# Load the hyphenated CLI script by path.
+_SCRIPT = Path(__file__).resolve().parents[1] / "claude" / "scripts" / "coverage-summary.py"
+_spec = importlib.util.spec_from_file_location("coverage_summary", _SCRIPT)
+cs = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(cs)
+
+
+def write_report(root: Path, files: dict) -> Path:
+ """Write a coverage.py-shaped JSON report.
+
+ files maps a project-relative path to (covered, total). Mirrors the real
+ schema: files[path] = {"summary": {"covered_lines", "num_statements", ...}}.
+ """
+ payload = {"meta": {"version": "7.0"}, "files": {}, "totals": {}}
+ for rel, (cov, tot) in files.items():
+ payload["files"][rel] = {
+ "executed_lines": list(range(1, cov + 1)),
+ "summary": {
+ "covered_lines": cov,
+ "num_statements": tot,
+ "percent_covered": (100.0 * cov / tot) if tot else 100.0,
+ "missing_lines": tot - cov,
+ "excluded_lines": 0,
+ },
+ "missing_lines": list(range(cov + 1, tot + 1)),
+ "excluded_lines": [],
+ }
+ p = root / "coverage.json"
+ p.write_text(json.dumps(payload))
+ return p
+
+
+@pytest.fixture
+def project(tmp_path):
+ """A package tree under src/ plus a place for the report. Returns (root, src)."""
+ src = tmp_path / "src"
+ (src / "sub").mkdir(parents=True)
+ return tmp_path, src
+
+
+# --- parse: covered / total ------------------------------------------------
+
+def test_parse_counts_covered_and_statements(project):
+ root, src = project
+ report = write_report(root, {"src/a.py": (3, 5)})
+ table = cs.parse_report(report)
+ key = str((root / "src/a.py").resolve())
+ assert table[key] == (3, 5)
+
+
+def test_parse_missing_report_raises(project):
+ root, _ = project
+ with pytest.raises(SystemExit):
+ cs.parse_report(root / "nope.json")
+
+
+def test_parse_malformed_json_raises(project):
+ root, _ = project
+ bad = root / "coverage.json"
+ bad.write_text("{not json")
+ with pytest.raises(SystemExit):
+ cs.parse_report(bad)
+
+
+# --- file percentage -------------------------------------------------------
+
+def test_file_pct_normal():
+ assert cs.file_pct(1, 2) == 50.0
+
+
+def test_file_pct_no_statements_is_100():
+ assert cs.file_pct(0, 0) == 100.0
+
+
+def test_file_pct_fully_covered():
+ assert cs.file_pct(4, 4) == 100.0
+
+
+# --- missing-file detection (the kernel) -----------------------------------
+
+def test_missing_finds_ondisk_file_absent_from_report(project):
+ root, src = project
+ (src / "tracked.py").write_text("x = 1\n")
+ (src / "untested.py").write_text("y = 2\n")
+ report = write_report(root, {"src/tracked.py": (1, 1)})
+ table = cs.under_dir(cs.parse_report(report), src, root)
+ missing = cs.missing(list(table), src, root)
+ assert missing == ["src/untested.py"]
+
+
+def test_missing_recurses_into_subpackages(project):
+ root, src = project
+ (src / "top.py").write_text("a = 1\n")
+ (src / "sub" / "deep.py").write_text("b = 2\n")
+ report = write_report(root, {"src/top.py": (1, 1)})
+ table = cs.under_dir(cs.parse_report(report), src, root)
+ missing = cs.missing(list(table), src, root)
+ assert "src/sub/deep.py" in missing
+
+
+def test_missing_empty_when_all_tracked(project):
+ root, src = project
+ (src / "a.py").write_text("x = 1\n")
+ report = write_report(root, {"src/a.py": (1, 1)})
+ table = cs.under_dir(cs.parse_report(report), src, root)
+ assert cs.missing(list(table), src, root) == []
+
+
+# --- project number (file-weighted, missing as 0%) -------------------------
+
+def test_project_pct_unit_weighted_with_missing_as_zero(project):
+ root, src = project
+ for name in ("full.py", "half.py", "untested.py"):
+ (src / name).write_text("x = 1\n")
+ report = write_report(root, {"src/full.py": (2, 2), "src/half.py": (1, 2)})
+ # full=100, half=50, untested missing=0 -> (100+50+0)/3 = 50.0
+ assert cs.project_pct(report, src, root) == 50.0
+
+
+def test_project_pct_no_missing(project):
+ root, src = project
+ for name in ("full.py", "half.py"):
+ (src / name).write_text("x = 1\n")
+ report = write_report(root, {"src/full.py": (2, 2), "src/half.py": (1, 2)})
+ assert cs.project_pct(report, src, root) == 75.0
+
+
+# --- end-to-end text -------------------------------------------------------
+
+def test_summary_text_reports_missing_and_project_number(project):
+ root, src = project
+ (src / "a.py").write_text("x = 1\n")
+ (src / "orphan.py").write_text("y = 2\n")
+ report = write_report(root, {"src/a.py": (1, 2)})
+ text = cs.summary_text(report, src, root)
+ assert "orphan.py" in text
+ assert "0%" in text
+ # a.py = 50%, orphan missing = 0% -> 25.0%
+ assert "25.0%" in text