diff options
| author | Craig Jennings <c@cjennings.net> | 2026-04-19 16:15:03 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-04-19 16:15:03 -0500 |
| commit | f926e6921311826b668e3e3a9fc2156b35cb1e77 (patch) | |
| tree | 872a72d62651003abb50a6e79786313d54fea953 | |
| parent | b11cfd66b185a253fecf10ad06080ae165f32a74 (diff) | |
| download | rulesets-f926e6921311826b668e3e3a9fc2156b35cb1e77.tar.gz rulesets-f926e6921311826b668e3e3a9fc2156b35cb1e77.zip | |
chore(add-tests): sharpen the pairwise-vs-exhaustive prompt
Previous wording — 'pairwise or continue with normal category coverage' —
muddled the choice. Category coverage (Normal/Boundary/Error) tests each
parameter individually and doesn't meaningfully address combinatorial
interaction questions.
Sharper framing: when a function has parameters that combine, the real
question is pairwise vs exhaustive (the M^N full factorial). Category
coverage is a separate axis — it handles the error-case and boundary-case
testing that parameter-value combinations don't.
New prompt cites specific counts:
'Function X has N parameters (M^N = Y exhaustive combinations).
Pairwise or exhaustive?'
Pairwise stays the pragmatic default; exhaustive only when the user names
a regulatory / safety-critical reason.
| -rw-r--r-- | add-tests/SKILL.md | 8 |
1 files changed, 7 insertions, 1 deletions
diff --git a/add-tests/SKILL.md b/add-tests/SKILL.md index 5803dd8..45ab2ac 100644 --- a/add-tests/SKILL.md +++ b/add-tests/SKILL.md @@ -28,7 +28,13 @@ Work through four phases in order. Do not skip phases. 6. Present a test plan as a markdown table with columns: File, Functions, Current Coverage, Proposed Tests, Priority (P0/P1/P2). 7. For each proposed test, specify: target function, category (Normal / Boundary / Error), one-line description, and any mocking required. All three categories are required per function — see `testing.md` for category definitions. -8. **If a function has 3+ parameters with multiple values each, surface `/pairwise-tests`.** For that function, hand-writing exhaustive or even category-based cases risks either an explosion (N × M exhaustive) or under-coverage (3-5 ad-hoc tests miss pair interactions). Tell the user: "Function `<name>` has N parameters — would you rather generate a pairwise-covering matrix via `/pairwise-tests` for that function, or continue with normal `/add-tests` category coverage?" Default to continuing unless they pick pairwise. When in doubt, suggest pairwise for the combinatorial function and fall back to `/add-tests` for the rest. +8. **Surface the coverage choice for parameter-heavy functions.** If a function has 3+ parameters with multiple values each, category-based cases (Normal/Boundary/Error per parameter individually) miss interaction bugs, while exhaustive cases explode combinatorially. Ask the user, citing specific numbers: + + > "Function `<name>` has N parameters (M^N = Y exhaustive combinations). Pairwise or exhaustive? + > - **Pairwise** (`/pairwise-tests`): ~X cases covering every 2-way parameter interaction; catches 60-90% of combinatorial bugs. + > - **Exhaustive**: Y hand-written cases covering every combination; only needed in regulated / provably-complete-coverage contexts." + + Pairwise is the pragmatic default. Exhaustive only when the user names a specific reason (regulatory, safety-critical, audit evidence). If the user doesn't pick explicitly, proceed with pairwise for that function and continue with category coverage for the rest. 9. For large codebases, group the plan into batches (e.g., "Batch 1: core domain models, Batch 2: API endpoints"). 10. **Stop and ask the user for confirmation.** The user may add, remove, or reprioritize tests before you proceed. Do not write any test files until the user approves. |
