diff options
| author | Craig Jennings <c@cjennings.net> | 2026-05-22 14:38:16 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-05-22 14:38:16 -0500 |
| commit | 4c3dbdb1e4a50835c0588e0ede568c06c31aadec (patch) | |
| tree | 14d7f6c9865193683d32ce4ff2eddbb8b0c32907 /pairwise-tests | |
| parent | 9480b424cb5069ace4979a9efd49a983c5526481 (diff) | |
| download | rulesets-4c3dbdb1e4a50835c0588e0ede568c06c31aadec.tar.gz rulesets-4c3dbdb1e4a50835c0588e0ede568c06c31aadec.zip | |
docs(skills): add t-way escalation and honest generator path to pairwise-tests
Two audit fixes. A t-way escalation section says to start pairwise, then raise specific high-risk clusters to 3-way or higher when history, safety, security, or coupling warrants it, with the sub-model order syntax shown. A second note explains PICT's ~ marker (a negative or invalid-value tag, not an operator) and adds an honest stop-at-the-model rule: if no PICT generator is installed, produce the model and stop rather than hand-writing rows and calling them generated.
Diffstat (limited to 'pairwise-tests')
| -rw-r--r-- | pairwise-tests/SKILL.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/pairwise-tests/SKILL.md b/pairwise-tests/SKILL.md index 5e45b40..ff83e98 100644 --- a/pairwise-tests/SKILL.md +++ b/pairwise-tests/SKILL.md @@ -129,6 +129,8 @@ python scripts/pict_helper.py parse output.txt - https://pairwise.teremokgames.com/ 3. Or install PICT locally (see references/pict_syntax.md) +**If no generator is available, stop at the model.** Generating the actual rows needs either the `pict` binary or the `pypict` Python binding present in the environment. Check first (`pict` on PATH, or `python -c "import pypict"`). If neither is installed, produce the PICT model and stop there — hand the user the model plus the online-tool and install paths above. Do not hand-write a table of "generated" rows and present them as PICT output; a hand-rolled matrix is not guaranteed to satisfy pairwise coverage, and claiming it does is wrong. Either a generator produced the rows or it didn't — say which. + ### 4. Determine Expected Outputs For each generated test case, determine the expected outcome based on: @@ -187,6 +189,8 @@ IF [Parameter1] = "Value1" THEN [Parameter2] = "ValueA"; - Include boundary values: `Age: 0, 17, 18, 65, 66` - Add negative values for error testing: `Amount: ~-1, 0, 100, ~999999` +**On the `~` prefix:** the leading `~` is a PICT marker that tags a value as a *negative* (invalid / error-path) input — not an arithmetic operator. In `~-1` the `~` is the marker and `-1` is the value; the minus sign belongs to the number. PICT treats marked values specially: it pairs each negative value with otherwise-valid values exactly once rather than combining negatives with each other, so a single test isn't rejected for two unrelated reasons at the same time. The `~` is stripped before the value reaches the system under test — it labels intent in the model, nothing more. + **Avoid:** - Generic names: `Param1`, `Value1`, `V1` - Too many values without partitioning @@ -224,6 +228,17 @@ For large parameter sets: - Start with order 2 (pairwise), increase for critical combinations - Typical pairwise testing reduces test cases by 80-90% vs exhaustive +### Escalating Beyond Pairwise (t-way) + +Pairwise (2-way) is the default because it catches most interaction faults at a fraction of the exhaustive cost. NIST combinatorial-testing research shows some faults only surface when three or more factors interact, so higher-strength (t-way) coverage has its place — but applied selectively, not across the board. + +Start with pairwise for the whole parameter space. Then escalate specific high-risk parameter clusters to 3-way or higher when history, safety, security, or domain reasoning says a fault likely needs more than two interacting factors: +- A past defect that only reproduced with three settings aligned +- A safety- or security-critical path where a missed interaction is expensive +- Domain knowledge that three parameters are tightly coupled + +Raise the order on those clusters via a sub-model with its own order (`{ A, B, C } @ 3`) or by running a higher-order pass over just those parameters. PICT's command-line order option (`/o:3`) raises the strength for the whole model — reserve a blanket bump for the rare case where the entire space is critical, since the test count climbs fast with order. The point is targeted escalation of the clusters that warrant it, not a uniform higher strength. + ## Common Patterns ### Web Form Testing |
