diff options
| author | Craig Jennings <c@cjennings.net> | 2026-07-24 12:28:18 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-07-24 12:28:18 -0500 |
| commit | f0c1bc40708615d5b423922c08c1f27e6cf96259 (patch) | |
| tree | 5fc8ebe7fde6381f4cd91b5de15fe4a2521889c8 /working/hook-fail-open/test-validate-el-hook.bats | |
| parent | 781fa0786e2096797e630dcb81ffbc62ed98460b (diff) | |
| download | rulesets-f0c1bc40708615d5b423922c08c1f27e6cf96259.tar.gz rulesets-f0c1bc40708615d5b423922c08c1f27e6cf96259.zip | |
fix(hooks): the secret scan no longer passes when git fails
Every bundle built its scan input as `git diff --cached ... | grep ... || true`. The `|| true` has to stay, since grep exits 1 when it matches nothing and that is the ordinary case. But with no pipefail it also swallowed a failure of git itself, so an empty result made the scan search nothing, find nothing, and report clean with a real credential staged. A gate that passes without having looked.
.emacs.d found it in elisp. It was in all five: bash, elisp, go, python and typescript, eleven sites once each bundle's staged-file list is counted. Two of those bundles are ones I wrote yesterday by copying bash, so I propagated it while closing an unrelated gap in the same file. Each site now reads the diff on its own and aborts if git fails, leaving the greps their `|| true`.
Verified per bundle on three axes: refuses when the diff cannot be read, still blocks a real staged secret, still passes a clean commit.
The cross-bundle test suite had its own version of the same disease. Its VARIANTS list read "elisp bash go" while python and typescript also shipped hooks, so every "in every variant" assertion had quietly skipped two bundles since the day they were added. VARIANTS is now discovered from the tree. Two fail-closed assertions join it, and .emacs.d's elisp suite is adopted here beside the canonical hook, because a test living in the consuming project cannot fail when the canonical regresses.
Also removes the validate-el auto-test cap, Craig's call. Above 20 matching test files the runner skipped everything and exited 0 with no output. The premise was speed and it did not hold: a whole family runs in about a second. The cap was also hiding a real cross-test pollution bug that only surfaces when a family runs in one process.
Diffstat (limited to 'working/hook-fail-open/test-validate-el-hook.bats')
| -rw-r--r-- | working/hook-fail-open/test-validate-el-hook.bats | 97 |
1 files changed, 0 insertions, 97 deletions
diff --git a/working/hook-fail-open/test-validate-el-hook.bats b/working/hook-fail-open/test-validate-el-hook.bats deleted file mode 100644 index 43c3569..0000000 --- a/working/hook-fail-open/test-validate-el-hook.bats +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env bats -# Tests for .claude/hooks/validate-el.sh — the auto-test runner. -# -# The runner used to skip entirely above MAX_AUTO_TEST_FILES=20, with no else -# branch: nothing printed, exit 0, indistinguishable from a passing run. That -# was live for the three largest families here (calendar-sync 63 test files, -# music 45, ai-term 35), so every edit to those ran parens and byte-compile and -# zero tests, silently. -# -# The cap was removed rather than made loud, because its premise did not hold. -# Measured on this machine, running a whole family takes about a second: -# ai-term 208 tests in 1.0s, music 403 in 1.7s, calendar-sync 633 in 0.9s. It -# was also concealing a real cross-test pollution bug in calendar-sync that -# only appears when that family runs in one process. -# -# These tests pin that no file count is skipped. Each builds a synthetic -# project in BATS_TEST_TMPDIR and points CLAUDE_PROJECT_DIR at it, so nothing -# runs against the real tree. - -setup() { - HOOK="${BATS_TEST_DIRNAME}/../.claude/hooks/validate-el.sh" - PROJ="${BATS_TEST_TMPDIR}/proj" - mkdir -p "$PROJ/modules" "$PROJ/tests" - export CLAUDE_PROJECT_DIR="$PROJ" - printf '(provide (quote widget))\n' > "$PROJ/modules/widget.el" -} - -# N green test files matching the widget stem. -make_tests() { - local n="$1" i - for ((i = 1; i <= n; i++)); do - printf '(require (quote ert))\n(ert-deftest test-widget-%d () (should t))\n' \ - "$i" > "$PROJ/tests/test-widget-${i}.el" - done -} - -# One failing test file, to prove the run is real rather than merely quiet. -make_failing_test() { - printf '(require (quote ert))\n(ert-deftest test-widget-bad () (should nil))\n' \ - > "$PROJ/tests/test-widget-bad.el" -} - -hook_input() { - printf '{"tool_input":{"file_path":"%s"}}' "$PROJ/modules/widget.el" -} - -run_hook() { - run bash -c "$(printf '%q' "$HOOK") <<< '$(hook_input)'" -} - -# ------------------------------- Normal cases ------------------------------- - -@test "a small family runs and passes quietly" { - make_tests 3 - run_hook - [ "$status" -eq 0 ] -} - -@test "a failing test blocks, so a quiet pass means the tests really ran" { - make_tests 3 - make_failing_test - run_hook - [ "$status" -eq 2 ] - [[ "$output" == *"TESTS FAILED"* ]] -} - -# ------------------------------ Boundary cases ------------------------------ - -@test "at the old cap of 20 files: runs" { - make_tests 20 - run_hook - [ "$status" -eq 0 ] -} - -@test "past the old cap: still runs, no longer skipped" { - make_tests 21 - run_hook - [ "$status" -eq 0 ] - [[ "${output,,}" != *"skipped"* ]] -} - -@test "well past the old cap: a failure in file 63 is still caught" { - # The regression this guards: at 63 files the runner used to skip, so a red - # test in a big family reported clean. calendar-sync is exactly this size. - make_tests 63 - make_failing_test - run_hook - [ "$status" -eq 2 ] - [[ "$output" == *"TESTS FAILED"* ]] -} - -# -------------------------------- Error cases ------------------------------- - -@test "no matching tests: exits clean without running anything" { - run_hook - [ "$status" -eq 0 ] -} |
