aboutsummaryrefslogtreecommitdiff
path: root/scripts/tests/pre-commit-secret-scan.bats
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-07-24 12:28:18 -0500
committerCraig Jennings <c@cjennings.net>2026-07-24 12:28:18 -0500
commitf0c1bc40708615d5b423922c08c1f27e6cf96259 (patch)
tree5fc8ebe7fde6381f4cd91b5de15fe4a2521889c8 /scripts/tests/pre-commit-secret-scan.bats
parent781fa0786e2096797e630dcb81ffbc62ed98460b (diff)
downloadrulesets-f0c1bc40708615d5b423922c08c1f27e6cf96259.tar.gz
rulesets-f0c1bc40708615d5b423922c08c1f27e6cf96259.zip
fix(hooks): the secret scan no longer passes when git fails
Every bundle built its scan input as `git diff --cached ... | grep ... || true`. The `|| true` has to stay, since grep exits 1 when it matches nothing and that is the ordinary case. But with no pipefail it also swallowed a failure of git itself, so an empty result made the scan search nothing, find nothing, and report clean with a real credential staged. A gate that passes without having looked. .emacs.d found it in elisp. It was in all five: bash, elisp, go, python and typescript, eleven sites once each bundle's staged-file list is counted. Two of those bundles are ones I wrote yesterday by copying bash, so I propagated it while closing an unrelated gap in the same file. Each site now reads the diff on its own and aborts if git fails, leaving the greps their `|| true`. Verified per bundle on three axes: refuses when the diff cannot be read, still blocks a real staged secret, still passes a clean commit. The cross-bundle test suite had its own version of the same disease. Its VARIANTS list read "elisp bash go" while python and typescript also shipped hooks, so every "in every variant" assertion had quietly skipped two bundles since the day they were added. VARIANTS is now discovered from the tree. Two fail-closed assertions join it, and .emacs.d's elisp suite is adopted here beside the canonical hook, because a test living in the consuming project cannot fail when the canonical regresses. Also removes the validate-el auto-test cap, Craig's call. Above 20 matching test files the runner skipped everything and exited 0 with no output. The premise was speed and it did not hold: a whole family runs in about a second. The cap was also hiding a real cross-test pollution bug that only surfaces when a family runs in one process.
Diffstat (limited to 'scripts/tests/pre-commit-secret-scan.bats')
-rw-r--r--scripts/tests/pre-commit-secret-scan.bats55
1 files changed, 54 insertions, 1 deletions
diff --git a/scripts/tests/pre-commit-secret-scan.bats b/scripts/tests/pre-commit-secret-scan.bats
index 013129e..4647556 100644
--- a/scripts/tests/pre-commit-secret-scan.bats
+++ b/scripts/tests/pre-commit-secret-scan.bats
@@ -14,7 +14,13 @@
# random base64 blob matched. Measured at ~6% of 100KB blobs; case-sensitive
# matching drops it to 0 across ~10MB.
-VARIANTS="elisp bash go"
+# Discovered, never enumerated. This list read "elisp bash go" while python and
+# typescript also shipped pre-commit hooks, so every "in every variant" test
+# below silently skipped two bundles from the day they were added — the same
+# enumerate-instead-of-discover failure these tests exist to catch. A new bundle
+# is now covered the moment it has a hook.
+VARIANTS="$(cd "${BATS_TEST_DIRNAME}/../../languages" && \
+ for d in */githooks/pre-commit; do [ -f "$d" ] && printf '%s ' "${d%%/*}"; done)"
setup() {
REPO="$(mktemp -d)"
@@ -142,3 +148,50 @@ run_hook() {
[ "$status" -eq 0 ] || { echo "$v blocked a removal: $output"; return 1; }
done
}
+
+# ---- Fail-closed: a broken git must never read as "nothing to scan" ----
+
+# Put a stub `git` ahead of the real one that fails only the staged-diff call
+# and delegates everything else, so just the pipeline under test breaks.
+break_git() {
+ mkdir -p "$REPO/bin"
+ cat > "$REPO/bin/git" <<'STUB'
+#!/usr/bin/env bash
+if [ "${1:-}" = "diff" ] && [ "${2:-}" = "--cached" ]; then
+ echo "simulated git failure" >&2
+ exit 128
+fi
+exec /usr/bin/git "$@"
+STUB
+ chmod +x "$REPO/bin/git"
+}
+
+@test "secret-scan: a broken git refuses rather than passing blind, in every variant" {
+ # The scan built its input as `git diff ... | grep ... || true`. With no
+ # pipefail, a git failure yielded an empty string, so the scan searched
+ # nothing, found nothing, and reported clean with a real secret staged.
+ # Found by .emacs.d in elisp 2026-07-24; all five variants had it.
+ stage 'aws_key = "AKIAIOSFODNN7EXAMPLE"'
+ break_git
+ for v in $VARIANTS; do
+ run env PATH="$REPO/bin:$PATH" bash \
+ "${BATS_TEST_DIRNAME}/../../languages/$v/githooks/pre-commit"
+ [ "$status" -ne 0 ] || {
+ echo "$v FAILED OPEN: exited 0 with a secret staged and git broken"
+ return 1
+ }
+ done
+}
+
+@test "secret-scan: the refusal says why, in every variant" {
+ stage 'aws_key = "AKIAIOSFODNN7EXAMPLE"'
+ break_git
+ for v in $VARIANTS; do
+ run env PATH="$REPO/bin:$PATH" bash \
+ "${BATS_TEST_DIRNAME}/../../languages/$v/githooks/pre-commit"
+ [[ "$output" == *"cannot read"* ]] || {
+ echo "$v refused without naming the cause: $output"
+ return 1
+ }
+ done
+}