test(archsetup): make Testinfra the authoritative validator (P3 cutover)

run-test.sh no longer runs the shell run_all_validations sweep; the Testinfra
pytest sweep now drives the run's pass/fail. run_testinfra_validation returns
pytest's exit code (and treats "could not run" as a failure, not a silent
pass), surfaces the pass/skip/fail counts through the shared VALIDATION_*
counters, and parses the attribution file so generate_issue_report still
buckets failures into archsetup / base_install / unknown.

The shell-sweep functions stay in validation.sh for now because
run-test-baremetal.sh still calls them; removing them (after migrating the
bare-metal runner) is filed as a follow-up.

1 files changed, 9 insertions, 15 deletions

diff --git a/scripts/testing/run-test.sh b/scripts/testing/run-test.sh
index 90022d3..6e51fc2 100755
--- a/scripts/testing/run-test.sh
+++ b/scripts/testing/run-test.sh
@@ -316,23 +316,17 @@ copy_from_vm "/var/log/archsetup-installed-packages.txt" "$TEST_RESULTS_DIR/" "$
 # Capture post-install state
 capture_post_install_state "$TEST_RESULTS_DIR"
 
-# Run comprehensive validation
-# This uses the validation.sh library for all checks.
+# Run comprehensive validation (Testinfra/pytest is the primary validator; the
+# old shell run_all_validations sweep was retired once pytest reached parity).
 #
 # From here to the end of the script, errexit is disabled on purpose: the
-# validation functions are designed to fail-and-count (see VALIDATION_FAILED)
-# rather than abort, and the analysis/report-generation steps below can also
-# legitimately return non-zero. With `set -e` active, a single failed check
-# would kill the run before the test report is written or the VM is cleaned
-# up. Pass/fail is signalled explicitly by the exit code at the bottom.
+# analysis/report-generation steps below can legitimately return non-zero, and
+# with `set -e` active a single failed check would kill the run before the test
+# report is written or the VM is cleaned up. Pass/fail is signalled explicitly
+# by the exit code at the bottom.
 set +e
-run_all_validations
-validate_all_services
-
-# Advisory Testinfra sweep alongside the shell sweep (P1). Compare the two on a
-# real run to confirm parity before pytest becomes primary. Does not affect
-# pass/fail yet.
 run_testinfra_validation "$TEST_RESULTS_DIR"
+testinfra_rc=$?
 
 # Analyze log differences (pre vs post install)
 analyze_log_diff "$TEST_RESULTS_DIR"
@@ -341,8 +335,8 @@ analyze_log_diff "$TEST_RESULTS_DIR"
 # If base install issues found and archzfs inbox exists, create issue file
 generate_issue_report "$TEST_RESULTS_DIR" "$ARCHZFS_INBOX"
 
-# Set validation result based on failure count
-if [ "$VALIDATION_FAILED" -eq 0 ]; then
+# The run passes only if the Testinfra sweep passed.
+if [ "$testinfra_rc" -eq 0 ]; then
     TEST_PASSED=true
 else
     TEST_PASSED=false