archsetup/scripts/testing/lib, branch main

refactor(test): share runner and scenario contracts

2026-07-25T20:00:44+00:00

fix(test): harden four VM-framework paths from the S5 audit

2026-07-21T04:40:56+00:00

init_vm_paths suffixed the disk and NVRAM by FS_PROFILE but left PID_FILE, MONITOR_SOCK, and SERIAL_LOG shared. A concurrent btrfs and zfs run read each other's PID file, so the second run's stop logic could kill the first run's VM. All runtime paths now carry the suffix. kill_qemu sent kill -9 and returned without waiting, so the force-kill fallback could run qemu-img snapshot against a qcow2 the dying qemu still held locked (the restore failed silently behind || true, leaving the base image dirty). It now reaps or polls the process to death before returning. debug-vm.sh hardcoded the btrfs base-disk name, so FS_PROFILE=zfs booted the wrong base or fatalled. It now takes DISK_PATH from init_vm_paths. Both runners reported a completion-marker grep as "ArchSetup Exit Code," but the installer runs detached with set -e off, so its real exit status was never captured -- a run that errored mid-way and still reached its last line reported exit 0. The flag is now ARCHSETUP_COMPLETED with the report labeled honestly; Testinfra stays the pass/fail authority.

test(vm): collect network evidence before failing in pre-flight diagnostics

2026-07-04T21:23:57+00:00

run_network_diagnostics tested HTTP before DNS and returned on the first failure, so a DNS failure surfaced as a generic "no internet" and the IP/route/resolver evidence was never reached. It now collects read-only facts first (ip -brief addr, default route, resolv.conf) and prints them regardless of outcome, runs every check, and reports all failures together in a summary. Generic checks (DNS, egress, TLS) are split from Arch-specific ones (mirror, AUR) so a DNS failure is named as DNS, not a mirror problem. Raw fact output is saved to the results dir when one is set.

fix(test): bump default VM RAM to 8 GiB to stop AUR-build OOM kills

2026-06-28T01:44:55+00:00

The zfs green run OOM-killed cc1plus three times during AUR C++ builds: makepkg runs -j$VM_CPUS (4), and parallel compiles at ~700 MB each overran the 4 GiB default. The install still passed (yay retries), but the kills showed up as attributed issues. 8 GiB gives the four jobs headroom. Overridable via VM_RAM as before.

fix(test): give each filesystem profile its own OVMF NVRAM file

2026-06-27T21:20:58+00:00

init_vm_paths suffixed the disk image per profile but shared one OVMF_VARS.fd across btrfs and zfs. NVRAM holds the UEFI boot entries and lives outside the qcow2, so a disk-snapshot revert can't restore it. A zfs run's ZFSBootMenu entries clobbered the btrfs GRUB entry, and with no removable ESP fallback the btrfs base then booted to "no bootable device" and timed out before archsetup ran. NVRAM now carries the same per-profile suffix as the disk image, so the two profiles keep separate boot state. Validated by a full green zfs run (ArchSetup exit 0, Testinfra 96 passed / 0 failed).

refactor(testing): delete the dead validation.sh shell sweep

2026-06-26T03:24:13+00:00

Both runners now validate through run_testinfra_validation, so the shell sweep validation.sh ran is dead. Delete run_all_validations, validate_all_services, run_full_validation, the ~35 validate_* checks, and validation_pass/fail/warn/skip (called only by those checks). Keep the live helpers the runners and testinfra.sh still use: ssh_cmd, attribute_issue, capture_pre/post_install_state, analyze_log_diff, categorize_errors, generate_issue_report, and the VALIDATION_* counters plus issue arrays. The file drops from 1156 lines to 314. Closes the P5 follow-up from the Testinfra cutover.

test(archsetup): migrate bare-metal runner to key auth + Testinfra

2026-06-26T03:12:51+00:00

run-test-baremetal.sh SSHed to the target as root by password throughout, which archsetup's sshd hardening (PermitRootLogin prohibit-password) kills mid-install, the same break the VM runner already fixed. It also still called the validation.sh shell sweep (run_all_validations, validate_all_services, validate_zfs_services), the last caller keeping those functions alive. It now mirrors the VM runner. After the first SSH, and after any genesis rollback so the key survives it, inject_root_key authorizes a throwaway root key, and every later ssh_cmd plus the raw scp transfers and log-copies thread SSH_KEY_OPT to survive the hardening. The shell sweep is replaced with run_testinfra_validation, now the authoritative validator on both runners. A --port option, threaded through every SSH and scp, lets the runner target a test VM on 2222 instead of only real hardware on 22. inject_root_key now authorizes root@$VM_IP instead of root@localhost, so one helper serves both runners (the VM runner sets VM_IP=localhost). Validated against the ZFS VM (--validate-only, localhost:2222): connectivity, the ZFS check, key authorization, and the Testinfra sweep all connect and run over the key-based ssh-config. A green bare-metal install still needs real ZFS hardware.

test(archsetup): add FS_PROFILE selector for ZFS VM coverage

2026-06-25T16:21:15+00:00

The VM harness only built one btrfs base image, so every ZFS-conditional check in the Testinfra suite skipped and the ZFS install path went untested in automation. I added an FS_PROFILE selector (btrfs default, zfs) so `make test FS_PROFILE=zfs` can target a ZFS root. init_vm_paths derives the image name from FS_PROFILE and validates it. btrfs keeps the legacy unsuffixed archsetup-base.qcow2 so existing images and invocations are untouched. The zfs profile gets archsetup-base-zfs.qcow2. create-base-vm.sh picks archsetup-test.conf vs the new archsetup-test-zfs.conf (FILESYSTEM=zfs, NO_ENCRYPT=yes for an unattended install), and the Makefile resolves the matching image for its base-VM check. The archsetup run config stays shared. archsetup reads no filesystem key. It detects ZFS from the live root via is_zfs_root, so the ZFS branch fires on its own once the base image is ZFS. The design doc is reconciled to that: no separate archsetup-vm-zfs.conf, and the non-ZFS profile is btrfs, not ext4. Building the ZFS base image and running the ZFS sweep green is next.

test(archsetup): make Testinfra the authoritative validator (P3 cutover)

2026-06-25T07:39:10+00:00

run-test.sh no longer runs the shell run_all_validations sweep; the Testinfra pytest sweep now drives the run's pass/fail. run_testinfra_validation returns pytest's exit code (and treats "could not run" as a failure, not a silent pass), surfaces the pass/skip/fail counts through the shared VALIDATION_* counters, and parses the attribution file so generate_issue_report still buckets failures into archsetup / base_install / unknown. The shell-sweep functions stay in validation.sh for now because run-test-baremetal.sh still calls them; removing them (after migrating the bare-metal runner) is filed as a follow-up.

fix(testing): authorize a root key so make test survives sshd hardening

2026-06-25T05:24:33+00:00

The VM test SSHes into the guest as root with a password for the whole run. archsetup hardens sshd to PermitRootLogin prohibit-password and reloads it partway through the install, so every SSH after that step failed with "Permission denied" and the run aborted before any validation — make test had been silently broken since the hardening landed. inject_root_key authorizes a throwaway root key right after the first SSH (before archsetup runs) and the ssh/scp helpers now add -i via SSH_KEY_OPT. prohibit-password still allows root key auth, so the harness survives the very hardening it validates. Password stays as the fallback, so the change is additive.