diff options
| author | Craig Jennings <c@cjennings.net> | 2026-06-25 03:33:59 -0400 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-06-25 03:33:59 -0400 |
| commit | 2d63802e77617e4840c81baceb709260341c251a (patch) | |
| tree | 9002fa58f5d00dd0838411828c625d5f1a1870b2 /todo.org | |
| parent | 08844e730f9acd0874f596bb9906f1f264824eba (diff) | |
| download | archsetup-2d63802e77617e4840c81baceb709260341c251a.tar.gz archsetup-2d63802e77617e4840c81baceb709260341c251a.zip | |
test(archsetup): expand validation coverage + fix ParallelDownloads (P4)
Add post-install checks beyond the original shell sweep, validated against a
live VM: test_hardening (sshd prohibit-password, quiet-printk sysctl, emptied
/etc/issue, console font, EFI mount perms), test_config_applied (pacman
ParallelDownloads/Color/multilib, makepkg flags, NetworkManager drop-ins,
fail2ban jail, reflector), and test_backups (the .archsetup.bak files
backup_system_file leaves behind — end-to-end proof of that feature).
The new tests caught a real bug: ParallelDownloads stayed at Arch's default 5
because the sed only matched a commented "#ParallelDownloads", but current Arch
ships it uncommented. Match both (^#?ParallelDownloads) so the intended 10 takes
effect.
Verified against a kept VM: 95 passed, 10 skipped (the one remaining failure was
the pre-fix ParallelDownloads on the already-built VM, which the sed fix
resolves on the next fresh install).
Diffstat (limited to 'todo.org')
| -rw-r--r-- | todo.org | 2 |
1 files changed, 2 insertions, 0 deletions
@@ -544,6 +544,8 @@ Built the Testinfra harness skeleton: =scripts/testing/tests/= (conftest.py with Ported the whole shell sweep to pytest: test_users (exists/shell/15 groups parametrized), test_packages (yay+functional, pacman, terminus-font, emacs+config readable, git, 5 dev tools), test_services (required enabled/active, enabled-only, timers, optional skip-if-absent, DoT drop-in, fail2ban/nmcli responds, log-cleanup cron, syncthing lingering, DNS/mDNS/docker skips), test_desktop (Hyprland tools+configs+portal+socket gated on install/compositor, DWM suckless, autologin), test_boot (grub, mkinitcpio hooks branched on zfs_root, console-font-in-initramfs, nvme gated, zfs/sanoid), test_keyring (dir 700/owner/default=login), test_archsetup (log no Error:, ≥12 state markers). conftest fixtures: target_user/home/zfs_root/has_nvme/hyprland_installed/dwm_installed/compositor_running/on_slirp. 88 tests collected, py_compile clean. Correctness fix vs the shell sweep: check =awww= not the stale =swww=. Installed python-pytest-testinfra on velox so the harness gate passes. Next: VM run to diff pytest vs shell sweep for parity. *** 2026-06-25 Thu @ 01:24:11 -0400 Fixed: sshd hardening had silently broken =make test= VM run #1 aborted ~6 min in (Error 5), before any validation ran. Root cause (pre-existing, not the Testinfra work): the 2026-06-24 sshd hardening sets =PermitRootLogin prohibit-password= + reloads sshd mid-install, and the harness SSHes as root by *password* throughout — so every op after that step got "Permission denied" and run-test.sh fataled before validations. Fix: =inject_root_key= authorizes a throwaway root key right after first SSH (before archsetup runs) and all helpers (=wait_for_ssh=/=vm_exec=/=copy_to_vm=/=copy_from_vm=/=ssh_cmd=) gained =$SSH_KEY_OPT= so they use key auth, which =prohibit-password= still allows. testinfra.sh reuses that key. Additive (password stays as fallback). bash -n + shellcheck clean. Re-running the VM suite to confirm it now reaches the validation + pytest phases. +*** 2026-06-25 Thu @ 03:33:33 -0400 Parity proven + P4 expansion validated on a live VM +VM run #3 (=make test-keep=, kept VM up): pytest parity = 78 passed / 10 skipped / 0 fail / 0 err — matches & exceeds the shell sweep (53/0/0). Then built P4 expansion against the live VM (iterating in ~30s, no rebuild): test_hardening (sshd prohibit-password, sysctl printk, /etc/issue emptied, vconsole font, /efi fmask), test_config_applied (pacman ParallelDownloads/Color/multilib, makepkg MAKEFLAGS/OPTIONS, NM dns+wifi-privacy drop-ins, fail2ban jail, reflector), test_backups (=.archsetup.bak= present for pacman.conf/makepkg.conf/sudoers/mkinitcpio.conf — end-to-end proof of the backup feature). Full suite vs live VM: 95 passed / 10 skipped / 1 fail. The 1 fail = a REAL archsetup bug the tests caught: =ParallelDownloads= stayed at the Arch default 5 because the sed only matched a commented =#ParallelDownloads=, but current Arch ships it uncommented — fixed the sed to match both (=^#\?ParallelDownloads=). Also fixed a test bug (=grep -qx '[multilib]'= → =grep -Fxq=, the brackets were a regex char class). Remaining: P3 cutover (pytest authoritative) + P5 retire shell sweep, then a final fresh =make test=. Create comprehensive integration tests using Testinfra (Python + pytest) to validate archsetup installations Tests should cover: |