diff options
| author | Craig Jennings <c@cjennings.net> | 2025-11-13 23:26:21 -0600 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2025-11-13 23:26:21 -0600 |
| commit | 2e10a8856d0bdd4c8f77c53320221ad1b8deaa13 (patch) | |
| tree | 95832c3b74fc523fe9d8319e25c5ea5bf1d40433 /scripts/testing/README.org | |
| parent | fd9cce59993556400b635256d712a65d87f5d72d (diff) | |
fix(archsetup): implement critical bug fixes and test improvements
This commit addresses several high-priority bugs and enhances the testing infrastructure:
**Bug Fixes:**
1. Add root permission check at script start to fail fast with clear error message
2. Disable debug package installation by adding --nodebug flag to all yay calls
3. Replace unsafe `git pull --force` with safe rm + fresh clone to prevent data loss
4. Add geoclue package with correct systemd service configuration for geolocation
5. Add completion marker for reliable automated test detection
**Testing Infrastructure:**
- Add comprehensive VM-based testing framework in scripts/testing/
- Fix test script pgrep infinite loop using grep bracket self-exclusion pattern
- Add network diagnostics and pre-flight checks
- Support snapshot-based testing for reproducible test runs
**Package Management:**
- Remove anki (build hangs 98+ minutes)
- Remove adwaita-color-schemes (CMake build issues)
Test Results: 0 errors, 1,363 packages installed in 40 minutes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'scripts/testing/README.org')
| -rw-r--r-- | scripts/testing/README.org | 508 |
1 files changed, 508 insertions, 0 deletions
diff --git a/scripts/testing/README.org b/scripts/testing/README.org new file mode 100644 index 0000000..a52dbea --- /dev/null +++ b/scripts/testing/README.org @@ -0,0 +1,508 @@ +#+TITLE: ArchSetup Testing Infrastructure +#+AUTHOR: Craig Jennings +#+DATE: 2025-11-08 + +* Overview + +This directory contains the complete testing infrastructure for archsetup, built on QEMU/KVM virtualization. It provides automated, reproducible testing of the archsetup installation script in isolated virtual machines. + +** Philosophy + +*Realism over speed.* We use full VMs (not containers) to test everything archsetup does: user creation, systemd services, X11/DWM, hardware drivers, and boot process. + +* Quick Start + +** One-Time Setup + +#+begin_src bash +# Install required packages and configure environment +./scripts/testing/setup-testing-env.sh + +# Log out and back in (for libvirt group membership) + +# Create the base VM (minimal Arch installation) +./scripts/testing/create-base-vm.sh +# Follow on-screen instructions to complete installation +# Run finalize-base-vm.sh when done +#+end_src + +** Run a Test + +#+begin_src bash +# Test the current archsetup script +./scripts/testing/run-test.sh + +# Test a specific version +./scripts/testing/run-test.sh --script /path/to/archsetup + +# Keep VM running for debugging +./scripts/testing/run-test.sh --keep +#+end_src + +** Debug Interactively + +#+begin_src bash +# Clone base VM for debugging +./scripts/testing/debug-vm.sh + +# Use existing test disk +./scripts/testing/debug-vm.sh test-results/20251108-143000/test.qcow2 + +# Use base VM (read-only) +./scripts/testing/debug-vm.sh --base +#+end_src + +** Clean Up + +#+begin_src bash +# Interactive cleanup (prompts for confirmation) +./scripts/testing/cleanup-tests.sh + +# Keep last 10 test results +./scripts/testing/cleanup-tests.sh --keep 10 + +# Force cleanup without prompts +./scripts/testing/cleanup-tests.sh --force +#+end_src + +* Scripts + +** setup-testing-env.sh + +*Purpose:* One-time setup of testing infrastructure + +*What it does:* +- Installs QEMU/KVM, libvirt, and related tools +- Configures libvirt networking +- Adds user to libvirt group +- Verifies KVM support +- Creates directories for artifacts + +*When to run:* Once per development machine + +*Usage:* +#+begin_src bash +./scripts/testing/setup-testing-env.sh +#+end_src + +** create-base-vm.sh + +*Purpose:* Create the "golden image" minimal Arch VM + +*What it does:* +- Downloads latest Arch ISO +- Creates VM and boots from ISO +- Opens virt-viewer for you to complete installation manually + +*When to run:* Once (or when you want to refresh base image) + +*Usage:* +#+begin_src bash +./scripts/testing/create-base-vm.sh +#+end_src + +*Process:* +1. Script creates VM and boots from Arch ISO +2. virt-viewer opens automatically showing VM display +3. You complete installation manually using archinstall: + - Login as root (no password) + - Run =archinstall= + - Configure: hostname=archsetup-test, root password=archsetup + - Install packages: openssh git vim sudo + - Enable services: sshd, dhcpcd + - Configure SSH to allow root login + - Poweroff when done +4. Run =./scripts/testing/finalize-base-vm.sh= to complete + +*See also:* [[file:../../docs/base-vm-installation-checklist.org][Base VM Installation Checklist]] + +*Result:* Base VM image at =vm-images/archsetup-base.qcow2= + +** run-test.sh + +*Purpose:* Execute a full test run of archsetup + +*What it does:* +- Clones base VM (copy-on-write, fast) +- Starts test VM +- Transfers archsetup script and dotfiles +- Executes archsetup inside VM +- Captures logs and results +- Runs validation checks +- Generates test report +- Cleans up (unless =--keep=) + +*When to run:* Every time you want to test archsetup + +*Usage:* +#+begin_src bash +# Test current archsetup +./scripts/testing/run-test.sh + +# Test specific version +./scripts/testing/run-test.sh --script /path/to/archsetup + +# Keep VM for debugging +./scripts/testing/run-test.sh --keep +#+end_src + +*Time:* 30-60 minutes (mostly downloading packages) + +*Results:* Saved to =test-results/TIMESTAMP/= +- =test.log= - Complete log output +- =test-report.txt= - Summary of results +- =archsetup-*.log= - Log from archsetup script +- =*.txt= - Package lists from VM + +** debug-vm.sh + +*Purpose:* Launch VM for interactive debugging + +*What it does:* +- Creates VM from base image or existing test disk +- Configures console and SSH access +- Provides connection instructions + +*When to run:* When you need to manually test or debug + +*Usage:* +#+begin_src bash +# Clone base VM for debugging +./scripts/testing/debug-vm.sh + +# Use existing test disk +./scripts/testing/debug-vm.sh vm-images/archsetup-test-20251108-143000.qcow2 + +# Use base VM (read-only) +./scripts/testing/debug-vm.sh --base +#+end_src + +*Connect via:* +- Console: =virsh console archsetup-debug-TIMESTAMP= +- SSH: =ssh root@IP_ADDRESS= (password: archsetup) +- VNC: =virt-viewer archsetup-debug-TIMESTAMP= + +** cleanup-tests.sh + +*Purpose:* Clean up old test VMs and artifacts + +*What it does:* +- Lists all test VMs and destroys them +- Removes test disk images +- Keeps last N test results, deletes rest + +*When to run:* Periodically to free disk space + +*Usage:* +#+begin_src bash +# Interactive cleanup +./scripts/testing/cleanup-tests.sh + +# Keep last 3 test results +./scripts/testing/cleanup-tests.sh --keep 3 + +# Force without prompts +./scripts/testing/cleanup-tests.sh --force +#+end_src + +* Directory Structure + +#+begin_example +archsetup/ +├── scripts/ +│ └── testing/ +│ ├── README.org # This file +│ ├── setup-testing-env.sh # Setup infrastructure +│ ├── create-base-vm.sh # Create base VM +│ ├── run-test.sh # Run tests +│ ├── debug-vm.sh # Interactive debugging +│ ├── cleanup-tests.sh # Clean up +│ ├── finalize-base-vm.sh # Finalize base (generated) +│ ├── archinstall-config.json # Archinstall config +│ └── lib/ +│ ├── logging.sh # Logging utilities +│ └── vm-utils.sh # VM management +├── vm-images/ # VM disk images (gitignored) +│ ├── archsetup-base.qcow2 # Golden image +│ ├── arch-latest.iso # Arch ISO +│ └── archsetup-test-*.qcow2 # Test VMs +├── test-results/ # Test results (gitignored) +│ ├── TIMESTAMP/ +│ │ ├── test.log +│ │ ├── test-report.txt +│ │ └── archsetup-*.log +│ └── latest -> TIMESTAMP/ # Symlink to latest +└── docs/ + └── testing-strategy.org # Complete strategy doc +#+end_example + +* Configuration + +** VM Specifications + +All test VMs use: +- *CPUs:* 2 vCPUs +- *RAM:* 4GB (matches archsetup tmpfs build directory) +- *Disk:* 50GB (thin provisioned qcow2) +- *Network:* NAT via libvirt default network +- *Boot:* UEFI (systemd-boot bootloader) +- *Display:* Serial console + VNC + +Set environment variables to customize: +#+begin_src bash +VM_CPUS=4 VM_RAM=8192 ./scripts/testing/run-test.sh +#+end_src + +** Base VM Specifications + +The base VM contains a minimal Arch installation: +- Base system packages +- Linux kernel and firmware +- OpenSSH server (for automation) +- dhcpcd (for networking) +- git, vim, sudo (essentials) +- Root password: "archsetup" + +This matches the documented prerequisites for archsetup. + +* Validation Checks + +Each test run performs these validation checks: + +** Critical (Must Pass) +1. archsetup exits with code 0 +2. User 'cjennings' was created +3. Dotfiles are stowed (symlinks exist) +4. yay (AUR helper) is installed +5. DWM is built and installed + +** Additional (Future) +- All expected packages installed +- X11 can start +- systemd services enabled +- Firewall configured + +* Troubleshooting + +** VM fails to start + +Check if libvirtd is running: +#+begin_src bash +sudo systemctl status libvirtd +sudo systemctl start libvirtd +#+end_src + +** Cannot get VM IP address + +The VM may not have booted completely or networking failed: +#+begin_src bash +# Check VM status +virsh domstate VM_NAME + +# Connect to console to debug +virsh console VM_NAME + +# Check if dhcpcd is running in VM +systemctl status dhcpcd +#+end_src + +** SSH connection refused + +Wait longer for VM to boot, or check if sshd is enabled: +#+begin_src bash +virsh console VM_NAME +# Inside VM: +systemctl status sshd +systemctl start sshd +#+end_src + +** KVM not available + +Check if virtualization is enabled in BIOS and KVM modules loaded: +#+begin_src bash +# Check for /dev/kvm +ls -l /dev/kvm + +# Load KVM module (Intel) +sudo modprobe kvm-intel + +# Or for AMD +sudo modprobe kvm-amd + +# Verify +lsmod | grep kvm +#+end_src + +** Disk space issues + +Clean up old tests: +#+begin_src bash +./scripts/testing/cleanup-tests.sh --force +#+end_src + +Check disk usage: +#+begin_src bash +du -sh vm-images/ test-results/ +#+end_src + +** Base VM Installation Issues + +*** Firewall blocking VM network + +If the VM cannot reach the internet (100% packet loss when pinging), check the host firewall: + +#+begin_src bash +# Check UFW status on host +sudo ufw status + +# Check libvirt NAT rules on host +sudo iptables -t nat -L -n -v | grep -i libvirt +#+end_src + +*Solution:* Temporarily disable UFW during base VM creation: +#+begin_src bash +sudo ufw disable +# Create base VM +sudo ufw enable +#+end_src + +Or add libvirt rules to UFW: +#+begin_src bash +sudo ufw allow in on virbr0 +sudo ufw allow out on virbr0 +#+end_src + +*** VM network not working after boot + +If =dhcpcd= isn't running or network isn't configured: + +#+begin_src bash +# In the VM - restart network from scratch +killall dhcpcd + +# Bring interface down and up (replace enp1s0 with your interface) +ip link set enp1s0 down +ip link set enp1s0 up + +# Start dhcpcd +dhcpcd enp1s0 + +# Wait and verify +sleep 3 +ip addr show enp1s0 +ip route + +# Test connectivity +ping -c 3 192.168.122.1 # Gateway +ping -c 3 8.8.8.8 # Google DNS +#+end_src + +*** DNS not working (127.0.0.53 in resolv.conf) + +The Live ISO uses systemd-resolved stub resolver which may not work: + +#+begin_src bash +# In the VM - set real DNS servers +echo "nameserver 8.8.8.8" > /etc/resolv.conf +echo "nameserver 1.1.1.1" >> /etc/resolv.conf + +# Test +ping -c 3 archlinux.org +#+end_src + +*** Cannot paste into virt-viewer terminal + +Clipboard integration doesn't work well with virt-viewer. Use HTTP server instead: + +#+begin_src bash +# On host - serve the installation script +cd vm-images +python -m http.server 8000 + +# In VM - download and run +curl http://192.168.122.1:8000/auto-install.sh | bash + +# Or download first to review +curl http://192.168.122.1:8000/auto-install.sh -o install.sh +cat install.sh +bash install.sh +#+end_src + +*** Partitions "in use" error + +If re-running installation after a failed attempt: + +#+begin_src bash +# In the VM - unmount and wipe partitions +mount | grep vda +umount /mnt/boot 2>/dev/null +umount /mnt 2>/dev/null +umount -l /mnt # Lazy unmount if still busy + +# Wipe partition table completely +wipefs -a /dev/vda + +# Run install script again +bash install.sh +#+end_src + +*** Alternative: Use archinstall + +Instead of the auto-install.sh script, you can use Arch's built-in installer: + +#+begin_src bash +# In the VM +archinstall +#+end_src + +*Recommended settings:* +- Disk: =/dev/vda= +- Filesystem: =ext4= +- Bootloader: =systemd-boot= +- Hostname: =archsetup-test= +- Root password: =archsetup= +- Profile: =minimal= +- Additional packages: =openssh dhcpcd git vim sudo= +- Network: =NetworkManager= or =systemd-networkd= + +*After installation, before rebooting:* +#+begin_src bash +# Chroot into new system +arch-chroot /mnt + +# Enable services +systemctl enable sshd +systemctl enable dhcpcd # or NetworkManager + +# Allow root SSH login +sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config +sed -i 's/#PasswordAuthentication yes/PasswordAuthentication yes/' /etc/ssh/sshd_config + +# Exit and poweroff +exit +poweroff +#+end_src + +* Future Enhancements + +** Planned Improvements +- [ ] Fully automated base VM creation (expect-based console automation) +- [ ] Parallel test execution (multiple VMs) +- [ ] Screenshot capture of X11 desktop +- [ ] CI/CD integration (GitHub Actions / GitLab CI) +- [ ] Performance benchmarking over time +- [ ] Cloud-init based base image (faster provisioning) + +** Test Scenarios +- [ ] Idempotency test (run archsetup twice) +- [ ] Network failure recovery +- [ ] Offline installation (local package cache) +- [ ] Different hardware profiles (CPU/RAM variations) + +* References + +- [[file:../../docs/testing-strategy.org][Testing Strategy Document]] +- [[https://wiki.archlinux.org/title/Libvirt][Arch Wiki: libvirt]] +- [[https://wiki.archlinux.org/title/QEMU][Arch Wiki: QEMU]] +- [[file:../../archsetup][Main archsetup script]] +- [[file:../../TODO.org][Project TODO]] |
