path: root/scripts/testing/README.org

#+TITLE: ArchSetup Testing Infrastructure
#+AUTHOR: Craig Jennings
#+DATE: 2025-11-08

* Overview

This directory contains the complete testing infrastructure for archsetup, built on QEMU/KVM virtualization. It provides automated, reproducible testing of the archsetup installation script in isolated virtual machines.

** Philosophy

*Realism over speed.* We use full VMs (not containers) to test everything archsetup does: user creation, systemd services, X11/DWM, hardware drivers, and boot process.

* Quick Start

** One-Time Setup

#+begin_src bash
# Install required packages and configure environment
./scripts/testing/setup-testing-env.sh

# Log out and back in (for libvirt group membership)

# Create the base VM (minimal Arch installation)
./scripts/testing/create-base-vm.sh
# Follow on-screen instructions to complete installation
# Run finalize-base-vm.sh when done
#+end_src

** Run a Test

#+begin_src bash
# Test the current archsetup script
./scripts/testing/run-test.sh

# Test a specific version
./scripts/testing/run-test.sh --script /path/to/archsetup

# Keep VM running for debugging
./scripts/testing/run-test.sh --keep
#+end_src

** Debug Interactively

#+begin_src bash
# Clone base VM for debugging
./scripts/testing/debug-vm.sh

# Use existing test disk
./scripts/testing/debug-vm.sh test-results/20251108-143000/test.qcow2

# Use base VM (read-only)
./scripts/testing/debug-vm.sh --base
#+end_src

** Clean Up

#+begin_src bash
# Interactive cleanup (prompts for confirmation)
./scripts/testing/cleanup-tests.sh

# Keep last 10 test results
./scripts/testing/cleanup-tests.sh --keep 10

# Force cleanup without prompts
./scripts/testing/cleanup-tests.sh --force
#+end_src

* Scripts

** setup-testing-env.sh

*Purpose:* One-time setup of testing infrastructure

*What it does:*
- Installs QEMU/KVM, libvirt, and related tools
- Configures libvirt networking
- Adds user to libvirt group
- Verifies KVM support
- Creates directories for artifacts

*When to run:* Once per development machine

*Usage:*
#+begin_src bash
./scripts/testing/setup-testing-env.sh
#+end_src

** create-base-vm.sh

*Purpose:* Create the "golden image" minimal Arch VM

*What it does:*
- Downloads latest Arch ISO
- Creates VM and boots from ISO
- Opens virt-viewer for you to complete installation manually

*When to run:* Once (or when you want to refresh base image)

*Usage:*
#+begin_src bash
./scripts/testing/create-base-vm.sh
#+end_src

*Process:*
1. Script creates VM and boots from Arch ISO
2. virt-viewer opens automatically showing VM display
3. You complete installation manually using archinstall:
   - Login as root (no password)
   - Run =archinstall=
   - Configure: hostname=archsetup-test, root password=archsetup
   - Install packages: openssh git vim sudo
   - Enable services: sshd, dhcpcd
   - Configure SSH to allow root login
   - Poweroff when done
4. Run =./scripts/testing/finalize-base-vm.sh= to complete

*See also:* [[file:../../docs/base-vm-installation-checklist.org][Base VM Installation Checklist]]

*Result:* Base VM image at =vm-images/archsetup-base.qcow2=

** run-test.sh

*Purpose:* Execute a full test run of archsetup

*What it does:*
- Clones base VM (copy-on-write, fast)
- Starts test VM
- Transfers archsetup script and dotfiles
- Executes archsetup inside VM
- Captures logs and results
- Runs validation checks
- Generates test report
- Cleans up (unless =--keep=)

*When to run:* Every time you want to test archsetup

*Usage:*
#+begin_src bash
# Test current archsetup
./scripts/testing/run-test.sh

# Test specific version
./scripts/testing/run-test.sh --script /path/to/archsetup

# Keep VM for debugging
./scripts/testing/run-test.sh --keep
#+end_src

*Time:* 30-60 minutes (mostly downloading packages)

*Results:* Saved to =test-results/TIMESTAMP/=
- =test.log= - Complete log output
- =test-report.txt= - Summary of results
- =archsetup-*.log= - Log from archsetup script
- =*.txt= - Package lists from VM

** debug-vm.sh

*Purpose:* Launch VM for interactive debugging

*What it does:*
- Creates VM from base image or existing test disk
- Configures console and SSH access
- Provides connection instructions

*When to run:* When you need to manually test or debug

*Usage:*
#+begin_src bash
# Clone base VM for debugging
./scripts/testing/debug-vm.sh

# Use existing test disk
./scripts/testing/debug-vm.sh vm-images/archsetup-test-20251108-143000.qcow2

# Use base VM (read-only)
./scripts/testing/debug-vm.sh --base
#+end_src

*Connect via:*
- Console: =virsh console archsetup-debug-TIMESTAMP=
- SSH: =ssh root@IP_ADDRESS= (password: archsetup)
- VNC: =virt-viewer archsetup-debug-TIMESTAMP=

** cleanup-tests.sh

*Purpose:* Clean up old test VMs and artifacts

*What it does:*
- Lists all test VMs and destroys them
- Removes test disk images
- Keeps last N test results, deletes rest

*When to run:* Periodically to free disk space

*Usage:*
#+begin_src bash
# Interactive cleanup
./scripts/testing/cleanup-tests.sh

# Keep last 3 test results
./scripts/testing/cleanup-tests.sh --keep 3

# Force without prompts
./scripts/testing/cleanup-tests.sh --force
#+end_src

* Directory Structure

#+begin_example
archsetup/
├── scripts/
│   └── testing/
│       ├── README.org                    # This file
│       ├── setup-testing-env.sh          # Setup infrastructure
│       ├── create-base-vm.sh             # Create base VM
│       ├── run-test.sh                   # Run tests
│       ├── debug-vm.sh                   # Interactive debugging
│       ├── cleanup-tests.sh              # Clean up
│       ├── finalize-base-vm.sh           # Finalize base (generated)
│       ├── archinstall-config.json       # Archinstall config
│       └── lib/
│           ├── logging.sh                # Logging utilities
│           └── vm-utils.sh               # VM management
├── vm-images/                            # VM disk images (gitignored)
│   ├── archsetup-base.qcow2             # Golden image
│   ├── arch-latest.iso                   # Arch ISO
│   └── archsetup-test-*.qcow2           # Test VMs
├── test-results/                         # Test results (gitignored)
│   ├── TIMESTAMP/
│   │   ├── test.log
│   │   ├── test-report.txt
│   │   └── archsetup-*.log
│   └── latest -> TIMESTAMP/              # Symlink to latest
└── docs/
    └── testing-strategy.org              # Complete strategy doc
#+end_example

* Configuration

** VM Specifications

All test VMs use:
- *CPUs:* 2 vCPUs
- *RAM:* 4GB (matches archsetup tmpfs build directory)
- *Disk:* 50GB (thin provisioned qcow2)
- *Network:* NAT via libvirt default network
- *Boot:* UEFI (systemd-boot bootloader)
- *Display:* Serial console + VNC

Set environment variables to customize:
#+begin_src bash
VM_CPUS=4 VM_RAM=8192 ./scripts/testing/run-test.sh
#+end_src

** Base VM Specifications

The base VM contains a minimal Arch installation:
- Base system packages
- Linux kernel and firmware
- OpenSSH server (for automation)
- dhcpcd (for networking)
- git, vim, sudo (essentials)
- Root password: "archsetup"

This matches the documented prerequisites for archsetup.

* Validation Checks

Each test run performs these validation checks:

** Critical (Must Pass)
1. archsetup exits with code 0
2. User 'cjennings' was created
3. Dotfiles are stowed (symlinks exist)
4. yay (AUR helper) is installed
5. DWM is built and installed

** Additional (Future)
- All expected packages installed
- X11 can start
- systemd services enabled
- Firewall configured

* Troubleshooting

** VM fails to start

Check if libvirtd is running:
#+begin_src bash
sudo systemctl status libvirtd
sudo systemctl start libvirtd
#+end_src

** Cannot get VM IP address

The VM may not have booted completely or networking failed:
#+begin_src bash
# Check VM status
virsh domstate VM_NAME

# Connect to console to debug
virsh console VM_NAME

# Check if dhcpcd is running in VM
systemctl status dhcpcd
#+end_src

** SSH connection refused

Wait longer for VM to boot, or check if sshd is enabled:
#+begin_src bash
virsh console VM_NAME
# Inside VM:
systemctl status sshd
systemctl start sshd
#+end_src

** KVM not available

Check if virtualization is enabled in BIOS and KVM modules loaded:
#+begin_src bash
# Check for /dev/kvm
ls -l /dev/kvm

# Load KVM module (Intel)
sudo modprobe kvm-intel

# Or for AMD
sudo modprobe kvm-amd

# Verify
lsmod | grep kvm
#+end_src

** Disk space issues

Clean up old tests:
#+begin_src bash
./scripts/testing/cleanup-tests.sh --force
#+end_src

Check disk usage:
#+begin_src bash
du -sh vm-images/ test-results/
#+end_src

** Base VM Installation Issues

*** Firewall blocking VM network

If the VM cannot reach the internet (100% packet loss when pinging), check the host firewall:

#+begin_src bash
# Check UFW status on host
sudo ufw status

# Check libvirt NAT rules on host
sudo iptables -t nat -L -n -v | grep -i libvirt
#+end_src

*Solution:* Temporarily disable UFW during base VM creation:
#+begin_src bash
sudo ufw disable
# Create base VM
sudo ufw enable
#+end_src

Or add libvirt rules to UFW:
#+begin_src bash
sudo ufw allow in on virbr0
sudo ufw allow out on virbr0
#+end_src

*** VM network not working after boot

If =dhcpcd= isn't running or network isn't configured:

#+begin_src bash
# In the VM - restart network from scratch
killall dhcpcd

# Bring interface down and up (replace enp1s0 with your interface)
ip link set enp1s0 down
ip link set enp1s0 up

# Start dhcpcd
dhcpcd enp1s0

# Wait and verify
sleep 3
ip addr show enp1s0
ip route

# Test connectivity
ping -c 3 192.168.122.1  # Gateway
ping -c 3 8.8.8.8        # Google DNS
#+end_src

*** DNS not working (127.0.0.53 in resolv.conf)

The Live ISO uses systemd-resolved stub resolver which may not work:

#+begin_src bash
# In the VM - set real DNS servers
echo "nameserver 8.8.8.8" > /etc/resolv.conf
echo "nameserver 1.1.1.1" >> /etc/resolv.conf

# Test
ping -c 3 archlinux.org
#+end_src

*** Cannot paste into virt-viewer terminal

Clipboard integration doesn't work well with virt-viewer. Use HTTP server instead:

#+begin_src bash
# On host - serve the installation script
cd vm-images
python -m http.server 8000

# In VM - download and run
curl http://192.168.122.1:8000/auto-install.sh | bash

# Or download first to review
curl http://192.168.122.1:8000/auto-install.sh -o install.sh
cat install.sh
bash install.sh
#+end_src

*** Partitions "in use" error

If re-running installation after a failed attempt:

#+begin_src bash
# In the VM - unmount and wipe partitions
mount | grep vda
umount /mnt/boot 2>/dev/null
umount /mnt 2>/dev/null
umount -l /mnt  # Lazy unmount if still busy

# Wipe partition table completely
wipefs -a /dev/vda

# Run install script again
bash install.sh
#+end_src

*** Alternative: Use archinstall

Instead of the auto-install.sh script, you can use Arch's built-in installer:

#+begin_src bash
# In the VM
archinstall
#+end_src

*Recommended settings:*
- Disk: =/dev/vda=
- Filesystem: =ext4=
- Bootloader: =systemd-boot=
- Hostname: =archsetup-test=
- Root password: =archsetup=
- Profile: =minimal=
- Additional packages: =openssh dhcpcd git vim sudo=
- Network: =NetworkManager= or =systemd-networkd=

*After installation, before rebooting:*
#+begin_src bash
# Chroot into new system
arch-chroot /mnt

# Enable services
systemctl enable sshd
systemctl enable dhcpcd  # or NetworkManager

# Allow root SSH login
sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication yes/' /etc/ssh/sshd_config

# Exit and poweroff
exit
poweroff
#+end_src

* Future Enhancements

** Planned Improvements
- [ ] Fully automated base VM creation (expect-based console automation)
- [ ] Parallel test execution (multiple VMs)
- [ ] Screenshot capture of X11 desktop
- [ ] CI/CD integration (GitHub Actions / GitLab CI)
- [ ] Performance benchmarking over time
- [ ] Cloud-init based base image (faster provisioning)

** Test Scenarios
- [ ] Idempotency test (run archsetup twice)
- [ ] Network failure recovery
- [ ] Offline installation (local package cache)
- [ ] Different hardware profiles (CPU/RAM variations)

* References

- [[file:../../docs/testing-strategy.org][Testing Strategy Document]]
- [[https://wiki.archlinux.org/title/Libvirt][Arch Wiki: libvirt]]
- [[https://wiki.archlinux.org/title/QEMU][Arch Wiki: QEMU]]
- [[file:../../archsetup][Main archsetup script]]
- [[file:../../TODO.org][Project TODO]]