path: root/scripts/testing/README.org

#+TITLE: ArchSetup Testing Infrastructure
#+AUTHOR: Craig Jennings
#+DATE: 2025-11-08

* Overview

This directory contains the complete testing infrastructure for archsetup, built on QEMU/KVM virtualization. It provides automated, reproducible testing of the archsetup installation script in isolated virtual machines.

** Philosophy

*Realism over speed.* We use full VMs (not containers) to test everything archsetup does: user creation, systemd services, X11/DWM, hardware drivers, and boot process.

** Architecture

Uses direct QEMU with user-mode networking and SSH port forwarding. No libvirt/virsh dependency. Base VM creation is fully automated via the archangel ISO (unattended Arch Linux installer).

* Quick Start

** One-Time Setup

#+begin_src bash
# Install required packages (qemu-full, sshpass, edk2-ovmf, socat)
./scripts/testing/setup-testing-env.sh

# Copy an archangel ISO to vm-images/
cp /path/to/archzfs-*.iso vm-images/

# Create the base VM (fully automated, no manual steps)
./scripts/testing/create-base-vm.sh
#+end_src

** Run a Test

#+begin_src bash
# Test the current archsetup script
./scripts/testing/run-test.sh

# Test a specific version
./scripts/testing/run-test.sh --script /path/to/archsetup

# Keep VM running for debugging
./scripts/testing/run-test.sh --keep
#+end_src

** Debug Interactively

#+begin_src bash
# Launch base VM with graphical display (copy-on-write overlay, safe)
./scripts/testing/debug-vm.sh --base

# Use existing test disk directly
./scripts/testing/debug-vm.sh vm-images/some-disk.qcow2
#+end_src

** Clean Up

#+begin_src bash
# Interactive cleanup (prompts for confirmation)
./scripts/testing/cleanup-tests.sh

# Keep last 10 test results
./scripts/testing/cleanup-tests.sh --keep 10

# Force cleanup without prompts
./scripts/testing/cleanup-tests.sh --force
#+end_src

* Scripts

** setup-testing-env.sh

*Purpose:* One-time setup of testing infrastructure

*What it does:*
- Installs QEMU/KVM, sshpass, OVMF firmware, and socat
- Verifies KVM support
- Creates directories for artifacts

*When to run:* Once per development machine

** create-base-vm.sh

*Purpose:* Create the "golden image" minimal Arch VM

*What it does:*
- Boots an archangel ISO in QEMU with SSH port forwarding
- Waits for SSH to become available on the live ISO
- Copies archangel config file (=archsetup-test.conf=) into the VM
- Runs =archangel --config-file= for fully unattended installation
- Boots the installed system and verifies SSH/services
- Creates a "clean-install" qemu-img snapshot

*When to run:* Once (or when you want to refresh base image)

*Result:* Base VM image at =vm-images/archsetup-base.qcow2= with "clean-install" snapshot

** run-test.sh

*Purpose:* Execute a full test run of archsetup

*What it does:*
- Reverts base VM to clean-install snapshot
- Boots the VM via QEMU
- Transfers archsetup script and dotfiles via git bundle
- Executes archsetup inside VM
- Captures logs and results
- Runs comprehensive validation checks
- Generates test report
- Reverts to clean snapshot (unless =--keep=)

*When to run:* Every time you want to test archsetup

*Results:* Saved to =test-results/TIMESTAMP/=
- =test.log= - Complete log output
- =test-report.txt= - Summary of results
- =archsetup-*.log= - Log from archsetup script
- =*.txt= - Package lists from VM

** debug-vm.sh

*Purpose:* Launch VM for interactive debugging

*What it does:*
- Creates a copy-on-write overlay disk (instant, protects base image)
- Boots QEMU with GTK graphical display
- Provides SSH connection info
- Cleans up overlay when the GTK window is closed

*When to run:* When you need to manually test or debug

*Connect via:*
- GTK window (opens automatically)
- SSH: =sshpass -p 'archsetup' ssh -p 2222 root@localhost=

** cleanup-tests.sh

*Purpose:* Clean up old test VMs and artifacts

*What it does:*
- Stops running QEMU processes
- Kills orphaned QEMU processes
- Removes temporary disk images (overlays, test clones)
- Keeps last N test results, deletes rest

*When to run:* Periodically to free disk space

** archsetup-test.conf

*Purpose:* Archangel config file for automated base VM creation

Defines the base system: btrfs filesystem, no encryption, SSH enabled, root password "archsetup". Used by =create-base-vm.sh=.

* Directory Structure

#+begin_example
archsetup/
├── scripts/
│   └── testing/
│       ├── README.org                    # This file
│       ├── setup-testing-env.sh          # Setup infrastructure
│       ├── create-base-vm.sh             # Create base VM (automated)
│       ├── run-test.sh                   # Run tests
│       ├── run-test-baremetal.sh         # Bare-metal testing
│       ├── debug-vm.sh                   # Interactive debugging
│       ├── cleanup-tests.sh              # Clean up
│       ├── archsetup-test.conf           # Archangel config for test VMs
│       └── lib/
│           ├── logging.sh                # Logging utilities
│           ├── vm-utils.sh               # QEMU VM management
│           ├── validation.sh             # Test validation checks
│           └── network-diagnostics.sh    # Network pre-flight checks
├── vm-images/                            # VM disk images (gitignored)
│   ├── archsetup-base.qcow2             # Golden image with snapshot
│   ├── archzfs-*.iso                    # Archangel ISO
│   ├── OVMF_VARS.fd                     # UEFI variables (per-VM)
│   └── debug-overlay-*.qcow2            # Temp debug overlays
├── test-results/                         # Test results (gitignored)
│   └── TIMESTAMP/
│       ├── test.log
│       ├── test-report.txt
│       └── archsetup-*.log
└── docs/
    ├── testing-strategy.org              # Complete strategy doc
    └── archangel-vm-testing-guide.org    # Archangel integration guide
#+end_example

* Configuration

** VM Specifications

All test VMs use:
- *CPUs:* 4 vCPUs
- *RAM:* 4GB
- *Disk:* 50GB (thin provisioned qcow2)
- *Network:* User-mode networking with SSH port forwarding (localhost:2222)
- *Boot:* UEFI (via OVMF firmware)
- *Display:* Headless (automated) or GTK (debug)

Set environment variables to customize:
#+begin_src bash
VM_CPUS=8 VM_RAM=8192 ./scripts/testing/run-test.sh
#+end_src

** Base VM Specifications

The base VM is created automatically by archangel with:
- Btrfs filesystem (no encryption)
- GRUB bootloader
- OpenSSH server (root login enabled)
- NetworkManager
- Root password: "archsetup"

* Validation Checks

Each test run performs these validation checks:

** Critical (Must Pass)
1. archsetup exits with code 0
2. User 'cjennings' was created
3. Dotfiles are stowed (symlinks exist)
4. yay (AUR helper) is installed
5. DWM or Hyprland is configured

** Additional
- All expected packages installed
- systemd services enabled
- Firewall configured
- Developer tools present

* Troubleshooting

** VM fails to start

Check if KVM is available and port 2222 is free:
#+begin_src bash
ls -l /dev/kvm
ss -tln | grep 2222
#+end_src

** SSH connection refused

Wait longer for VM to boot, or check serial log:
#+begin_src bash
cat vm-images/qemu-serial.log
#+end_src

** KVM not available

Check if virtualization is enabled in BIOS and KVM modules loaded:
#+begin_src bash
ls -l /dev/kvm
sudo modprobe kvm-amd   # or kvm-intel
lsmod | grep kvm
#+end_src

** Disk space issues

Clean up old tests:
#+begin_src bash
./scripts/testing/cleanup-tests.sh --force
du -sh vm-images/ test-results/
#+end_src

** Port 2222 already in use

Another QEMU instance may be running:
#+begin_src bash
# Find what's using the port
ss -tlnp | grep 2222

# Kill orphaned QEMU processes
pkill -f "qemu-system.*archsetup-test"
#+end_src

* Future Enhancements

** Planned Improvements
- [ ] Parallel test execution (multiple VMs on different ports)
- [ ] Screenshot capture of X11/Wayland desktop
- [ ] CI/CD integration (GitHub Actions / GitLab CI)
- [ ] Performance benchmarking over time

** Test Scenarios
- [ ] Idempotency test (run archsetup twice)
- [ ] Network failure recovery
- [ ] Offline installation (local package cache)
- [ ] Different hardware profiles (CPU/RAM variations)

* References

- [[file:../../docs/testing-strategy.org][Testing Strategy Document]]
- [[file:../../docs/archangel-vm-testing-guide.org][Archangel VM Testing Guide]]
- [[https://wiki.archlinux.org/title/QEMU][Arch Wiki: QEMU]]
- [[file:../../archsetup][Main archsetup script]]
- [[file:../../todo.org][Project TODO]]