* Open Work
** TODO [#A] Build AUR packages and include in ISO as local repository
Build AUR packages during ISO creation and include them in a local pacman repository.
This allows AUR software to work both in the live environment AND be installable to target systems.

*** Implementation Plan

**** 1. Create build infrastructure
Add to build.sh or separate script (build-aur.sh):
#+BEGIN_SRC bash
build_aur_packages() {
    local aur_packages=(downgrade yay sanoid informant rate-mirrors)
    local repo_dir="$PROJECT_DIR/aur-packages"
    local build_dir="/tmp/aur-build"

    mkdir -p "$repo_dir" "$build_dir"

    for pkg in "${aur_packages[@]}"; do
        info "Building AUR package: $pkg"
        git clone --depth 1 "https://aur.archlinux.org/${pkg}.git" "$build_dir/${pkg}"
        (cd "$build_dir/${pkg}" && makepkg -s --noconfirm --needed)
        cp "$build_dir/${pkg}"/*.pkg.tar.zst "$repo_dir/"
    done

    # Create/update repo database
    repo-add "$repo_dir/aur.db.tar.gz" "$repo_dir"/*.pkg.tar.zst
}
#+END_SRC

**** 2. Add local repo to ISO's pacman.conf
In profile/pacman.conf, add:
#+BEGIN_SRC ini
[aur]
SigLevel = Optional TrustAll
Server = file:///usr/share/aur-packages
#+END_SRC

**** 3. Copy repo into ISO
In build.sh, copy aur-packages/ to profile/airootfs/usr/share/aur-packages/

**** 4. Add packages to packages.x86_64
Once in local repo, packages can be listed normally and pacman will find them.

**** 5. Make available during installation
In install-archzfs, copy local repo to target or mount it:
#+BEGIN_SRC bash
# Copy AUR repo to target for installation
cp -r /usr/share/aur-packages /mnt/usr/share/
# Add repo to target's pacman.conf temporarily
#+END_SRC

*** AUR Packages to Include

**** Essential (Priority A)
| Package | Description | Why needed |
|---------+-------------+------------|
| downgrade | Roll back to previous package versions | Essential for recovery when updates break |
| yay | AUR helper | Users can install additional AUR packages |
| informant | Check Arch news before upgrading | Prevents breaking changes from surprises |

**** ZFS Management (Priority A)
| Package | Description | Why needed |
|---------+-------------+------------|
| sanoid | ZFS snapshot policy management | Automated snapshot creation/pruning |
| syncoid | ZFS replication tool (part of sanoid) | Backup to remote systems |
| zrepl | ZFS replication daemon | Alternative to sanoid for replication |

**** System Maintenance (Priority B)
| Package | Description | Why needed |
|---------+-------------+------------|
| rate-mirrors | Fast Arch mirror selection | Better than reflector for speed |
| paru | Alternative AUR helper (Rust) | Some prefer over yay |
| pacman-cleanup-hook | Auto-remove old package cache | Disk space management |
| arch-audit | CVE security monitoring | Check for vulnerable packages |

**** Recovery Tools (Priority B)
| Package | Description | Why needed |
|---------+-------------+------------|
| ventoy-bin | Create multiboot USB drives | Useful rescue tool |
| topgrade | Universal upgrade tool | Update everything at once |
| mkinitcpio-firmware | Suppress firmware warnings | Cleaner initramfs builds |

**** Nice to Have (Priority C)
| Package | Description | Why needed |
|---------+-------------+------------|
| zfs-auto-snapshot | Automatic ZFS snapshots | Simple cron-based snapshots |
| btop | Modern resource monitor | Better than htop |
| duf | Modern disk usage viewer | Better than df |
| dust | Modern du replacement | Intuitive disk usage |
| procs | Modern ps replacement | Better process viewer |

*** Considerations
- Build must run on Arch Linux (or in Arch container)
- Some AUR packages have dependencies that are also AUR - need to handle build order
- Package versions will be frozen at ISO build time
- Consider caching built packages to speed rebuilds
- May want to GPG sign the local repo for security

*** Size Estimate
Most AUR packages are small (<5MB each). Estimate ~50-100MB for full suite.
Significantly less than pre-cloning git repos.

** TODO [#A] Install Arch Wiki on ISO for offline package help
Include offline Arch Wiki on the ISO for package documentation and troubleshooting.
Would be invaluable for rescue scenarios without network access.

Options to research:
- arch-wiki-docs package (official, ~500MB compressed)
- arch-wiki-lite for terminal viewing
- Custom subset focusing on ZFS, recovery, and commonly needed topics

Reference in RESCUE-GUIDE.txt and README.org.

** TODO [#B] Research mkosi as alternative to mkarchiso
Investigate whether mkosi offers advantages over mkarchiso for building the ISO.

*** Reference
- https://wiki.archlinux.org/title/Mkosi

** TODO [#B] Set up CI/CD pipeline for automated ISO builds
*** Options to evaluate
- Self-hosted on TrueNAS (primary target)
  - Gitea + Gitea Actions or Drone CI
  - Jenkins in a jail/VM
  - Woodpecker CI (lightweight Drone fork)
- GitHub Actions (if repo mirrored to GitHub)
- GitLab CI (self-hosted or gitlab.com)

*** Requirements
- Arch Linux build environment (container or VM)
- Sudo/root access for mkarchiso
- ~10GB disk space per build
- Caching for pacman packages to speed builds

*** Considerations
- Trigger builds on push to main
- Scheduled builds (weekly?) to catch upstream updates
- Store artifacts (ISO) with retention policy
- Notifications on build failure
- Test automation (boot ISO in QEMU, run checks)

*** TrueNAS-specific tips
- Use a jail or VM for the CI runner
- Consider bhyve VM with Arch Linux for native builds
- Mount dataset for build artifacts and cache
- Snapshot before/after builds for easy cleanup

** TODO [#B] Consider Avahi for USB boot disk discoverability
Make the live ISO discoverable on the network by name (e.g., archzfs.local) when booted.
This would make SSH access easier without needing to find the IP address.

Already have avahi package installed - just need to configure/enable it.

** TODO [#B] Consider adding bootable archzfs ISO to GRUB boot menu
Store the archzfs ISO on disk and add a GRUB menu entry to boot it directly - no USB drive needed for recovery/reinstall.

*** Benefits
- Always have a rescue environment available
- Can reinstall or rollback without external media
- Useful for remote/headless servers

*** Challenges
1. Storage location - ISO is 5GB. Can't live on ZFS (GRUB can't read it). Options:
   - EFI partition (currently 1GB - would need to be larger)
   - Dedicated recovery partition (ext4 or FAT32)
   - Second EFI partition just for the ISO

2. GRUB loopback boot - Arch ISOs support this with the right kernel params:
   #+BEGIN_SRC
   menuentry "Archzfs Recovery" {
       loopback loop /path/to/archzfs.iso
       linux (loop)/arch/boot/x86_64/vmlinuz-linux archisolabel=ARCHZFS
       initrd (loop)/arch/boot/x86_64/initramfs-linux.img
   }
   #+END_SRC

3. Keeping it updated - Would need a mechanism to update the ISO when rebuilding

*** Questions to resolve
- Is this for recovery scenarios, or would you actually reinstall from it?
- Would you want this integrated into the installer (auto-create recovery partition)?
- Or just document how to set it up manually?

** TODO [#C] Check for methods to abstract

** Ideas from arch-linux-live-cd-iso-with-zfs project
Reference: /home/cjennings/code/arch-linux-live-cd-iso-with-zfs
GitHub: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

*** TODO [#A] Integrate ZFSBootMenu as alternative boot manager
ZFSBootMenu is a powerful boot manager specifically designed for ZFS root systems.
It provides boot environment selection, snapshot rollback from boot menu, and recovery options.

Benefits:
- Boot directly into any ZFS snapshot
- Rollback to previous boot environment without booting first
- Better ZFS-native boot experience than GRUB
- Single EFI file (~15MB) that can be updated independently

Implementation:
- Download portable EFI from https://get.zfsbootmenu.org/efi
- Include in ISO's /root/software/zfsbootmenu/
- Add update script for post-installation updates
- Consider as GRUB alternative or supplement

Reference: https://zfsbootmenu.org/

*** TODO [#B] Add Docker/Podman container support for builds
The reference project uses containers with minimal capabilities instead of full root/privileged mode.
This improves security and reproducibility.

Capabilities needed for mkarchiso:
- DAC_OVERRIDE
- SYS_ADMIN
- SYS_CHROOT
- SYS_MODULE
- Device: /dev/loop-control

Benefits:
- Reproducible builds across different host systems
- No need to install archiso on host
- Cleaner build environment
- Easier CI/CD integration

*** TODO [#B] Support building against Arch Linux Archive snapshots
When archzfs lags behind the main Arch repos, builds can fail due to kernel version mismatch.
The reference project supports pinning to historical repo snapshots.

Implementation:
- Add -r/--repo-date flag to build.sh
- Use archive.archlinux.org for historical packages
- Example: ./build.sh -r 2026/01/15 or ./build.sh -r week

This solves the common problem of ZFS packages not being available for the latest kernel.

*** TODO [#B] Add build logging with tee
Capture all build output to a log file for debugging and CI artifact collection.

Implementation:
#+BEGIN_SRC bash
exec &> >(tee "build-$(date +%Y%m%d-%H%M%S).log")
#+END_SRC

Also useful: Check log for known error patterns (e.g., DKMS failures) and fail fast.

*** TODO [#C] Support multi-variant ISO builds
The reference project builds 8 variants automatically:
- linux vs linux-lts kernel
- DKMS vs native ZFS packages
- Default vs experimental archzfs repos

Could be useful for offering users choice between stability and bleeding-edge.
Lower priority since we currently focus on linux-lts + DKMS for maximum compatibility.

*** TODO [#C] Pre-clone useful tools and documentation into ISO
The reference project bundles Git repos (without .git dirs) into /root:
- archinstall (official installer)
- downgrade (package rollback)
- ZFS howtos and documentation
- Recovery scripts

Already partially implemented (have rescue tools), but could expand with:
- Pre-cloned arch-linux-configuration scripts
- ZFS administration cheatsheets
- Offline troubleshooting guides

*** TODO [#C] Add environment file configuration (.env pattern)
Allow build customization via .env file instead of command-line flags.
Useful for CI/CD and reproducible builds.

Example .env:
#+BEGIN_SRC
KERNEL=linux-lts
USE_DKMS=1
BE_VERBOSE=0
PACKAGES_TO_ADD=git,vim
PACKAGES_TO_REMOVE=b43-fwcutter
#+END_SRC

*** TODO [#C] Add dry-run mode to build.sh
Support --dry-run flag that shows what would be done without executing.
Useful for testing configuration changes and debugging.

* Resolved
** DONE [#B] Add config file information to README
Config file format documented in README.org with full reference and examples.

** DONE [#B] Add CI/CD test infrastructure
Added Makefile, test-install.sh, and test configs for automated VM testing.

** CANCELLED [#C] Consider Dialog-Based Interface for Status, Information, and Questions
Using fzf instead.

** DONE [#C] Consider fzf interface for choices
Implemented fzf for timezone, locale, keymap, disk, RAID, and WiFi selection.
** DONE [#A] Create comprehensive project documentation (README.org)
CLOSED: [2026-01-18 Sun 02:01]
- State "DONE"       from "TODO"       [2026-01-18 Sun 02:01]
*** Proposed sections in order
1. Overview - What this project is
2. Features - Key capabilities
3. Quick Start - Minimal steps to build/boot/install
4. Prerequisites - Arch Linux host, sudo, dependencies
5. Building the ISO - build.sh usage and output
6. Project Structure - File/script locations and descriptions
7. Testing with VMs - test-vm.sh options, SSH access, multi-disk
8. Development Workflow - Rollback to genesis, re-run VM, iterative testing
9. Installation Walkthrough - fzf interface, RAID options, each step explained
10. Bare Metal Installation - Laptop/server installs, WiFi, SSH access after reboot
11. Post-Installation - Genesis snapshot, rollback script, archsetup
12. Keeping Up-to-Date - Rebuilding for new kernel/ZFS versions
13. Troubleshooting - Common pitfalls, known issues, tips
14. Links - archzfs.com, ZFS documentation, Arch Wiki
15. License - GPL

*** Also needed
- Add GPL header to all scripts (build.sh, install-archzfs, test-vm.sh)
- Create LICENSE file in project root

** DONE [#C] Add date/timestamp to install-archzfs log
Log filename now includes timestamp: /tmp/install-archzfs-YYYY-MM-DD-HH-MM-SS.log
Also includes header with start time inside the log file.

** DONE [#B] Add common recovery tools to archzfs ISO
CLOSED: [2026-01-18 Sat]
Make the ISO double as a general-purpose recovery disk.
See custom/RESCUE-GUIDE.txt for comprehensive documentation of all tools.

*** Research Sources
- [[https://www.system-rescue.org/Detailed-packages-list/][SystemRescue Package List]] - Arch-based rescue distro with 1000+ packages
- [[https://alternativeto.net/software/hiren39s-bootcd/][Hiren's BootCD Alternatives]] - Popular rescue disk comparisons
- [[https://www.chntpw.com/][chntpw]] - Windows password reset tool
- [[https://rtech.support/disks/encryption/dislocker/][Dislocker]] - BitLocker decryption on Linux

*** Currently on archzfs ISO
- ZFS tools (zpool, zfs, zfs-utils)
- Base Arch system (coreutils, util-linux, systemd)
- Network (networkmanager, openssh, curl, wget)
- Editors (vim)
- Partitioning (parted, gdisk)
- Filesystem (dosfstools, e2fsprogs)

*** Proposed Additions by Category

**** Data Recovery & Forensics
| Package | Description | Size |
|---------+-------------+------|
| testdisk | Partition/file recovery, fixes partition tables | ~2MB |
| ddrescue | Copy data from failing drives with retries | ~1MB |
| foremost | File carving based on headers/footers | ~1MB |
| sleuthkit | Forensic analysis toolkit (icat, fls, etc.) | ~5MB |
| safecopy | Low-level data recovery from damaged media | ~1MB |

**** Disk Cloning & Imaging
| Package | Description | Size |
|---------+-------------+------|
| partclone | Efficient partition cloning (used by Clonezilla) | ~2MB |
| fsarchiver | Filesystem archiver with compression | ~1MB |
| partimage | Partition imaging (legacy, but useful) | ~1MB |

**** Windows Recovery
| Package | Description | Size |
|---------+-------------+------|
| chntpw | Reset Windows passwords, edit registry | ~500KB |
| dislocker | Read/write BitLocker encrypted volumes | ~1MB |
| ntfs-3g | Full NTFS read/write support | (likely included) |
| hivex | Windows registry hive extraction | ~500KB |
| ms-sys | Write Windows MBR/boot records | ~100KB |

**** Filesystem Tools
| Package | Description | Size |
|---------+-------------+------|
| ntfs-3g | NTFS read/write | ~2MB |
| exfatprogs | exFAT support | ~500KB |
| btrfs-progs | Btrfs tools | ~5MB |
| xfsprogs | XFS tools | ~3MB |
| f2fs-tools | Flash-Friendly FS tools | ~1MB |
| hfsprogs | HFS/HFS+ (Mac) support | ~1MB |

**** Hardware Diagnostics
| Package | Description | Size |
|---------+-------------+------|
| smartmontools | SMART disk health monitoring | ~1MB |
| hdparm | HDD/SSD parameter tuning | ~500KB |
| nvme-cli | NVMe drive management | ~1MB |
| lshw | Detailed hardware lister | ~1MB |
| dmidecode | DMI/SMBIOS decoder | ~500KB |
| memtester | Userspace memory testing | ~100KB |
| stress-ng | CPU/memory/IO stress testing | ~2MB |
| i2c-tools | I2C/SMBus tools | ~500KB |
| lm_sensors | Hardware monitoring (temps, fans, voltages) | ~500KB |

**** Network Diagnostics
| Package | Description | Size |
|---------+-------------+------|
| nmap | Network scanner and security auditing | ~25MB |
| tcpdump | Command-line packet analyzer | ~1MB |
| wireshark-cli | TShark packet analysis | ~50MB |
| iperf3 | Network bandwidth testing | ~500KB |
| mtr | Combined ping/traceroute | ~500KB |
| iftop | Network bandwidth monitor | ~500KB |
| nethogs | Per-process bandwidth monitor | ~500KB |
| ethtool | NIC configuration/diagnostics | ~500KB |
| bind | dig/nslookup DNS tools | ~5MB |

**** Security & Malware Scanning
| Package | Description | Size |
|---------+-------------+------|
| clamav | Open source antivirus engine | ~150MB |
| rkhunter | Rootkit detection | ~1MB |
| chkrootkit | Rootkit checker | ~500KB |
| lynis | Security auditing tool | ~1MB |

**** Boot Repair
| Package | Description | Size |
|---------+-------------+------|
| efibootmgr | UEFI boot manager (likely included) | ~100KB |
| grub | GRUB bootloader tools (included) | - |
| os-prober | Detect other OSes for GRUB | ~100KB |
| syslinux | Legacy bootloader tools | ~2MB |

**** File Management & Compression
| Package | Description | Size |
|---------+-------------+------|
| mc | Midnight Commander file manager | ~2MB |
| ncdu | NCurses disk usage analyzer | ~500KB |
| tree | Directory listing | ~100KB |
| p7zip | 7-Zip compression | ~3MB |
| unrar | RAR extraction | ~500KB |
| cabextract | Windows CAB extraction | ~100KB |
| cpio | Archive tool (initramfs) | ~500KB |

**** Remote Access
| Package | Description | Size |
|---------+-------------+------|
| remmina | RDP/VNC client (GUI) | ~10MB |
| tigervnc | VNC server/client | ~5MB |
| screen | Terminal multiplexer | ~1MB |
| tmux | Terminal multiplexer | ~1MB |

**** Monitoring & System Info
| Package | Description | Size |
|---------+-------------+------|
| htop | Interactive process viewer | ~500KB |
| iotop | I/O monitoring | ~100KB |
| atop | Advanced system monitor | ~500KB |
| inxi | System information script | ~500KB |
| neofetch | System info display | ~500KB |

**** GUI Tools (if keeping Xfce or adding minimal X)
| Package | Description | Size |
|---------+-------------+------|
| gparted | Graphical partition editor | ~5MB |
| baobab | Disk usage analyzer | ~3MB |
| gnome-disk-utility | GNOME Disks | ~5MB |

*** Summary Table (Quick Reference)
| Category | Key Tools | Est. Size |
|----------+-----------+-----------|
| Data Recovery | testdisk, ddrescue, foremost, sleuthkit | ~10MB |
| Disk Cloning | partclone, fsarchiver | ~4MB |
| Windows Recovery | chntpw, dislocker, hivex, ms-sys | ~3MB |
| Filesystem | ntfs-3g, exfatprogs, btrfs-progs, xfsprogs, hfsprogs | ~12MB |
| Hardware Diag | smartmontools, hdparm, nvme-cli, lshw, memtester, stress-ng, lm_sensors | ~8MB |
| Network | nmap, tcpdump, iperf3, mtr, iftop, bind | ~35MB |
| Security | clamav, rkhunter, chkrootkit | ~150MB |
| Boot Repair | os-prober, syslinux | ~2MB |
| File Mgmt | mc, ncdu, p7zip, unrar | ~6MB |
| Remote | screen, tmux, tigervnc | ~7MB |
| Monitoring | htop, iotop, inxi | ~2MB |

*** Size Estimate
- Core recovery tools (no ClamAV/Wireshark): ~90MB
- With ClamAV: +150MB
- With Wireshark CLI: +50MB
- GUI tools (optional): ~20MB
- Total estimate: ~150-300MB additional
- Current ISO size: 5.1GB
- Projected size with all tools: ~5.4GB

*** Discussion Notes
- SystemRescue is also Arch-based, making it a good reference for package names
- Hiren's BootCD PE is now Windows-based, but older Linux version had good tool selection
- Most recovery scenarios need: testdisk, ddrescue, chntpw, smartmontools, ntfs-3g
- For Windows malware scanning, ClamAV is the standard open-source option
- Consider: could offer two ISO variants (minimal vs full rescue)

*** Questions to Resolve
- Include ClamAV? Adds ~150MB but very useful for Windows malware scanning
- Include Wireshark CLI (tshark)? Adds ~50MB for network forensics
- Keep ISO under specific size target (e.g., 6GB for single-layer DVD)?
- Add GUI tools or keep it minimal/CLI-focused?
- Create separate "rescue" build profile or add to main ISO?
- Priority order for implementation?