path: root/todo.org

* Open Work

** TODO [#A] Fix mkinitcpio configuration in install-archzfs (causes boot failure)
After kernel updates or mkinitcpio regeneration, systems fail to boot because install-archzfs
leaves incorrect mkinitcpio configuration from the live ISO environment.

See [[file:docs/2026-01-22-mkinitcpio-config-boot-failure.org][bug report]] for full details.

*** Three issues to fix

1. *Wrong HOOKS in mkinitcpio.conf* - uses systemd init (incompatible with ZFS hook), missing zfs hook
   #+BEGIN_SRC bash
   sed -i 's/^HOOKS=.*/HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block zfs filesystems)/' /mnt/etc/mkinitcpio.conf
   #+END_SRC

2. *Leftover archiso.conf drop-in* - overrides HOOKS setting
   #+BEGIN_SRC bash
   rm -f /mnt/etc/mkinitcpio.conf.d/archiso.conf
   #+END_SRC

3. *Wrong preset file* - has archiso configuration instead of standard
   #+BEGIN_SRC bash
   cat > /mnt/etc/mkinitcpio.d/linux-lts.preset << 'EOF'
   PRESETS=(default fallback)
   ALL_kver="/boot/vmlinuz-linux-lts"
   default_image="/boot/initramfs-linux-lts.img"
   fallback_image="/boot/initramfs-linux-lts-fallback.img"
   fallback_options="-S autodetect"
   EOF
   #+END_SRC

4. *Rebuild initramfs after fixing*
   #+BEGIN_SRC bash
   arch-chroot /mnt mkinitcpio -P
   #+END_SRC

** TODO [#A] Build AUR packages and include in ISO as local repository
Build AUR packages during ISO creation and include them in a local pacman repository.
This allows AUR software to work both in the live environment AND be installable to target systems.

*** Implementation Plan

**** 1. Create build infrastructure
Add to build.sh or separate script (build-aur.sh):
#+BEGIN_SRC bash
build_aur_packages() {
    local aur_packages=(downgrade yay sanoid informant rate-mirrors)
    local repo_dir="$PROJECT_DIR/aur-packages"
    local build_dir="/tmp/aur-build"

    mkdir -p "$repo_dir" "$build_dir"

    for pkg in "${aur_packages[@]}"; do
        info "Building AUR package: $pkg"
        git clone --depth 1 "https://aur.archlinux.org/${pkg}.git" "$build_dir/${pkg}"
        (cd "$build_dir/${pkg}" && makepkg -s --noconfirm --needed)
        cp "$build_dir/${pkg}"/*.pkg.tar.zst "$repo_dir/"
    done

    # Create/update repo database
    repo-add "$repo_dir/aur.db.tar.gz" "$repo_dir"/*.pkg.tar.zst
}
#+END_SRC

**** 2. Add local repo to ISO's pacman.conf
In profile/pacman.conf, add:
#+BEGIN_SRC ini
[aur]
SigLevel = Optional TrustAll
Server = file:///usr/share/aur-packages
#+END_SRC

**** 3. Copy repo into ISO
In build.sh, copy aur-packages/ to profile/airootfs/usr/share/aur-packages/

**** 4. Add packages to packages.x86_64
Once in local repo, packages can be listed normally and pacman will find them.

**** 5. Make available during installation
In install-archzfs, copy local repo to target or mount it:
#+BEGIN_SRC bash
# Copy AUR repo to target for installation
cp -r /usr/share/aur-packages /mnt/usr/share/
# Add repo to target's pacman.conf temporarily
#+END_SRC

*** AUR Packages to Include

**** Essential (Priority A)
| Package | Description | Why needed |
|---------+-------------+------------|
| downgrade | Roll back to previous package versions | Essential for recovery when updates break |
| yay | AUR helper | Users can install additional AUR packages |
| informant | Check Arch news before upgrading | Prevents breaking changes from surprises |
| arch-wiki-lite | Offline Arch Wiki with CLI reader | Documentation when network is down |

**** ZFS Management (Priority A)
| Package | Description | Why needed |
|---------+-------------+------------|
| sanoid | ZFS snapshot policy management | Automated snapshot creation/pruning |
| syncoid | ZFS replication tool (part of sanoid) | Backup to remote systems |
| zrepl | ZFS replication daemon | Alternative to sanoid for replication |

**** System Maintenance (Priority B)
| Package | Description | Why needed |
|---------+-------------+------------|
| rate-mirrors | Fast Arch mirror selection | Better than reflector for speed |
| paru | Alternative AUR helper (Rust) | Some prefer over yay |
| pacman-cleanup-hook | Auto-remove old package cache | Disk space management |
| arch-audit | CVE security monitoring | Check for vulnerable packages |

**** Recovery Tools (Priority B)
| Package | Description | Why needed |
|---------+-------------+------------|
| ventoy-bin | Create multiboot USB drives | Useful rescue tool |
| topgrade | Universal upgrade tool | Update everything at once |
| mkinitcpio-firmware | Suppress firmware warnings | Cleaner initramfs builds |

**** Nice to Have (Priority C)
| Package | Description | Why needed |
|---------+-------------+------------|
| zfs-auto-snapshot | Automatic ZFS snapshots | Simple cron-based snapshots |
| btop | Modern resource monitor | Better than htop |
| duf | Modern disk usage viewer | Better than df |
| dust | Modern du replacement | Intuitive disk usage |
| procs | Modern ps replacement | Better process viewer |

*** Considerations
- Build must run on Arch Linux (or in Arch container)
- Some AUR packages have dependencies that are also AUR - need to handle build order
- Package versions will be frozen at ISO build time
- Consider caching built packages to speed rebuilds
- May want to GPG sign the local repo for security

*** Size Estimate
Most AUR packages are small (<5MB each). Estimate ~50-100MB for full suite.
Significantly less than pre-cloning git repos.

** TODO [#A] Install Arch Wiki on ISO for offline package help
Include offline Arch Wiki on the ISO for package documentation and troubleshooting.
Invaluable for rescue scenarios when networking is broken - exactly when you need docs most.

*** Reader Options

**** Option 1: arch-wiki-lite (AUR) - Recommended
Purpose-built CLI reader for offline Arch Wiki.
#+BEGIN_SRC bash
$ wiki-search zfs
$ wiki-search mkinitcpio
#+END_SRC
- Searches articles by keyword
- Displays content as plain text in terminal
- Add to our AUR package build list

**** Option 2: arch-wiki-docs (official) + text browser
- Package installs HTML to /usr/share/doc/arch-wiki/html/
- Use w3m, lynx, or elinks to browse
- Would need wrapper script for searching:
#+BEGIN_SRC bash
#!/bin/bash
# archwiki - search and display offline Arch Wiki
WIKI_DIR="/usr/share/doc/arch-wiki/html"
SEARCH="$1"
MATCHES=$(find "$WIKI_DIR" -iname "*${SEARCH}*" -type f)
if [[ -n "$MATCHES" ]]; then
    echo "$MATCHES" | fzf --preview 'w3m -dump {}' | xargs w3m
fi
#+END_SRC

*** Implementation
1. Add arch-wiki-lite to AUR package list (leverages existing AUR infrastructure)
2. Also install arch-wiki-docs for complete HTML version
3. Add w3m for HTML viewing as backup
4. Document in RESCUE-GUIDE.txt

*** Size
- arch-wiki-docs: ~500MB compressed
- arch-wiki-lite: ~200MB (text only, smaller)
- Could include both for ~600MB total

** TODO [#A] Fix ZFS rollback breaking boot (/boot not on ZFS)
ZFS rollbacks can leave the system unbootable because /boot is on a separate EFI partition
that doesn't get rolled back with the ZFS root filesystem.

*** The Problem
When rolling back ZFS:
- /usr/lib/modules/ (kernel modules) gets rolled back
- /var/lib/pacman/ (package database) gets rolled back
- Everything else on ZFS root gets rolled back

But /boot (EFI partition) does NOT roll back:
- Kernel images (vmlinuz-*) remain at newer version
- Initramfs images remain (may reference missing modules)
- GRUB config still lists kernels that may not have matching modules

Result: After rollback, GRUB shows kernels that can't boot because their modules
no longer exist on root. User gets kernel panic or missing module errors.

*** Why This Matters
- Kernel updates happen frequently and often go unnoticed
- User does ZFS rollback for unrelated reason
- System fails to boot with confusing errors
- Defeats the purpose of ZFS snapshots for easy recovery

*** Solutions

**** Option 1: ZFSBootMenu (Recommended)
Replace GRUB with ZFSBootMenu which is designed for ZFS boot environments.
- Boots directly from ZFS snapshots
- Kernel and initramfs stored on ZFS (rolled back together)
- Can select boot environment from boot menu
- See existing task below for implementation details

**** Option 2: Put /boot on ZFS
- GRUB can read ZFS (with limitations)
- Requires careful GRUB configuration
- May have issues with ZFS features GRUB doesn't support

**** Option 3: Sync /boot snapshots with ZFS
- Script to backup /boot before ZFS snapshot
- Restore /boot when rolling back ZFS
- More complex, error-prone

**** Option 4: Always rebuild initramfs after rollback
- Document this as required step
- Add helper script to automate
- Doesn't help if kernel package itself was rolled back

*** References
- https://zfsbootmenu.org/
- https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS
- https://openzfs.github.io/openzfs-docs/Getting%20Started/Arch%20Linux/index.html

** TODO [#A] Integrate ZFSBootMenu as alternative boot manager
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

ZFSBootMenu is a powerful boot manager specifically designed for ZFS root systems.
It provides boot environment selection, snapshot rollback from boot menu, and recovery options.

*** Benefits
- Boot directly into any ZFS snapshot
- Rollback to previous boot environment without booting first
- Better ZFS-native boot experience than GRUB
- Single EFI file (~15MB) that can be updated independently

*** Implementation
- Download portable EFI from https://get.zfsbootmenu.org/efi
- Include in ISO's /root/software/zfsbootmenu/
- Add update script for post-installation updates
- Consider as GRUB alternative or supplement

*** Reference
https://zfsbootmenu.org/

** TODO [#B] Set up CI/CD pipeline for automated ISO builds

*** Options to evaluate
- Self-hosted on TrueNAS (primary target)
  - Gitea + Gitea Actions or Drone CI
  - Jenkins in a jail/VM
  - Woodpecker CI (lightweight Drone fork)
- GitHub Actions (if repo mirrored to GitHub)
- GitLab CI (self-hosted or gitlab.com)

*** Requirements
- Arch Linux build environment (container or VM)
- Sudo/root access for mkarchiso
- ~10GB disk space per build
- Caching for pacman packages to speed builds

*** Considerations
- Trigger builds on push to main
- Scheduled builds (weekly?) to catch upstream updates
- Store artifacts (ISO) with retention policy
- Notifications on build failure
- Test automation (boot ISO in QEMU, run checks)

*** TrueNAS-specific tips
- Use a jail or VM for the CI runner
- Consider bhyve VM with Arch Linux for native builds
- Mount dataset for build artifacts and cache
- Snapshot before/after builds for easy cleanup

** TODO [#B] Add RAID configuration tests (mirror, raidz)
Currently full-test.sh only tests single-disk installations.
Add tests for multi-disk RAID configurations to catch regressions.

*** Test configurations to add
- 2-disk mirror (most common RAID setup)
- 3-disk raidz1 (if practical for VM testing)
- Verify pool redundancy after installation
- Test degraded pool boot scenario

*** Implementation
- Add --raid flag to full-test.sh or separate test-raid.sh
- Create multiple virtual disks for VM
- Verify zpool status shows correct redundancy
- Consider testing disk failure/resilience

** TODO [#B] Add pre-flight validation to install-archzfs
Validate configuration and environment before any destructive operations.
Fail fast with clear error messages rather than failing mid-install.

*** Validations to add
- Disk exists and is accessible
- Disk is not mounted or in use
- Sufficient disk space (minimum 20GB recommended)
- Network connectivity (for package downloads)
- Required commands available (zpool, zfs, etc.)
- Config file syntax valid (if using unattended mode)
- EFI variables accessible (for UEFI installs)

*** Benefits
- Prevents partial installations that leave system in bad state
- Clear error messages help users fix issues before starting
- Reduces support burden from avoidable failures
- Aligns with "fail fast" testing principle

*** Implementation
Add validate_environment() function called before any disk operations:
#+BEGIN_SRC bash
validate_environment() {
    local errors=0

    # Check disk exists
    [[ -b "$INSTALL_DISK" ]] || { error "Disk $INSTALL_DISK not found"; ((errors++)); }

    # Check not mounted
    mountpoint -q "$INSTALL_DISK"* && { error "Disk is mounted"; ((errors++)); }

    # Check ZFS tools
    command -v zpool >/dev/null || { error "zpool not found"; ((errors++)); }

    [[ $errors -gt 0 ]] && exit 1
}
#+END_SRC

** TODO [#B] Extract install-archzfs into testable functions
Current install-archzfs is a monolithic script that's hard to test in isolation.
Refactoring into focused functions enables unit testing and better maintainability.

*** Current pain points
- Can't test partition creation without running full install
- Can't test ZFS pool creation logic separately
- Failures mid-install are hard to diagnose
- No way to verify individual steps worked

*** Proposed refactoring
Extract into library of functions that can be sourced and tested:
- lib/partition.sh - Disk partitioning functions
- lib/zfs.sh - ZFS pool and dataset creation
- lib/bootloader.sh - GRUB installation and configuration
- lib/network.sh - Network configuration (mDNS, etc.)
- lib/packages.sh - Package installation via pacstrap

*** Testing approach
- Unit tests for pure functions (config parsing, validation)
- Integration tests using small loop devices
- Mock destructive operations for safety
- Test each function's error handling

*** Benefits
- Easier to debug failures (which function failed?)
- Can test edge cases without full VM boot
- Reusable functions for other projects
- Clearer code organization

** TODO [#B] Add Docker/Podman container support for builds
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

Use containers with minimal capabilities instead of full root/privileged mode.
This improves security and reproducibility.

*** Capabilities needed for mkarchiso
- DAC_OVERRIDE
- SYS_ADMIN
- SYS_CHROOT
- SYS_MODULE
- Device: /dev/loop-control

*** Benefits
- Reproducible builds across different host systems
- No need to install archiso on host
- Cleaner build environment
- Easier CI/CD integration

** TODO [#B] Support building against Arch Linux Archive snapshots
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

When archzfs lags behind the main Arch repos, builds can fail due to kernel version mismatch.
Pinning to historical repo snapshots solves this problem.

*** Implementation
- Add -r/--repo-date flag to build.sh
- Use archive.archlinux.org for historical packages
- Example: ./build.sh -r 2026/01/15 or ./build.sh -r week

This solves the common problem of ZFS packages not being available for the latest kernel.

** TODO [#B] Add build logging with tee
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

Capture all build output to a log file for debugging and CI artifact collection.

*** Implementation
See ~/code/archsetup for a reference implementation. The approach there provides a better
user experience by hiding confusing (sometimes scary-looking) messages on the console while
preserving complete build output in the log file. Users see clean progress indicators while
full diagnostic information is available for troubleshooting.

#+BEGIN_SRC bash
# Basic approach
exec &> >(tee "build-$(date +%Y%m%d-%H%M%S).log")

# Better UX: show progress on console, full output to log
exec 3>&1 4>&2
exec 1> >(tee -a "$LOG_FILE") 2>&1
# Then use fd 3 for user-facing output: echo "Installing..." >&3
#+END_SRC

*** Additional features
- Check log for known error patterns (e.g., DKMS failures) and fail fast
- Rotate old logs to prevent disk space issues
- Include system info header (date, kernel version, etc.)

** TODO [#C] Consider adding bootable archzfs ISO to GRUB boot menu
Store the archzfs ISO on disk and add a GRUB menu entry to boot it directly - no USB drive needed for recovery/reinstall.

*** Benefits
- Always have a rescue environment available
- Can reinstall or rollback without external media
- Useful for remote/headless servers

*** Challenges
1. Storage location - ISO is 5GB. Can't live on ZFS (GRUB can't read it). Options:
   - EFI partition (currently 1GB - would need to be larger)
   - Dedicated recovery partition (ext4 or FAT32)
   - Second EFI partition just for the ISO

2. GRUB loopback boot - Arch ISOs support this with the right kernel params:
   #+BEGIN_SRC
   menuentry "Archzfs Recovery" {
       loopback loop /path/to/archzfs.iso
       linux (loop)/arch/boot/x86_64/vmlinuz-linux archisolabel=ARCHZFS
       initrd (loop)/arch/boot/x86_64/initramfs-linux.img
   }
   #+END_SRC

3. Keeping it updated - Would need a mechanism to update the ISO when rebuilding

*** Questions to resolve
- Is this for recovery scenarios, or would you actually reinstall from it?
- Would you want this integrated into the installer (auto-create recovery partition)?
- Or just document how to set it up manually?

** TODO [#C] Research mkosi as alternative to mkarchiso
Investigate whether mkosi (systemd project) offers advantages over mkarchiso.

*** Comparison
| Aspect | mkarchiso | mkosi |
|--------|-----------|-------|
| Purpose | Live ISO images | Disk images, containers, ISOs |
| Config | Shell scripts + file structure | Declarative TOML files |
| Output | ISO9660 (USB/CD) | GPT disk images, tarballs, ISOs |
| Boot | GRUB/syslinux/systemd-boot | UKI (Unified Kernel Images) |
| Distros | Arch only | Arch, Fedora, Debian, Ubuntu |
| Build env | Host or chroot | Container-native, reproducible |

*** Where mkosi shines
- Reproducible builds - designed for CI/CD, hermetic builds
- Unified Kernel Images - modern secure boot (kernel+initrd+cmdline in one signed EFI)
- VM images - can output raw disk images directly (great for QEMU testing)
- Declarative - TOML config instead of shell scripts

*** Where mkarchiso is better for us
- Arch ecosystem - all docs, examples, community use it
- ZFS live environment - archiso has the hooks we need
- Proven - we know it works for our use case

*** Verdict
Keep mkarchiso for now. mkosi could be valuable for:
- VM test images instead of booting ISOs
- Future UKI boot (more secure boot chain)
- Reproducibility when CI/CD becomes important

*** References
- https://wiki.archlinux.org/title/Mkosi
- https://github.com/systemd/mkosi

** TODO [#C] Add 1-minute countdown timer before automatic reboot after installation
Display a countdown timer (1 minute) with red text after installation completes, before automatically rebooting the system.
Gives user time to review the installation summary and cancel if needed.

*** Implementation
In install-archzfs, after displaying the completion message:
#+BEGIN_SRC bash
# Red text countdown before reboot
echo -e "\n\033[0;31mSystem will reboot in 60 seconds. Press Ctrl+C to cancel.\033[0m"
for i in {60..1}; do
    printf "\r\033[0;31mRebooting in %2d seconds...\033[0m" "$i"
    sleep 1
done
echo
reboot
#+END_SRC

** TODO [#C] Add negative/failure test cases
Current tests only verify happy path (successful installation).
Add tests for error conditions to ensure graceful failure handling.

*** Test cases to add
- Installation with insufficient disk space
- Installation with disk that disappears mid-install
- Installation with network failure during pacstrap
- Installation with invalid config file
- Installation on already-mounted disk
- Verify error messages are helpful and actionable

*** Benefits
- Ensures failures don't leave system in corrupted state
- Validates error messages help users diagnose issues
- Catches regressions in error handling code
- Aligns with quality engineering "error cases" principle

** TODO [#C] Add install-archzfs --dry-run mode
Show what would be done without making any changes.
Useful for validating configuration before committing to installation.

*** What dry-run should show
- Disk partitioning plan (sizes, types)
- ZFS pool and dataset structure
- Packages that would be installed
- Services that would be enabled
- Bootloader configuration

*** Implementation approach
- Add DRY_RUN=1 flag checked before destructive operations
- Replace actual commands with echo statements showing what would run
- Validate all inputs and configuration
- Exit with success if everything validates

*** Benefits
- Users can verify configuration before destroying data
- Easier debugging of configuration issues
- Supports "measure twice, cut once" workflow
- Can be used in CI to validate config without full install

** TODO [#C] Pre-clone useful tools and documentation into ISO
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

Bundle Git repos (without .git dirs) into /root for offline access:
- archinstall (official installer)
- downgrade (package rollback)
- ZFS howtos and documentation
- Recovery scripts

Already partially implemented (have rescue tools), but could expand with:
- Pre-cloned arch-linux-configuration scripts
- ZFS administration cheatsheets
- Offline troubleshooting guides

** TODO [#C] Add environment file configuration (.env pattern)
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

Allow build customization via .env file instead of command-line flags.
Useful for CI/CD and reproducible builds.

*** Example .env
#+BEGIN_SRC
KERNEL=linux-lts
USE_DKMS=1
BE_VERBOSE=0
PACKAGES_TO_ADD=git,vim
PACKAGES_TO_REMOVE=b43-fwcutter
#+END_SRC

** TODO [#C] Add dry-run mode to build.sh
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

Support --dry-run flag that shows what would be done without executing.
Useful for testing configuration changes and debugging.

** TODO [#D] Support multi-variant ISO builds
Idea from: https://github.com/stevleibelt/arch-linux-live-cd-iso-with-zfs

The reference project builds 8 variants automatically:
- linux vs linux-lts kernel
- DKMS vs native ZFS packages
- Default vs experimental archzfs repos

Very low priority. We're focused on robustness and compatibility first, bleeding edge last.
The linux-lts + DKMS combination provides maximum stability and hardware compatibility.
Only consider this if there's clear user demand for bleeding-edge kernel support.

* Resolved

** DONE [#B] Add zfsrollback and zfssnapshot scripts to ISO
CLOSED: [2026-01-19 Sun]
Include dedicated ZFS snapshot management scripts in the archzfs ISO rather than archsetup.
These tools belong here since they're useful for rescue scenarios and post-install management.

** DONE [#C] Consider Avahi for USB boot disk discoverability
CLOSED: [2026-01-19 Sun]
Make the live ISO discoverable on the network by name (e.g., archzfs.local) when booted.

Implemented in commit 0bd172a:
- Added avahi and nss-mdns packages to ISO
- Enabled avahi-daemon.service
- Set hostname to "archzfs"
- Live ISO now accessible as root@archzfs.local

** DONE [#B] Add Avahi mDNS to installed systems
CLOSED: [2026-01-19 Sun]
Matches archsetup's implementation: install avahi + nss-mdns, enable avahi-daemon.

Added to install-archzfs:
- Packages: avahi, nss-mdns (in pacstrap)
- Service: avahi-daemon enabled

After installation, system will be accessible as <hostname>.local on the local network.

** DONE [#B] Add config file information to README
Config file format documented in README.org with full reference and examples.

** DONE [#B] Add CI/CD test infrastructure
Added Makefile, test-install.sh, and test configs for automated VM testing.

** CANCELLED [#C] Consider Dialog-Based Interface for Status, Information, and Questions
Using fzf instead.

** DONE [#C] Consider fzf interface for choices
Implemented fzf for timezone, locale, keymap, disk, RAID, and WiFi selection.

** DONE [#A] Create comprehensive project documentation (README.org)
CLOSED: [2026-01-18 Sun 02:01]

** DONE [#C] Add date/timestamp to install-archzfs log
Log filename now includes timestamp: /tmp/install-archzfs-YYYY-MM-DD-HH-MM-SS.log
Also includes header with start time inside the log file.

** DONE [#B] Add common recovery tools to archzfs ISO
CLOSED: [2026-01-18 Sat]
Make the ISO double as a general-purpose recovery disk.
See custom/RESCUE-GUIDE.txt for comprehensive documentation of all tools.