#+TITLE: Archangel
Arch Linux ISO and installer with ZFS and Btrfs support.
#+AUTHOR: Craig Jennings
#+OPTIONS: toc:3

* Overview

Archangel is a custom Arch Linux ISO build system.

~make build~ will begin generating a live ISO with OpenZFS kernel modules and an installer that supports Arch Linux on ZFS root along with Btrfs, encrypted filesystems, and multi-disk RAID configurations.

The ISO also doubles as a [[#rescue-disk][rescue disk]] with data recovery, boot repair, hardware diagnostics, and other tools pre-installed. An [[#ssh-access][SSH server]] starts automatically making remote connections dead easy.

* Features

- *Dual Filesystem Support* - Choose ZFS or Btrfs during installation ([[#filesystem-comparison][comparison]])
- *Native Encryption* - ZFS encryption or LUKS2 for Btrfs
- *Multi-Disk RAID* - Mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
- *Snapshot Boot* - ZFSBootMenu (ZFS) or grub-btrfs (Btrfs) for [[#post-installation][booting into snapshots]]
- *[[#ssh-access][SSH Ready]]* - Optional SSH with root login for headless servers
- *EFI Boot Redundancy* - Bootloader installed on all disks for resilience.
- *[[https://github.com/junegunn/fzf][fzf]]*-Based Interface - Fuzzy search for timezone, locale, keymap, disk, RAID, and WiFi.
- *NetworkManager* - WiFi configuration copied to installed system
- *[[#unattended-installation][Unattended Install]]* - Headless installation via a single plain text config file
- *[[#rescue-disk][Rescue Disk]]* - Data recovery, boot repair, hardware diagnostics, and more

Archangel currently uses linux-lts for stability. Choosing linux and linux-zen kernel varieties coming shortly.

** Filesystem Comparison

| Feature          | ZFS                        | Btrfs                |
|------------------+----------------------------+----------------------|
| Bootloader       | ZFSBootMenu                | GRUB + grub-btrfs    |
| Encryption       | Native ZFS encryption      | LUKS2                |
| Snapshot utility | zfssnapshot                | snapper              |
| Snapshot boot    | Built into ZFSBootMenu     | grub-btrfs menu      |
| RAID support     | mirror, raidz1/2/3, stripe | RAID0, RAID1, RAID10 |
| EFI size         | 512MB                      | 1GB                  |

* Prerequisites

The build script will report if you're missing any of these in a preflight check.

** Build Host Requirements
- Arch Linux (or Arch-based distribution)
- Root/sudo access
- ~archiso~ package (~pacman -S archiso~) — auto-installed if missing
- ~10GB free disk space for build

*Optional: pacoloco caching proxy* — set this up once and ~build.sh~
detects it automatically. Pacoloco caches Arch core/extra plus the
archzfs GitHub-releases URL, which mitigates the recurring archzfs
corruption that bites cold-cache builds. Install + enable:

#+BEGIN_SRC bash
yay -S pacoloco
sudo systemctl enable --now pacoloco
#+END_SRC

The default ~/etc/pacoloco.yaml~ ships an ~archlinux~ repo example;
add an ~archzfs~ entry pointing at
=https://github.com/archzfs/archzfs/releases/download/experimental=
and reload the service. When pacoloco isn't running, ~build.sh~ falls
back to the upstream URLs.

** Baked AUR Packages

~build.sh~ builds a fixed set of AUR packages at build time and bakes
them into the ISO as a local pacman repo at ~/usr/share/aur-packages~.
They work in the live environment (~pacman -Sl aur~) and install onto
the target offline, so the installer never needs the AUR or a build
toolchain on the target. The v1 set (re-audited 2026-06-16):
~downgrade~, ~yay~, ~informant~, ~zrepl~, ~pacman-cleanup-hook~,
~zfs-auto-snapshot~, ~topgrade~, ~ventoy-bin~.

Building these needs ~base-devel~ and ~git~ on the build host, and the
build must run under ~sudo~ so it can drop to your user for ~makepkg~
(makepkg refuses to run as root). Note that ~makepkg -s~ installs each
package's build dependencies onto the *build host* via pacman; v1
leaves those installed (a throwaway-container build env is planned).
Build deps never land on the ISO.

Each build writes a manifest pinning every package's version, AUR
commit, and SHA256 — at ~/usr/share/aur-packages/manifest.tsv~ in the
ISO and ~out/<iso-basename>-aur-manifest.tsv~ beside it — so a given
ISO's exact AUR set is auditable later. The repo is a point-in-time
snapshot (the AUR is fetched at HEAD), not a rebuild-reproducible
lockfile.

The baked packages are installed onto the target as normal packages,
but the ~[aur]~ repo is *not* retained in the installed system's
~/etc/pacman.conf~ — its ~/usr/share/aur-packages~ path exists only on
the live ISO. To reinstall a baked package after first boot, re-add a
repo or rebuild from the AUR.

Pass ~--skip-aur~ to skip the whole AUR path for a faster build when
you're iterating on something unrelated:

#+BEGIN_SRC bash
sudo ./build.sh --skip-aur
#+END_SRC

** Runtime Dependencies (included in ISO)
- ZFS kernel modules (via zfs-dkms)
- Btrfs tools
- NetworkManager
- fzf for interactive selection

* Building the ISO

** Basic Build

The makefile is your primary entry point for building, [[#testing][testing]], and cleaning the project ISO.

#+BEGIN_SRC bash
make build
#+END_SRC

The build script will:
1. Copy the base Arch releng profile
2. Switch to linux-lts kernel
3. Add the archzfs repository (for ZFS packages)
4. Add custom packages (ZFS, Btrfs, NetworkManager, fzf, etc.)
5. Build the baked AUR local repo (skip with ~--skip-aur~)
6. Copy the archangel installer script
7. Build the ISO using mkarchiso

** Build Output

- ISO location: ~out/archangel-YYYY-MM-DD-vmlinuz-{version}-lts-x86_64.iso~
- Example: ~archangel-2026-01-18-vmlinuz-6.12.65-lts-x86_64.iso~
- Build log: ~out/<iso-basename>.log~ on success (matches the ISO name), or
  ~out/build-YYYY-MM-DD-HHMM.log~ if the build failed before producing an ISO.
  Logs survive the ~work/~ cleanup and are tee'd from ~mkarchiso~ in real time.

** Writing to USB

#+BEGIN_SRC bash
sudo dd if=out/archangel-*.iso of=/dev/sdX bs=4M status=progress oflag=sync
#+END_SRC

Replace ~/dev/sdX~ with your USB drive (check with ~lsblk~).

** Clean Rebuild

#+BEGIN_SRC bash
make clean
make build
#+END_SRC

* Booting the ISO

1. Secure Boot must be disabled — ZFS kernel modules are unsigned. Check your
   BIOS/UEFI firmware documentation for how to disable it.
2. Boot from USB and wait for the live environment to load.

** SSH Access

The live environment automatically starts an SSH server, allowing remote access
for headless installations, rescue operations, or VM testing. Avahi is
pre-configured so you can connect by hostname from any machine on the same
network:

#+BEGIN_SRC bash
ssh root@archangel.local     # via mDNS (avahi)
#+END_SRC

The default root password is set in ~build.sh~ (variable ~LIVE_ROOT_PASSWORD~).

*Security Warning*: The live ISO has SSH root login enabled with the password
~archangel~. This is intended for testing, headless installations, and rescue
operations only. Do not expose the live environment to untrusted networks.

** Rescue Disk

The ISO serves as a general-purpose rescue disk with a comprehensive set of
recovery and diagnostic tools pre-installed:

- *Data Recovery* - ~ddrescue~, ~testdisk~, ~photorec~, ~foremost~
- *Boot Repair* - ~grub-install~, ~efibootmgr~, ~arch-chroot~, ~mkinitcpio~, ~syslinux~
- *Windows Recovery* - ~chntpw~ (password reset), ~ntfs-3g~, ~hivex~ (registry editing)
- *Hardware Diagnostics* - ~smartctl~, ~memtester~, ~stress-ng~, ~lm_sensors~, ~hdparm~, ~iotop~
- *Disk Operations* - ~partclone~, ~fsarchiver~, ~nwipe~ (secure erase), ~ncdu~
- *Network Diagnostics* - ~nmap~, ~tcpdump~, ~wireshark-cli~ (tshark), ~mtr~, ~iperf3~, ~iftop~, ~nethogs~
- *Encryption* - ~cryptsetup~ (LUKS), ~gpg~, ~dislocker~ (BitLocker)
- *System Tracing* - ~bpftrace~, ~bcc-tools~, ~perf~

A detailed rescue guide is included on the ISO at ~/root/RESCUE-GUIDE.txt~.
View it with ~less /root/RESCUE-GUIDE.txt~ or any text viewer.

~tealdeer~ (tldr) is also pre-installed with an offline cache, providing concise
command examples:

#+BEGIN_SRC bash
tldr zfs                     # quick ZFS command reference
tldr cryptsetup              # LUKS operations
tldr dd                      # disk imaging
#+END_SRC

* Installation

Run ~archangel~ to start the installer. It provides a guided installation with
fzf-based selection interfaces with helpful information displayed about the
choices.

*Note*: Archangel performs a minimal Arch Linux installation with root login
only. No additional user accounts are created — configure those after first boot.

** Phase 1: Configuration Gathering

1. *Filesystem* - Choose ZFS or Btrfs
2. *Hostname* - System hostname
3. *Timezone* - Fuzzy search through all timezones
4. *Locale* - All locales available
5. *Keymap* - Console keyboard layout
6. *Disk Selection* - Multi-select with TAB (preview shows disk details)
7. *RAID Level* - For multi-disk: mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
8. *Encryption* - Encryption passphrase (ZFS native or LUKS2)
9. *Root Password* - System root password
10. *SSH* - Enable SSH with root login (default: yes)

** Phase 2: Installation

After configuration, the installation runs without intervention:
- Disk partitioning (EFI + root on each disk)
- Filesystem creation with encryption
- Dataset/subvolume creation
- Base system installation via pacstrap
- System configuration (locale, timezone, hostname)
- Bootloader installation (ZFSBootMenu or GRUB)
- Genesis snapshot creation

** Unattended Installation

For automated or headless installations, you can specify a simple plain text config file containing the choices. 

*** Using a Config File

#+BEGIN_SRC bash
# Copy and edit the example config
cp /root/archangel.conf.example /root/my-install.conf
$EDITOR /root/my-install.conf

# Run with config file
archangel --config-file /root/my-install.conf
#+END_SRC

*Important*: The config file is ONLY used when explicitly specified with
~--config-file~. The installer will never automatically read a config file
to prevent accidental disk destruction.

*** Example Config File

#+BEGIN_SRC bash
# archangel.conf - Unattended Installation Configuration
#
# Copy this file and edit values.
# Usage: archangel --config-file /path/to/your-config.conf
#
# Required fields: HOSTNAME, TIMEZONE, DISKS, ROOT_PASSWORD
# For ZFS: also need ZFS_PASSPHRASE or NO_ENCRYPT=yes
# For Btrfs: also need LUKS_PASSPHRASE or NO_ENCRYPT=yes
# All other fields have sensible defaults.

FILESYSTEM=zfs
HOSTNAME=archangel
TIMEZONE=America/Los_Angeles
LOCALE=en_US.UTF-8
KEYMAP=us

DISKS=/dev/vda
RAID_LEVEL=

ZFS_PASSPHRASE=changeme
#LUKS_PASSPHRASE=changeme
#NO_ENCRYPT=no

ROOT_PASSWORD=changeme
ENABLE_SSH=yes
#+END_SRC

A complete example with all options is available at ~installer/archangel.conf.example~.

*** Config File Reference

| Field          | Required | Default     | Description                                |
|----------------+----------+-------------+--------------------------------------------|
| ~FILESYSTEM~     | No       | zfs         | Filesystem type (zfs or btrfs)             |
| ~HOSTNAME~       | Yes      | -           | System hostname                            |
| ~TIMEZONE~       | Yes      | -           | Timezone (Region/City format)              |
| ~DISKS~          | Yes      | -           | Comma-separated disk paths                 |
| ~ZFS_PASSPHRASE~ | Yes*     | -           | Encryption passphrase (*if not NO_ENCRYPT) |
| ~ROOT_PASSWORD~  | Yes      | -           | Root user password                         |
| ~LOCALE~         | No       | en_US.UTF-8 | System locale                              |
| ~KEYMAP~         | No       | us          | Console keyboard layout                    |
| ~RAID_LEVEL~     | No       | mirror      | RAID type for multi-disk                   |
| ~ENABLE_SSH~     | No       | yes         | Enable SSH server                          |
| ~NO_ENCRYPT~     | No       | no          | Skip encryption (testing only)             |

* Post-Reboot

** ZFS Systems
1. If encryption is enabled, enter ZFS passphrase at ZFSBootMenu prompt
2. Select boot environment (or wait for default)
3. Log in as root

** Btrfs Systems
1. GRUB menu appears
2. If encryption is enabled, enter LUKS passphrase when prompted
3. Log in as root

* Post-Installation

** ZFS Snapshot Management

#+BEGIN_SRC bash
# List snapshots
zfssnapshot list

# Create a snapshot
zfssnapshot create "before-experiment"

# Interactive rollback (fzf)
zfssnapshot rollback

# Interactive multi-select destroy (fzf)
zfssnapshot delete
#+END_SRC

** Btrfs Snapshot Management

#+BEGIN_SRC bash
# Create a snapshot (via snapper)
snapper -c root create -d "before-experiment"

# List snapshots
snapper -c root list

# Rollback (requires reboot)
snapper -c root rollback <number>
#+END_SRC

** Genesis Snapshot

Both filesystems create a "genesis" snapshot after installation, representing
the pristine post-install state.

#+BEGIN_SRC bash
# ZFS: View genesis snapshot
zfs list -t snapshot | grep genesis

# Btrfs: View genesis snapshot
snapper -c root list | grep genesis
#+END_SRC

** SSH on the Installed System

When ~ENABLE_SSH~ is ~yes~ (the default), the installer enables ~sshd~ on the
installed system and configures ~PermitRootLogin yes~. You are prompted during
installation and can decline, or set ~ENABLE_SSH=no~ in a config file.

*Important*: Harden SSH after installation — switch to key-based authentication
and consider installing ~fail2ban~.

* Testing

Two test layers: ~make test~ runs shellcheck + bats unit tests for the pure
logic in ~installer/lib/*.sh~ (fast, no VMs). ~make test-install~ runs the full
VM integration suite against a freshly built ISO. Use the former every commit;
the latter before releases or when touching install paths.

See [[file:testing-strategy.org][testing-strategy.org]] for the full testing strategy, unit-test coverage,
how to add new tests, and technical details on encryption testing.

- ~make test~ — Run shellcheck + bats unit tests (~1s, no VMs needed).
- ~make bats~ — Run bats unit tests only.
- ~make lint~ — Run shellcheck on all scripts.
- ~make test-vm~ — Create a 50GB virtual disk and boot the ISO in a
  single-disk QEMU VM. Opens a GTK window with the VM console. SSH in
  with ~ssh -p 2222 root@localhost~ (password: ~archangel~).
- ~make test-multi~ — Same as ~test-vm~ but with two 50GB disks for
  testing mirror and RAID configurations.
- ~make test-multi3~ — Same as ~test-vm~ but with three 50GB disks for
  testing raidz1 configurations.
- ~make test-boot~ — Boot from an existing installed disk (no ISO).
  Auto-detects multi-disk setups. Use after running an install in the
  VM to verify the system boots from disk.
- ~make test-clean~ — Remove all VM disks and OVMF vars to start fresh.
- ~make test-install~ — Run the full automated test suite (builds ISO
  first). Runs 12 unattended installs covering ZFS/Btrfs,
  single/multi-disk, encrypted/unencrypted, and custom locale. Each
  test installs, reboots from disk, verifies boot, and checks rollback.

* Project Structure

#+BEGIN_EXAMPLE
archangel/
├── build.sh                       # Main ISO build script
├── Makefile                       # Build, lint, test, and release targets
├── installer/
│   ├── archangel                  # Interactive installation script
│   ├── archangel.conf.example     # Example config for unattended install
│   ├── lib/                       # Modular installer components
│   │   ├── common.sh              # Shared utilities, password prompt, pacstrap list
│   │   ├── config.sh              # Argument parsing + config-file loading + validation
│   │   ├── disk.sh                # Disk partitioning and EFI formatting
│   │   ├── btrfs.sh               # Btrfs-specific functions (LUKS, subvolumes, GRUB)
│   │   └── raid.sh                # Pure RAID-level logic (levels, validation, usable space)
│   ├── zfssnapshot                # ZFS snapshot utility (list/create/rollback/delete)
│   └── RESCUE-GUIDE.txt           # Recovery tools documentation
├── tests/
│   └── unit/                      # Bats unit tests for installer/lib/*.sh
├── scripts/
│   ├── test-vm.sh                 # QEMU test VM launcher
│   ├── test-install.sh            # Automated install tests
│   ├── test-configs/              # Test configuration files
│   ├── full-test.sh               # Comprehensive test suite
│   ├── sanity-test.sh             # Quick ISO verification
│   ├── boot-vm.sh                 # Boot VM from disk or ISO
│   └── build-release              # Build and distribute ISO
├── vm/                            # VM disk images (created by test-vm.sh)
├── work/                          # Build working directory
└── out/                           # Built ISO output
#+END_EXAMPLE

** Script Descriptions

| Script              | Description                                                                                                  |
|---------------------+--------------------------------------------------------------------------------------------------------------|
| ~build.sh~            | Builds the ISO. Copies releng profile, adds packages, configures kernel, runs mkarchiso                      |
| ~installer/archangel~ | Interactive installer. Handles disk partitioning, filesystem creation, base system install, bootloader setup |
| ~scripts/test-vm.sh~  | Launches QEMU VM for testing. Supports single and multi-disk configurations                                  |

* Troubleshooting

** Build Fails with Package Conflicts

Clean the work directory and rebuild:
#+BEGIN_SRC bash
make clean
make build
#+END_SRC

** ZFS Module Not Loading

The ISO includes DKMS-built ZFS modules. If modules fail to load:
- Check ~dmesg | grep -i zfs~ for errors
- Ensure you're using the LTS kernel

** Disk Not Showing in Selection

- Ensure the disk is not mounted
- Check ~lsblk~ to verify disk visibility
- USB drives may need a moment to be detected

** Boot Fails After Installation

*** ZFS
- Check ZFSBootMenu appears (if not, check EFI boot order with ~efibootmgr~)
- Verify pool can import: boot ISO, ~zpool import -f zroot~

*** Btrfs
- Verify EFI boot entries: ~efibootmgr -v~
- Check GRUB config: ~/boot/grub/grub.cfg~

* Links

- [[https://github.com/archzfs/archzfs][archzfs Repository]] - ZFS packages for Arch Linux (GitHub Releases; archzfs.com was abandoned mid-2025)
- [[https://openzfs.github.io/openzfs-docs/][OpenZFS Documentation]] - Official ZFS documentation
- [[https://get.zfsbootmenu.org][ZFSBootMenu]] - ZFS boot manager
- [[https://wiki.archlinux.org/title/Btrfs][Arch Wiki - Btrfs]] - Btrfs information
- [[https://wiki.archlinux.org/title/Snapper][Arch Wiki - Snapper]] - Btrfs snapshot management
- [[https://github.com/Antynea/grub-btrfs][grub-btrfs]] - Boot Btrfs snapshots from GRUB

* License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).
See [[file:LICENSE][LICENSE]] file for the full license text.

Note: [[https://github.com/openzfs/zfs][OpenZFS]] is licensed separately under the
[[https://github.com/openzfs/zfs/blob/master/LICENSE][CDDL license]]. ZFS packages
are provided by the [[https://archzfs.com][archzfs]] third-party repository and are
not part of this project.