#+TITLE: Archangel
Arch Linux ISO and installer with ZFS and Btrfs support.
#+AUTHOR: Craig Jennings
#+OPTIONS: toc:3

* Overview

Archangel is a custom Arch Linux ISO build system.

~make build~ will begin generating a live ISO with OpenZFS kernel modules and an installer that supports Arch Linux on ZFS root along with Btrfs, encrypted filesystems, and multi-disk RAID configurations.

The ISO also doubles as a [[#rescue-disk][rescue disk]] with data recovery, boot repair, hardware diagnostics, and other tools pre-installed. An [[#ssh-access][SSH server]] starts automatically making remote connections dead easy.

* Features

- *Dual Filesystem Support* - Choose ZFS or Btrfs during installation ([[#filesystem-comparison][comparison]])
- *Native Encryption* - ZFS encryption or LUKS2 for Btrfs
- *Multi-Disk RAID* - Mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
- *Snapshot Boot* - ZFSBootMenu (ZFS) or grub-btrfs (Btrfs) for [[#post-installation][booting into snapshots]]
- *[[#ssh-access][SSH Ready]]* - Optional SSH with root login for headless servers
- *EFI Boot Redundancy* - Bootloader installed on all disks for resilience.
- *[[https://github.com/junegunn/fzf][fzf]]*-Based Interface - Fuzzy search for timezone, locale, keymap, disk, RAID, and WiFi.
- *NetworkManager* - WiFi configuration copied to installed system
- *[[#unattended-installation][Unattended Install]]* - Headless installation via a single plain text config file
- *[[#rescue-disk][Rescue Disk]]* - Data recovery, boot repair, hardware diagnostics, and more

Archangel currently uses linux-lts for stability. Choosing linux and linux-zen kernel varieties coming shortly.

** Filesystem Comparison

| Feature          | ZFS                        | Btrfs                |
|------------------+----------------------------+----------------------|
| Bootloader       | ZFSBootMenu                | GRUB + grub-btrfs    |
| Encryption       | Native ZFS encryption      | LUKS2                |
| Snapshot utility | zfssnapshot helper scripts | snapper              |
| Snapshot boot    | Built into ZFSBootMenu     | grub-btrfs menu      |
| RAID support     | mirror, raidz1/2/3, stripe | RAID0, RAID1, RAID10 |
| EFI size         | 512MB                      | 1GB                  |

* Prerequisites

The build script will report if you're missing any of these in a preflight check.

** Build Host Requirements
- Arch Linux (or Arch-based distribution)
- Root/sudo access
- ~archiso~ package (~pacman -S archiso~) — auto-installed if missing
- ~10GB free disk space for build

** Runtime Dependencies (included in ISO)
- ZFS kernel modules (via zfs-dkms)
- Btrfs tools
- NetworkManager
- fzf for interactive selection

* Building the ISO

** Basic Build

The makefile is your primary entry point for building, [[#testing-with-vms][testing]], and cleaning the project ISO.

#+BEGIN_SRC bash
make build
#+END_SRC

The build script will:
1. Copy the base Arch releng profile
2. Switch to linux-lts kernel
3. Add the archzfs repository (for ZFS packages)
4. Add custom packages (ZFS, Btrfs, NetworkManager, fzf, etc.)
5. Copy the archangel installer script
6. Build the ISO using mkarchiso

** Build Output

- ISO location: ~out/archangel-YYYY-MM-DD-vmlinuz-{version}-lts-x86_64.iso~
- Example: ~archangel-2026-01-18-vmlinuz-6.12.65-lts-x86_64.iso~
- Build logs: visible in terminal output (not saved to disk)

** Writing to USB

#+BEGIN_SRC bash
sudo dd if=out/archangel-*.iso of=/dev/sdX bs=4M status=progress oflag=sync
#+END_SRC

Replace ~/dev/sdX~ with your USB drive (check with ~lsblk~).

** Clean Rebuild

#+BEGIN_SRC bash
make clean
make build
#+END_SRC

* Booting the ISO

1. Secure Boot must be disabled — ZFS kernel modules are unsigned. Check your
   BIOS/UEFI firmware documentation for how to disable it.
2. Boot from USB and wait for the live environment to load.

** SSH Access

The live environment automatically starts an SSH server, allowing remote access
for headless installations, rescue operations, or VM testing. Avahi is
pre-configured so you can connect by hostname from any machine on the same
network:

#+BEGIN_SRC bash
ssh root@archangel.local     # via mDNS (avahi)
#+END_SRC

The default root password is set in ~build.sh~ (variable ~LIVE_ROOT_PASSWORD~).

*Security Warning*: The live ISO has SSH root login enabled with the password
~archangel~. This is intended for testing, headless installations, and rescue
operations only. Do not expose the live environment to untrusted networks.

** Rescue Disk

The ISO serves as a general-purpose rescue disk with a comprehensive set of
recovery and diagnostic tools pre-installed:

- *Data Recovery* - ~ddrescue~, ~testdisk~, ~photorec~, ~foremost~
- *Boot Repair* - ~grub-install~, ~efibootmgr~, ~arch-chroot~, ~mkinitcpio~, ~syslinux~
- *Windows Recovery* - ~chntpw~ (password reset), ~ntfs-3g~, ~hivex~ (registry editing)
- *Hardware Diagnostics* - ~smartctl~, ~memtester~, ~stress-ng~, ~lm_sensors~, ~hdparm~, ~iotop~
- *Disk Operations* - ~partclone~, ~fsarchiver~, ~nwipe~ (secure erase), ~ncdu~
- *Network Diagnostics* - ~nmap~, ~tcpdump~, ~wireshark-cli~ (tshark), ~mtr~, ~iperf3~, ~iftop~, ~nethogs~
- *Encryption* - ~cryptsetup~ (LUKS), ~gpg~, ~dislocker~ (BitLocker)
- *System Tracing* - ~bpftrace~, ~bcc-tools~, ~perf~

A detailed rescue guide is included on the ISO at ~/root/RESCUE-GUIDE.txt~.
View it with ~less /root/RESCUE-GUIDE.txt~ or any text viewer.

~tealdeer~ (tldr) is also pre-installed with an offline cache, providing concise
command examples:

#+BEGIN_SRC bash
tldr zfs                     # quick ZFS command reference
tldr cryptsetup              # LUKS operations
tldr dd                      # disk imaging
#+END_SRC

* Installation

Run ~archangel~ to start the installer. It provides a guided installation with
fzf-based selection interfaces with helpful information displayed about the
choices.

*Note*: Archangel performs a minimal Arch Linux installation with root login
only. No additional user accounts are created — configure those after first boot.

** Phase 1: Configuration Gathering

1. *Filesystem* - Choose ZFS or Btrfs
2. *Hostname* - System hostname
3. *Timezone* - Fuzzy search through all timezones
4. *Locale* - All locales available
5. *Keymap* - Console keyboard layout
6. *Disk Selection* - Multi-select with TAB (preview shows disk details)
7. *RAID Level* - For multi-disk: mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
8. *Encryption* - Encryption passphrase (ZFS native or LUKS2)
9. *Root Password* - System root password
10. *SSH* - Enable SSH with root login (default: yes)

** Phase 2: Installation

After configuration, the installation runs without intervention:
- Disk partitioning (EFI + root on each disk)
- Filesystem creation with encryption
- Dataset/subvolume creation
- Base system installation via pacstrap
- System configuration (locale, timezone, hostname)
- Bootloader installation (ZFSBootMenu or GRUB)
- Genesis snapshot creation

** Unattended Installation

For automated or headless installations, you can specify a simple plain text config file containing the choices. 

*** Using a Config File

#+BEGIN_SRC bash
# Copy and edit the example config
cp /root/archangel.conf.example /root/my-install.conf
$EDITOR /root/my-install.conf

# Run with config file
archangel --config-file /root/my-install.conf
#+END_SRC

*Important*: The config file is ONLY used when explicitly specified with
~--config-file~. The installer will never automatically read a config file
to prevent accidental disk destruction.

*** Example Config File

#+BEGIN_SRC bash
# archangel.conf - Unattended Installation Configuration
#
# Copy this file and edit values.
# Usage: archangel --config-file /path/to/your-config.conf
#
# Required fields: HOSTNAME, TIMEZONE, DISKS, ROOT_PASSWORD
# For ZFS: also need ZFS_PASSPHRASE or NO_ENCRYPT=yes
# For Btrfs: also need LUKS_PASSPHRASE or NO_ENCRYPT=yes
# All other fields have sensible defaults.

FILESYSTEM=zfs
HOSTNAME=archangel
TIMEZONE=America/Los_Angeles
LOCALE=en_US.UTF-8
KEYMAP=us

DISKS=/dev/vda
RAID_LEVEL=

ZFS_PASSPHRASE=changeme
#LUKS_PASSPHRASE=changeme
#NO_ENCRYPT=no

ROOT_PASSWORD=changeme
ENABLE_SSH=yes
#+END_SRC

A complete example with all options is available at ~installer/archangel.conf.example~.

*** Config File Reference

| Field          | Required | Default     | Description                                |
|----------------+----------+-------------+--------------------------------------------|
| ~FILESYSTEM~     | No       | zfs         | Filesystem type (zfs or btrfs)             |
| ~HOSTNAME~       | Yes      | -           | System hostname                            |
| ~TIMEZONE~       | Yes      | -           | Timezone (Region/City format)              |
| ~DISKS~          | Yes      | -           | Comma-separated disk paths                 |
| ~ZFS_PASSPHRASE~ | Yes*     | -           | Encryption passphrase (*if not NO_ENCRYPT) |
| ~ROOT_PASSWORD~  | Yes      | -           | Root user password                         |
| ~LOCALE~         | No       | en_US.UTF-8 | System locale                              |
| ~KEYMAP~         | No       | us          | Console keyboard layout                    |
| ~RAID_LEVEL~     | No       | mirror      | RAID type for multi-disk                   |
| ~ENABLE_SSH~     | No       | yes         | Enable SSH server                          |
| ~NO_ENCRYPT~     | No       | no          | Skip encryption (testing only)             |

* Post-Reboot

** ZFS Systems
1. If encryption is enabled, enter ZFS passphrase at ZFSBootMenu prompt
2. Select boot environment (or wait for default)
3. Log in as root

** Btrfs Systems
1. GRUB menu appears
2. If encryption is enabled, enter LUKS passphrase when prompted
3. Log in as root

* Post-Installation

** ZFS Snapshot Management

#+BEGIN_SRC bash
# Create a snapshot
zfssnapshot "before-experiment"

# Interactive rollback with fzf
zfsrollback

# List snapshots
zfs list -t snapshot
#+END_SRC

** Btrfs Snapshot Management

#+BEGIN_SRC bash
# Create a snapshot (via snapper)
snapper -c root create -d "before-experiment"

# List snapshots
snapper -c root list

# Rollback (requires reboot)
snapper -c root rollback <number>
#+END_SRC

** Genesis Snapshot

Both filesystems create a "genesis" snapshot after installation, representing
the pristine post-install state.

#+BEGIN_SRC bash
# ZFS: View genesis snapshot
zfs list -t snapshot | grep genesis

# Btrfs: View genesis snapshot
snapper -c root list | grep genesis
#+END_SRC

** SSH on the Installed System

When ~ENABLE_SSH~ is ~yes~ (the default), the installer enables ~sshd~ on the
installed system and configures ~PermitRootLogin yes~. You are prompted during
installation and can decline, or set ~ENABLE_SSH=no~ in a config file.

*Important*: Harden SSH after installation — switch to key-based authentication
and consider installing ~fail2ban~.

* Testing with VMs

QEMU VMs validate that the ISO boots correctly and that installations complete
successfully across all supported configurations (ZFS/Btrfs, single/multi-disk,
encrypted/unencrypted). Run these after modifying the installer or build scripts
to catch regressions.

See [[file:testing-strategy.org][testing-strategy.org]] for the full testing strategy,
including how to add new tests and technical details on encryption testing.

- ~make test-vm~ — Create a 50GB virtual disk and boot the ISO in a
  single-disk QEMU VM. Opens a GTK window with the VM console. SSH in
  with ~ssh -p 2222 root@localhost~ (password: ~archangel~).
- ~make test-multi~ — Same as ~test-vm~ but with two 50GB disks for
  testing mirror and RAID configurations.
- ~make test-multi3~ — Same as ~test-vm~ but with three 50GB disks for
  testing raidz1 configurations.
- ~make test-boot~ — Boot from an existing installed disk (no ISO).
  Auto-detects multi-disk setups. Use after running an install in the
  VM to verify the system boots from disk.
- ~make test-clean~ — Remove all VM disks and OVMF vars to start fresh.
- ~make test-install~ — Run the full automated test suite (builds ISO
  first). Runs 12 unattended installs covering ZFS/Btrfs,
  single/multi-disk, encrypted/unencrypted, and custom locale. Each
  test installs, reboots from disk, verifies boot, and checks rollback.
- ~make lint~ — Run shellcheck on all scripts.

* Project Structure

#+BEGIN_EXAMPLE
archangel/
├── build.sh                       # Main ISO build script
├── Makefile                       # Build, lint, test, and release targets
├── installer/
│   ├── archangel                  # Interactive installation script
│   ├── archangel.conf.example     # Example config for unattended install
│   ├── lib/                       # Modular installer components
│   │   ├── common.sh              # Shared utilities
│   │   ├── config.sh              # Configuration handling
│   │   ├── disk.sh                # Disk partitioning
│   │   ├── zfs.sh                 # ZFS-specific functions
│   │   └── btrfs.sh               # Btrfs-specific functions
│   ├── zfssnapshot                # ZFS snapshot utility
│   ├── zfsrollback                # ZFS rollback utility
│   └── RESCUE-GUIDE.txt           # Recovery tools documentation
├── scripts/
│   ├── test-vm.sh                 # QEMU test VM launcher
│   ├── test-install.sh            # Automated install tests
│   ├── test-configs/              # Test configuration files
│   ├── full-test.sh               # Comprehensive test suite
│   ├── sanity-test.sh             # Quick ISO verification
│   ├── boot-vm.sh                 # Boot VM from disk or ISO
│   └── build-release              # Build and distribute ISO
├── vm/                            # VM disk images (created by test-vm.sh)
├── work/                          # Build working directory
└── out/                           # Built ISO output
#+END_EXAMPLE

** Script Descriptions

| Script              | Description                                                                                                  |
|---------------------+--------------------------------------------------------------------------------------------------------------|
| ~build.sh~            | Builds the ISO. Copies releng profile, adds packages, configures kernel, runs mkarchiso                      |
| ~installer/archangel~ | Interactive installer. Handles disk partitioning, filesystem creation, base system install, bootloader setup |
| ~scripts/test-vm.sh~  | Launches QEMU VM for testing. Supports single and multi-disk configurations                                  |

* Troubleshooting

** Build Fails with Package Conflicts

Clean the work directory and rebuild:
#+BEGIN_SRC bash
make clean
make build
#+END_SRC

** ZFS Module Not Loading

The ISO includes DKMS-built ZFS modules. If modules fail to load:
- Check ~dmesg | grep -i zfs~ for errors
- Ensure you're using the LTS kernel

** Disk Not Showing in Selection

- Ensure the disk is not mounted
- Check ~lsblk~ to verify disk visibility
- USB drives may need a moment to be detected

** Boot Fails After Installation

*** ZFS
- Check ZFSBootMenu appears (if not, check EFI boot order with ~efibootmgr~)
- Verify pool can import: boot ISO, ~zpool import -f zroot~

*** Btrfs
- Verify EFI boot entries: ~efibootmgr -v~
- Check GRUB config: ~/boot/grub/grub.cfg~

* Links

- [[https://archzfs.com][archzfs Repository]] - ZFS packages for Arch Linux
- [[https://openzfs.github.io/openzfs-docs/][OpenZFS Documentation]] - Official ZFS documentation
- [[https://get.zfsbootmenu.org][ZFSBootMenu]] - ZFS boot manager
- [[https://wiki.archlinux.org/title/Btrfs][Arch Wiki - Btrfs]] - Btrfs information
- [[https://wiki.archlinux.org/title/Snapper][Arch Wiki - Snapper]] - Btrfs snapshot management
- [[https://github.com/Antynea/grub-btrfs][grub-btrfs]] - Boot Btrfs snapshots from GRUB

* License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).
See [[file:LICENSE][LICENSE]] file for the full license text.

Note: [[https://github.com/openzfs/zfs][OpenZFS]] is licensed separately under the
[[https://github.com/openzfs/zfs/blob/master/LICENSE][CDDL license]]. ZFS packages
are provided by the [[https://archzfs.com][archzfs]] third-party repository and are
not part of this project.