#+TITLE: Archangel
Arch Linux ISO and installer with ZFS and Btrfs support.
#+AUTHOR: Craig Jennings
#+OPTIONS: toc:3

* Overview

Archangel is a custom Arch Linux ISO build system. ~make build~ will begin generating a live ISO with OpenZFS kernel modules and an installer that supports Arch Linux on ZFS root along with Btrfs, encrypted filesystems, multi-disk RAID configurations. The ISO also doubles as a [[#rescue-disk][rescue disk]] with data recovery, boot repair, hardware diagnostics, and other tools pre-installed. An [[#ssh-server][SSH server]] starts automatically making remote connections dead easy. 

* Features

- *Dual Filesystem Support* - Choose ZFS or Btrfs during installation ([[#filesystem-comparison][comparison]])
- *Native Encryption* - ZFS encryption or LUKS2 for Btrfs
- *Multi-Disk RAID* - Mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
- *Snapshot Boot* - ZFSBootMenu (ZFS) or grub-btrfs (Btrfs) for [[#post-installation][booting into snapshots]]
- *[[#ssh-server][SSH Ready]]* - Optional SSH with root login for headless servers
- *EFI Boot Redundancy* - Bootloader installed on all disks for resilience.
- *[[https://github.com/junegunn/fzf][fzf]]*-Based Interface - Fuzzy search for timezone, locale, keymap, disk, RAID, and WiFi.
- *NetworkManager* - WiFi configuration copied to installed system
- *[[#unattended-installation][Unattended Install]]* - Headless installation via a single plain text config file
- *[[#rescue-disk][Rescue Disk]]* - Data recovery, boot repair, hardware diagnostics, and more

Archangel currently uses linux-lts for stability. Choosing linux and linux-zen kernel varieties coming shortly.

** Filesystem Comparison

| Feature          | ZFS                        | Btrfs                |
|------------------+----------------------------+----------------------|
| Bootloader       | ZFSBootMenu                | GRUB + grub-btrfs    |
| Encryption       | Native ZFS encryption      | LUKS2                |
| Snapshot utility | zfssnapshot helper scripts | snapper              |
| Snapshot boot    | Built into ZFSBootMenu     | grub-btrfs menu      |
| RAID support     | mirror, raidz1/2/3, stripe | RAID0, RAID1, RAID10 |
| EFI size         | 512MB                      | 1GB                  |

* Quick Start

#+BEGIN_SRC bash
# Build the ISO (requires root)
sudo ./build.sh

# Test in a VM
./scripts/test-vm.sh

# Or test with multiple disks for RAID
./scripts/test-vm.sh --multi-disk
#+END_SRC

Boot the ISO and run ~archangel~ to start the installation.

* Prerequisites

** Build Host Requirements
- Arch Linux (or Arch-based distribution)
- Root/sudo access
- ~archiso~ package (~pacman -S archiso~) — auto-installed if missing
- ~10GB free disk space for build

** Runtime Dependencies (included in ISO)
- ZFS kernel modules (via zfs-dkms)
- Btrfs tools
- NetworkManager
- fzf for interactive selection

* Building the ISO

** Basic Build

#+BEGIN_SRC bash
sudo ./build.sh
#+END_SRC

The build script will:
1. Copy the base Arch releng profile
2. Switch to linux-lts kernel
3. Add the archzfs repository (for ZFS packages)
4. Add custom packages (ZFS, Btrfs, NetworkManager, fzf, etc.)
5. Copy the archangel installer script
6. Build the ISO using mkarchiso

** Build Output

- ISO location: ~out/archangel-vmlinuz-{version}-lts-YYYY-MM-DD-x86_64.iso~
- Example: ~archangel-vmlinuz-6.12.65-lts-2026-01-18-x86_64.iso~
- Build logs: visible in terminal output (not saved to disk)

** Clean Rebuild

#+BEGIN_SRC bash
sudo rm -rf work out
sudo ./build.sh
#+END_SRC

* Project Structure

#+BEGIN_EXAMPLE
archangel/
├── build.sh                       # Main ISO build script
├── Makefile                       # Build, lint, test, and release targets
├── installer/
│   ├── archangel                  # Interactive installation script
│   ├── archangel.conf.example     # Example config for unattended install
│   ├── lib/                       # Modular installer components
│   │   ├── common.sh              # Shared utilities
│   │   ├── config.sh              # Configuration handling
│   │   ├── disk.sh                # Disk partitioning
│   │   ├── zfs.sh                 # ZFS-specific functions
│   │   └── btrfs.sh               # Btrfs-specific functions
│   ├── zfssnapshot                # ZFS snapshot utility
│   ├── zfsrollback                # ZFS rollback utility
│   └── RESCUE-GUIDE.txt           # Recovery tools documentation
├── scripts/
│   ├── test-vm.sh                 # QEMU test VM launcher
│   ├── test-install.sh            # Automated install tests
│   ├── test-configs/              # Test configuration files
│   ├── full-test.sh               # Comprehensive test suite
│   ├── sanity-test.sh             # Quick ISO verification
│   ├── boot-vm.sh                 # Boot VM from disk or ISO
│   └── build-release              # Build and distribute ISO
├── vm/                            # VM disk images (created by test-vm.sh)
├── work/                          # Build working directory
└── out/                           # Built ISO output
#+END_EXAMPLE

** Script Descriptions

| Script              | Description                                                                                                  |
|---------------------+--------------------------------------------------------------------------------------------------------------|
| ~build.sh~            | Builds the ISO. Copies releng profile, adds packages, configures kernel, runs mkarchiso                      |
| ~installer/archangel~ | Interactive installer. Handles disk partitioning, filesystem creation, base system install, bootloader setup |
| ~scripts/test-vm.sh~  | Launches QEMU VM for testing. Supports single and multi-disk configurations                                  |

* Testing with VMs

The test suite launches QEMU VMs to validate that the ISO boots correctly and
that installations complete successfully across all supported configurations
(ZFS/Btrfs, single/multi-disk, encrypted/unencrypted). Run these tests after
modifying the installer or build scripts to catch regressions.

** Basic VM Test

#+BEGIN_SRC bash
./scripts/test-vm.sh
#+END_SRC

This creates a 50GB virtual disk and boots the ISO.

** Multi-Disk RAID Test

#+BEGIN_SRC bash
# Two 50GB disks (for mirror)
./scripts/test-vm.sh --multi-disk

# Three 50GB disks (for raidz1)
./scripts/test-vm.sh --multi-disk=3
#+END_SRC

** SSH Access to VM

*Security Warning*: The live ISO has SSH root login enabled with the password ~archangel~. This is intended for testing, headless installations, and rescue operations only. Do not expose the live environment to untrusted networks.

#+BEGIN_SRC bash
# Password: archangel
ssh -p 2222 root@localhost

# Or with sshpass
sshpass -p archangel ssh -p 2222 root@localhost
#+END_SRC

** Clean VM State

#+BEGIN_SRC bash
./scripts/test-vm.sh --clean
#+END_SRC

** Boot from Installed Disk

#+BEGIN_SRC bash
./scripts/test-vm.sh --boot-disk
#+END_SRC

* Installation Walkthrough

The ~archangel~ script provides a guided installation with fzf-based selection interfaces with helpful information displayed about the choices. 

** Phase 1: Configuration Gathering

1. *Filesystem* - Choose ZFS or Btrfs
2. *Hostname* - System hostname
3. *Timezone* - Fuzzy search through all timezones
4. *Locale* - All locales available
5. *Keymap* - Console keyboard layout
6. *Disk Selection* - Multi-select with TAB (preview shows disk details)
7. *RAID Level* - For multi-disk: mirror, stripe, raidz1/2/3 (ZFS) or RAID0/1/10 (Btrfs)
8. *Encryption* - Encryption passphrase (ZFS native or LUKS2)
9. *Root Password* - System root password
10. *SSH* - Enable SSH with root login (default: yes)

** Phase 2: Unattended Installation

After configuration, the installation runs without intervention:
- Disk partitioning (EFI + root on each disk)
- Filesystem creation with encryption
- Dataset/subvolume creation
- Base system installation via pacstrap
- System configuration (locale, timezone, hostname)
- Bootloader installation (ZFSBootMenu or GRUB)
- Genesis snapshot creation

* Unattended Installation

For automated or headless installations, you can specify a simple plain text config file containing the choices. 

** Using a Config File

#+BEGIN_SRC bash
# Copy and edit the example config
cp /root/archangel.conf.example /root/my-install.conf
$EDITOR /root/my-install.conf

# Run with config file
archangel --config-file /root/my-install.conf
#+END_SRC

*Important*: The config file is ONLY used when explicitly specified with
~--config-file~. The installer will never automatically read a config file
to prevent accidental disk destruction.

** Example Config File

#+BEGIN_SRC bash
# archangel.conf - Unattended Installation Configuration
#
# Copy this file and edit values.
# Usage: archangel --config-file /path/to/your-config.conf
#
# Required fields: HOSTNAME, TIMEZONE, DISKS, ROOT_PASSWORD
# For ZFS: also need ZFS_PASSPHRASE or NO_ENCRYPT=yes
# For Btrfs: also need LUKS_PASSPHRASE or NO_ENCRYPT=yes
# All other fields have sensible defaults.

FILESYSTEM=zfs
HOSTNAME=archangel
TIMEZONE=America/Los_Angeles
LOCALE=en_US.UTF-8
KEYMAP=us

DISKS=/dev/vda
RAID_LEVEL=

ZFS_PASSPHRASE=changeme
#LUKS_PASSPHRASE=changeme
#NO_ENCRYPT=no

ROOT_PASSWORD=changeme
ENABLE_SSH=yes
#+END_SRC

A complete example with all options is available at ~installer/archangel.conf.example~.

** Config File Reference

| Field          | Required | Default     | Description                                |
|----------------+----------+-------------+--------------------------------------------|
| ~FILESYSTEM~     | No       | zfs         | Filesystem type (zfs or btrfs)             |
| ~HOSTNAME~       | Yes      | -           | System hostname                            |
| ~TIMEZONE~       | Yes      | -           | Timezone (Region/City format)              |
| ~DISKS~          | Yes      | -           | Comma-separated disk paths                 |
| ~ZFS_PASSPHRASE~ | Yes*     | -           | Encryption passphrase (*if not NO_ENCRYPT) |
| ~ROOT_PASSWORD~  | Yes      | -           | Root user password                         |
| ~LOCALE~         | No       | en_US.UTF-8 | System locale                              |
| ~KEYMAP~         | No       | us          | Console keyboard layout                    |
| ~RAID_LEVEL~     | No       | mirror      | RAID type for multi-disk                   |
| ~ENABLE_SSH~     | No       | yes         | Enable SSH server                          |
| ~NO_ENCRYPT~     | No       | no          | Skip encryption (testing only)             |

* Bare Metal Installation

** Preparing Installation Media

#+BEGIN_SRC bash
# Write ISO to USB drive (replace /dev/sdX)
sudo dd if=out/archangel-vmlinuz-*.iso of=/dev/sdX bs=4M status=progress oflag=sync
#+END_SRC

** Booting

1. Boot from USB — Secure Boot must be disabled since ZFS kernel modules are
   unsigned. Check your BIOS/UEFI firmware documentation for how to disable it.
2. Wait for live environment to load
3. Run ~archangel~

*Note*: Archangel performs a minimal Arch Linux installation with root login
only. No additional user accounts are created — configure those after first boot.

** Post-Reboot

*** ZFS Systems
1. If encryption is enabled, enter ZFS passphrase at ZFSBootMenu prompt
2. Select boot environment (or wait for default)
3. Log in as root

*** Btrfs Systems
1. GRUB menu appears
2. If encryption is enabled, enter LUKS passphrase when prompted
3. Log in as root

* Post-Installation

** ZFS Snapshot Management

#+BEGIN_SRC bash
# Create a snapshot
zfssnapshot "before-experiment"

# Interactive rollback with fzf
zfsrollback

# List snapshots
zfs list -t snapshot
#+END_SRC

** Btrfs Snapshot Management

#+BEGIN_SRC bash
# Create a snapshot (via snapper)
snapper -c root create -d "before-experiment"

# List snapshots
snapper -c root list

# Rollback (requires reboot)
snapper -c root rollback <number>
#+END_SRC

** Genesis Snapshot

Both filesystems create a "genesis" snapshot after installation, representing
the pristine post-install state.

#+BEGIN_SRC bash
# ZFS: View genesis snapshot
zfs list -t snapshot | grep genesis

# Btrfs: View genesis snapshot
snapper -c root list | grep genesis
#+END_SRC

** SSH on the Installed System

When ~ENABLE_SSH~ is ~yes~ (the default), the installer enables ~sshd~ on the
installed system and configures ~PermitRootLogin yes~. You are prompted during
installation and can decline, or set ~ENABLE_SSH=no~ in a config file.

*Important*: Harden SSH after installation — switch to key-based authentication
and consider installing ~fail2ban~.

* Rescue Disk

The ISO serves as a general-purpose rescue disk with a comprehensive set of
recovery and diagnostic tools pre-installed:

- *Data Recovery* - ~ddrescue~, ~testdisk~, ~photorec~, ~foremost~
- *Boot Repair* - ~grub-install~, ~efibootmgr~, ~arch-chroot~, ~mkinitcpio~, ~syslinux~
- *Windows Recovery* - ~chntpw~ (password reset), ~ntfs-3g~, ~hivex~ (registry editing)
- *Hardware Diagnostics* - ~smartctl~, ~memtester~, ~stress-ng~, ~lm_sensors~, ~hdparm~, ~iotop~
- *Disk Operations* - ~partclone~, ~fsarchiver~, ~nwipe~ (secure erase), ~ncdu~
- *Network Diagnostics* - ~nmap~, ~tcpdump~, ~wireshark-cli~ (tshark), ~mtr~, ~iperf3~, ~iftop~, ~nethogs~
- *Encryption* - ~cryptsetup~ (LUKS), ~gpg~, ~dislocker~ (BitLocker)
- *System Tracing* - ~bpftrace~, ~bcc-tools~, ~perf~

A detailed rescue guide is included on the ISO at ~/root/RESCUE-GUIDE.txt~.

* SSH Server

The live environment automatically starts an SSH server, allowing remote access
for headless installations, rescue operations, or VM testing. Connect from
another machine on the same network:

#+BEGIN_SRC bash
ssh root@archangel.local     # via mDNS (avahi)
#+END_SRC

The default root password is set in ~build.sh~ (variable ~LIVE_ROOT_PASSWORD~).

*Security Warning*: The live ISO has SSH root login enabled with the password
~archangel~. This is intended for testing, headless installations, and rescue
operations only. Do not expose the live environment to untrusted networks.

* Troubleshooting

** Build Fails with Package Conflicts

Clean the work directory and rebuild:
#+BEGIN_SRC bash
sudo rm -rf work
sudo ./build.sh
#+END_SRC

** ZFS Module Not Loading

The ISO includes DKMS-built ZFS modules. If modules fail to load:
- Check ~dmesg | grep -i zfs~ for errors
- Ensure you're using the LTS kernel

** Disk Not Showing in Selection

- Ensure the disk is not mounted
- Check ~lsblk~ to verify disk visibility
- USB drives may need a moment to be detected

** Boot Fails After Installation

*** ZFS
- Check ZFSBootMenu appears (if not, check EFI boot order with ~efibootmgr~)
- Verify pool can import: boot ISO, ~zpool import -f zroot~

*** Btrfs
- Verify EFI boot entries: ~efibootmgr -v~
- Check GRUB config: ~/boot/grub/grub.cfg~

* Links

- [[https://archzfs.com][archzfs Repository]] - ZFS packages for Arch Linux
- [[https://openzfs.github.io/openzfs-docs/][OpenZFS Documentation]] - Official ZFS documentation
- [[https://get.zfsbootmenu.org][ZFSBootMenu]] - ZFS boot manager
- [[https://wiki.archlinux.org/title/Btrfs][Arch Wiki - Btrfs]] - Btrfs information
- [[https://wiki.archlinux.org/title/Snapper][Arch Wiki - Snapper]] - Btrfs snapshot management
- [[https://github.com/Antynea/grub-btrfs][grub-btrfs]] - Boot Btrfs snapshots from GRUB

* License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).
See [[file:LICENSE][LICENSE]] file for the full license text.

Note: [[https://github.com/openzfs/zfs][OpenZFS]] is licensed separately under the
[[https://github.com/openzfs/zfs/blob/master/LICENSE][CDDL license]]. ZFS packages
are provided by the [[https://archzfs.com][archzfs]] third-party repository and are
not part of this project.