session: first bare metal install on velox, multiple fixes

ZFS and Btrfs tested on bare metal. Fixed archzfs repo URL, LUKS pbkdf2
for GRUB, no-color default, and missing inetutils. Tagged v0.8.

1 files changed, 510 insertions, 0 deletions

diff --git a/docs/notes.org b/docs/notes.org
new file mode 100644
index 0000000..df795b0
--- /dev/null
+++ b/docs/notes.org
@@ -0,0 +1,510 @@
+#+TITLE: Claude Code Notes - archangel
+#+AUTHOR: Craig Jennings & Claude
+#+DATE: 2026-01-17
+
+* About This File
+
+This file contains project-specific information for this project.
+
+**When to read this:**
+- At the start of EVERY session (after reading protocols.org)
+- When needing project context or history
+- When checking reminders or pending decisions
+
+**What's in this file:**
+- Project-specific context and goals
+- Available workflows for this project
+- Active reminders
+- Pending decisions
+- Session history
+
+**For protocols and conventions, see:** [[file:protocols.org][protocols.org]]
+
+* Project-Specific Context
+
+** Overview
+
+Build system for creating a custom Arch Linux installation ISO with ZFS support. The goal is to have a bootable ISO that can install Arch Linux on ZFS root without needing to manually compile ZFS or deal with kernel version mismatches.
+
+** Repository
+
+- Remote: =cjennings@cjennings.net:git/archangel.git=
+- Branch: =main=
+- docs/ is committed (not private)
+
+** Key Components
+
+- =build.sh= - Main build script (runs as root)
+  - Downloads ZFS packages from archzfs.com repository
+  - Creates custom archiso profile based on releng
+  - Adds custom packages (nodejs, npm, jq, zsh, htop, ripgrep, etc.)
+  - Copies custom installer scripts into ISO
+  - Builds ISO with mkarchiso
+
+- =custom/= - Custom scripts included in ISO
+  - =archangel= - Main installer script
+  - =install-claude= - Claude Code installer
+  - =archsetup-zfs= - ZFS-specific Arch setup
+  - =zfs-setup= - Installs ZFS packages and loads module (generated by build.sh)
+
+- =scripts/test-vm.sh= - QEMU VM for testing the ISO
+
+** Current State
+
+TESTING: archangel installer supports both ZFS and Btrfs.
+
+- ISO builds successfully with linux-lts + zfs-dkms
+- ZFS installations use ZFSBootMenu
+- Btrfs installations use GRUB + grub-btrfs for snapshot boot
+- Both filesystems support multi-disk RAID configurations
+
+** Goals
+
+Create a bootable Arch Linux installation ISO that:
+1. Installs Arch on ZFS root with native encryption
+2. Uses sane defaults for dataset layout
+3. Configures automatic snapshots (sanoid)
+4. Sets up replication to TrueNAS for backups
+5. Includes Claude Code on live ISO for emergency troubleshooting
+
+** Design Decisions
+
+*** Kernel Strategy
+- Use =linux-lts= + =zfs-dkms= from archzfs.com repo
+- DKMS builds ZFS from source, guaranteeing kernel compatibility
+- Slower build time but eliminates version mismatch issues entirely
+- LTS kernel provides stability, DKMS provides flexibility
+
+*** ZFS Pool Configuration
+| Setting | Value | Rationale |
+|---------+-------+-----------|
+| Pool name | =zroot= | Standard convention |
+| Encryption | AES-256-GCM, passphrase | Required at every boot |
+| Compression | =zstd= (default) | Good balance of speed/ratio |
+| Ashift | 12 (4K sectors) | Modern drives |
+| Root reservation | 50GB | Prevents pool from filling |
+
+*** Dataset Layout
+| Dataset | Mountpoint | Special Settings | Purpose |
+|---------+------------+------------------+---------|
+| zroot/ROOT/default | / | reservation=50G | Root filesystem |
+| zroot/home | /home | | Home directories (archsetup creates user subdataset) |
+| zroot/media | /media | compression=off | Pre-compressed media files |
+| zroot/vms | /vms | recordsize=64K | VM disk images (qemu/libvirt + virtualbox) |
+| zroot/var/log | /var/log | | System logs |
+| zroot/var/cache | /var/cache | | Package cache |
+| zroot/var/lib/pacman | /var/lib/pacman | | Package database |
+| zroot/var/lib/docker | /var/lib/docker | | Docker storage |
+| zroot/tmp | /tmp | auto-snapshot=false | Temp files |
+| zroot/var/tmp | /var/tmp | auto-snapshot=false | Temp files |
+
+*** Snapshot Policy (Sanoid)
+Less aggressive since TrueNAS handles long-term backups:
+
+| Template | Hourly | Daily | Weekly | Monthly | Used For |
+|----------+--------+-------+--------+---------+----------|
+| production | 6 | 7 | 2 | 1 | root, home, var/log, pacman |
+| backup | 0 | 3 | 2 | 1 | media, vms |
+| none | 0 | 0 | 0 | 0 | tmp, cache |
+
+Plus: Pacman hook creates snapshot before every transaction.
+
+*** TrueNAS Replication
+- Primary: =truenas.local= (local network)
+- Fallback: =truenas= (tailscale)
+- Destination pool: =vault/[TBD]=
+- Schedule: Nightly at 2:00 AM
+- Datasets: ROOT/default, home, media, vms
+
+*** Included Packages
+- Base system + development tools
+- =nodejs=, =npm=, =jq= (for Claude Code)
+- =zsh=, =htop=, =ripgrep=, =eza=, =fd=, =fzf=
+- =sanoid= (snapshot management)
+- =dialog= (installer UI)
+
+*** Installation UX
+- All questions asked upfront, then unattended installation
+- WiFi tested before installation begins (if provided)
+- User can walk away during install and come back
+- Summary + final confirmation before starting
+
+*** User Account Strategy
+- install-archzfs creates root account only (asks for root password)
+- No user account created during install
+- Just create =zroot/home= dataset (no user-specific subdataset)
+- archsetup creates user account + home dataset post-reboot
+
+*** GRUB HiDPI Support
+- Generate 32px DejaVuSansMono font during install
+- Set =GRUB_FONT= to use custom font
+- Works well on HiDPI and regular displays
+
+*** WiFi Configuration
+- Ask for SSID + password during install (optional)
+- Test connection before installation starts
+- Copy connection profile to installed system
+- Auto-connects after reboot
+
+*** Post-Install Workflow
+1. install-archzfs: Minimal ZFS system + root account
+2. Reboot, login as root
+3. Run archsetup manually for full workstation setup
+
+*** Testing/Debugging (VM)
+- SSH access on live ISO: sshd enabled, known root password
+- Serial console: =-serial mon:stdio= in QEMU for terminal copy/paste
+- Port forwarding: 2222→22 (already configured)
+- Allows easy copy/paste of error messages during testing
+
+** Open Questions
+
+- [ ] TrueNAS destination dataset path (vault/???)
+
+* AVAILABLE WORKFLOWS
+
+This section lists all documented workflows for this project. Update this section whenever a new workflow is created.
+
+** create-workflow
+File: [[file:workflows/create-workflow.org][docs/workflows/create-workflow.org]]
+
+Meta-workflow for creating new workflows. Use this when identifying repetitive workflows that would benefit from documentation.
+
+Workflow:
+1. Q&A discovery (4 core questions)
+2. Assess completeness
+3. Name the workflow
+4. Document it
+5. Update notes.org
+6. Validate by execution
+
+Created: [Date when workflow was created]
+
+** create-v2mom
+File: [[file:workflows/create-v2mom.org][docs/workflows/create-v2mom.org]]
+
+Workflow for creating a V2MOM (Vision, Values, Methods, Obstacles, Metrics) strategic framework for any project or goal.
+
+Workflow:
+1. Understand V2MOM framework
+2. Create document structure
+3. Define Vision (aspirational picture of success)
+4. Define Values (2-4 principles with concrete definitions)
+5. Define Methods (4-7 approaches ordered by priority)
+6. Identify Obstacles (honest personal/technical challenges)
+7. Define Metrics (measurable outcomes)
+8. Review and refine
+9. Commit and use immediately
+
+Time: ~2-3 hours total
+Applicable to: Any project (health, finance, software, personal infrastructure, etc.)
+
+Created: 2025-11-05
+
+** startup
+File: [[file:workflows/startup.org][docs/workflows/startup.org]]
+
+Workflow for beginning a Claude Code session with proper context and priorities.
+
+Triggered by: **Automatically at the start of EVERY session**
+
+Workflow:
+1. Add session start timestamp (check for interrupted sessions)
+2. Sync with templates (exclude notes.org and previous-session-history.org)
+3. Scan workflows directory for available workflows
+4. Read key notes.org sections (NOT entire file)
+5. Process inbox (mandatory)
+6. Ask about priorities (urgent work vs what's-next workflow)
+
+Ensures: Full context, current templates, processed inbox, clear session direction
+
+Created: 2025-11-14
+
+** wrap-it-up
+File: [[file:workflows/wrap-it-up.org][docs/workflows/wrap-it-up.org]]
+
+Workflow for ending a Claude Code session cleanly with proper documentation and version control.
+
+Triggered by: "wrap it up," "that's a wrap," "let's call it a wrap," or similar phrases
+
+Workflow:
+1. Write session notes to notes.org Session History section
+2. Archive sessions older than 5 sessions to previous-session-history.org
+3. Git commit and push all changes (NO Claude attribution)
+4. Provide brief valediction with accomplishments and next steps
+
+Ensures: Clean handoff between sessions, nothing lost, clear git history, proper documentation
+
+Created: 2025-11-14
+
+** [Add more workflows as they are created]
+
+Format for new entries:
+#+begin_example
+** workflow-name
+File: [[file:workflows/workflow-name.org][docs/workflows/workflow-name.org]]
+
+Brief description of what this workflow does.
+
+Workflow:
+1. Step 1
+2. Step 2
+3. Step 3
+
+Created: YYYY-MM-DD
+#+end_example
+
+* PENDING DECISIONS
+
+This section tracks decisions that need Craig's input before work can proceed.
+
+**Instructions:**
+- Add pending decisions as they arise during sessions
+- Format: =** [Topic/Feature Name]=
+- Include: What needs to be decided, options available, why it matters
+- Remove decisions once resolved (document resolution in Session History)
+
+**Example format:**
+#+begin_example
+** Feature Name or Topic
+
+Craig needs to decide on [specific question].
+
+Options:
+1. Option A - [brief description, pros/cons]
+2. Option B - [brief description, pros/cons]
+
+Why this matters: [impact on project]
+
+Implementation is ready - just need Craig's preference.
+#+end_example
+
+** Current Pending Decisions
+
+(None currently - will be added as they arise)
+
+* Active Reminders
+
+** Current Reminders
+
+None.
+
+** Instructions for This Section
+
+When Craig says "remind me" about something:
+1. Add it here with timestamp and description
+2. If it's a TODO, also add to =/home/cjennings/sync/org/roam/inbox.org= scheduled for today
+3. Check this section at start of every session
+4. Remove reminders once addressed
+
+Format:
+- =[YYYY-MM-DD]= Description of what to remind Craig about
+
+* Session History
+
+This section contains notes from each session with Craig. Sessions are logged in reverse chronological order (most recent first).
+
+**Note:** Sessions older than 5 sessions are archived in [[file:previous-session-history.org][Previous Session History]]
+
+** Format for Session History Entries
+
+Each entry should use this format:
+
+- **Timestamp:** =*** YYYY-MM-DD Day @ HH:MM TZ= (get TZ with =date +%z=)
+- **Time estimate:** How long the session took
+- **Status:** COMPLETE / IN PROGRESS / PAUSED
+- **What We Completed:** Bulleted list of accomplishments
+- **Key Decisions:** Any important decisions made
+- **Files Modified:** Links to changed files (use relative paths)
+- **Next Steps:** What to do next session (if applicable)
+
+**Best practices:**
+- Keep entries concise but informative
+- Include enough context to resume work later
+- Document important technical insights
+- Note any new patterns or preferences discovered
+- Link to files using org-mode =file:= links
+
+** Session Entries
+
+*** 2026-04-09 Thu @ 21:30-22:31 -0500
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Attempted ZFS install on bare metal machine "velox" (7.3T NVMe) — failed due to archzfs.com repo being stale (ZFS 2.3.3, max kernel 6.15)
+- Fell back to Btrfs/LUKS install — completed but GRUB rejected the correct LUKS passphrase at boot
+  - Root cause: LUKS2 created with argon2id PBKDF, GRUB only supports pbkdf2
+  - Fixed in installer/lib/btrfs.sh (both single-disk and multi-disk paths)
+- Processed inbox: archzfs repo migration notice
+  - archzfs.com abandoned mid-2025, project moved to GitHub Releases
+  - Updated build.sh and installer/archangel to use new URL: github.com/archzfs/archzfs/releases/download/experimental
+  - ZFS 2.4.1 now available for kernel 6.18.21
+- Rebuilt ISO with all fixes (archzfs URL, LUKS pbkdf2, no-color default)
+- Successfully installed velox with ZFS encrypted root — boots and unlocks via ZFSBootMenu
+- Added --color flag to archangel (default: no color, opt-in with flag)
+- Added inetutils to installed system packages (hostname was missing)
+- Tagged v0.8 and created GitHub release
+- Distributed ISO to Ventoy USB and truenas.local:/mnt/vault/isos/ (cleaned up old ISOs)
+- Created archsetup ZFS snapshot on velox
+- Created velox-zfs.conf and velox-btrfs.conf config files for unattended installs
+
+*Key Decisions:*
+- archzfs repo migrated to GitHub Releases permanently
+- LUKS2 containers now use pbkdf2 instead of argon2id for GRUB compatibility
+- Installer output is colorless by default; --color flag enables it
+
+*Bugs Found and Fixed:*
+- GRUB LUKS2 argon2id incompatibility (installer/lib/btrfs.sh)
+- archzfs.com stale repo causing ZFS/kernel mismatch (build.sh, installer/archangel)
+- inetutils missing from target system packages (installer/archangel)
+
+*Files Modified:*
+- [[file:../build.sh][build.sh]] — archzfs URL, comment update
+- [[file:../installer/archangel][installer/archangel]] — archzfs URL (2 places), added inetutils to both pacstrap lists
+- [[file:../installer/lib/btrfs.sh][installer/lib/btrfs.sh]] — argon2id → pbkdf2
+- [[file:../installer/lib/common.sh][installer/lib/common.sh]] — no-color default + enable_color()
+- [[file:../installer/lib/config.sh][installer/lib/config.sh]] — --color flag parsing
+
+*Files Created:*
+- [[file:../installer/velox-zfs.conf][installer/velox-zfs.conf]] — ZFS unattended config for velox
+- [[file:../installer/velox-btrfs.conf][installer/velox-btrfs.conf]] — Btrfs unattended config for velox
+
+*Next Steps:*
+- Add build-time check to prevent ZFS/kernel incompatibility in build.sh
+- Continue open-sourcing prep
+
+*** 2026-03-28 Sat @ 13:24-13:42 -0500
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Synced templates from claude-templates (protocols, workflows, scripts, announcements)
+- Moved 3 logo images from inbox/ to new assets/ directory (renamed space-in-filename)
+- Added TODO [#C] task to todo.org: choose a project logo
+- Built new archangel ISO for linux-lts 6.18.20 kernel
+  - ISO: archangel-2026-03-28-vmlinuz-6.18.20-lts-x86_64.iso (2.0G)
+  - First build failed (mkarchiso /proc cleanup race condition), retry succeeded
+- Mounted Ventoy USB, copied ISO to /mnt/ventoy/
+- Copied ISO to truenas.local:/mnt/vault/isos/ via rsync
+- Unmounted Ventoy USB
+
+*Files Modified:*
+- [[file:../todo.org][todo.org]] — added logo selection task
+
+*Files Added:*
+- [[file:../assets/archangel-logo.png][assets/archangel-logo.png]]
+- [[file:../assets/archangel-logo2.png][assets/archangel-logo2.png]]
+- [[file:../assets/archangel-logo-samples.png][assets/archangel-logo-samples.png]]
+
+*Next Steps:*
+- Choose a project logo from candidates in assets/
+- Build AUR packages as local repo for ISO (todo.org [#A] task)
+- Continue open-sourcing prep
+
+*** 2026-02-24 Tue @ 08:51-09:12 -0600
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Built new archangel ISO for linux-lts 6.12.74-1 kernel
+- ISO: archangel-2026-02-24-vmlinuz-6.12.74-lts-x86_64.iso (1.8G)
+- ZFS DKMS 2.3.3 compiled successfully against new kernel
+- Sanity test passed: 26/26 checks (ZFS, Btrfs, networking, scripts)
+- Distributed locally to ~/archangel-isos/
+- Pushed ISO to truenas.local:/mnt/vault/isos/ (TrueNAS now reachable)
+- Cleared TrueNAS ISO hash reminder from 2026-02-12 (connectivity restored, ISO pushed)
+
+*Files Modified:*
+- None (build artifacts only, no source changes)
+
+*Next Steps:*
+- Continue with open-sourcing prep or other todo.org items
+- Manual LUKS reboot verification when hardware available
+
+*** 2026-02-24 Tue @ 08:42 -0600
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Reorganized README.org sections to follow logical user journey (build → boot → install → post-reboot)
+- Eliminated "Bare Metal Installation" section, redistributing content:
+  - =dd= command → Building the ISO > Writing to USB
+  - Secure Boot/boot steps → new "Booting the ISO" section
+  - SSH/Avahi content → "SSH Access" subsection under Booting the ISO
+  - =archangel= invocation + minimal install note → "Installation" intro
+  - ZFS/Btrfs first-boot steps → new "Post-Reboot" section
+- Renamed sections: "Connecting via SSH Server" → "Booting the ISO", "Arch Linux Install Walkthrough" → "Installation"
+- Updated internal org links (#ssh-server → #ssh-access)
+
+*Files Modified:*
+- [[file:../README.org][README.org]] — section reorganization, no content lost
+
+*Next Steps:*
+- Continue with open-sourcing prep or other todo.org items
+- Verify TrueNAS ISO hash (still pending from 2026-02-12)
+
+*** 2026-02-19 Thu @ 16:11-16:14 -0600
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Template sync from claude-templates (protocols, workflows, scripts, announcements)
+- Processed 4 announcements:
+  1. Calendar workflows updated with cross-calendar visibility
+  2. gcalcli now available for Google Calendar CLI access
+  3. New open-tasks workflow — updated todo.org headers to project-named convention (Archangel Open Work / Archangel Resolved)
+  4. New summarize-emails workflow added
+- New workflows synced: add-calendar-event, delete-calendar-event, edit-calendar-event, read-calendar-events, open-tasks, summarize-emails
+- New script synced: maildir-flag-manager.py
+
+*Files Modified:*
+- [[file:../todo.org][todo.org]] — renamed headers to project-named convention
+
+*Files Added (from template):*
+- docs/workflows/{add,delete,edit,read}-calendar-event.org
+- docs/workflows/open-tasks.org, summarize-emails.org
+- docs/scripts/maildir-flag-manager.py
+- docs/announcements/inbox-gitkeep.txt
+
+*Outstanding Reminder:*
+- [2026-02-12] Verify TrueNAS ISO hash — still pending
+
+*** 2026-02-12 Thu @ 08:23-16:08 -0600
+
+*Status:* COMPLETE
+
+*What We Completed:*
+- Rebuilt archangel ISO for linux-lts 6.12.70-1 kernel
+- ISO: archangel-vmlinuz-6.12.70-lts-2026-02-12-x86_64.iso (2.3G)
+- All tests passed: sanity (26/26), single-disk, mirror, raidz1
+- Fixed archzfs GPG key prompt hanging unattended installs (SigLevel → Never)
+- Fixed pgrep false positive in full-test.sh (avahi matched hostname pattern)
+- Bumped INSTALL_TIMEOUT from 900s to 1800s for DKMS builds
+- Added local distribution to build-release (~/downloads/isos + archsetup inbox notification)
+- Distributed ISO to ~/downloads/isos and truenas.local:/mnt/vault/isos
+- Audited codebase for open-source readiness, added todo.org task with full checklist
+- Dropped SSH access info and test VM rebuild notice in archsetup inbox
+
+*Key Decisions:*
+- archzfs SigLevel changed to Never (HTTPS provides transport security; GPG key management kept breaking unattended installs)
+- USB drives removed as distribution target
+- build-release now handles ~/downloads/isos and archsetup inbox automatically
+
+*Bugs Found and Fixed:*
+1. archzfs GPG key prompt: pacstrap -K creates empty keyring, pacman-key -r silently fails, pacman prompts interactively → changed SigLevel to Never in custom/archangel (2 locations)
+2. Test pgrep false positive: pgrep -f 'archangel' matched avahi-daemon's "running [archangel.local]" → changed to pgrep -f '/usr/local/bin/archangel'
+3. Install timeout: 15 min too short for DKMS compile in VM → bumped to 30 min
+
+*Files Modified:*
+- [[file:../custom/archangel][custom/archangel]] — SigLevel fix (install_base + configure_system)
+- [[file:../scripts/full-test.sh][scripts/full-test.sh]] — pgrep fix, timeout bump
+- [[file:../scripts/build-release][scripts/build-release]] — local distribution + archsetup inbox
+- [[file:../todo.org][todo.org]] — open-sourcing prep task
+
+*Next Steps:*
+- Verify TrueNAS ISO hash (was unreachable at session end)
+- Fix TrueNAS connectivity issues
+- Continue with open-sourcing prep or other todo.org items
+
+