path: root/docs/notes.org

#+TITLE: Claude Code Notes - archangel
#+AUTHOR: Craig Jennings & Claude
#+DATE: 2026-01-17

* About This File

This file contains project-specific information for this project.

**When to read this:**
- At the start of EVERY session (after reading protocols.org)
- When needing project context or history
- When checking reminders or pending decisions

**What's in this file:**
- Project-specific context and goals
- Available workflows for this project
- Active reminders
- Pending decisions
- Session history

**For protocols and conventions, see:** [[file:protocols.org][protocols.org]]

* Project-Specific Context

** Overview

Build system for creating a custom Arch Linux installation ISO with ZFS support. The goal is to have a bootable ISO that can install Arch Linux on ZFS root without needing to manually compile ZFS or deal with kernel version mismatches.

** Repository

- Remote: =cjennings@cjennings.net:git/archangel.git=
- Branch: =main=
- docs/ is committed (not private)

** Key Components

- =build.sh= - Main build script (runs as root)
  - Downloads ZFS packages from archzfs.com repository
  - Creates custom archiso profile based on releng
  - Adds custom packages (nodejs, npm, jq, zsh, htop, ripgrep, etc.)
  - Copies custom installer scripts into ISO
  - Builds ISO with mkarchiso

- =custom/= - Custom scripts included in ISO
  - =archangel= - Main installer script
  - =install-claude= - Claude Code installer
  - =archsetup-zfs= - ZFS-specific Arch setup
  - =zfs-setup= - Installs ZFS packages and loads module (generated by build.sh)

- =scripts/test-vm.sh= - QEMU VM for testing the ISO

** Current State

TESTING: archangel installer supports both ZFS and Btrfs.

- ISO builds successfully with linux-lts + zfs-dkms
- ZFS installations use ZFSBootMenu
- Btrfs installations use GRUB + grub-btrfs for snapshot boot
- Both filesystems support multi-disk RAID configurations

** Goals

Create a bootable Arch Linux installation ISO that:
1. Installs Arch on ZFS root with native encryption
2. Uses sane defaults for dataset layout
3. Configures automatic snapshots (sanoid)
4. Sets up replication to TrueNAS for backups
5. Includes Claude Code on live ISO for emergency troubleshooting

** Design Decisions

*** Kernel Strategy
- Use =linux-lts= + =zfs-dkms= from archzfs.com repo
- DKMS builds ZFS from source, guaranteeing kernel compatibility
- Slower build time but eliminates version mismatch issues entirely
- LTS kernel provides stability, DKMS provides flexibility

*** ZFS Pool Configuration
| Setting | Value | Rationale |
|---------+-------+-----------|
| Pool name | =zroot= | Standard convention |
| Encryption | AES-256-GCM, passphrase | Required at every boot |
| Compression | =zstd= (default) | Good balance of speed/ratio |
| Ashift | 12 (4K sectors) | Modern drives |
| Root reservation | 50GB | Prevents pool from filling |

*** Dataset Layout
| Dataset | Mountpoint | Special Settings | Purpose |
|---------+------------+------------------+---------|
| zroot/ROOT/default | / | reservation=50G | Root filesystem |
| zroot/home | /home | | Home directories (archsetup creates user subdataset) |
| zroot/media | /media | compression=off | Pre-compressed media files |
| zroot/vms | /vms | recordsize=64K | VM disk images (qemu/libvirt + virtualbox) |
| zroot/var/log | /var/log | | System logs |
| zroot/var/cache | /var/cache | | Package cache |
| zroot/var/lib/pacman | /var/lib/pacman | | Package database |
| zroot/var/lib/docker | /var/lib/docker | | Docker storage |
| zroot/tmp | /tmp | auto-snapshot=false | Temp files |
| zroot/var/tmp | /var/tmp | auto-snapshot=false | Temp files |

*** Snapshot Policy (Sanoid)
Less aggressive since TrueNAS handles long-term backups:

| Template | Hourly | Daily | Weekly | Monthly | Used For |
|----------+--------+-------+--------+---------+----------|
| production | 6 | 7 | 2 | 1 | root, home, var/log, pacman |
| backup | 0 | 3 | 2 | 1 | media, vms |
| none | 0 | 0 | 0 | 0 | tmp, cache |

Plus: Pacman hook creates snapshot before every transaction.

*** TrueNAS Replication
- Primary: =truenas.local= (local network)
- Fallback: =truenas= (tailscale)
- Destination pool: =vault/[TBD]=
- Schedule: Nightly at 2:00 AM
- Datasets: ROOT/default, home, media, vms

*** Included Packages
- Base system + development tools
- =nodejs=, =npm=, =jq= (for Claude Code)
- =zsh=, =htop=, =ripgrep=, =eza=, =fd=, =fzf=
- =sanoid= (snapshot management)
- =dialog= (installer UI)

*** Installation UX
- All questions asked upfront, then unattended installation
- WiFi tested before installation begins (if provided)
- User can walk away during install and come back
- Summary + final confirmation before starting

*** User Account Strategy
- install-archzfs creates root account only (asks for root password)
- No user account created during install
- Just create =zroot/home= dataset (no user-specific subdataset)
- archsetup creates user account + home dataset post-reboot

*** GRUB HiDPI Support
- Generate 32px DejaVuSansMono font during install
- Set =GRUB_FONT= to use custom font
- Works well on HiDPI and regular displays

*** WiFi Configuration
- Ask for SSID + password during install (optional)
- Test connection before installation starts
- Copy connection profile to installed system
- Auto-connects after reboot

*** Post-Install Workflow
1. install-archzfs: Minimal ZFS system + root account
2. Reboot, login as root
3. Run archsetup manually for full workstation setup

*** Testing/Debugging (VM)
- SSH access on live ISO: sshd enabled, known root password
- Serial console: =-serial mon:stdio= in QEMU for terminal copy/paste
- Port forwarding: 2222→22 (already configured)
- Allows easy copy/paste of error messages during testing

** Open Questions

- [ ] TrueNAS destination dataset path (vault/???)

* AVAILABLE WORKFLOWS

This section lists all documented workflows for this project. Update this section whenever a new workflow is created.

** create-workflow
File: [[file:workflows/create-workflow.org][docs/workflows/create-workflow.org]]

Meta-workflow for creating new workflows. Use this when identifying repetitive workflows that would benefit from documentation.

Workflow:
1. Q&A discovery (4 core questions)
2. Assess completeness
3. Name the workflow
4. Document it
5. Update notes.org
6. Validate by execution

Created: [Date when workflow was created]

** create-v2mom
File: [[file:workflows/create-v2mom.org][docs/workflows/create-v2mom.org]]

Workflow for creating a V2MOM (Vision, Values, Methods, Obstacles, Metrics) strategic framework for any project or goal.

Workflow:
1. Understand V2MOM framework
2. Create document structure
3. Define Vision (aspirational picture of success)
4. Define Values (2-4 principles with concrete definitions)
5. Define Methods (4-7 approaches ordered by priority)
6. Identify Obstacles (honest personal/technical challenges)
7. Define Metrics (measurable outcomes)
8. Review and refine
9. Commit and use immediately

Time: ~2-3 hours total
Applicable to: Any project (health, finance, software, personal infrastructure, etc.)

Created: 2025-11-05

** startup
File: [[file:workflows/startup.org][docs/workflows/startup.org]]

Workflow for beginning a Claude Code session with proper context and priorities.

Triggered by: **Automatically at the start of EVERY session**

Workflow:
1. Add session start timestamp (check for interrupted sessions)
2. Sync with templates (exclude notes.org and previous-session-history.org)
3. Scan workflows directory for available workflows
4. Read key notes.org sections (NOT entire file)
5. Process inbox (mandatory)
6. Ask about priorities (urgent work vs what's-next workflow)

Ensures: Full context, current templates, processed inbox, clear session direction

Created: 2025-11-14

** wrap-it-up
File: [[file:workflows/wrap-it-up.org][docs/workflows/wrap-it-up.org]]

Workflow for ending a Claude Code session cleanly with proper documentation and version control.

Triggered by: "wrap it up," "that's a wrap," "let's call it a wrap," or similar phrases

Workflow:
1. Write session notes to notes.org Session History section
2. Archive sessions older than 5 sessions to previous-session-history.org
3. Git commit and push all changes (NO Claude attribution)
4. Provide brief valediction with accomplishments and next steps

Ensures: Clean handoff between sessions, nothing lost, clear git history, proper documentation

Created: 2025-11-14

** [Add more workflows as they are created]

Format for new entries:
#+begin_example
** workflow-name
File: [[file:workflows/workflow-name.org][docs/workflows/workflow-name.org]]

Brief description of what this workflow does.

Workflow:
1. Step 1
2. Step 2
3. Step 3

Created: YYYY-MM-DD
#+end_example

* PENDING DECISIONS

This section tracks decisions that need Craig's input before work can proceed.

**Instructions:**
- Add pending decisions as they arise during sessions
- Format: =** [Topic/Feature Name]=
- Include: What needs to be decided, options available, why it matters
- Remove decisions once resolved (document resolution in Session History)

**Example format:**
#+begin_example
** Feature Name or Topic

Craig needs to decide on [specific question].

Options:
1. Option A - [brief description, pros/cons]
2. Option B - [brief description, pros/cons]

Why this matters: [impact on project]

Implementation is ready - just need Craig's preference.
#+end_example

** Current Pending Decisions

(None currently - will be added as they arise)

* Active Reminders

** Current Reminders

None.

** Instructions for This Section

When Craig says "remind me" about something:
1. Add it here with timestamp and description
2. If it's a TODO, also add to =/home/cjennings/sync/org/roam/inbox.org= scheduled for today
3. Check this section at start of every session
4. Remove reminders once addressed

Format:
- =[YYYY-MM-DD]= Description of what to remind Craig about

* Session History

This section contains notes from each session with Craig. Sessions are logged in reverse chronological order (most recent first).

**Note:** Sessions older than 5 sessions are archived in [[file:previous-session-history.org][Previous Session History]]

** Format for Session History Entries

Each entry should use this format:

- **Timestamp:** =*** YYYY-MM-DD Day @ HH:MM TZ= (get TZ with =date +%z=)
- **Time estimate:** How long the session took
- **Status:** COMPLETE / IN PROGRESS / PAUSED
- **What We Completed:** Bulleted list of accomplishments
- **Key Decisions:** Any important decisions made
- **Files Modified:** Links to changed files (use relative paths)
- **Next Steps:** What to do next session (if applicable)

**Best practices:**
- Keep entries concise but informative
- Include enough context to resume work later
- Document important technical insights
- Note any new patterns or preferences discovered
- Link to files using org-mode =file:= links

** Session Entries

*** 2026-04-09 Thu @ 21:30-22:31 -0500

*Status:* COMPLETE

*What We Completed:*
- Attempted ZFS install on bare metal machine "velox" (7.3T NVMe) — failed due to archzfs.com repo being stale (ZFS 2.3.3, max kernel 6.15)
- Fell back to Btrfs/LUKS install — completed but GRUB rejected the correct LUKS passphrase at boot
  - Root cause: LUKS2 created with argon2id PBKDF, GRUB only supports pbkdf2
  - Fixed in installer/lib/btrfs.sh (both single-disk and multi-disk paths)
- Processed inbox: archzfs repo migration notice
  - archzfs.com abandoned mid-2025, project moved to GitHub Releases
  - Updated build.sh and installer/archangel to use new URL: github.com/archzfs/archzfs/releases/download/experimental
  - ZFS 2.4.1 now available for kernel 6.18.21
- Rebuilt ISO with all fixes (archzfs URL, LUKS pbkdf2, no-color default)
- Successfully installed velox with ZFS encrypted root — boots and unlocks via ZFSBootMenu
- Added --color flag to archangel (default: no color, opt-in with flag)
- Added inetutils to installed system packages (hostname was missing)
- Tagged v0.8 and created GitHub release
- Distributed ISO to Ventoy USB and truenas.local:/mnt/vault/isos/ (cleaned up old ISOs)
- Created archsetup ZFS snapshot on velox
- Created velox-zfs.conf and velox-btrfs.conf config files for unattended installs

*Key Decisions:*
- archzfs repo migrated to GitHub Releases permanently
- LUKS2 containers now use pbkdf2 instead of argon2id for GRUB compatibility
- Installer output is colorless by default; --color flag enables it

*Bugs Found and Fixed:*
- GRUB LUKS2 argon2id incompatibility (installer/lib/btrfs.sh)
- archzfs.com stale repo causing ZFS/kernel mismatch (build.sh, installer/archangel)
- inetutils missing from target system packages (installer/archangel)

*Files Modified:*
- [[file:../build.sh][build.sh]] — archzfs URL, comment update
- [[file:../installer/archangel][installer/archangel]] — archzfs URL (2 places), added inetutils to both pacstrap lists
- [[file:../installer/lib/btrfs.sh][installer/lib/btrfs.sh]] — argon2id → pbkdf2
- [[file:../installer/lib/common.sh][installer/lib/common.sh]] — no-color default + enable_color()
- [[file:../installer/lib/config.sh][installer/lib/config.sh]] — --color flag parsing

*Files Created:*
- [[file:../installer/velox-zfs.conf][installer/velox-zfs.conf]] — ZFS unattended config for velox
- [[file:../installer/velox-btrfs.conf][installer/velox-btrfs.conf]] — Btrfs unattended config for velox

*Next Steps:*
- Add build-time check to prevent ZFS/kernel incompatibility in build.sh
- Continue open-sourcing prep

*** 2026-03-28 Sat @ 13:24-13:42 -0500

*Status:* COMPLETE

*What We Completed:*
- Synced templates from claude-templates (protocols, workflows, scripts, announcements)
- Moved 3 logo images from inbox/ to new assets/ directory (renamed space-in-filename)
- Added TODO [#C] task to todo.org: choose a project logo
- Built new archangel ISO for linux-lts 6.18.20 kernel
  - ISO: archangel-2026-03-28-vmlinuz-6.18.20-lts-x86_64.iso (2.0G)
  - First build failed (mkarchiso /proc cleanup race condition), retry succeeded
- Mounted Ventoy USB, copied ISO to /mnt/ventoy/
- Copied ISO to truenas.local:/mnt/vault/isos/ via rsync
- Unmounted Ventoy USB

*Files Modified:*
- [[file:../todo.org][todo.org]] — added logo selection task

*Files Added:*
- [[file:../assets/archangel-logo.png][assets/archangel-logo.png]]
- [[file:../assets/archangel-logo2.png][assets/archangel-logo2.png]]
- [[file:../assets/archangel-logo-samples.png][assets/archangel-logo-samples.png]]

*Next Steps:*
- Choose a project logo from candidates in assets/
- Build AUR packages as local repo for ISO (todo.org [#A] task)
- Continue open-sourcing prep

*** 2026-02-24 Tue @ 08:51-09:12 -0600

*Status:* COMPLETE

*What We Completed:*
- Built new archangel ISO for linux-lts 6.12.74-1 kernel
- ISO: archangel-2026-02-24-vmlinuz-6.12.74-lts-x86_64.iso (1.8G)
- ZFS DKMS 2.3.3 compiled successfully against new kernel
- Sanity test passed: 26/26 checks (ZFS, Btrfs, networking, scripts)
- Distributed locally to ~/archangel-isos/
- Pushed ISO to truenas.local:/mnt/vault/isos/ (TrueNAS now reachable)
- Cleared TrueNAS ISO hash reminder from 2026-02-12 (connectivity restored, ISO pushed)

*Files Modified:*
- None (build artifacts only, no source changes)

*Next Steps:*
- Continue with open-sourcing prep or other todo.org items
- Manual LUKS reboot verification when hardware available

*** 2026-02-24 Tue @ 08:42 -0600

*Status:* COMPLETE

*What We Completed:*
- Reorganized README.org sections to follow logical user journey (build → boot → install → post-reboot)
- Eliminated "Bare Metal Installation" section, redistributing content:
  - =dd= command → Building the ISO > Writing to USB
  - Secure Boot/boot steps → new "Booting the ISO" section
  - SSH/Avahi content → "SSH Access" subsection under Booting the ISO
  - =archangel= invocation + minimal install note → "Installation" intro
  - ZFS/Btrfs first-boot steps → new "Post-Reboot" section
- Renamed sections: "Connecting via SSH Server" → "Booting the ISO", "Arch Linux Install Walkthrough" → "Installation"
- Updated internal org links (#ssh-server → #ssh-access)

*Files Modified:*
- [[file:../README.org][README.org]] — section reorganization, no content lost

*Next Steps:*
- Continue with open-sourcing prep or other todo.org items
- Verify TrueNAS ISO hash (still pending from 2026-02-12)

*** 2026-02-19 Thu @ 16:11-16:14 -0600

*Status:* COMPLETE

*What We Completed:*
- Template sync from claude-templates (protocols, workflows, scripts, announcements)
- Processed 4 announcements:
  1. Calendar workflows updated with cross-calendar visibility
  2. gcalcli now available for Google Calendar CLI access
  3. New open-tasks workflow — updated todo.org headers to project-named convention (Archangel Open Work / Archangel Resolved)
  4. New summarize-emails workflow added
- New workflows synced: add-calendar-event, delete-calendar-event, edit-calendar-event, read-calendar-events, open-tasks, summarize-emails
- New script synced: maildir-flag-manager.py

*Files Modified:*
- [[file:../todo.org][todo.org]] — renamed headers to project-named convention

*Files Added (from template):*
- docs/workflows/{add,delete,edit,read}-calendar-event.org
- docs/workflows/open-tasks.org, summarize-emails.org
- docs/scripts/maildir-flag-manager.py
- docs/announcements/inbox-gitkeep.txt

*Outstanding Reminder:*
- [2026-02-12] Verify TrueNAS ISO hash — still pending

*** 2026-02-12 Thu @ 08:23-16:08 -0600

*Status:* COMPLETE

*What We Completed:*
- Rebuilt archangel ISO for linux-lts 6.12.70-1 kernel
- ISO: archangel-vmlinuz-6.12.70-lts-2026-02-12-x86_64.iso (2.3G)
- All tests passed: sanity (26/26), single-disk, mirror, raidz1
- Fixed archzfs GPG key prompt hanging unattended installs (SigLevel → Never)
- Fixed pgrep false positive in full-test.sh (avahi matched hostname pattern)
- Bumped INSTALL_TIMEOUT from 900s to 1800s for DKMS builds
- Added local distribution to build-release (~/downloads/isos + archsetup inbox notification)
- Distributed ISO to ~/downloads/isos and truenas.local:/mnt/vault/isos
- Audited codebase for open-source readiness, added todo.org task with full checklist
- Dropped SSH access info and test VM rebuild notice in archsetup inbox

*Key Decisions:*
- archzfs SigLevel changed to Never (HTTPS provides transport security; GPG key management kept breaking unattended installs)
- USB drives removed as distribution target
- build-release now handles ~/downloads/isos and archsetup inbox automatically

*Bugs Found and Fixed:*
1. archzfs GPG key prompt: pacstrap -K creates empty keyring, pacman-key -r silently fails, pacman prompts interactively → changed SigLevel to Never in custom/archangel (2 locations)
2. Test pgrep false positive: pgrep -f 'archangel' matched avahi-daemon's "running [archangel.local]" → changed to pgrep -f '/usr/local/bin/archangel'
3. Install timeout: 15 min too short for DKMS compile in VM → bumped to 30 min

*Files Modified:*
- [[file:../custom/archangel][custom/archangel]] — SigLevel fix (install_base + configure_system)
- [[file:../scripts/full-test.sh][scripts/full-test.sh]] — pgrep fix, timeout bump
- [[file:../scripts/build-release][scripts/build-release]] — local distribution + archsetup inbox
- [[file:../todo.org][todo.org]] — open-sourcing prep task

*Next Steps:*
- Verify TrueNAS ISO hash (was unreachable at session end)
- Fix TrueNAS connectivity issues
- Continue with open-sourcing prep or other todo.org items