#+TITLE: Claude Code Notes - archangel
#+AUTHOR: Craig Jennings & Claude
#+DATE: 2026-01-17

* About This File

This file contains project-specific information for this project.

**When to read this:**
- At the start of EVERY session (after reading protocols.org)
- When needing project context or history
- When checking reminders or pending decisions

**What's in this file:**
- Project-specific context and goals
- Available workflows for this project
- Active reminders
- Pending decisions
- Session history

**For protocols and conventions, see:** [[file:protocols.org][protocols.org]]

* Project-Specific Context

** Overview

Build system for creating a custom Arch Linux installation ISO with ZFS support. The goal is to have a bootable ISO that can install Arch Linux on ZFS root without needing to manually compile ZFS or deal with kernel version mismatches.

** Repository

- Remote: =cjennings@cjennings.net:git/archangel.git=
- Branch: =main=
- docs/ is committed (not private)

** Key Components

- =build.sh= - Main build script (runs as root)
  - Downloads ZFS packages from archzfs.com repository
  - Creates custom archiso profile based on releng
  - Adds custom packages (nodejs, npm, jq, zsh, htop, ripgrep, etc.)
  - Copies custom installer scripts into ISO
  - Builds ISO with mkarchiso

- =custom/= - Custom scripts included in ISO
  - =archangel= - Main installer script
  - =install-claude= - Claude Code installer
  - =archsetup-zfs= - ZFS-specific Arch setup
  - =zfs-setup= - Installs ZFS packages and loads module (generated by build.sh)

- =scripts/test-vm.sh= - QEMU VM for testing the ISO

** Current State

TESTING: archangel installer supports both ZFS and Btrfs.

- ISO builds successfully with linux-lts + zfs-dkms
- ZFS installations use ZFSBootMenu
- Btrfs installations use GRUB + grub-btrfs for snapshot boot
- Both filesystems support multi-disk RAID configurations

** Goals

Create a bootable Arch Linux installation ISO that:
1. Installs Arch on ZFS root with native encryption
2. Uses sane defaults for dataset layout
3. Configures automatic snapshots (sanoid)
4. Sets up replication to TrueNAS for backups
5. Includes Claude Code on live ISO for emergency troubleshooting

** Design Decisions

*** Kernel Strategy
- Use =linux-lts= + =zfs-dkms= from archzfs.com repo
- DKMS builds ZFS from source, guaranteeing kernel compatibility
- Slower build time but eliminates version mismatch issues entirely
- LTS kernel provides stability, DKMS provides flexibility

*** ZFS Pool Configuration
| Setting | Value | Rationale |
|---------+-------+-----------|
| Pool name | =zroot= | Standard convention |
| Encryption | AES-256-GCM, passphrase | Required at every boot |
| Compression | =zstd= (default) | Good balance of speed/ratio |
| Ashift | 12 (4K sectors) | Modern drives |
| Root reservation | 50GB | Prevents pool from filling |

*** Dataset Layout
| Dataset | Mountpoint | Special Settings | Purpose |
|---------+------------+------------------+---------|
| zroot/ROOT/default | / | reservation=50G | Root filesystem |
| zroot/home | /home | | Home directories (archsetup creates user subdataset) |
| zroot/media | /media | compression=off | Pre-compressed media files |
| zroot/vms | /vms | recordsize=64K | VM disk images (qemu/libvirt + virtualbox) |
| zroot/var/log | /var/log | | System logs |
| zroot/var/cache | /var/cache | | Package cache |
| zroot/var/lib/pacman | /var/lib/pacman | | Package database |
| zroot/var/lib/docker | /var/lib/docker | | Docker storage |
| zroot/tmp | /tmp | auto-snapshot=false | Temp files |
| zroot/var/tmp | /var/tmp | auto-snapshot=false | Temp files |

*** Snapshot Policy (Sanoid)
Less aggressive since TrueNAS handles long-term backups:

| Template | Hourly | Daily | Weekly | Monthly | Used For |
|----------+--------+-------+--------+---------+----------|
| production | 6 | 7 | 2 | 1 | root, home, var/log, pacman |
| backup | 0 | 3 | 2 | 1 | media, vms |
| none | 0 | 0 | 0 | 0 | tmp, cache |

Plus: Pacman hook creates snapshot before every transaction.

*** TrueNAS Replication
- Primary: =truenas.local= (local network)
- Fallback: =truenas= (tailscale)
- Destination pool: =vault/[TBD]=
- Schedule: Nightly at 2:00 AM
- Datasets: ROOT/default, home, media, vms

*** Included Packages
- Base system + development tools
- =nodejs=, =npm=, =jq= (for Claude Code)
- =zsh=, =htop=, =ripgrep=, =eza=, =fd=, =fzf=
- =sanoid= (snapshot management)
- =dialog= (installer UI)

*** Installation UX
- All questions asked upfront, then unattended installation
- WiFi tested before installation begins (if provided)
- User can walk away during install and come back
- Summary + final confirmation before starting

*** User Account Strategy
- install-archzfs creates root account only (asks for root password)
- No user account created during install
- Just create =zroot/home= dataset (no user-specific subdataset)
- archsetup creates user account + home dataset post-reboot

*** GRUB HiDPI Support
- Generate 32px DejaVuSansMono font during install
- Set =GRUB_FONT= to use custom font
- Works well on HiDPI and regular displays

*** WiFi Configuration
- Ask for SSID + password during install (optional)
- Test connection before installation starts
- Copy connection profile to installed system
- Auto-connects after reboot

*** Post-Install Workflow
1. install-archzfs: Minimal ZFS system + root account
2. Reboot, login as root
3. Run archsetup manually for full workstation setup

*** Testing/Debugging (VM)
- SSH access on live ISO: sshd enabled, known root password
- Serial console: =-serial mon:stdio= in QEMU for terminal copy/paste
- Port forwarding: 2222→22 (already configured)
- Allows easy copy/paste of error messages during testing

** Open Questions

- [ ] TrueNAS destination dataset path (vault/???)

* AVAILABLE WORKFLOWS

This section lists all documented workflows for this project. Update this section whenever a new workflow is created.

** create-workflow
File: [[file:workflows/create-workflow.org][docs/workflows/create-workflow.org]]

Meta-workflow for creating new workflows. Use this when identifying repetitive workflows that would benefit from documentation.

Workflow:
1. Q&A discovery (4 core questions)
2. Assess completeness
3. Name the workflow
4. Document it
5. Update notes.org
6. Validate by execution

Created: [Date when workflow was created]

** create-v2mom
File: [[file:workflows/create-v2mom.org][docs/workflows/create-v2mom.org]]

Workflow for creating a V2MOM (Vision, Values, Methods, Obstacles, Metrics) strategic framework for any project or goal.

Workflow:
1. Understand V2MOM framework
2. Create document structure
3. Define Vision (aspirational picture of success)
4. Define Values (2-4 principles with concrete definitions)
5. Define Methods (4-7 approaches ordered by priority)
6. Identify Obstacles (honest personal/technical challenges)
7. Define Metrics (measurable outcomes)
8. Review and refine
9. Commit and use immediately

Time: ~2-3 hours total
Applicable to: Any project (health, finance, software, personal infrastructure, etc.)

Created: 2025-11-05

** startup
File: [[file:workflows/startup.org][docs/workflows/startup.org]]

Workflow for beginning a Claude Code session with proper context and priorities.

Triggered by: **Automatically at the start of EVERY session**

Workflow:
1. Add session start timestamp (check for interrupted sessions)
2. Sync with templates (exclude notes.org and previous-session-history.org)
3. Scan workflows directory for available workflows
4. Read key notes.org sections (NOT entire file)
5. Process inbox (mandatory)
6. Ask about priorities (urgent work vs what's-next workflow)

Ensures: Full context, current templates, processed inbox, clear session direction

Created: 2025-11-14

** wrap-it-up
File: [[file:workflows/wrap-it-up.org][docs/workflows/wrap-it-up.org]]

Workflow for ending a Claude Code session cleanly with proper documentation and version control.

Triggered by: "wrap it up," "that's a wrap," "let's call it a wrap," or similar phrases

Workflow:
1. Write session notes to notes.org Session History section
2. Archive sessions older than 5 sessions to previous-session-history.org
3. Git commit and push all changes (NO Claude attribution)
4. Provide brief valediction with accomplishments and next steps

Ensures: Clean handoff between sessions, nothing lost, clear git history, proper documentation

Created: 2025-11-14

** [Add more workflows as they are created]

Format for new entries:
#+begin_example
** workflow-name
File: [[file:workflows/workflow-name.org][docs/workflows/workflow-name.org]]

Brief description of what this workflow does.

Workflow:
1. Step 1
2. Step 2
3. Step 3

Created: YYYY-MM-DD
#+end_example

* PENDING DECISIONS

This section tracks decisions that need Craig's input before work can proceed.

**Instructions:**
- Add pending decisions as they arise during sessions
- Format: =** [Topic/Feature Name]=
- Include: What needs to be decided, options available, why it matters
- Remove decisions once resolved (document resolution in Session History)

**Example format:**
#+begin_example
** Feature Name or Topic

Craig needs to decide on [specific question].

Options:
1. Option A - [brief description, pros/cons]
2. Option B - [brief description, pros/cons]

Why this matters: [impact on project]

Implementation is ready - just need Craig's preference.
#+end_example

** Current Pending Decisions

(None currently - will be added as they arise)

* Active Reminders

** Current Reminders

- [2026-02-12] Verify TrueNAS ISO hash matches local: =d17351445e4110ed2cf7190c25dc5fa91ec7325bb34644bbca1515fcd876d251=. TrueNAS was unreachable at end of session. Push hash file to truenas isos directory after verifying.

** Instructions for This Section

When Craig says "remind me" about something:
1. Add it here with timestamp and description
2. If it's a TODO, also add to =/home/cjennings/sync/org/roam/inbox.org= scheduled for today
3. Check this section at start of every session
4. Remove reminders once addressed

Format:
- =[YYYY-MM-DD]= Description of what to remind Craig about

* Session History

This section contains notes from each session with Craig. Sessions are logged in reverse chronological order (most recent first).

**Note:** Sessions older than 5 sessions are archived in [[file:previous-session-history.org][Previous Session History]]

** Format for Session History Entries

Each entry should use this format:

- **Timestamp:** =*** YYYY-MM-DD Day @ HH:MM TZ= (get TZ with =date +%z=)
- **Time estimate:** How long the session took
- **Status:** COMPLETE / IN PROGRESS / PAUSED
- **What We Completed:** Bulleted list of accomplishments
- **Key Decisions:** Any important decisions made
- **Files Modified:** Links to changed files (use relative paths)
- **Next Steps:** What to do next session (if applicable)

**Best practices:**
- Keep entries concise but informative
- Include enough context to resume work later
- Document important technical insights
- Note any new patterns or preferences discovered
- Link to files using org-mode =file:= links

** Session Entries

*** 2026-02-19 Thu @ 16:11-16:14 -0600

*Status:* COMPLETE

*What We Completed:*
- Template sync from claude-templates (protocols, workflows, scripts, announcements)
- Processed 4 announcements:
  1. Calendar workflows updated with cross-calendar visibility
  2. gcalcli now available for Google Calendar CLI access
  3. New open-tasks workflow — updated todo.org headers to project-named convention (Archangel Open Work / Archangel Resolved)
  4. New summarize-emails workflow added
- New workflows synced: add-calendar-event, delete-calendar-event, edit-calendar-event, read-calendar-events, open-tasks, summarize-emails
- New script synced: maildir-flag-manager.py

*Files Modified:*
- [[file:../todo.org][todo.org]] — renamed headers to project-named convention

*Files Added (from template):*
- docs/workflows/{add,delete,edit,read}-calendar-event.org
- docs/workflows/open-tasks.org, summarize-emails.org
- docs/scripts/maildir-flag-manager.py
- docs/announcements/inbox-gitkeep.txt

*Outstanding Reminder:*
- [2026-02-12] Verify TrueNAS ISO hash — still pending

*** 2026-02-12 Thu @ 08:23-16:08 -0600

*Status:* COMPLETE

*What We Completed:*
- Rebuilt archangel ISO for linux-lts 6.12.70-1 kernel
- ISO: archangel-vmlinuz-6.12.70-lts-2026-02-12-x86_64.iso (2.3G)
- All tests passed: sanity (26/26), single-disk, mirror, raidz1
- Fixed archzfs GPG key prompt hanging unattended installs (SigLevel → Never)
- Fixed pgrep false positive in full-test.sh (avahi matched hostname pattern)
- Bumped INSTALL_TIMEOUT from 900s to 1800s for DKMS builds
- Added local distribution to build-release (~/downloads/isos + archsetup inbox notification)
- Distributed ISO to ~/downloads/isos and truenas.local:/mnt/vault/isos
- Audited codebase for open-source readiness, added todo.org task with full checklist
- Dropped SSH access info and test VM rebuild notice in archsetup inbox

*Key Decisions:*
- archzfs SigLevel changed to Never (HTTPS provides transport security; GPG key management kept breaking unattended installs)
- USB drives removed as distribution target
- build-release now handles ~/downloads/isos and archsetup inbox automatically

*Bugs Found and Fixed:*
1. archzfs GPG key prompt: pacstrap -K creates empty keyring, pacman-key -r silently fails, pacman prompts interactively → changed SigLevel to Never in custom/archangel (2 locations)
2. Test pgrep false positive: pgrep -f 'archangel' matched avahi-daemon's "running [archangel.local]" → changed to pgrep -f '/usr/local/bin/archangel'
3. Install timeout: 15 min too short for DKMS compile in VM → bumped to 30 min

*Files Modified:*
- [[file:../custom/archangel][custom/archangel]] — SigLevel fix (install_base + configure_system)
- [[file:../scripts/full-test.sh][scripts/full-test.sh]] — pgrep fix, timeout bump
- [[file:../scripts/build-release][scripts/build-release]] — local distribution + archsetup inbox
- [[file:../todo.org][todo.org]] — open-sourcing prep task

*Next Steps:*
- Verify TrueNAS ISO hash (was unreachable at session end)
- Fix TrueNAS connectivity issues
- Continue with open-sourcing prep or other todo.org items

*** 2026-02-07 Sat @ 21:36 -0600

*Status:* COMPLETE

*What We Completed:*
- Synced templates from claude-templates (protocols.org, workflows, scripts updated)
- Executed announcements:
  1. Deleted old renamed workflows (session-wrap-up.org, retrospective-workflow.org, session-start.org) - replaced by wrap-it-up.org, retrospective.org, startup.org
  2. Confirmed no docs/templates/ cache to remove
  3. Renamed NOTES.org to notes.org, updated all internal references
- Updated workflow catalog in notes.org to reflect renamed workflows

*Files Modified:*
- [[file:notes.org][docs/notes.org]] - Renamed from NOTES.org, updated references and workflow catalog
- [[file:protocols.org][docs/protocols.org]] - Synced from template
- [[file:previous-session-history.org][docs/previous-session-history.org]] - Updated NOTES.org references
- [[file:PLAN-archangel-btrfs.org][docs/PLAN-archangel-btrfs.org]] - Updated NOTES.org reference

*Files Deleted:*
- docs/workflows/session-wrap-up.org (replaced by wrap-it-up.org)
- docs/workflows/retrospective-workflow.org (replaced by retrospective.org)
- docs/workflows/session-start.org (replaced by startup.org)

*Files Added (from template):*
- New workflows: wrap-it-up.org, retrospective.org, startup.org, set-alarm.org, status-check.org, send-email.org, find-email.org, extract-email.org, sync-email.org, email-assembly.org
- docs/scripts/ updates

*** 2026-01-25 Sun @ 00:15-08:34 -0600

*Status:* COMPLETE

*What We Completed:*
- Phase 4.3 validation testing for btrfs installations
- Non-LUKS btrfs tests all PASS: btrfs-single, btrfs-mirror, btrfs-stripe
- Attempted LUKS automated reboot testing (sendkey + keyfile hybrid approach)
- GRUB passphrase via sendkey WORKS (Slot 0 opened)
- Initramfs encrypt hook does NOT receive sendkey - accepted as limitation
- Fixed configure_btrfs_initramfs() - was overwriting HOOKS and removing encrypt hook
- Added setup_luks_testing_keyfile() function for keyfile-based testing
- Created TESTING-STRATEGY.org documenting the LUKS automation limitation
- Copied ISO to Ventoy flash drive
- Added manual LUKS verification task to todo.org (priority A)

*Key Decisions:*
- LUKS reboot automation is a known limitation - installation tests pass, reboot verification requires manual testing
- Hybrid approach (sendkey for GRUB, keyfile for initramfs) was correct direction but initramfs encrypt hook reads input differently than GRUB
- Documented in TESTING-STRATEGY.org for future reference

*Files Modified:*
- [[file:../custom/lib/btrfs.sh][custom/lib/btrfs.sh]] - LUKS keyfile support, encrypt hook fix
- [[file:../custom/archangel][custom/archangel]] - Integrated keyfile setup call
- [[file:../scripts/test-configs/btrfs-luks.conf][scripts/test-configs/btrfs-luks.conf]] - Added TESTING=yes
- [[file:../scripts/test-configs/btrfs-mirror-luks.conf][scripts/test-configs/btrfs-mirror-luks.conf]] - Added TESTING=yes
- [[file:TESTING-STRATEGY.org][docs/TESTING-STRATEGY.org]] - New file documenting approach
- [[file:../todo.org][todo.org]] - Added manual LUKS verification task

*Test Results:*
| Config | Installation | Reboot |
|--------+--------------+--------|
| btrfs-single | PASS | PASS |
| btrfs-mirror | PASS | PASS |
| btrfs-stripe | PASS | PASS |
| btrfs-luks | PASS | MANUAL |
| btrfs-mirror-luks | PASS | MANUAL |

*Next Steps:*
- Review Phase 5 (CLI tools: archangel-snapshot, archangel-rollback, archangel-list)
- Manual LUKS reboot verification when hardware available

*** 2026-01-23 Fri @ 02:12 -0600

*Status:* COMPLETE

*What We Completed:*
- Validated ZFSBootMenu installation on bare metal (ratio - Framework Desktop)
- Monitored full install-archzfs run via SSH to archzfs.local
- Installation completed successfully on 2-disk NVMe mirror (2x 7.3TB)
- Verified ZFSBootMenu installed on both EFI partitions (redundancy working)
- Confirmed both EFI boot entries created (disk1 and disk2)
- Genesis snapshot created, system ready to reboot

*Validation Results:*
| Component | Status |
|-----------+--------|
| ZFS pool (mirror) | Exported correctly |
| EFI partition nvme0n1p1 | ZFSBootMenu installed |
| EFI partition nvme1n1p1 | ZFSBootMenu installed |
| Boot0001 ZFSBootMenu-disk1 | Created, first priority |
| Boot0006 ZFSBootMenu-disk2 | Created, fallback |
| Initramfs with ZFS hooks | Built successfully |
| Genesis snapshot | Created |

*System Configuration:*
- Hostname: ratio
- Pool: zroot (2-disk mirror, unencrypted)
- Kernel: linux-lts 6.12.66-1
- ZFSBootMenu timeout: 3 seconds

*Next Steps:*
- Reboot ratio into ZFSBootMenu
- Run archsetup for full workstation configuration
- Merge zfsbootmenu branch to main after successful boot

*** 2026-01-23 Fri @ 01:21 -0600

*Status:* COMPLETE

*What We Completed:*
- Replaced GRUB with ZFSBootMenu bootloader in install-archzfs
- Researched 5 open-source ZFS installation projects for best practices
- Created implementation plan (PLAN-zfsbootmenu-implementation.org)
- Deleted GRUB snapshot tooling: grub-zfs-snap, 40_zfs_snapshots, zz-grub-zfs-snap.hook, zfs-snap-prune
- Updated build.sh to remove GRUB file copying
- Updated zfssnapshot and zfsrollback to remove grub-zfs-snap calls
- Built and tested ISO successfully:
  - Single disk installation: PASSED
  - 2-disk mirror (mirror-0): PASSED
  - 3-disk RAIDZ1 (raidz1-0): PASSED
- Committed to zfsbootmenu branch, pushed to origin
- Copied ISO to Ventoy flash drive (/dev/sda1)

*Key Technical Changes:*
- EFI partition reduced from 1GB to 512MB (only holds ZFSBootMenu)
- EFI mounts at /efi instead of /boot
- Kernel/initramfs live on ZFS root (enables snapshot boot with matching kernel)
- Downloads pre-built ZFSBootMenu EFI from get.zfsbootmenu.org
- Creates boot entries for all disks in multi-disk configs
- Syncs ZFSBootMenu to all EFI partitions for redundancy
- Sets org.zfsbootmenu:commandline on zroot/ROOT for kernel cmdline inheritance
- AMD GPU workarounds (pg_mask, cwsr_enable) auto-added when AMD detected

*Key Decisions:*
- /boot must NOT be a separate ZFS dataset (per research from all 5 projects)
- Download pre-built ZFSBootMenu rather than building from AUR
- Use --unicode parameter for ZFSBootMenu cmdline in efibootmgr

*Files Modified:*
- [[file:../build.sh][build.sh]] - Removed grub-zfs-snap copying
- [[file:../custom/archangel][custom/install-archzfs]] - Major ZFSBootMenu rewrite
- [[file:../custom/zfssnapshot][custom/zfssnapshot]] - Removed grub-zfs-snap call
- [[file:../custom/zfsrollback][custom/zfsrollback]] - Removed grub-zfs-snap call

*Files Deleted:*
- custom/grub-zfs-snap
- custom/40_zfs_snapshots
- custom/zz-grub-zfs-snap.hook
- custom/zfs-snap-prune

*Files Created:*
- [[file:../PLAN-zfsbootmenu-implementation.org][PLAN-zfsbootmenu-implementation.org]] - Implementation plan
- [[file:2026-01-22-ratio-amd-gpu-freeze-fix-instructions.org][docs/2026-01-22-ratio-amd-gpu-freeze-fix-instructions.org]] - Filed from inbox
- [[file:research-sandreas-zarch.org][docs/research-sandreas-zarch.org]] - Research notes
- [[file:session-context.org][docs/session-context.org]] - Session tracking

*Commits Made:*
- 2ad560b: Replace GRUB with ZFSBootMenu bootloader

*Next Steps:*
- Test ZFSBootMenu on physical hardware (ratio or new install)
- Merge zfsbootmenu branch to main after hardware validation
- Consider encrypted pool test (requires interactive passphrase)

*** 2026-01-22 Thu @ 15:44 -0600

*Status:* COMPLETE

*What We Completed:*
- Diagnosed system freeze on ratio - same VPE power gating bug from earlier
- Researched VPE_IDLE_TIMEOUT patch status - NOT merged, AMD maintainer skeptical
- Discovered kernel 6.18.x has critical CWSR bugs for Strix Halo - do NOT upgrade
- Identified two workarounds: amdgpu.pg_mask=0 and amdgpu.cwsr_enable=0
- Verified neither workaround is currently applied on ratio
- Created detailed fix instructions in inbox/instructions.txt for live ISO application

*Key Decisions:*
- Stay on kernel 6.15.2 (pinned) - 6.18.x has worse bugs for Strix Halo
- Apply both pg_mask=0 and cwsr_enable=0 workarounds
- Will apply fixes from velox via live ISO (mkinitcpio triggers freeze on live system)

*Key Technical Findings:*
- VPE_IDLE_TIMEOUT patch (1s→2s) submitted Aug 2025 but not merged
- Framework Community recommends 6.15.x-6.17.x for Strix Halo, avoid 6.18+
- Current ratio state: pg_mask=4294967295 (bad), cwsr_enable=1 (bad)

*Files Created:*
- [[file:../inbox/instructions.txt][inbox/instructions.txt]] - Detailed fix instructions for live ISO

*Next Steps:*
- Boot ratio from archzfs live ISO (from velox)
- Apply both workarounds per inbox/instructions.txt
- Rebuild initramfs from live ISO
- Verify fixes active after reboot