chore(todo): velox reinstall runbook and the tasks it surfaced

The runbook drove today's rebuild: preflight, a required salvage pass, the install, and post-install verification. It stays as the working record until velox is settled.

Filed from the rebuild: reseating the input-cover ribbon, which is [#A] for tomorrow because phantom power-button presses have killed the machine four times and the last one cut power below systemd, where no setting can catch it. The rest are hibernate in the settings dial, a truenas VM for long-running sessions, the keyboard-backlight binding, and the screen-lock suite that fails 21 of 23 on ratio.

1 files changed, 154 insertions, 0 deletions

diff --git a/working/velox-reinstall/velox-reinstall-runbook.org b/working/velox-reinstall/velox-reinstall-runbook.org
new file mode 100644
index 0000000..02671e2
--- /dev/null
+++ b/working/velox-reinstall/velox-reinstall-runbook.org
@@ -0,0 +1,154 @@
+#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-08-13
+
+Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon
+890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry,
+and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the
+pool. Decision: full reinstall via archangel + archsetup, run deliberately as
+a disaster-recovery test of the ISO and scripts before the Sunday flight.
+Recent backup in hand; ratio available as the working machine.
+
+Fallback ordering if the test finds a real gap:
+- Before partitioning starts: the old system is intact — the ZBM repair
+  route (efibootmgr entry pointing at the ZBM loader on the ESP, then
+  amd-ucode swap in a chroot) is still available.
+- After partitioning: the floor is a manual Arch install; the backup makes
+  that survivable.
+
+* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk)
+
+- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates
+  the microcode vendor-detection fix (archsetup, 2026-08-08) and was built
+  without ARCHSETUP_DIR at all.
+  #+begin_src sh
+  cd ~/code/archangel && sudo ARCHSETUP_DIR=~/code/archsetup ./build.sh
+  #+end_src
+- [ ] build.sh fixes before the final rebuild (archangel repo):
+  - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the
+    credential audit — off the portable USB stick).
+  - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile
+    is on the ISO at =/root/velox-zfs.conf=.
+- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=),
+  =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs
+  spot-check.
+- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy
+  drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16
+  archangel ISOs removed. Boot the stick and pick
+  =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu.
+
+* Phase 1 — UEFI setup on velox (BIOS screen, before any boot)
+
+- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned;
+  the new board ships with it enforced by factory default.
+- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01;
+  a wrong clock breaks TLS and pacman signature checks in the live env.
+  Rough accuracy is fine — NTP tightens it once networked.
+- [ ] While you're in setup: check boot-order UI shows the USB.
+
+* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED
+
+NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is
+DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke
+around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio
+and mybitch backed up today). Everything since Jul 6 exists only on the old
+SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO
+backup at all. This pass also keeps the repair fallback alive until
+partitioning starts.
+
+- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=.
+- [ ] Import the old pool read-only and unlock:
+  #+begin_src sh
+  zpool import -N -o readonly=on -R /mnt zroot
+  zfs load-key zroot          # passphrase prompt
+  zfs mount zroot/ROOT/default
+  zfs mount -a 2>/dev/null    # home datasets etc.; ignore failures
+  #+end_src
+- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout
+  the backup job uses (etc + home), into a clearly-named one-off dir:
+  #+begin_src sh
+  rsync -aHAX --info=progress2 /mnt/etc /mnt/home \
+      truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/
+  #+end_src
+  (=/usr= is in the regular backups but is all reinstallable — skip unless
+  paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.)
+- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest
+  files in =~/documents= and =~/downloads=.
+- [ ] Check for uncommitted repo work and either push or note it:
+  =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge),
+  =~/.dotfiles=, anything under =~/code=.
+- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=.
+
+* Phase 3 — Install (the actual DR test)
+
+- [ ] Review the profile, then run the installer:
+  #+begin_src sh
+  less /root/velox-zfs.conf     # FILESYSTEM=zfs, HOSTNAME=velox, single nvme
+  archangel --config-file /root/velox-zfs.conf
+  #+end_src
+  Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome=
+  placeholders — fine for install; both change post-install (=zfs change-key
+  zroot= for the pool, =passwd= for root).
+- [ ] Record every rough edge as a DR-test finding — that's the point of
+  running it this way. Anything that needs a manual nudge gets a todo entry
+  in archangel or archsetup afterward.
+- [ ] Reboot into ZBM → boot the new environment.
+
+* Phase 4 — archsetup (first boot of the installed system)
+
+- [ ] Log in as root, network up, then verify the clock synced.
+- [ ] Get archsetup — two paths, test the offline one since this is a DR
+  drill (the online curl path is the everyday alternative):
+  #+begin_src sh
+  # offline: mount the install USB and copy the baked tree
+  mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt
+  cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup
+  ./archsetup
+  #+end_src
+- [ ] Expected on the new board: =install_cpu_microcode= detects
+  AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests).
+  Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all
+  in the installer now — none need manual application afterward.
+- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel
+  assumptions (swept 2026-08-13); maint's capability probe runtime-detects
+  amd-pstate.
+
+* Phase 5 — Post-install restore + verification
+
+- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The
+  secrets-bundle-in-ISO design is not built yet — manual restore is the
+  known gap, not a test failure.
+- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=;
+  re-import the NM profile (autoconnect off, as before).
+- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=.
+- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known
+  idle instability (Panel Self Refresh hangs/reboots the machine; hit during
+  the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the
+  installed system's kernel command line — velox boots via ZBM, so set it on
+  the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default=
+  (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a
+  kernel that fixes PSR on Strix Point — track via the Framework issue
+  tracker (SoftwareFirmwareIssueTracker #110).
+- [ ] New-hardware spot-checks:
+  - =journalctl -k | grep -i microcode= — amd-ucode applied.
+  - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect
+    amd-pstate(-epp).
+  - wifi + bluetooth up (new board radios), touchpad behavior, camera.
+  - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV.
+- [ ] Fresh clones automatically carry the post-purge rewritten git history —
+  closes the clone-reconcile rider from 2026-08-11 without action.
+- [ ] Fix and verify the backup timer on the fresh install — it was silently
+  broken since ~Jul 6. After the first manual run succeeds, confirm a new
+  DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it
+  broke (timer unit dead? mount failure? credential?) if the old journal
+  survives in the salvage copy.
+- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate),
+  both daily drivers AMD. Fix the stale =intel_pstate= comment in
+  =airplane-mode= line 6 while at it (cosmetic).
+- [ ] File every DR-test finding in the owning project's todo.
+
+* Timing
+
+Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or
+Friday, leaving Saturday as pure buffer. If the install stalls past Friday
+evening, cut losses to the manual-install floor.