1 files changed, 61 insertions, 0 deletions
diff --git a/docs/design/2026-07-15-velox-boot-failure-handoff.org b/docs/design/2026-07-15-velox-boot-failure-handoff.org
new file mode 100644
index 0000000..5ec996f
--- /dev/null
+++ b/docs/design/2026-07-15-velox-boot-failure-handoff.org
@@ -0,0 +1,61 @@
+#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-07-15
+
+* Why this is coming to archsetup
+
+Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below.
+
+A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox.
+
+* Timeline
+
+- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50).
+- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times).
+- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel.
+- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session.
+
+* Facts established so far (from the ZBM recovery shell)
+
+- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem.
+- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective.
+- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be.
+- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone.
+
+* Working hypothesis
+
+A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent.
+
+* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell)
+
+1. Read pacman's log (on the zroot/var dataset):
+   #+begin_src sh
+   mkdir -p /mnt/var
+   mount -t zfs -o zfsutil zroot/var /mnt/var
+   tail -60 /mnt/var/log/pacman.log
+   #+end_src
+   Expect the failed/interrupted kernel transaction near the end; note its timestamp.
+2. List recovery candidates:
+   #+begin_src sh
+   zfs list -t snapshot zroot/ROOT/default | tail -20
+   #+end_src
+   Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox.
+
+* Recovery plan (agreed with Craig, pending the log read)
+
+1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction.
+2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot.
+3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched).
+4. zpool export zroot, reboot — ZBM should now see the kernel.
+5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown.
+
+* Retrospective candidates for archsetup
+
+- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion.
+- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically.
+- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above.
+- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors.
+
+* Related loose end already in your inbox
+
+A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again.