#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached
#+AUTHOR: Craig Jennings
#+DATE: 2026-07-15

* Why this is coming to archsetup

Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below.

A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox.

* Timeline

- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50).
- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times).
- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel.
- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session.

* Facts established so far (from the ZBM recovery shell)

- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem.
- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective.
- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be.
- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone.

* Working hypothesis

A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent.

* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell)

1. Read pacman's log (on the zroot/var dataset):
   #+begin_src sh
   mkdir -p /mnt/var
   mount -t zfs -o zfsutil zroot/var /mnt/var
   tail -60 /mnt/var/log/pacman.log
   #+end_src
   Expect the failed/interrupted kernel transaction near the end; note its timestamp.
2. List recovery candidates:
   #+begin_src sh
   zfs list -t snapshot zroot/ROOT/default | tail -20
   #+end_src
   Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox.

* Recovery plan (agreed with Craig, pending the log read)

1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction.
2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot.
3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched).
4. zpool export zroot, reboot — ZBM should now see the kernel.
5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown.

* Retrospective candidates for archsetup

- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion.
- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically.
- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above.
- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors.

* Related loose end already in your inbox

A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again.