1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached
#+AUTHOR: Craig Jennings
#+DATE: 2026-07-15
* Why this is coming to archsetup
Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below.
A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox.
* Timeline
- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50).
- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times).
- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel.
- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session.
* Facts established so far (from the ZBM recovery shell)
- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem.
- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective.
- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be.
- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone.
* Working hypothesis
A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent.
* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell)
1. Read pacman's log (on the zroot/var dataset):
#+begin_src sh
mkdir -p /mnt/var
mount -t zfs -o zfsutil zroot/var /mnt/var
tail -60 /mnt/var/log/pacman.log
#+end_src
Expect the failed/interrupted kernel transaction near the end; note its timestamp.
2. List recovery candidates:
#+begin_src sh
zfs list -t snapshot zroot/ROOT/default | tail -20
#+end_src
Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox.
* Recovery plan (agreed with Craig, pending the log read)
1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction.
2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot.
3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched).
4. zpool export zroot, reboot — ZBM should now see the kernel.
5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown.
* Retrospective candidates for archsetup
- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion.
- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically.
- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above.
- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors.
* Related loose end already in your inbox
A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again.
|
