aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-09-13 07:22:10 -0500
committerCraig Jennings <c@cjennings.net>2026-09-13 07:22:10 -0500
commitc9f9a9881548114e49752d5bc4feba4a83ed4d84 (patch)
tree7c00b7fc248ed99627e8cdaab0c67daec1f1be37 /docs
parentdc00a625906eafd52bc791fe3ba117225e7851da (diff)
downloadarchsetup-c9f9a9881548114e49752d5bc4feba4a83ed4d84.tar.gz
archsetup-c9f9a9881548114e49752d5bc4feba4a83ed4d84.zip
chore: file the legacy inbox references and two unfiled captures
The inbox held 64 processed-but-kept handoffs from July and August. The inbox discipline no longer allows that state: a file is a task, reference content in a tracked home, or gone. I read each one. Fifty-six were acknowledgments, FYIs, lint pipeline files or requests whose work had already landed. Those are deleted. Eight carried content worth keeping. The 2026-07-15 velox boot-failure diagnosis moves to docs/design, linked from the retrospective task. The three 2026-08-14 reinstall-gap reports move into the velox reinstall working dir, linked from the post-rebuild task. The two Maeda applets and their notes join the clock display references. Two captures had never landed anywhere: the waybar module separator I captured on 2026-07-20 ([#B]) and the window-configuration research idea from 2026-07-24 ([#C]). Both are filed. The velox reinstall task records the two live steps (wsdd off, passim masked) applied over ssh once velox came back, and the health-check log now says velox is masked.
Diffstat (limited to 'docs')
-rw-r--r--docs/design/2026-07-15-velox-boot-failure-handoff.org61
-rw-r--r--docs/workflows/system-health-check.org2
2 files changed, 62 insertions, 1 deletions
diff --git a/docs/design/2026-07-15-velox-boot-failure-handoff.org b/docs/design/2026-07-15-velox-boot-failure-handoff.org
new file mode 100644
index 0000000..5ec996f
--- /dev/null
+++ b/docs/design/2026-07-15-velox-boot-failure-handoff.org
@@ -0,0 +1,61 @@
+#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-07-15
+
+* Why this is coming to archsetup
+
+Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below.
+
+A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox.
+
+* Timeline
+
+- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50).
+- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times).
+- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel.
+- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session.
+
+* Facts established so far (from the ZBM recovery shell)
+
+- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem.
+- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective.
+- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be.
+- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone.
+
+* Working hypothesis
+
+A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent.
+
+* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell)
+
+1. Read pacman's log (on the zroot/var dataset):
+ #+begin_src sh
+ mkdir -p /mnt/var
+ mount -t zfs -o zfsutil zroot/var /mnt/var
+ tail -60 /mnt/var/log/pacman.log
+ #+end_src
+ Expect the failed/interrupted kernel transaction near the end; note its timestamp.
+2. List recovery candidates:
+ #+begin_src sh
+ zfs list -t snapshot zroot/ROOT/default | tail -20
+ #+end_src
+ Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox.
+
+* Recovery plan (agreed with Craig, pending the log read)
+
+1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction.
+2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot.
+3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched).
+4. zpool export zroot, reboot — ZBM should now see the kernel.
+5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown.
+
+* Retrospective candidates for archsetup
+
+- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion.
+- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically.
+- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above.
+- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors.
+
+* Related loose end already in your inbox
+
+A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again.
diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org
index 777e724..37c1932 100644
--- a/docs/workflows/system-health-check.org
+++ b/docs/workflows/system-health-check.org
@@ -1059,7 +1059,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi
** 2026-09-12: any — fwupdmgr activates passim, a public LAN listener
:host: any
- Symptom: running =fwupdmgr= (refresh, update) D-Bus-activates =passim.service=, fwupd's LAN metadata-sharing daemon, which listens on =0.0.0.0:27500= and trips the maint listeners check to crit.
-- The unit is static (no =[Install]= section), so =systemctl disable= is a no-op and it comes back on the next fwupdmgr run. Masking is what holds: =systemctl mask passim.service=. Ratio has been masked since 2026-07-21; velox was only stopped and disabled on 2026-09-12 and still needs the mask. The installer masks it as part of installing fwupd. =P2pPolicy=nothing= under =[fwupd]= in =/etc/fwupd/fwupd.conf= also works, but that file is pacman-owned and invites pacnew churn, so the mask is the form in use.
+- The unit is static (no =[Install]= section), so =systemctl disable= is a no-op and it comes back on the next fwupdmgr run. Masking is what holds: =systemctl mask passim.service=. Ratio has been masked since 2026-07-21; velox was stopped and disabled on 2026-09-12 (the disable being the no-op) and got the mask on 2026-09-13. The installer masks it as part of installing fwupd. =P2pPolicy=nothing= under =[fwupd]= in =/etc/fwupd/fwupd.conf= also works, but that file is pacman-owned and invites pacnew churn, so the mask is the form in use.
- Classification: KNOWN — a passim listener means a machine that predates the mask or lost it; mask it, don't allowlist it.
** 2026-09-12: velox — topgrade containers step fails on locally built images