diff options
| -rw-r--r-- | docs/workflows/system-health-check.org | 35 | ||||
| -rw-r--r-- | working/velox-reinstall/velox-uefi-boot-entry-reference.org | 33 |
2 files changed, 68 insertions, 0 deletions
diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org index b4f34a5..777e724 100644 --- a/docs/workflows/system-health-check.org +++ b/docs/workflows/system-health-check.org @@ -235,6 +235,16 @@ Updates are separate from issue investigation. After all issues are addressed (o 5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at [[file:strix-soak-watch.org][docs/workflows/strix-soak-watch.org]] before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired. 6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=). On maint hosts (ratio, velox) plain =topgrade= resolves to the dotfiles PATH wrapper, which stamps the console's topgrade-freshness metric on success — no extra step. If the run happened outside the wrapper somehow, =maint stamp topgrade= records it by hand. 7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot +8. On a ZFS-root host, after a kernel bump confirm the new initramfs carries the zfs module before rebooting: =sudo lsinitcpio /boot/initramfs-linux-lts.img | grep -c 'zfs.ko'=. The =sudo= is load-bearing: the images are 0600, so an unprivileged =lsinitcpio= exits 1 with "Unable to read file" on stderr and nothing on stdout, and once piped into =grep -c= that empty stdout reads as a count of 0 and looks exactly like a missing module (velox, 2026-09-12). + +*** Before a firmware (BIOS) update + +Firmware stays a manual step (=topgrade.toml= keeps =[firmware] upgrade = false=); =fwupdmgr update= stages it and the next reboot flashes it. Before staging, capture the two things a bad reboot will make you guess at: + +1. =sudo efibootmgr -v= — every boot entry with its loader path and command line, pasted into the session's context file. +2. Secure Boot state — =bootctl status 2>/dev/null | grep -i 'secure boot'=. + +After the flash, if the machine reports no bootable device, check Secure Boot *first*. The Framework 04.02 update on velox re-enabled it, which rejects the unsigned ZFSBootMenu loader and reads as "Default Boot Device Missing" rather than a security violation; the boot entries were untouched (see the Known Issues Log, 2026-09-12). Only when Secure Boot is off and =efibootmgr -v= from a stick shows the entry gone does the boot-entry recreate apply (velox: =working/velox-reinstall/velox-uefi-boot-entry-reference.org= in archsetup). *** Two-Stage Reboot Pattern (MANDATORY if Phase 3 installed kernel / iproute2 / systemd / NetworkManager) @@ -1036,3 +1046,28 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi - Functional status at 2026-06-13 check: no coredumps yet that day; Telegram scans still worked from cached chat state. Treat as an app/server-container crash, not a machine-health fault. - Classification: KNOWN — annotate future =telega-server= coredumps on ratio as =KNOWN — dockerized telega-server musl SIGSEGV= if the signature matches =tdat_plist_value= / unexpected plist value or otherwise stays inside the Telega container. Escalate only if crashes become continuous, break Telegram workflows, or appear after moving off the Docker musl build. - Deferred remediation options, in order of least disruption: update the Emacs =telega= package, rebuild/pull a newer =telega-server= image, pin a known-good pre-2026-06 image digest, build =telega-server= natively, or report upstream with =coredumpctl= and log evidence. + +** 2026-09-12: velox — Framework BIOS update re-enabled Secure Boot (reads as "no bootable device") +:host: velox +- Symptom: after fwupd staged system firmware 0.0.3.5 → 0.0.4.2 and the reboot flashed it, the BIOS reported "Default Boot Device Missing / no bootable drive". Indistinguishable from the NVRAM wipe that forced the 2026-08-13 reinstall. +- Actual cause: the update set Enforce Secure Boot = Enabled. The unsigned ZFSBootMenu loader was rejected and the firmware reported it as a missing device, not a security violation. Boot0001 ZFSBootMenu survived intact with its command line; no =efibootmgr= was needed. +- The tell: booting the Ventoy stick showed shim's MOK management screen. +- Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. Boots straight into ZBM. Firmware confirmed at 04.02, pools healthy. +- Prevention: the pre-firmware-update checklist in Phase 3 (record =efibootmgr -v= and the Secure Boot state before staging). Check Secure Boot before assuming NVRAM loss. +- Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT= absent), so a real NVRAM wipe would still need the stick. Filed in archangel, which owns the ZBM install, as [#C] "Installed systems have no removable-media boot fallback on the ESP" (2026-09-12); it ships with the next ISO rebuild after it lands. + +** 2026-09-12: any — fwupdmgr activates passim, a public LAN listener +:host: any +- Symptom: running =fwupdmgr= (refresh, update) D-Bus-activates =passim.service=, fwupd's LAN metadata-sharing daemon, which listens on =0.0.0.0:27500= and trips the maint listeners check to crit. +- The unit is static (no =[Install]= section), so =systemctl disable= is a no-op and it comes back on the next fwupdmgr run. Masking is what holds: =systemctl mask passim.service=. Ratio has been masked since 2026-07-21; velox was only stopped and disabled on 2026-09-12 and still needs the mask. The installer masks it as part of installing fwupd. =P2pPolicy=nothing= under =[fwupd]= in =/etc/fwupd/fwupd.conf= also works, but that file is pacman-owned and invites pacnew churn, so the mask is the form in use. +- Classification: KNOWN — a passim listener means a machine that predates the mask or lost it; mask it, don't allowlist it. + +** 2026-09-12: velox — topgrade containers step fails on locally built images +:host: velox (ratio has the same shape with its own local images) +- Symptom: topgrade exits 1 after a clean package run because the containers step tries to =docker pull= images that were built locally (=cj/telega-server=, =telega-server-glycin=) and gets "pull access denied". The wrapper then never writes the topgrade-freshness stamp; =maint stamp topgrade= by hand after confirming the package steps succeeded. +- Classification: KNOWN — the step cannot succeed while local-only images exist. The fix (disable the containers step, or list the images under =ignored_containers=) is tracked on the topgrade guarded-upgrade task in archsetup's todo. + +** 2026-09-12: velox — mkinitcpio "Possibly missing firmware" for xhci_pci_renesas and qat_6xxx +:host: velox +- Stock Arch mkinitcpio noise on a kernel rebuild: =xhci_pci_renesas= wants the Renesas USB controller blob (AUR =upd72020x-fw=) and =qat_6xxx= is Intel QuickAssist firmware. Neither is hardware this machine has. +- Classification: KNOWN — harmless; annotate and move on unless the named hardware appears. diff --git a/working/velox-reinstall/velox-uefi-boot-entry-reference.org b/working/velox-reinstall/velox-uefi-boot-entry-reference.org index 8d677da..53e6511 100644 --- a/working/velox-reinstall/velox-uefi-boot-entry-reference.org +++ b/working/velox-reinstall/velox-uefi-boot-entry-reference.org @@ -8,6 +8,39 @@ this kind of empty NVRAM, which is what forced the reinstall — so a cleared boot entry is the specific failure worth being able to undo in one command rather than reconstruct. +* Before recreating anything: check Secure Boot first + +The 04.02 update landed on 2026-09-12 (staged via fwupd, flashed on the next +reboot). It did NOT clear NVRAM: Boot0001 survived with its command line +intact. What it did was re-enable Secure Boot (Enforce Secure Boot = +Enabled), so the unsigned ZBM loader was rejected and the Framework BIOS +reported it as "Default Boot Device Missing / no bootable drive" rather than +a security violation. The symptom is indistinguishable from the NVRAM wipe +this document was written for. + +The tell: booting the Ventoy stick shows shim's MOK management screen. + +Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. The +machine then boots straight into ZBM. No efibootmgr needed. + +So when velox says no bootable device after a firmware update, check Secure +Boot before touching the boot entries. Only if Secure Boot is already off +and =efibootmgr -v= (from the stick) shows Boot0001 gone does the recreate +below apply. + +Before any future firmware update, record both =efibootmgr -v= and the +Secure Boot state so the post-reboot diagnosis is a comparison, not a guess. +The pre-firmware-update checklist in +[[file:../../docs/workflows/system-health-check.org][docs/workflows/system-health-check.org]] +(Phase 3) carries the steps. + +Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT= +does not exist), so a real NVRAM wipe would still need the stick. Copying +=zfsbootmenu.efi= to =/efi/EFI/BOOT/BOOTX64.EFI= would let it boot unaided. +That belongs to archangel's ZBM install, where it is filed as [#C] +"Installed systems have no removable-media boot fallback on the ESP" +(2026-09-12) and ships with the next ISO rebuild after it lands. + * State at capture - BIOS: 03.05 (2025-10-30) |