aboutsummaryrefslogtreecommitdiff
path: root/docs/workflows/system-health-check.org
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-07-09 12:58:56 -0500
committerCraig Jennings <c@cjennings.net>2026-07-09 12:58:56 -0500
commit76de64ac803c336252ac225f1f493906a7eacd07 (patch)
treef5de1a8590e9cb344187fc419825f13fe0bb22dd /docs/workflows/system-health-check.org
parent4563af6702bcc44defbca22baeb87ab8261572c6 (diff)
downloadarchsetup-76de64ac803c336252ac225f1f493906a7eacd07.tar.gz
archsetup-76de64ac803c336252ac225f1f493906a7eacd07.zip
docs: move the host inventories into the org-roam knowledge base
The four host capability inventories, the TrueNAS hardware specs, and the ratio USB/xHCI record now live as roam nodes under ~/org/roam/hardware/, linked from the Homelab Hardware Inventory index. The copies here are gone. Carried rather than pointed at. A pointer keeps one canonical copy, but a third project that discovers a durable hardware fact would then have to write into this repo to record it, which the cross-project rules forbid — so the fact would land in an inbox instead of on the device's page. Moving keeps one canonical copy without closing the write path. system-health-check.org resolves a host's inventory by its #+HOSTNAME: keyword under ${ROAM_DIR:-$HOME/org/roam}/hardware/, never by filename: a node's timestamp prefix does not survive a rename. A host with no roam clone takes the same NO INVENTORY FILE path it always did. strix-soak-watch.org lands here from home, beside the workflow whose Phase 3 invokes it. It is a workflow, not an inventory page.
Diffstat (limited to 'docs/workflows/system-health-check.org')
-rw-r--r--docs/workflows/system-health-check.org38
1 files changed, 21 insertions, 17 deletions
diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org
index b66a186..98fa17f 100644
--- a/docs/workflows/system-health-check.org
+++ b/docs/workflows/system-health-check.org
@@ -8,7 +8,7 @@ This workflow performs a comprehensive diagnostic scan of whatever host it runs
Run this when Craig asks "how is <host> doing?", "check my system health", or as a periodic maintenance check.
-Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=.
+Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live in the org-roam knowledge base at =~/org/roam/hardware/= (moved there 2026-07-09).
Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3.
@@ -23,20 +23,24 @@ Read the TOML at the start of Phase 1 and grade against its values. Where a chec
* Hosts & Capabilities
-The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift.
+The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory nodes in the org-roam knowledge base declare the expected capabilities in a property drawer and act as a cross-check for drift.
-Host inventory files (match by =#+HOSTNAME:= keyword inside each):
+The inventories live at =~/org/roam/hardware/= (2026-07-09), alongside the rest of the hardware inventory. They moved out of this repo so any project's session can read a host's capabilities — and write back what a diagnosis discovers — without reaching across a project boundary. The index node "Homelab Hardware Inventory" links them all.
-| Host | Inventory file | Notes |
-|---------+-------------------------------------------+----------------------------------|
-| ratio | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]] | Arch, Btrfs RAID1, pacman |
-|---------+-------------------------------------------+----------------------------------|
-| velox | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]] | Arch, ZFS (re-installed 2026-04) |
-|---------+-------------------------------------------+----------------------------------|
-| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt |
-|---------+-------------------------------------------+----------------------------------|
-| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM |
-|---------+-------------------------------------------+----------------------------------|
+Host inventory nodes (match by =#+HOSTNAME:= keyword inside each):
+
+| Host | Inventory node | Notes |
+|---------+-----------------------------------------------+----------------------------------|
+| ratio | =hardware/…-ratio_desktop_workstation.org= | Arch, Btrfs RAID1, pacman |
+|---------+-----------------------------------------------+----------------------------------|
+| velox | =hardware/…-velox_laptop.org= | Arch, ZFS (re-installed 2026-04) |
+|---------+-----------------------------------------------+----------------------------------|
+| mybitch | =hardware/…-mybitch_christines_laptop.org= | Mint, ext4, apt |
+|---------+-----------------------------------------------+----------------------------------|
+| truenas | =hardware/…-truenas_nas_storage_server.org= | TrueNAS SCALE, ZFS, no PM |
+|---------+-----------------------------------------------+----------------------------------|
+
+Never hardcode a node's filename — its timestamp prefix is not stable across a rename. Resolve by the =#+HOSTNAME:= keyword, as Phase 0 does.
Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift.
@@ -108,10 +112,10 @@ Present the probe result as a one-line capability summary at the top of the repo
Locate the inventory file for this host:
#+begin_src bash
-inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1)
+inv=$(grep -l "^#+HOSTNAME: $host\b" "${ROAM_DIR:-$HOME/org/roam}"/hardware/*.org 2>/dev/null | head -1)
#+end_src
-If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check.
+If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. A host without the roam clone (mybitch, truenas) takes that path, exactly as it did when the inventories lived in this repo.
If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe.
@@ -228,7 +232,7 @@ Updates are separate from issue investigation. After all issues are addressed (o
2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware)
3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices
4. Check =/var/log/pacman.log= for recent failed transactions
-5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
+5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at [[file:strix-soak-watch.org][docs/workflows/strix-soak-watch.org]] before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=). On maint hosts (ratio, velox) plain =topgrade= resolves to the dotfiles PATH wrapper, which stamps the console's topgrade-freshness metric on success — no extra step. If the run happened outside the wrapper somehow, =maint stamp topgrade= records it by hand.
7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot
@@ -996,7 +1000,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi
- Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986.
- Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot.
- Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean.
-- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.)
+- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in the =mybitch - Christine's Laptop= roam node → Operational Changes Log.)
- *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=.
- *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug).
- Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111.