aboutsummaryrefslogtreecommitdiff
path: root/docs/workflows
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-07-09 12:58:56 -0500
committerCraig Jennings <c@cjennings.net>2026-07-09 12:58:56 -0500
commit76de64ac803c336252ac225f1f493906a7eacd07 (patch)
treef5de1a8590e9cb344187fc419825f13fe0bb22dd /docs/workflows
parent4563af6702bcc44defbca22baeb87ab8261572c6 (diff)
downloadarchsetup-76de64ac803c336252ac225f1f493906a7eacd07.tar.gz
archsetup-76de64ac803c336252ac225f1f493906a7eacd07.zip
docs: move the host inventories into the org-roam knowledge base
The four host capability inventories, the TrueNAS hardware specs, and the ratio USB/xHCI record now live as roam nodes under ~/org/roam/hardware/, linked from the Homelab Hardware Inventory index. The copies here are gone. Carried rather than pointed at. A pointer keeps one canonical copy, but a third project that discovers a durable hardware fact would then have to write into this repo to record it, which the cross-project rules forbid — so the fact would land in an inbox instead of on the device's page. Moving keeps one canonical copy without closing the write path. system-health-check.org resolves a host's inventory by its #+HOSTNAME: keyword under ${ROAM_DIR:-$HOME/org/roam}/hardware/, never by filename: a node's timestamp prefix does not survive a rename. A host with no roam clone takes the same NO INVENTORY FILE path it always did. strix-soak-watch.org lands here from home, beside the workflow whose Phase 3 invokes it. It is a workflow, not an inventory page.
Diffstat (limited to 'docs/workflows')
-rw-r--r--docs/workflows/strix-soak-watch.org146
-rw-r--r--docs/workflows/system-health-check.org38
2 files changed, 167 insertions, 17 deletions
diff --git a/docs/workflows/strix-soak-watch.org b/docs/workflows/strix-soak-watch.org
new file mode 100644
index 0000000..e2fbf68
--- /dev/null
+++ b/docs/workflows/strix-soak-watch.org
@@ -0,0 +1,146 @@
+#+TITLE: Strix Halo SMU Deadlock — Kernel Update Watch
+#+AUTHOR: Craig Jennings & Claude
+#+DATE: 2026-05-10
+
+* Purpose
+
+Companion to =system-health-check.org=. Triggered when linux / linux-lts / linux-firmware land in =checkupdates= on ratio, this addendum checks whether the upstream side of the Strix Halo SMU deadlock has moved — so we know when to retire the locally-patched =linux-lts-strix= kernel and switch back to stock.
+
+This is a one-time hardware/kernel issue. Delete this file when the strix kernel is retired.
+
+* Current State (as of 2026-05-10)
+
+- *Boot kernel:* =linux-lts-strix 6.18.25-1= (custom AUR-style local build), GRUB default pinned
+- *SMC firmware:* =100.6.0= (from =/sys/kernel/debug/dri/*/amdgpu_firmware_info= SMC line)
+- *Cmdline workaround:* =amdgpu.no_vpe_idle_pg=1= active
+- *Soak verdict:* HOLDING since 2026-04-30 — zero SMU/dcn35/VPE freeze signature hits (see =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project))
+
+* Background (one-paragraph summary so future sessions don't have to reconstruct it)
+
+Strix Halo (gfx_v11_5_0, smu_v14_0_2 APU) has an SMU firmware deadlock triggered during VPE (Video Processing Engine) idle power-gating: hard freeze, no journal output, requires power-cycle. Crupi posted a workaround patch on the Framework community forum that disables VPE idle power-gating via =amdgpu.no_vpe_idle_pg=. Real fix has to come from AMD as a PMFW/SMU firmware update (delivered via =linux-firmware=). Until then, ratio runs a custom kernel build that carries the patch + two supporting amdgpu allocation-in-critical-path patches. Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795
+
+* Local Patch Set
+
+Located at =~/build/linux-lts-patched/linux-lts/=:
+
+| File | Strix-relevant? |
+|---------------------------------------------------------------------------------+-----------------|
+| =0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-CLONE_NEWUSER.patch= | No (security) |
+| =0002-drm-amdgpu-avoid-memory-allocation-in-the-critical-code-path-v3.patch= | Yes |
+| =0003-drm-amdgpu-use-GFP_ATOMIC-instead-of-NOWAIT-in-the-critical-path.patch= | Yes |
+| =0004-drm-amdgpu-add-no_vpe_idle_pg-strix-halo-smu-deadlock-workaround.patch= | Yes (Crupi's) |
+
+* When This Watch Fires
+
+Phase 3 of =system-health-check.org= invokes this addendum when *any* of the following appear in =checkupdates=:
+
+- =linux= (any version bump)
+- =linux-lts= (any version bump)
+- =linux-firmware= (any version bump — this is the most likely vector for the real fix)
+- =mesa= (major version only, e.g. 26.x.y → 27.x.y — display-stack timing can shift the deadlock window)
+
+* What To Check On Each Fire
+
+** 1. Stable kernel changelogs (per bumped version)
+
+Fetch the per-version changelog and search for the keyword set below. The 2026-05-10 baseline pass found *no matches* in 6.18.27, 6.18.28, 7.0.4, 7.0.5 — so anything that does match is news.
+
+#+begin_src
+https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-<NEW-LTS-VERSION>
+https://cdn.kernel.org/pub/linux/kernel/v7.x/ChangeLog-<NEW-MAINLINE-VERSION>
+#+end_src
+
+Keyword set (case-insensitive):
+- =no_vpe_idle_pg=, =vpe_idle=, =VPE idle=, =VPE power gating=
+- =Strix Halo=, =STRIX_HALO=, =strix_halo=
+- =gfx_v11_5= (the GFX block id for Strix Halo)
+- =smu_v14= (the SMU IP block driver, =drivers/gpu/drm/amd/pm/swsmu/smu14=)
+- =dcn351= (Strix Halo display controller; =dcn35= is Phoenix-class — adjacent but not the same)
+- =drm/amdgpu/vpe=, =drm/amdgpu/swsmu/smu14=, =amdgpu/pm/swsmu=
+- =SMU deadlock=, =critical path= (combined with amdgpu/drm)
+
+The two adjacent allocation-fix patches (0002, 0003) may land separately from Crupi's patch. They have distinctive subject lines: =avoid memory allocation in the critical code path= and =use GFP_ATOMIC instead of NOWAIT in the critical path=. Search for those too.
+
+** 2. linux-firmware delta (if linux-firmware is the bumped package)
+
+The canonical fix arrives as a PMFW/SMU firmware update, NOT a kernel patch.
+
+#+begin_src bash
+# Pre-bump SMC version (from running system, BEFORE topgrade)
+sudo cat /sys/kernel/debug/dri/*/amdgpu_firmware_info | grep "SMC feature"
+
+# After topgrade installs the new linux-firmware, examine the delta.
+# Watch for these files in the upgraded package:
+# amdgpu/smu_14_0_2.bin — Strix Halo SMU firmware
+# amdgpu/psp_14_0_*.bin — PSP/secure-os blobs
+# amdgpu/strix* — anything Strix-tagged
+pacman -Ql linux-firmware | grep -iE "smu_14_0_2|psp_14_0|strix"
+
+# Reboot is required to load the new firmware. Re-check SMC version post-reboot.
+#+end_src
+
+A bump from =100.6.0= to anything higher is the canonical Strix-fix signal — that means AMD shipped new SMU firmware. Once that lands, the workaround kernel is no longer needed (in theory; see "Outcomes" below for the validation path).
+
+** 3. Crupi's forum thread
+
+#+begin_src
+https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795
+#+end_src
+
+Skim recent posts for:
+- "Merged upstream" / "Landed in 6.x.y" announcements
+- AMD acknowledgements or workaround retirement notes
+- Other affected users reporting fix verification on stock kernels
+
+** 4. amd-pm subsystem mailing list (optional, only if 1-3 are inconclusive)
+
+#+begin_src
+https://lore.kernel.org/amd-gfx/?q=strix+halo+smu+deadlock
+https://lore.kernel.org/amd-gfx/?q=no_vpe_idle_pg
+#+end_src
+
+* Outcomes & Actions
+
+** Nothing Strix-adjacent in the bump (default — also the 2026-05-10 outcome)
+
+- Continue running =linux-lts-strix 6.18.25-1=
+- Do *not* rebuild the strix kernel against the newer base purely for the version bump. Same four patches against a newer base = same workaround behavior + new risk of unrelated base-version regressions. Only rebuild when there's a security-pressing reason.
+- Topgrade is safe; the new stock =linux-lts= installs to disk but isn't booted.
+
+** linux-firmware bumps SMC past =100.6.0=
+
+This is the most likely real-fix path.
+
+1. Note the new SMC version (capture pre and post).
+2. Schedule a controlled test boot:
+ - Backup current =/etc/default/grub= GRUB_DEFAULT and =linux-lts-strix= package
+ - Drop =amdgpu.no_vpe_idle_pg=1= from the cmdline on a test boot of stock =linux-lts=
+ - Re-run the soak-check script for 1 week (same harness used for the original verdict)
+3. If zero signature hits over the soak window → retire =linux-lts-strix=, switch GRUB default back to stock =linux-lts=, remove the custom build dir.
+4. If any freeze recurs → restore strix-lts as default and re-engage the workaround. Note the recurrence in this file and on the forum thread.
+
+** Crupi's patch (or equivalent) lands in 6.18.x stable
+
+1. Check the merged form: is the =no_vpe_idle_pg= behavior the default-on, or still gated behind a cmdline?
+2. If default-on: rebuild =linux-lts-strix= against the new base with patch 0004 dropped (keep 0002, 0003, 0001 as long as they remain non-redundant).
+3. If still cmdline-gated: keep the strix kernel as-is until linux-firmware fix arrives — the in-tree path is equivalent to the local one.
+
+** Patches 0002/0003 land upstream but Crupi's doesn't
+
+Drop the matching patches from the PKGBUILD on next strix rebuild; keep 0004 until the SMU firmware fix or kernel merge.
+
+* Reference Artifacts
+
+- Soak verdict (archived): =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project)
+- VPE-patch deployment session: =.ai/sessions/2026-04-29-23-35-linux-lts-strix-vpe-patch-deployed.org=
+- Soak fix + housekeeping session: =.ai/sessions/2026-05-03-06-48-soak-fix-and-velox-sync-housekeeping.org=
+- Verdict-holds session: =.ai/sessions/2026-05-05-01-48-strix-verdict-holds-mybitch-soak-deferred.org=
+- Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795
+- Build dir: =~/build/linux-lts-patched/linux-lts/=
+- Inventory: the =ratio - Desktop Workstation= node in =~/org/roam/hardware/= (resolve by =#+HOSTNAME: ratio=)
+
+* Out of Scope
+
+- Generalizing this to other AMD APUs. The deadlock signature is specific to Strix Halo (gfx_v11_5_0, smu_v14_0_2). Other Phoenix / Hawk Point / RDNA-N APUs are unaffected and shouldn't be checked here.
+- Tracking AMD's internal bug tracker. Stick to public signals (kernel.org changelogs, lore.kernel.org, the Framework forum thread).
+- This is not a soak-check runner. The soak-check harness (=at=-job + scripts) lives separately under =scripts/= or =inbox/=. This file watches for the *retirement signal* for that harness.
diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org
index b66a186..98fa17f 100644
--- a/docs/workflows/system-health-check.org
+++ b/docs/workflows/system-health-check.org
@@ -8,7 +8,7 @@ This workflow performs a comprehensive diagnostic scan of whatever host it runs
Run this when Craig asks "how is <host> doing?", "check my system health", or as a periodic maintenance check.
-Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=.
+Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live in the org-roam knowledge base at =~/org/roam/hardware/= (moved there 2026-07-09).
Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3.
@@ -23,20 +23,24 @@ Read the TOML at the start of Phase 1 and grade against its values. Where a chec
* Hosts & Capabilities
-The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift.
+The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory nodes in the org-roam knowledge base declare the expected capabilities in a property drawer and act as a cross-check for drift.
-Host inventory files (match by =#+HOSTNAME:= keyword inside each):
+The inventories live at =~/org/roam/hardware/= (2026-07-09), alongside the rest of the hardware inventory. They moved out of this repo so any project's session can read a host's capabilities — and write back what a diagnosis discovers — without reaching across a project boundary. The index node "Homelab Hardware Inventory" links them all.
-| Host | Inventory file | Notes |
-|---------+-------------------------------------------+----------------------------------|
-| ratio | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]] | Arch, Btrfs RAID1, pacman |
-|---------+-------------------------------------------+----------------------------------|
-| velox | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]] | Arch, ZFS (re-installed 2026-04) |
-|---------+-------------------------------------------+----------------------------------|
-| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt |
-|---------+-------------------------------------------+----------------------------------|
-| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM |
-|---------+-------------------------------------------+----------------------------------|
+Host inventory nodes (match by =#+HOSTNAME:= keyword inside each):
+
+| Host | Inventory node | Notes |
+|---------+-----------------------------------------------+----------------------------------|
+| ratio | =hardware/…-ratio_desktop_workstation.org= | Arch, Btrfs RAID1, pacman |
+|---------+-----------------------------------------------+----------------------------------|
+| velox | =hardware/…-velox_laptop.org= | Arch, ZFS (re-installed 2026-04) |
+|---------+-----------------------------------------------+----------------------------------|
+| mybitch | =hardware/…-mybitch_christines_laptop.org= | Mint, ext4, apt |
+|---------+-----------------------------------------------+----------------------------------|
+| truenas | =hardware/…-truenas_nas_storage_server.org= | TrueNAS SCALE, ZFS, no PM |
+|---------+-----------------------------------------------+----------------------------------|
+
+Never hardcode a node's filename — its timestamp prefix is not stable across a rename. Resolve by the =#+HOSTNAME:= keyword, as Phase 0 does.
Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift.
@@ -108,10 +112,10 @@ Present the probe result as a one-line capability summary at the top of the repo
Locate the inventory file for this host:
#+begin_src bash
-inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1)
+inv=$(grep -l "^#+HOSTNAME: $host\b" "${ROAM_DIR:-$HOME/org/roam}"/hardware/*.org 2>/dev/null | head -1)
#+end_src
-If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check.
+If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. A host without the roam clone (mybitch, truenas) takes that path, exactly as it did when the inventories lived in this repo.
If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe.
@@ -228,7 +232,7 @@ Updates are separate from issue investigation. After all issues are addressed (o
2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware)
3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices
4. Check =/var/log/pacman.log= for recent failed transactions
-5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
+5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at [[file:strix-soak-watch.org][docs/workflows/strix-soak-watch.org]] before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=). On maint hosts (ratio, velox) plain =topgrade= resolves to the dotfiles PATH wrapper, which stamps the console's topgrade-freshness metric on success — no extra step. If the run happened outside the wrapper somehow, =maint stamp topgrade= records it by hand.
7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot
@@ -996,7 +1000,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi
- Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986.
- Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot.
- Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean.
-- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.)
+- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in the =mybitch - Christine's Laptop= roam node → Operational Changes Log.)
- *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=.
- *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug).
- Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111.