docs: adopt system-health-check workflow and host inventories

The workflow and the four host capability inventories move here from the home project. System maintenance is archsetup's domain, and this puts the thresholds TOML and both of its consumers in one repo.

The workflow's prose severity rules now cite the installed TOML keys, and the TOML wins on disagreement. Inventory paths point at docs/homelab-inventory/. A note routes routine ratio/velox checks to the maint CLI, leaving this workflow for the non-Arch hosts and forensic deep dives.

Only the #+HOSTNAME: capability inventories moved (ratio, velox, mybitch, truenas, plus the truenas hardware-specs asset they link). The personal gear records stay in home, and the home-side removal is handed off through its inbox.

6 files changed, 1948 insertions, 0 deletions

diff --git a/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org b/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org
new file mode 100644
index 0000000..42d33ad
--- /dev/null
+++ b/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org
@@ -0,0 +1,229 @@
+#+TITLE: TrueNAS Server Hardware Specifications
+#+AUTHOR: Auto-generated via SSH
+#+DATE: 2026-01-05
+
+* Package Management Note
+
+**IMPORTANT:** Package management tools (apt) are disabled on TrueNAS appliances. Attempting to install packages with apt or methods other than the TrueNAS web interface can result in a nonfunctional system.
+
+All software installation and updates must be done through the TrueNAS web interface.
+
+* System Information
+
+** Hostname: truenas
+** IP Address: 192.168.86.5 (static)
+** Network Interface: eno1
+** Kernel: Linux 6.12.33-production+truenas
+** Build Date: Wed Dec 17 21:17:21 UTC 2025
+** Architecture: x86_64 GNU/Linux
+
+* CPU Specifications
+
+** Model: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
+** Architecture: x86_64 (Coffee Lake S)
+** Cores: 8 physical cores
+** Threads: 16 (2 threads per core)
+** Base Clock: 3.60 GHz
+** Max Turbo: 5.00 GHz
+** Min Clock: 800 MHz
+
+** Cache:
+- L1d cache: 256 KiB (8 instances)
+- L1i cache: 256 KiB (8 instances)
+- L2 cache: 2 MiB (8 instances)
+- L3 cache: 16 MiB (1 instance)
+
+** Features:
+- Virtualization: VT-x
+- CPU family: 6
+- Model: 158
+- Stepping: 13
+
+* Memory Specifications
+
+** Total RAM: 64 GiB (62 GiB usable)
+** Swap: 0 B (no swap configured)
+** Current Usage: ~3.0 GiB used, 59 GiB free
+
+* Storage Controllers
+
+** SATA Controllers:
+1. Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
+2. ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02)
+
+** NVMe Controllers:
+1. Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01)
+
+* Block Devices
+
+** Hard Drives:
+- sda: 10.9 TB (ZFS member)
+- sdb: 10.9 TB (ZFS member)
+- sdc: 10.9 TB (ZFS member)
+- sdd: 10.9 TB (ZFS member)
+- sdg: 18.2 TB (ZFS member)
+- sdh: 18.2 TB (ZFS member)
+- sdi: 18.2 TB (ZFS member)
+- sdj: 18.2 TB (ZFS member)
+
+** System Drives:
+- sde: 1.9 TB (Boot pool, ZFS member)
+  - sde1: 1M partition
+  - sde2: 512M VFAT partition
+  - sde3: 1.9T ZFS partition
+- sdf: 1.9 TB (ZFS member)
+
+** NVMe:
+- nvme0n1: 1.8 TB
+
+** Total Raw Capacity:
+- 4× 10.9 TB = 43.6 TB
+- 4× 18.2 TB = 72.8 TB
+- 2× 1.9 TB = 3.8 TB
+- 1× 1.8 TB NVMe = 1.8 TB
+- **Total: ~122 TB raw storage**
+
+* Network Interfaces
+
+** eno1 (Primary - Active):
+- Type: Ethernet controller - Intel Corporation Ethernet Connection (7) I219-V (rev 10)
+- MAC: 70:85:c2:db:9d:94
+- IP: 192.168.86.5/21
+- Subnet broadcast: 192.168.87.255
+- State: UP
+- MTU: 1500
+- Alternative name: enp0s31f6
+
+** enp2s0 (Secondary - Down):
+- Type: Ethernet controller - Intel Corporation I211 Gigabit Network Connection (rev 03)
+- MAC: 70:85:c2:db:9d:94
+- State: DOWN (NO-CARRIER)
+
+** Wireless:
+- Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10)
+- Status: Not configured
+
+** Docker Bridge:
+- docker0: 172.16.0.1/24
+- State: DOWN (NO-CARRIER)
+
+* Motherboard/Chipset
+
+** Chipset: Intel Z390
+- Host Bridge: 8th/9th Gen Core 8-core Desktop Processor (Coffee Lake S)
+- ISA Bridge: Z390 Chipset LPC/eSPI Controller (rev 10)
+- PCIe Root Ports: Multiple Cannon Lake PCH PCI Express ports
+
+** Integrated Components:
+- Graphics: Intel CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
+- Audio: Intel Cannon Lake PCH cAVS (rev 10)
+- USB: Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
+- Thermal: Cannon Lake PCH Thermal Controller (rev 10)
+- HECI: Cannon Lake PCH HECI Controller (rev 10)
+- SMBus: Cannon Lake PCH SMBus Controller (rev 10)
+- SPI: Cannon Lake PCH SPI Controller (rev 10)
+
+* ZFS Pool Configuration
+
+** boot-pool (1.86 TB):
+- Allocated: 3.06 GB
+- Free: 1.86 TB
+- Capacity: 0%
+- Health: ONLINE
+
+** sysdata (1.86 TB):
+- Allocated: 314 MB
+- Free: 1.86 TB
+- Capacity: 0%
+- Health: ONLINE
+- Purpose: System data and applications
+
+** tank (72.8 TB):
+- Allocated: 1.17 MB
+- Free: 72.7 TB
+- Capacity: 0%
+- Health: ONLINE
+- Purpose: Large storage pool (empty, ready for expansion)
+
+** vault (43.6 TB):
+- Allocated: 24.5 TB
+- Free: 19.1 TB
+- Capacity: 56%
+- Health: ONLINE
+- Purpose: Main media storage
+
+* Vault Datasets
+
+** Media (Movies, Music, TV Shows):
+- Size: 30 TB (allocated)
+- Used: 17 TB
+- Available: 14 TB
+- Capacity: 54%
+
+** Lectures:
+- Size: 15 TB (allocated)
+- Used: 878 GB
+- Available: 14 TB
+- Capacity: 6%
+
+** Audiobooks:
+- Size: 14 TB (allocated)
+- Used: 89 GB
+- Available: 14 TB
+- Capacity: 1%
+
+** Magic:
+- Size: 15 TB (allocated)
+- Used: 543 GB
+- Available: 14 TB
+- Capacity: 4%
+
+** Books:
+- Size: 14 TB (allocated)
+- Used: 177 GB
+- Available: 14 TB
+- Capacity: 2%
+
+* Security Mitigations
+
+** CPU Vulnerabilities (Status):
+- Gather data sampling: Mitigated (Microcode)
+- Indirect target selection: Mitigated (Aligned branch/return thunks)
+- Itlb multihit: KVM Mitigation (Split huge pages)
+- L1tf: Not affected
+- Mds: Not affected
+- Meltdown: Not affected
+- Mmio stale data: Mitigated (Clear CPU buffers; SMT vulnerable)
+- Reg file data sampling: Not affected
+- Retbleed: Mitigated (Enhanced IBRS)
+- Spec rstack overflow: Not affected
+- Spec store bypass: Mitigated (Disabled via prctl)
+- Spectre v1: Mitigated (usercopy/swapgs barriers)
+- Spectre v2: Mitigated (Enhanced/Automatic IBRS)
+- Srbds: Mitigated (Microcode)
+- Tsx async abort: Mitigated (TSX disabled)
+
+* Full PCI Device List
+
+#+BEGIN_EXAMPLE
+00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
+00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d)
+00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
+00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
+00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
+00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
+00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
+00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
+00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0)
+00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0)
+00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
+00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10)
+00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
+00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
+00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
+00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
+01:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02)
+02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
+03:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10)
+04:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01)
+#+END_EXAMPLE
diff --git a/docs/homelab-inventory/mybitch-laptop.org b/docs/homelab-inventory/mybitch-laptop.org
new file mode 100644
index 0000000..e522c40
--- /dev/null
+++ b/docs/homelab-inventory/mybitch-laptop.org
@@ -0,0 +1,238 @@
+#+TITLE: mybitch - Christine's Laptop
+#+DATE: 2026-01-31
+#+HOSTNAME: mybitch
+
+* Automated Capabilities
+
+# Maintained by the system-health-check workflow. Manual edits may be
+# overwritten when the workflow detects drift between this drawer and the
+# live system. Machine-readable capability signals used to dispatch which
+# checks run on this host.
+
+:PROPERTIES:
+:FS:         ext4
+:PM:         apt
+:ORCH:       none
+:SNAPSHOT:   timeshift
+:MESH:       tailscale
+:BACKUP:     source
+:VIRT:       libvirt
+:INIT:       systemd
+:LAST_AUDIT: 2026-06-23
+:END:
+
+* Overview
+
+| Field    | Value                             |
+|----------+-----------------------------------|
+| Hostname | mybitch                           |
+|----------+-----------------------------------|
+| Type     | Laptop                            |
+|----------+-----------------------------------|
+| Owner    | Christine                         |
+|----------+-----------------------------------|
+| Maker    | Framework                         |
+|----------+-----------------------------------|
+| Model    | Laptop 16 (AMD Ryzen 7040 Series) |
+|----------+-----------------------------------|
+| OS       | Linux Mint 22.3 (Zena)            |
+|----------+-----------------------------------|
+| Kernel   | 6.17.0-23-generic                 |
+|----------+-----------------------------------|
+| Status   | In use                            |
+|----------+-----------------------------------|
+
+* Hardware Specifications
+
+** CPU
+
+| Field        | Value                             |
+|--------------+-----------------------------------|
+| Model        | AMD Ryzen 9 7940HS w/ Radeon 780M |
+|--------------+-----------------------------------|
+| Architecture | Zen 4 (TSMC 5nm)                  |
+|--------------+-----------------------------------|
+| Cores        | 8                                 |
+|--------------+-----------------------------------|
+| Threads      | 16                                |
+|--------------+-----------------------------------|
+| Base Clock   | 4.0 GHz                           |
+|--------------+-----------------------------------|
+| Boost Clock  | 5.25 GHz                          |
+|--------------+-----------------------------------|
+| L2 Cache     | 8 MiB (8x1024 KiB)                |
+|--------------+-----------------------------------|
+| L3 Cache     | 16 MiB                            |
+|--------------+-----------------------------------|
+
+** Memory
+
+| Field     | Value                        |
+|-----------+------------------------------|
+| Total     | 32 GB                        |
+|-----------+------------------------------|
+| Type      | DDR5-5600                    |
+|-----------+------------------------------|
+| Installed | 1x 32GB (A-DATA AD5S560032G) |
+|-----------+------------------------------|
+| Slots     | 2 (1 free)                   |
+|-----------+------------------------------|
+| Max       | 64 GB                        |
+|-----------+------------------------------|
+
+** Graphics
+
+| Device     | Model               | Driver | Notes           |
+|------------+---------------------+--------+-----------------|
+| Dedicated  | AMD Radeon RX 7700S | amdgpu | RDNA 3, Navi 33 |
+|------------+---------------------+--------+-----------------|
+| Integrated | AMD Radeon 780M     | amdgpu | RDNA 3, Phoenix |
+|------------+---------------------+--------+-----------------|
+
+** Storage
+
+| Device       | Model               | Size     | Interface    |
+|--------------+---------------------+----------+--------------|
+| /dev/nvme0n1 | WD BLACK SN770M 2TB | 1.82 TiB | PCIe Gen4 x4 |
+|--------------+---------------------+----------+--------------|
+| /dev/nvme1n1 | WD BLACK SN770 2TB  | 1.82 TiB | PCIe Gen4 x4 |
+|--------------+---------------------+----------+--------------|
+| Total        |                     | 3.64 TiB |              |
+|--------------+---------------------+----------+--------------|
+
+*** Partitions
+
+| Partition      | Size     | Used        | Filesystem | Mount     | Label  |
+|----------------+----------+-------------+------------+-----------+--------|
+| /dev/nvme1n1p2 | 1.82 TiB | 17.4%       | ext4       | /         |        |
+|----------------+----------+-------------+------------+-----------+--------|
+| /dev/nvme1n1p1 | 512 MiB  | 1.2%        | vfat       | /boot/efi |        |
+|----------------+----------+-------------+------------+-----------+--------|
+| /dev/nvme0n1p1 | 1.82 TiB | (unmounted) | ext4       |           | Backup |
+|----------------+----------+-------------+------------+-----------+--------|
+
+** Display
+
+| Field      | Value              |
+|------------+--------------------|
+| Panel      | BOE Display 0x0bc9 |
+|------------+--------------------|
+| Resolution | 2560x1600          |
+|------------+--------------------|
+| Size       | 16" (345x215mm)    |
+|------------+--------------------|
+| Ratio      | 16:10              |
+|------------+--------------------|
+| DPI        | 188                |
+|------------+--------------------|
+
+** Battery
+
+| Field    | Value            |
+|----------+------------------|
+| Model    | NVT FRANDBA      |
+|----------+------------------|
+| Capacity | 85.1 Wh (design) |
+|----------+------------------|
+| Current  | 87.5 Wh (102.9%) |
+|----------+------------------|
+| Cycles   | 13               |
+|----------+------------------|
+| Type     | Li-ion           |
+|----------+------------------|
+
+** Network
+
+| Interface | Type   | Chipset                  |
+|-----------+--------+--------------------------|
+| wlp5s0    | WiFi   | MediaTek MT7922 802.11ax |
+|-----------+--------+--------------------------|
+| Bluetooth | BT 5.2 | MediaTek Wireless_Device |
+|-----------+--------+--------------------------|
+
+** Expansion Cards (Framework)
+
+| Slot | Card                |
+|------+---------------------|
+| 1    | HDMI Expansion Card |
+|------+---------------------|
+
+** Input Modules (Framework)
+
+| Module   | Notes                   |
+|----------+-------------------------|
+| Keyboard | ANSI layout             |
+|----------+-------------------------|
+| Numpad   | Laptop 16 Numpad Module |
+|----------+-------------------------|
+
+* Connected Peripherals
+
+| Device                    | Connection              |
+|---------------------------+-------------------------|
+| Logitech MX Master 3      | Unifying Receiver (USB) |
+|---------------------------+-------------------------|
+| Realtek Laptop Camera     | Internal USB            |
+|---------------------------+-------------------------|
+| Goodix Fingerprint Reader | Internal USB            |
+|---------------------------+-------------------------|
+
+* Network Access
+
+| Method | Address       |
+|--------+---------------|
+| mDNS   | mybitch.local |
+|--------+---------------|
+| SSH    | Yes (OpenSSH) |
+|--------+---------------|
+
+* Notes
+
+- Framework Laptop 16 with modular expansion card system
+- Discrete AMD GPU (RX 7700S) in addition to integrated Radeon 780M
+- Second NVMe slot contains 2TB backup drive (unmounted, labeled "Backup")
+- Uptime at inventory: 4 days
+
+* Operational Changes Log
+
+Deliberate, non-default config changes applied to mybitch (so they can be found and reverted later). Most recent first.
+
+** 2026-05-12 — amdgpu.mes=0 added (escalation of the MES-hang freeze mitigation)
+
+The 2026-05-11 =amdgpu.cwsr_enable=0= mitigation didn't reduce the iGPU "MES failed to respond" errors (14 hits in 14.5h after the fix vs. 38 over 10 days before — rate went *up*), so escalated per the documented path: added =amdgpu.mes=0= to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= (now =quiet splash amdgpu.cwsr_enable=0 amdgpu.mes=0=). Backup: =/etc/default/grub.bak-2026-05-12-amdgpu-mes=. =update-grub= run; param verified in all 6 =/boot/grub/grub.cfg= entries. Takes effect on next reboot (planned for after the in-progress first rsyncshot backup). =amdgpu.mes=0= disables the GFX11 hardware MES scheduler and falls back to the legacy KIQ path — sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks); irrelevant for Christine's light desktop workload. =cwsr_enable=0= kept (harmless alongside mes=0). *Revert:* restore =/etc/default/grub.bak-2026-05-12-amdgpu-mes= (or sed out = amdgpu.mes=0=), =update-grub=, reboot. After the reboot, check =journalctl -k -b | grep -c 'MES failed'= → should be 0. See the 2026-05-11 entry in =.ai/project-workflows/system-health-check.org='s Known Issues Log for the full background.
+
+** 2026-05-12 — rsyncshot backups + travel keep-awake mode
+
+Set up the same =rsyncshot= → TrueNAS backup mybitch's siblings use, and made mybitch never suspend so backups keep running while Christine travels with it.
+
+*** rsyncshot install (mirrors ratio's setup)
+- =/usr/local/bin/rsyncshot= — the script (source of truth: =~/code/rsyncshot/rsyncshot=).
+- =/etc/rsyncshot/config= — =REMOTE_HOST="cjennings@truenas.tailf3bb8c.ts.net"=, =REMOTE_PATH="/mnt/vault/backups"=, =SSH_IDENTITY_FILE="/root/.ssh/id_ed25519"=. (See "destination pinned to Tailscale" below for why the full MagicDNS name.)
+- =/etc/rsyncshot/include.txt= — =/home /etc /usr/local/bin=. =/etc/rsyncshot/exclude.txt= — copied verbatim from ratio.
+- =/etc/logrotate.d/rsyncshot= — copied verbatim from ratio. =/var/log/rsyncshot.log= — the log.
+- Root crontab: hourly at :30 (hours 0-1,3-23, keep 23) + daily at 2:30 (keep 30), each wrapped in =flock -x /tmp/rsyncshot.lock=. Identical to ratio's schedule.
+- =/root/.ssh/id_ed25519= (+ =.pub=) — new keypair generated on mybitch, no passphrase, comment =root@mybitch-rsyncshot=. mybitch's =cjennings= account had no outbound key (it's a receive-only account), so the backup runs with a dedicated root key instead. The pubkey was appended to =~cjennings/.ssh/authorized_keys= on TrueNAS (the line tagged =root@mybitch-rsyncshot=).
+- =/root/.ssh/config= — =Host truenas truenas.tailf3bb8c.ts.net 100.67.22.65= → =User cjennings=, that IdentityFile, =ServerAliveInterval 60 / ServerAliveCountMax 10 / TCPKeepAlive yes=. Host keys for those names added to =/root/.ssh/known_hosts=.
+- TrueNAS side: =/mnt/vault/backups/mybitch/= directory created, owned =cjennings:cjennings= (=vault/backups= is a ZFS dataset; per-host backups are just subdirectories, like =ratio/= and =velox/=).
+- *Revert:* =sudo crontab -r= (or remove the rsyncshot lines), =sudo rm -rf /usr/local/bin/rsyncshot /etc/rsyncshot /etc/logrotate.d/rsyncshot /var/log/rsyncshot.log /root/.ssh/id_ed25519* /root/.ssh/config=, restore =/root/.ssh/known_hosts= if desired, remove the =root@mybitch-rsyncshot= line from =~cjennings/.ssh/authorized_keys= on TrueNAS, and =rm -rf /mnt/vault/backups/mybitch= on TrueNAS.
+
+*** destination pinned to Tailscale
+=REMOTE_HOST= uses the full Tailscale MagicDNS name =truenas.tailf3bb8c.ts.net= (not the bare =truenas=) so the backup always routes over the tailnet regardless of what a hotel/away-from-home router's DNS resolves =truenas= to. Tailscale uses a direct LAN path between same-LAN peers, so this is near-zero perf cost when mybitch is on the home network. *Revert:* set =REMOTE_HOST="cjennings@truenas"= in =/etc/rsyncshot/config=.
+
+*** keep-awake (never suspend) — three layers
+1. =systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target= — the hard guarantee; suspend can't happen even if requested. *Revert:* =sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target=.
+2. Cinnamon power gsettings for user =cciarm= set to ='nothing'=: =sleep-inactive-ac-type=, =sleep-inactive-battery-type=, =lid-close-ac-action=, =lid-close-battery-action= (schema =org.cinnamon.settings-daemon.plugins.power=). Were ='suspend'/'suspend'/'nothing'/'suspend'=. *Revert:* set them back, or run =gsettings reset= on each key as =cciarm=.
+3. =/etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= — =HandleLidSwitch=ignore=, =HandleLidSwitchExternalPower=ignore=, =HandleLidSwitchDocked=ignore=, =IdleAction=ignore=. (Written but not applied live — =systemd-logind= wasn't restarted because sessions were active; takes effect on next reboot. Layers 1-2 cover the gap.) *Revert:* =sudo rm /etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= then reboot or =systemctl restart systemd-logind=.
+
+When mybitch goes back to being a stationary machine (or this whole arrangement is no longer wanted), reverse layers 1-3. The rsyncshot backup itself is worth keeping regardless.
+
+*** IPv6 set to link-local-only (fixes the tailscale dropout)
+mybitch dropped off the Tailscale tailnet on 2026-05-12 AM. Root cause: the home WiFi ("dupre") has device(s) advertising ULA-only IPv6 RAs (prefixes =fdc4:c4c6:3fb0:16cc::/64= and =fd67:8615:6780:1::/64=) with no default route. NetworkManager (=ipv6.method=auto=) SLAAC'd a "global"-scope ULA address onto wlp5s0 with no v6 internet path, so tailscaled kept trying IPv6 to controlplane/DERP, failed "network is unreachable", flapped, and lost its control connection. mybitch has no working global IPv6 anywhere (no ISP v6), so the fix is to ignore underlay v6:
+- =/etc/NetworkManager/conf.d/ipv6-link-local-only.conf= → =[connection] ipv6.method=link-local= (connection default — covers home WiFi, travel WiFi, phone hotspot).
+- =nmcli con modify dupre ipv6.method link-local= (explicit on the home connection).
+- Effect: wlp5s0 keeps only fe80:: (mDNS still works), no SLAAC, no v6 routes; tailscale0's overlay v6 untouched; tailscaled uses the IPv4 underlay. Activates on reboot or =nmcli dev reapply wlp5s0=.
+- *Revert if mybitch ever gets real IPv6:* =sudo rm /etc/NetworkManager/conf.d/ipv6-link-local-only.conf=; =sudo nmcli con modify dupre ipv6.method auto=; reboot or =nmcli dev reapply wlp5s0=.
+
+*** caveats (relayed to Craig for Christine)
+- mybitch must stay plugged into AC for backups to keep running — masking suspend doesn't stop a battery dying with the lid closed.
+- mybitch must be on WiFi at the destination and stay on the Tailscale tailnet.
diff --git a/docs/homelab-inventory/ratio-desktop.org b/docs/homelab-inventory/ratio-desktop.org
new file mode 100644
index 0000000..86dcd4d
--- /dev/null
+++ b/docs/homelab-inventory/ratio-desktop.org
@@ -0,0 +1,175 @@
+#+TITLE: ratio - Desktop Workstation
+#+DATE: 2026-01-27
+#+HOSTNAME: ratio
+
+* Automated Capabilities
+
+# Maintained by the system-health-check workflow. Manual edits may be
+# overwritten when the workflow detects drift between this drawer and the
+# live system. Machine-readable capability signals used to dispatch which
+# checks run on this host.
+
+:PROPERTIES:
+:FS:         btrfs
+:PM:         pacman
+:ORCH:       topgrade
+:SNAPSHOT:   snapper
+:MESH:       tailscale
+:BACKUP:     source
+:VIRT:       libvirt,docker,podman
+:INIT:       systemd
+:LAST_AUDIT: 2026-05-25
+:END:
+
+* Overview
+
+| Field       | Value                                    |
+|-------------+------------------------------------------|
+| Hostname    | ratio                                    |
+|-------------+------------------------------------------|
+| Type        | Desktop workstation                      |
+|-------------+------------------------------------------|
+| Maker       | Framework                                |
+|-------------+------------------------------------------|
+| Model       | Desktop (AMD Ryzen AI Max 300 Series)    |
+|-------------+------------------------------------------|
+| Board       | FRANMFCP06 v A6                          |
+|-------------+------------------------------------------|
+| Firmware    | INSYDE UEFI v03.03 (2025-09-16)          |
+|-------------+------------------------------------------|
+| OS          | Arch Linux                               |
+|-------------+------------------------------------------|
+| Kernel      | 6.12.67-1-lts                            |
+|-------------+------------------------------------------|
+| Desktop     | Hyprland (Wayland)                       |
+|-------------+------------------------------------------|
+| Previous ID | cogito (repurposed and renamed to ratio) |
+|-------------+------------------------------------------|
+
+* CPU
+
+| Field         | Value                                 |
+|---------------+---------------------------------------|
+| Model         | AMD Ryzen AI MAX+ 395 w/ Radeon 8060S |
+|---------------+---------------------------------------|
+| Architecture  | Zen 5 (TSMC 4nm)                      |
+|---------------+---------------------------------------|
+| Cores/Threads | 16 cores / 32 threads                 |
+|---------------+---------------------------------------|
+| Base/Boost    | 3.0 GHz / 5.15 GHz                    |
+|---------------+---------------------------------------|
+| Cache L1      | 1.2 MiB (d-16x48 KiB; i-16x32 KiB)    |
+|---------------+---------------------------------------|
+| Cache L2      | 16 MiB (16x1024 KiB)                  |
+|---------------+---------------------------------------|
+| Cache L3      | 64 MiB (2x32 MiB)                     |
+|---------------+---------------------------------------|
+| NPU           | XDNA 2, 50+ peak AI TOPS              |
+|---------------+---------------------------------------|
+| Peak Perf     | 59.4 FP16/BF16 TFLOPS @ 2.9GHz        |
+|---------------+---------------------------------------|
+
+* GPU
+
+| Field        | Value                                   |
+|--------------+-----------------------------------------|
+| Model        | AMD Radeon 8060S (Strix Halo)           |
+|--------------+-----------------------------------------|
+| Architecture | RDNA 3.5                                |
+|--------------+-----------------------------------------|
+| CUs          | 40 Compute Units                        |
+|--------------+-----------------------------------------|
+| Driver       | amdgpu (kernel)                         |
+|--------------+-----------------------------------------|
+| Max VRAM     | 96GB (via AMD Variable Graphics Memory) |
+|--------------+-----------------------------------------|
+| GPU Arch ID  | gfx1151                                 |
+|--------------+-----------------------------------------|
+| PCIe         | Gen 4, 16 GT/s, 16 lanes                |
+|--------------+-----------------------------------------|
+
+* Memory
+
+| Field        | Value                    |
+|--------------+--------------------------|
+| Total        | 128 GiB unified (LPDDR5) |
+|--------------+--------------------------|
+| Speed        | 8000 MT/s                |
+|--------------+--------------------------|
+| Channels     | 8 (8x 16 GiB)            |
+|--------------+--------------------------|
+| Manufacturer | Micron Technology        |
+|--------------+--------------------------|
+| Part Number  | MT62F4G32D8DV-023 WT     |
+|--------------+--------------------------|
+
+* Storage
+
+| Device       | Model                  | Size      | Interface |
+|--------------+------------------------+-----------+-----------|
+| /dev/nvme0n1 | WD BLACK SN850X 8000GB | 7.28 TiB  | NVMe      |
+|--------------+------------------------+-----------+-----------|
+| /dev/nvme1n1 | WD BLACK SN850X 8000GB | 7.28 TiB  | NVMe      |
+|--------------+------------------------+-----------+-----------|
+| *Total*      |                        | 14.55 TiB |           |
+|--------------+------------------------+-----------+-----------|
+
+Filesystem: btrfs with subvolumes (@, @home, @snapshots, @log, @pkg)
+
+* Network
+
+| Interface | Chipset         | Speed   | Type     |
+|-----------+-----------------+---------+----------|
+| enp191s0  | Realtek RTL8126 | 5 GbE   | Wired    |
+|-----------+-----------------+---------+----------|
+| wlp192s0  | MediaTek MT7925 | Wi-Fi 7 | Wireless |
+|-----------+-----------------+---------+----------|
+
+Also has: docker0, tailscale0, virbr0 (virtual interfaces)
+
+* Audio
+
+- AMD Radeon High Definition Audio (HDMI, snd_hda_intel)
+- AMD Ryzen HD Audio (3.5mm, snd_hda_intel)
+- PipeWire v1.4.10 (pipewire-pulse, wireplumber)
+
+* PCI Slots
+
+| Slot | Info       | Status    |
+|------+------------+-----------|
+| 1    | M.2 JWLAN  | Available |
+|------+------------+-----------|
+| 2    | M.2 JSSD1  | In use    |
+|------+------------+-----------|
+| 3    | M.2 JSSD2  | In use    |
+|------+------------+-----------|
+| 4    | PCIe Gen 4 | Available |
+|------+------------+-----------|
+
+* Connected Peripherals
+
+- Monitor: Dell U3419W (HDMI-A-1, 3440x1440@60Hz)
+- Keyboard: Das Keyboard 4 (Cherry MX Blue) — replacing with Keychron Q6 Pro
+- Mouse: Logitech M650
+- DAC/Amp: JDS Labs Element IV (ordered)
+
+* AI/LLM Capability
+
+Primary purpose includes local AI/LLM inference via Ollama.
+
+| Model            | Size  | Purpose              |
+|------------------+-------+----------------------|
+| deepseek-r1:70b  | 42 GB | Main reasoning model |
+|------------------+-------+----------------------|
+| qwen3:30b        | 18 GB | MoE chat (256K ctx)  |
+|------------------+-------+----------------------|
+| deepseek-r1:14b  | 9 GB  | Light reasoning      |
+|------------------+-------+----------------------|
+| nomic-embed-text | 274MB | RAG embeddings       |
+|------------------+-------+----------------------|
+
+Inference benchmarks (AMD testing, LM Studio 0.3.11):
+- Small models (1-3B): ~100+ tok/s
+- Medium models (7-8B): ~60-80 tok/s
+- Large models (20B): ~58 tok/s
+- Very large models (120B): ~38 tok/s
diff --git a/docs/homelab-inventory/truenas-server.org b/docs/homelab-inventory/truenas-server.org
new file mode 100644
index 0000000..5ce8613
--- /dev/null
+++ b/docs/homelab-inventory/truenas-server.org
@@ -0,0 +1,114 @@
+#+TITLE: truenas - NAS/Storage Server
+#+DATE: 2026-01-27
+#+HOSTNAME: truenas
+
+* Automated Capabilities
+
+# Maintained by the system-health-check workflow. Manual edits may be
+# overwritten when the workflow detects drift between this drawer and the
+# live system. Machine-readable capability signals used to dispatch which
+# checks run on this host.
+#
+# NOTE: TrueNAS SCALE is middleware-managed. No user-level package manager
+# or topgrade orchestration. Snapshot and update operations go through the
+# TrueNAS UI / API, not pacman/apt.
+
+:PROPERTIES:
+:FS:         zfs
+:PM:         none
+:ORCH:       none
+:SNAPSHOT:   zfs-native
+:MESH:       tailscale
+:BACKUP:     target
+:VIRT:       docker
+:INIT:       systemd
+:LAST_AUDIT: 2026-04-21
+:END:
+
+* Overview
+
+| Field    | Value                                   |
+|----------+-----------------------------------------|
+| Hostname | truenas (truenas.local / 192.168.86.5)  |
+|----------+-----------------------------------------|
+| Type     | NAS / storage server                    |
+|----------+-----------------------------------------|
+| OS       | TrueNAS                                 |
+|----------+-----------------------------------------|
+| Purpose  | Media storage, backups, Plex, Syncthing |
+|----------+-----------------------------------------|
+
+See [[file:2026-01-05-truenas-hardware-specs.org][TrueNAS Hardware Specs]] for full sysinfo output.
+
+* CPU
+
+| Field         | Value                |
+|---------------+----------------------|
+| Model         | Intel Core i9-9900K  |
+|---------------+----------------------|
+| Architecture  | Coffee Lake          |
+|---------------+----------------------|
+| Cores/Threads | 8 cores / 16 threads |
+|---------------+----------------------|
+| Base/Boost    | 3.60 GHz / 5.00 GHz  |
+|---------------+----------------------|
+
+* Memory
+
+| Field | Value       |
+|-------+-------------|
+| Total | 64 GiB DDR4 |
+|-------+-------------|
+
+* Storage
+
+| Pool      | Config          | Raw Size   | Purpose       |
+|-----------+-----------------+------------+---------------|
+| boot-pool | Mirror          | ~1.9 TB x2 | System        |
+|-----------+-----------------+------------+---------------|
+| sysdata   | NVMe            | ~1.8 TB    | System data   |
+|-----------+-----------------+------------+---------------|
+| vault     | RAIDZ1 (4 disk) | ~43.6 TB   | Primary media |
+|-----------+-----------------+------------+---------------|
+| tank      | RAIDZ1 (4 disk) | ~72.8 TB   | Expansion     |
+|-----------+-----------------+------------+---------------|
+
+Total raw: ~122 TB
+
+*** Vault Datasets
+| Dataset    | Size  |
+|------------+-------|
+| Media      | 30 TB |
+|------------+-------|
+| Lectures   | 15 TB |
+|------------+-------|
+| Audiobooks | 14 TB |
+|------------+-------|
+| Magic      | 15 TB |
+|------------+-------|
+| Books      | 14 TB |
+|------------+-------|
+
+* Network
+
+| Interface | Chipset      | Speed  | Status       |
+|-----------+--------------+--------+--------------|
+| eno1      | Intel I219-V | 1 GbE  | Current      |
+|-----------+--------------+--------+--------------|
+| enp2s0    | Intel I211   | 1 GbE  | Available    |
+|-----------+--------------+--------+--------------|
+| (pending) | Intel I226-V | 2.5GbE | Ordered, NIC |
+|-----------+--------------+--------+--------------|
+
+* Motherboard
+
+| Field   | Value      |
+|---------+------------|
+| Chipset | Intel Z390 |
+|---------+------------|
+
+* Services
+
+- Plex Media Server
+- Syncthing
+- Tailscale
diff --git a/docs/homelab-inventory/velox-laptop.org b/docs/homelab-inventory/velox-laptop.org
new file mode 100644
index 0000000..6e4f540
--- /dev/null
+++ b/docs/homelab-inventory/velox-laptop.org
@@ -0,0 +1,158 @@
+#+TITLE: velox - Laptop
+#+DATE: 2026-01-27
+#+HOSTNAME: velox
+
+* Automated Capabilities
+
+# Maintained by the system-health-check workflow. Manual edits may be
+# overwritten when the workflow detects drift between this drawer and the
+# live system. Machine-readable capability signals used to dispatch which
+# checks run on this host.
+#
+# NOTE: 2026-04 velox was reinstalled Arch-on-ZFS. Values below reflect
+# that reinstall but remain TBD where a live probe has not yet confirmed.
+
+:PROPERTIES:
+:FS:         zfs
+:PM:         pacman
+:ORCH:       topgrade
+:SNAPSHOT:   sanoid
+:MESH:       tailscale
+:BACKUP:     source
+:VIRT:       libvirt,docker,podman
+:INIT:       systemd
+:LAST_AUDIT: 2026-05-26
+:END:
+
+* Overview
+
+| Field    | Value                           |
+|----------+---------------------------------|
+| Hostname | velox                           |
+|----------+---------------------------------|
+| Type     | Laptop                          |
+|----------+---------------------------------|
+| Maker    | Framework                       |
+|----------+---------------------------------|
+| Model    | Laptop (13th Gen Intel Core)    |
+|----------+---------------------------------|
+| Board    | FRANMCCP07 v A7                 |
+|----------+---------------------------------|
+| Firmware | INSYDE UEFI v03.07 (2024-12-26) |
+|----------+---------------------------------|
+| OS       | Arch Linux                      |
+|----------+---------------------------------|
+| Kernel   | 6.18.6-arch1-1                  |
+|----------+---------------------------------|
+| Desktop  | startx (X11)                    |
+|----------+---------------------------------|
+
+* CPU
+
+| Field         | Value                                   |
+|---------------+-----------------------------------------|
+| Model         | 13th Gen Intel Core i7-1370P            |
+|---------------+-----------------------------------------|
+| Architecture  | Raptor Lake (Intel 7 / 10nm)            |
+|---------------+-----------------------------------------|
+| Cores/Threads | 14 cores / 20 threads (6P + 8E, hybrid) |
+|---------------+-----------------------------------------|
+| Base/Boost    | 990 MHz / 5.2 GHz                       |
+|---------------+-----------------------------------------|
+| Cache L1      | 1.2 MiB (d-8x32K,6x48K; i-6x32K,8x64K)  |
+|---------------+-----------------------------------------|
+| Cache L2      | 11.5 MiB (6x1.2M, 2x2M)                 |
+|---------------+-----------------------------------------|
+| Cache L3      | 24 MiB (1x24M)                          |
+|---------------+-----------------------------------------|
+| Socket        | BGA1744 (U3E1)                          |
+|---------------+-----------------------------------------|
+
+* GPU
+
+| Field        | Value                  |
+|--------------+------------------------|
+| Model        | Intel Iris Xe Graphics |
+|--------------+------------------------|
+| Architecture | Xe (Intel 7)           |
+|--------------+------------------------|
+| Driver       | i915 / modesetting     |
+|--------------+------------------------|
+
+* Memory
+
+| Field    | Value                              |
+|----------+------------------------------------|
+| Total    | 64 GiB (DDR4)                      |
+|----------+------------------------------------|
+| Speed    | 3200 MT/s                          |
+|----------+------------------------------------|
+| Slots    | 2 (both populated)                 |
+|----------+------------------------------------|
+| Device 1 | 32 GiB Crucial CT32G4SFD832A.C16FB |
+|----------+------------------------------------|
+| Device 2 | 32 GiB Crucial CT32G4SFD832A.C16FE |
+|----------+------------------------------------|
+
+* Storage
+
+| Device       | Model                | Size     | Interface |
+|--------------+----------------------+----------+-----------|
+| /dev/nvme0n1 | Sabrent SB-RKT4P-8TB | 7.28 TiB | NVMe      |
+|--------------+----------------------+----------+-----------|
+
+- Filesystem: ZFS on root (zroot pool on nvme0n1p2; reinstalled 2026-04)
+- Datasets: zroot/ROOT/default, zroot/home, zroot/var/log, zroot/var/lib/pacman, zroot/var/lib/docker (per-layer), zroot/vms, zroot/media
+- Snapshots: sanoid hourly + zfs-scrub-weekly@zroot
+- SMART: PASSED (re-verified 2026-04-26); composite 43 °C; 0 media errors
+- Power-on history pre-dates reinstall — refresh on next full re-inventory
+
+* Display
+
+| Field      | Value                           |
+|------------+---------------------------------|
+| Panel      | BOE Display 0x0bca (built 2022) |
+|------------+---------------------------------|
+| Resolution | 2256x1504                       |
+|------------+---------------------------------|
+| Size       | 13.5" (285x190mm)               |
+|------------+---------------------------------|
+| DPI        | 201                             |
+|------------+---------------------------------|
+| Ratio      | 3:2                             |
+|------------+---------------------------------|
+
+* Network
+
+| Interface   | Chipset                        | Speed   | Type     |
+|-------------+--------------------------------+---------+----------|
+| wlp170s0    | Qualcomm Atheros AR9462        | Wi-Fi   | Wireless |
+|-------------+--------------------------------+---------+----------|
+| enp0s13f0u3 | Realtek USB 10/100/1G/2.5G LAN | 2.5 GbE | USB      |
+|-------------+--------------------------------+---------+----------|
+
+Also has: docker0, tailscale0, virbr0 (virtual interfaces)
+
+* Audio
+
+- Intel Raptor Lake-P/U/H cAVS (snd_hda_intel)
+- PipeWire v1.4.10 (pipewire-pulse, wireplumber)
+
+* Battery
+
+| Field     | Value                 |
+|-----------+-----------------------|
+| Capacity  | 55 Wh (design)        |
+|-----------+-----------------------|
+| Condition | 44/55 Wh (80% health) |
+|-----------+-----------------------|
+| Cycles    | 146                   |
+|-----------+-----------------------|
+| Type      | Li-ion                |
+|-----------+-----------------------|
+
+* Other Hardware
+
+- Webcam: Realtek Laptop Camera (UVC)
+- Fingerprint: Shenzhen Goodix USB Device
+- Bluetooth: Foxconn (BT 4.0)
diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org
new file mode 100644
index 0000000..8c3823e
--- /dev/null
+++ b/docs/workflows/system-health-check.org
@@ -0,0 +1,1034 @@
+#+TITLE: System Health Check Workflow
+#+AUTHOR: Craig Jennings & Claude
+#+DATE: 2026-02-27
+
+* Overview
+
+This workflow performs a comprehensive diagnostic scan of whatever host it runs on (ratio, velox, mybitch, or truenas via SSH). Unlike status-check.org (which monitors a single long-running job in real-time), this scans the entire system, produces a severity-ranked report, then investigates and proposes fixes one by one.
+
+Run this when Craig asks "how is <host> doing?", "check my system health", or as a periodic maintenance check.
+
+Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=.
+
+Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3.
+
+* Severity thresholds — the TOML is authoritative
+
+The graded severity thresholds in this workflow are defined in the installed thresholds file, shared with the =maint= console so the two can never drift:
+
+- Installed (read this): =~/.config/archsetup/maintenance-thresholds.toml=
+- Canonical seed: =configs/maintenance-thresholds.toml= in the archsetup repo
+
+Read the TOML at the start of Phase 1 and grade against its values. Where a check below names a threshold, it cites the TOML key (e.g. =storage.df_warn_pct=); any inline number is the seeded default kept for readability, and the TOML wins on disagreement. User overrides live in =~/.config/maint/curation.toml= and merge over the shipped layer.
+
+* Hosts & Capabilities
+
+The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift.
+
+Host inventory files (match by =#+HOSTNAME:= keyword inside each):
+
+| Host    | Inventory file                            | Notes                            |
+|---------+-------------------------------------------+----------------------------------|
+| ratio   | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]]  | Arch, Btrfs RAID1, pacman        |
+|---------+-------------------------------------------+----------------------------------|
+| velox   | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]]   | Arch, ZFS (re-installed 2026-04) |
+|---------+-------------------------------------------+----------------------------------|
+| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt                  |
+|---------+-------------------------------------------+----------------------------------|
+| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM        |
+|---------+-------------------------------------------+----------------------------------|
+
+Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift.
+
+* The Workflow
+
+** Phase 0: Capability Detection
+
+Before running any health checks, probe the live system to determine which checks apply, then cross-reference against the inventory file for this host.
+
+*** Probe the live system
+
+#+begin_src bash
+# Hostname — prefer hostnamectl (always present on systemd hosts), fall back to uname
+host=$(hostnamectl --static 2>/dev/null || uname -n)
+
+# Filesystem of /
+root_fs=$(findmnt -n -o FSTYPE /)
+
+# Any btrfs present?
+has_btrfs=$(findmnt -t btrfs -n 2>/dev/null | wc -l)
+
+# Any zfs pool imported?
+has_zfs=$(command -v zpool >/dev/null 2>&1 && zpool list -H -o name 2>/dev/null | wc -l || echo 0)
+
+# Package manager
+pm=none
+command -v pacman >/dev/null && pm=pacman
+command -v apt    >/dev/null && pm=apt
+command -v dnf    >/dev/null && pm=dnf
+
+# Orchestrator
+orch=none
+command -v topgrade >/dev/null && orch=topgrade
+
+# Snapshot tool
+snapshot=none
+command -v snapper >/dev/null && snapshot=snapper
+# ZFS-native snapshots present?
+[ "$has_zfs" -gt 0 ] && zfs list -t snapshot -H 2>/dev/null | head -1 | grep -q . && snapshot=zfs-native
+command -v sanoid >/dev/null && snapshot=sanoid
+
+# Mesh
+mesh=none
+command -v tailscale >/dev/null && mesh=tailscale
+
+# Backup role
+backup=none
+[ -f /var/log/rsyncshot.log ] && backup=source
+# Target role is declared by inventory, not probed (target hosts are discovered by their role, not a local artifact).
+
+# Virt / containers
+virt=""
+command -v virsh   >/dev/null && virt="${virt}libvirt,"
+command -v docker  >/dev/null && virt="${virt}docker,"
+command -v podman  >/dev/null && virt="${virt}podman,"
+virt="${virt%,}"
+[ -z "$virt" ] && virt=none
+
+# Init
+init=$(ps -p 1 -o comm=)
+
+echo "host=$host fs=$root_fs btrfs=$has_btrfs zfs=$has_zfs pm=$pm orch=$orch snapshot=$snapshot mesh=$mesh backup=$backup virt=$virt init=$init"
+#+end_src
+
+Present the probe result as a one-line capability summary at the top of the report.
+
+*** Cross-reference against inventory
+
+Locate the inventory file for this host:
+
+#+begin_src bash
+inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1)
+#+end_src
+
+If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check.
+
+If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe.
+
+*Canonical parser* (the naive awk range =/^\* Automated Capabilities/,/^\* /= does NOT work — the end pattern matches the start line, truncating the range to one line):
+
+#+begin_src bash
+get_inv() {
+  local key="$1" inv="$2"
+  awk -v key="$key" '
+    /^\* Automated Capabilities/ { in_section=1; next }
+    /^\* / && in_section { in_section=0 }
+    in_section && $0 ~ "^:"key":" {
+      sub("^:"key":[[:space:]]*", "")
+      sub("[[:space:]]+$", "")
+      print; exit
+    }
+  ' "$inv"
+}
+#+end_src
+
+Key-to-probed-variable mapping:
+
+| Key          | Probed variable |
+|--------------+-----------------|
+| =:FS:=       | =root_fs=       |
+|--------------+-----------------|
+| =:PM:=       | =pm=            |
+|--------------+-----------------|
+| =:ORCH:=     | =orch=          |
+|--------------+-----------------|
+| =:SNAPSHOT:= | =snapshot=      |
+|--------------+-----------------|
+| =:MESH:=     | =mesh=          |
+|--------------+-----------------|
+| =:BACKUP:=   | =backup=        |
+|--------------+-----------------|
+| =:VIRT:=     | =virt=          |
+|--------------+-----------------|
+| =:INIT:=     | =init=          |
+|--------------+-----------------|
+
+*** Announce-then-update on drift
+
+For each mismatch (excluding =TBD= on the inventory side, which means "not yet probed"):
+
+1. Print in the Phase 0 summary: =DRIFT: <host> <key>: inventory=X probed=Y — updating=
+2. Edit the inventory file: replace the drawer value with the probed value.
+3. Update =:LAST_AUDIT:= to today's date.
+
+Skip the write-back if the workflow is running on a host where the repo isn't checked out (e.g., truenas accessed via SSH). In that case, print the drift but flag it as =DRIFT (read-only host): update <file> manually=.
+
+*** Capability summary dictates which checks run
+
+Each check in the Reference section declares =Applies when: <capability>=. Phase 1 runs the check only if the capability is present; otherwise it prints =N/A — <capability> not present= for that check and moves on.
+
+** Phase 1: Scan & Report
+
+Run all checks from the Health Checks Reference below. Collect every finding into a ranked table sorted by severity:
+
+| Severity | Meaning                                                     |
+|----------+-------------------------------------------------------------|
+| CRITICAL | Service down, disk failing, backups not running             |
+|----------+-------------------------------------------------------------|
+| WARNING  | Disk space low, stale logs, failed units, overdue maint     |
+|----------+-------------------------------------------------------------|
+| INFO     | Informational (uptime, versions, counts) — no action needed |
+|----------+-------------------------------------------------------------|
+
+*Before presenting findings:* cross-reference each finding against the Known Issues Log at the bottom of this file. If a finding matches a known issue with a decided resolution (won't-fix, accepted-noise, fixed-in-config), annotate it as =KNOWN — <short note>= rather than presenting it as a fresh WARNING. This prevents re-investigating previously-decided items and keeps the report focused on actionable new state.
+
+Present the report as described in Report Format, then proceed to Phase 2.
+
+** Phase 2: Investigate & Fix
+
+After presenting the ranked report:
+
+1. Start with the highest-severity issue
+2. Investigate root cause with detailed commands
+3. Propose a fix, explain what it does
+4. Wait for Craig's approval before applying any changes
+5. Verify the fix worked
+6. Move to next issue
+7. Repeat until all CRITICAL and WARNING items are addressed
+8. INFO items are presented for awareness only (no action unless Craig wants to dig in)
+
+Handle issues one at a time — don't batch fixes together. Craig prefers to approve each individually.
+
+*** Defer Transient Warnings Until After Phase 3 Reboot
+
+If the Phase 1 pending-updates list includes any of these packages, =DEFER= Phase 2 deep-dive on network/DNS/VPN/uptime-accumulation warnings — investigate only if they survive the reboot:
+
+| Package (any of)          | Defers these warning categories                               |
+|---------------------------+---------------------------------------------------------------|
+| kernel (linux, linux-lts) | bgscan/wifi driver spam, firmware-related dmesg noise         |
+|---------------------------+---------------------------------------------------------------|
+| iproute2                  | VPN daemon failures, ip-rule/routing-related service failures |
+|---------------------------+---------------------------------------------------------------|
+| systemd                   | resolved/networkd/logind service anomalies                    |
+|---------------------------+---------------------------------------------------------------|
+| NetworkManager            | DNS reachability health checks, Tailscale DNS warnings        |
+|---------------------------+---------------------------------------------------------------|
+| wpa_supplicant            | WiFi connection/scan errors                                   |
+|---------------------------+---------------------------------------------------------------|
+
+*Rationale:* Warnings on a long-uptime system (>7 days) frequently reflect accumulated mid-transition state from earlier package updates that weren't rebooted into. A fresh reboot resolves these automatically. Investigation pre-reboot is wasted effort.
+
+*Procedure:* Mark the deferred warnings in the Phase 1 report as =DEFER-TO-POST-REBOOT=. Proceed directly from Phase 2 (investigating any non-deferrable warnings) to Phase 3. After reboot (see two-stage pattern in Phase 3), re-evaluate the deferred warnings — most will have vanished. Any that survive get the full Phase 2 treatment.
+
+** Phase 3: System Update
+
+Updates are separate from issue investigation. After all issues are addressed (or deferred per Phase 2's transient-warning rule):
+
+1. Review the pending update list (already gathered in Phase 1)
+2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware)
+3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices
+4. Check =/var/log/pacman.log= for recent failed transactions
+5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
+6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=)
+7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot
+
+*** Two-Stage Reboot Pattern (MANDATORY if Phase 3 installed kernel / iproute2 / systemd / NetworkManager)
+
+When a core networking or kernel package is updated, the workflow formally splits into two sessions:
+
+*Stage 1 — Pause before reboot:*
+1. Update =docs/session-context.org= with:
+   - What's been completed (Phase 1 findings, Phase 3 summary, packages bumped)
+   - What's deferred to post-reboot (the =DEFER-TO-POST-REBOOT= warnings from Phase 2)
+   - Remaining work: Phase 4, wrap-up
+2. Tell Craig to reboot
+3. End session (do NOT delete session-context.org — its presence signals an in-progress health check)
+
+*Stage 2 — Resume in new session post-reboot:*
+1. Read =docs/session-context.org= first
+2. Run post-reboot verification:
+   - =uname -r= → confirm new kernel is running
+   - =systemctl --failed= → confirm zero failed services
+   - =journalctl -p err -b= → count and compare to pre-reboot (sharp drop = transient warnings were real, accumulated noise)
+3. Re-evaluate the deferred warnings — most are gone without action
+4. Full Phase 2 treatment for any survivors
+5. Complete Phase 4, then wrap up (writes session entry to notes.org covering both halves as one unified session, spanning the reboot)
+6. Delete =session-context.org= only after wrap-up is committed
+
+*Rationale:* Trying to cram the whole flow into one session risks either (a) wasting time investigating transient warnings that would have self-healed, or (b) losing context if a crash happens during the reboot gap. The two-stage pattern is explicit about the pause and survives interruptions.
+
+** Phase 4: Resolve .pacnew Files
+
+After updates, check for and resolve .pacnew files:
+
+#+begin_src bash
+find /etc -name "*.pacnew" 2>/dev/null
+#+end_src
+
+For each .pacnew file:
+1. Diff the current config against the .pacnew version
+2. Determine if the current config is actively managed (e.g., by Reflector, or manually customized)
+3. If the current config is correct and the .pacnew adds nothing useful → delete the .pacnew
+4. If the .pacnew has meaningful changes → merge them into the current config, then delete the .pacnew
+5. Always explain the diff to Craig before deleting
+
+Common .pacnew files on ratio:
+- =/etc/pacman.d/mirrorlist.pacnew= — safe to delete; mirrorlist is managed by Reflector
+- =/etc/locale.gen.pacnew= — safe to delete if =en_US.UTF-8= is already uncommented
+
+* Health Checks Reference
+
+Run these checks in order. Each check produces one or more findings for the report.
+
+** 1. System Basics
+
+#+begin_src bash
+echo "Date:     $(date)"
+echo "Hostname: $(hostnamectl hostname)"
+echo "Kernel:   $(uname -r)"
+echo "Uptime:   $(uptime -p)"
+#+end_src
+
+Severity: INFO (always).
+
+** 2. Failed systemd Units
+
+#+begin_src bash
+systemctl --failed
+#+end_src
+
+- Any failed units → WARNING (CRITICAL if essential service like NetworkManager, fail2ban, cronie)
+- No failed units → INFO "All units healthy"
+
+** 3. Journal Errors (Current Boot)
+
+#+begin_src bash
+# Get total error count first
+journalctl -p err -b --no-pager | wc -l
+
+# Then separate known-noisy services from real errors
+journalctl -p err -b --no-pager | grep -v "Hands-Free Voice gateway" | tail -50
+#+end_src
+
+Known noisy patterns to filter when counting:
+- bluetoothd "Hands-Free Voice gateway" — paired device out of range, polls every 60s
+- pixman "_pixman_log_error" — benign Hyprland rendering debug message
+- xkbcomp "Errors from xkbcomp are not fatal" — X compatibility noise
+
+When bluetooth spam is found, identify the device with =bluetoothctl devices Paired= and offer to remove stale pairings.
+
+- Recurring or service-impacting errors → WARNING
+- Hardware errors (MCE, disk I/O) → CRITICAL
+- Noise-level errors (known benign) → skip, but note the noise source
+
+** 4. Kernel & Hardware Events (journalctl -k)
+
+Check #3 catches error-priority journal messages, but many hardware issues show up as
+patterns of lower-severity kernel messages (e.g., repeated USB disconnects, I/O retries).
+This check scans the kernel log for those patterns.
+
+*Why =journalctl -k -b= and not =dmesg=:* on systems that suspend (laptops especially), =dmesg -T= computes wall-clock timestamps using =current_realtime − CLOCK_MONOTONIC=, but kernel printk timestamps include suspend time. After a suspend-heavy uptime, dmesg displays timestamps drifted forward by the cumulative suspend duration (e.g., May 17 displayed for messages logged on May 10). =journalctl -k= stamps CLOCK_REALTIME at message-write time, so dates are always correct. Performance difference is ~100 ms vs ~10 ms — negligible at this cadence. (See the 2026-05-10 entry in the Known Issues Log.)
+
+#+begin_src bash
+# USB disconnect/reconnect cycles (repeated = hardware or power management issue)
+journalctl -k -b --no-pager | grep -c "USB disconnect"
+
+# If disconnects found, show the devices and timestamps
+journalctl -k -b --no-pager | grep "USB disconnect" | tail -20
+
+# USB protocol errors (error -71 = EPROTO, error -110 = timeout)
+journalctl -k -b --no-pager | grep -iE "usbhid.*error|usb.*error -[0-9]+" | tail -10
+
+# I/O errors on block devices
+journalctl -k -b --no-pager | grep -iE "I/O error|blk_update_request|Buffer I/O error" | tail -10
+
+# PCIe errors
+journalctl -k -b --no-pager | grep -iE "pcie.*error|AER|corrected error|uncorrected error" | tail -10
+
+# Machine Check Exceptions (CPU hardware errors)
+journalctl -k -b --no-pager | grep -iE "mce:|hardware error" | tail -10
+
+# GPU faults or resets
+journalctl -k -b --no-pager | grep -iE "amdgpu.*error|amdgpu.*fault|gpu reset|gpu hang" | tail -10
+
+# Thermal throttling
+journalctl -k -b --no-pager | grep -iE "thermal.*throttl|cpu.*throttl|prochot" | tail -10
+
+# ACPI errors (power management issues)
+journalctl -k -b --no-pager | grep -iE "acpi.*error|acpi.*warning" | tail -10
+
+# USB autosuspend state on internal hubs (Framework 16 specific)
+for d in /sys/bus/usb/devices/1-[234]; do
+    if [ -d "$d" ]; then
+        echo "$d: $(cat $d/product 2>/dev/null) | control=$(cat $d/power/control) | delay=$(cat $d/power/autosuspend_delay_ms)"
+    fi
+done
+#+end_src
+
+What to look for:
+- *USB disconnects*: A few at boot or after resume is normal. >5 during a session, or repeated disconnect/reconnect of the same device = problem. On Framework 16, the keyboard and numpad modules are internal USB — disconnects here mean input loss.
+- *Error -71 (EPROTO)*: USB protocol error, often caused by aggressive hub autosuspend. Check hub power settings.
+- *I/O errors*: Any I/O error on NVMe → immediate CRITICAL.
+- *MCE/hardware errors*: Always CRITICAL.
+- *GPU faults*: Occasional corrected errors are INFO. Resets or hangs are WARNING.
+- *Thermal throttling*: WARNING if active. Check if cooling is obstructed.
+- *USB hub autosuspend*: Internal hubs with =control=auto= and =delay=0= → WARNING (causes input disconnects on Framework 16).
+
+- Repeated USB disconnects of input devices → WARNING
+- USB protocol errors (EPROTO) → WARNING
+- I/O errors, MCE, or hardware errors → CRITICAL
+- GPU resets or hangs → WARNING
+- Thermal throttling active → WARNING
+- Aggressive hub autosuspend on internal devices → WARNING
+- Clean dmesg → INFO
+
+** 5. Disk Health (SMART)
+
+#+begin_src bash
+sudo smartctl -H /dev/nvme0n1
+sudo smartctl -H /dev/nvme1n1
+sudo smartctl -A /dev/nvme0n1 | grep -i "critical\|temperature\|wear\|error"
+sudo smartctl -A /dev/nvme1n1 | grep -i "critical\|temperature\|wear\|error"
+#+end_src
+
+- SMART overall health != PASSED → CRITICAL
+- NVMe wear > =storage.smart_wear_warn_pct= or temp > =storage.smart_temp_warn_c= (TOML; seeded 80% / 70°C) → WARNING
+- All healthy → INFO
+
+** 6. Disk Usage
+
+Dispatched by filesystem. Run every sub-check for which the capability is present — a host with both Btrfs and ZFS (hypothetical: ratio with a ZFS backup pool) runs 6a AND 6b.
+
+*** 6a. Btrfs Usage
+
+*Applies when:* =has_btrfs > 0=
+
+#+begin_src bash
+sudo btrfs filesystem usage /
+/usr/bin/df -h / /boot 2>/dev/null
+#+end_src
+
+Btrfs reports two numbers that can look confusing:
+- *Data allocated %* (e.g., 99.43%) — how full the allocated data chunks are. Btrfs allocates in chunks and can allocate more from the unallocated pool.
+- *df %* (e.g., 69%) — actual used space vs total device size. This is what matters.
+
+Check the "Device unallocated" line — as long as there's unallocated space, Btrfs can grow its data chunks.
+
+Severity rules:
+- df > =storage.df_crit_pct= (TOML; seeded 90%) → CRITICAL
+- df > =storage.df_warn_pct= (seeded 80%) → WARNING
+- df under the warn line → INFO (report both df% and unallocated space)
+
+*** 6b. ZFS Pool Health
+
+*Applies when:* =has_zfs > 0=
+
+#+begin_src bash
+# Quick health summary across all pools
+sudo zpool status -x                  # expect "all pools are healthy"
+
+# Full detail — state, errors, scrub, resilver
+sudo zpool status
+
+# Capacity, fragmentation, health per pool
+sudo zpool list -o name,size,alloc,free,capacity,health,frag
+
+# Per-dataset usage (top-level only; -d1 depth)
+sudo zfs list -o name,used,avail,refer -s used
+#+end_src
+
+Severity rules:
+- Pool state != ONLINE → CRITICAL
+- Any read/write/cksum error > 0 → CRITICAL
+- Capacity > =storage.zfs_capacity_crit_pct= (TOML; seeded 90%) → CRITICAL
+- Capacity > =storage.zfs_capacity_warn_pct= (seeded 80%) → WARNING (ZFS performance degrades)
+- Scrub age > =storage.zfs_scrub_crit_days= (seeded 60) → CRITICAL
+- Scrub age > =storage.zfs_scrub_warn_days= (seeded 35) → WARNING
+- Resilver in progress → INFO (but prominent in the report — mention drive being replaced)
+- Fragmentation > =storage.zfs_frag_info_pct= (seeded 50%) → INFO
+
+Parse scrub age from =zpool status= output: look for the =scan:= line — either =scrub repaired … in … with … errors on <date>= (completed) or =scrub in progress …= (running).
+
+*** 6c. Generic filesystem fallback
+
+*Applies when:* =root_fs= is ext4, xfs, or any other non-Btrfs non-ZFS filesystem
+
+#+begin_src bash
+/usr/bin/df -h / /home /boot 2>/dev/null
+#+end_src
+
+Severity rules:
+- > =storage.df_crit_pct= on any mounted partition → CRITICAL
+- > =storage.df_warn_pct= → WARNING
+- Normal → INFO
+
+** 7. Snapshots
+
+Dispatched by snapshot tool.
+
+*** 7a. Snapper (Btrfs)
+
+*Applies when:* =snapshot = snapper=
+
+#+begin_src bash
+snapper -c home list --columns number,date,description | head -5
+snapper -c home list --columns number,date,description | tail -5
+snapper -c home list | wc -l
+snapper -c home get-config | grep -iE "cleanup|TIMELINE_LIMIT|NUMBER_LIMIT"
+# oldest snapshot age (pile-up signal)
+oldest=$(sudo ls /home/.snapshots/ 2>/dev/null | grep -E '^[0-9]+$' | sort -n | head -1)
+[ -n "$oldest" ] && echo "oldest: $(sudo grep -oP '(?<=<date>).*(?=</date>)' "/home/.snapshots/$oldest/info.xml" 2>/dev/null | head -1)"
+#+end_src
+
+Severity rules:
+- Zero snapshots → WARNING
+- Cleanup disabled → WARNING
+- Snapshot count very high, OR oldest snapshot far older than intended retention (e.g. months back) → WARNING (pile-up)
+- =TIMELINE_LIMIT_MONTHLY= / =_YEARLY= set high (e.g. 10) on a large, churny subvolume → WARNING: that keeps ~10 months of monthly snapshots, which silently hoard space (this bit /home on 2026-05-26 — monthly snapshots back to Feb were the top space holders). Sane values are the TOML's =[snapshots]= =timeline_*= keys (seeded HOURLY 6, DAILY 7, WEEKLY 2, MONTHLY 2, QUARTERLY 0, YEARLY 0). Manual (=single=) snapshots are NOT touched by timeline cleanup — delete stale ones explicitly.
+- Normal range → INFO with count and date range
+
+*Disk-space deep-dive (on demand, only when /home is actually filling).* Per-snapshot reclaimable space is invisible without btrfs quotas. TEMPORARILY enable quota, rank by exclusive space, then DISABLE it again — quotas slow every snapshot deletion, so never leave them on:
+#+begin_src bash
+sudo btrfs quota enable /home && sudo btrfs quota rescan -w /home
+sudo btrfs qgroup show -re --sort=-excl /home | head -20   # top exclusive (reclaimable) holders
+sudo btrfs quota disable /home
+#+end_src
+Reality check: data deleted from the live fs but still in many old snapshots shows ~0 *exclusive* per snapshot (it's shared across them), and only frees once the whole chain holding it is pruned. So reclaiming a large deletion means pruning the old snapshots, not just the newest one.
+
+*** 7b. ZFS Snapshots
+
+*Applies when:* =snapshot = zfs-native= or =sanoid=
+
+#+begin_src bash
+# All snapshots, newest first
+sudo zfs list -t snapshot -o name,creation,used -s creation | tail -20
+
+# Count per pool
+sudo zfs list -t snapshot -H -o name | awk -F'@' '{print $1}' | sort | uniq -c
+
+# Oldest snapshot age per dataset (detect runaway retention)
+sudo zfs list -t snapshot -H -o name,creation -s creation | head -20
+
+# Auto-snapshot service status (zfs-auto-snapshot / sanoid / zrepl)
+systemctl list-timers --all 2>/dev/null | grep -Ei "zfs|sanoid|zrepl" || echo "no auto-snapshot timer"
+#+end_src
+
+Severity rules:
+- Zero snapshots on a dataset that should have them → WARNING
+- Snapshot count on any dataset > =snapshots.zfs_count_warn= (seeded 1000) → WARNING (runaway retention)
+- Snapshot space used > =snapshots.zfs_space_warn_pct= (seeded 20%) of pool capacity → WARNING (heavy divergence)
+- Auto-snapshot timer not running → WARNING
+- Normal → INFO (counts per pool, oldest snapshot age)
+
+*** 7c. No snapshot system
+
+*Applies when:* =snapshot = none=
+
+Report =INFO: no snapshot system configured on this host= and skip.
+
+** 8. Memory and Swap
+
+#+begin_src bash
+free -h
+journalctl -b --no-pager | grep -i "out of memory\|oom-kill\|killed process" | tail -10
+#+end_src
+
+Ratio has 128GB RAM and no swap configured. No swap is expected — don't flag it.
+
+- OOM kills found → WARNING
+- Normal → INFO
+
+** 9. CPU Temperatures
+
+#+begin_src bash
+sensors 2>/dev/null || echo "lm_sensors not installed or not configured"
+#+end_src
+
+Key sensors on ratio:
+- =k10temp= → CPU (Tctl)
+- =amdgpu= → GPU edge temp
+- =nvme= → NVMe drives (2 drives)
+- =cros_ec= → mainboard (power, memory, ambient, CPU)
+
+- CPU temp > =power.cpu_temp_warn_c= or GPU temp > =power.gpu_temp_warn_c= (TOML; seeded 90/95°C) → WARNING, CRITICAL if sustained
+- Normal → INFO (report CPU and GPU temps)
+
+** 10. NTP Sync
+
+#+begin_src bash
+chronyc tracking
+#+end_src
+
+Ratio uses chrony for NTP.
+
+- Clock not synchronized → WARNING
+- NTP service inactive → WARNING
+- Last offset > =network.ntp_offset_warn_ms= (TOML; seeded 100 ms) → WARNING
+- Synchronized → INFO
+
+** 11. rsyncshot Backups
+
+*Applies when:* =backup = source=
+
+#+begin_src bash
+tail -30 /var/log/rsyncshot.log 2>/dev/null
+#+end_src
+
+rsyncshot runs via root crontab:
+- Hourly: =30 0-1,3-23 * * *= (every hour except 2:30)
+- Daily: =30 2 * * *= (2:30 AM)
+
+Backups go to TrueNAS via rsync. Look for "rsyncshot completed successfully" and check the timestamp.
+
+- No daily backup in =backups.rsyncshot_daily_crit_hours= (seeded 48) → CRITICAL
+- No hourly backup in =backups.rsyncshot_hourly_warn_hours= (seeded 3) → WARNING (only on hosts whose log shows hourly runs)
+- Errors in log → WARNING
+- Recent and healthy → INFO
+
+** 12. Tailscale
+
+*Applies when:* =mesh = tailscale=
+
+#+begin_src bash
+tailscale status
+#+end_src
+
+Expected peers: ratio, cjennings, truenas, velox, worker. pixel6 is a phone and is often offline — don't flag it.
+
+- Tailscale not running → WARNING
+- Server peers (truenas, velox, worker) offline → WARNING
+- All expected peers online → INFO
+
+** 13. fail2ban
+
+#+begin_src bash
+sudo fail2ban-client status
+sudo fail2ban-client status sshd 2>/dev/null
+#+end_src
+
+- fail2ban not running → WARNING
+- Running → INFO (note any recent bans for awareness)
+
+** 14. Package Maintenance
+
+Dispatched by package manager.
+
+*** 14a. pacman (Arch)
+
+*Applies when:* =pm = pacman=
+
+#+begin_src bash
+# Orphaned packages
+pacman -Qtdq 2>/dev/null
+
+# Pending updates
+checkupdates 2>/dev/null
+
+# Package cache timer + current size
+systemctl list-timers paccache.timer --no-pager
+du -sh /var/cache/pacman/pkg 2>/dev/null
+
+# .pacnew files
+find /etc -name "*.pacnew" 2>/dev/null
+#+end_src
+
+Notes:
+- =paccache.timer= runs weekly keeping 3 versions — fine for routine pruning, but it only trims *old versions*. A cache can still balloon from *breadth* (many packages, ≤3 versions each), where =paccache -r= finds nothing to prune. On 2026-05-26 the cache was 17 GB and =paccache -r= pruned 0; the space came from deeper levers:
+  - =sudo paccache -ruk0= — remove cache for *uninstalled* packages (safe, re-downloadable).
+  - =sudo paccache -rk1= — keep only 1 version of installed packages (frees the most; less downgrade headroom).
+  Together those took 17 GB → 5.8 GB.
+- When removing orphans, pass package names as arguments (don't pipe through stdin — breaks snap-pac's snapper hook).
+- Review orphans before removing — some (like =rust=) warrant discussion (Craig switched to =rustup=).
+
+Severity rules:
+- Orphaned packages beyond the curated =curation.kept_orphans= set, in bulk (> ~20) → WARNING
+- Pending updates > =updates.pending_warn= (TOML; seeded 50) → WARNING
+- Unreviewed .pacnew files → WARNING
+- Package cache > =packages.cache_warn_gb= (TOML; seeded 10 GB) → INFO: suggest =paccache -ruk0= (uninstalled) and/or =paccache -rk1= (keep 1) for a deeper reclaim beyond the weekly keep-3.
+- Normal counts → INFO
+
+*** 14b. apt (Debian/Ubuntu/Mint)
+
+*Applies when:* =pm = apt=
+
+#+begin_src bash
+# Pending updates
+apt list --upgradable 2>/dev/null | tail -n +2
+
+# Auto-removable packages
+apt -s autoremove 2>&1 | grep -E "^Remv|packages will be REMOVED"
+
+# .dpkg-dist / .dpkg-new files (the apt equivalent of .pacnew)
+find /etc -name "*.dpkg-dist" -o -name "*.dpkg-new" 2>/dev/null
+
+# unattended-upgrades timer
+systemctl list-timers apt-daily.timer apt-daily-upgrade.timer --no-pager 2>/dev/null
+#+end_src
+
+Severity rules:
+- Pending security updates > 0 → WARNING
+- Auto-removable packages > 30 → WARNING
+- Unreviewed .dpkg-dist files → WARNING
+- Normal → INFO
+
+*** 14c. none (TrueNAS SCALE, other appliances)
+
+*Applies when:* =pm = none=
+
+Report =N/A — package maintenance handled by middleware= and skip. On TrueNAS SCALE, OS updates go through the TrueNAS UI (System → Update); user-level =apt= exists but is unsupported and shouldn't be used.
+
+*** 14d. Orchestrator (topgrade)
+
+*Applies when:* =orch = topgrade=
+
+If topgrade is present, Phase 3 uses it rather than the raw package manager. Phase 1 still runs 14a/14b/14c above — topgrade is for the update run, not the status scan.
+
+** 15. App Logs (~/.local/var/log)
+
+#+begin_src bash
+for log in ~/.local/var/log/*.log; do
+    if [ -f "$log" ]; then
+        errors=$(grep -ci "error\|critical\|fatal" "$log" 2>/dev/null)
+        if [ "$errors" -gt 0 ]; then
+            echo "$(basename $log): $errors error lines"
+            grep -i "error\|critical\|fatal" "$log" | tail -3
+            echo ""
+        fi
+    fi
+done
+#+end_src
+
+Known noise patterns:
+- *waybar*: LIBDBUSMENU-GLIB-WARNING "Unable to replace properties on 0" — caused by insync and zoom tray icons with broken dbusmenu implementations. Harmless. Filtered in waybar's log output via =grep -v= in hyprland.conf.
+- *hyprland*: "_pixman_log_error" and "xkbcomp" — benign
+- *gammastep*: "Wayland connection experienced a fatal error" — happens on session restart, self-recovers
+- *dunst*: gdk_pixbuf assertion failures — usually a malformed notification icon, one-off
+
+Look for patterns that are NOT in the known noise list.
+
+- Coredumps within =logs.coredump_window_days= (TOML; seeded 14 — =coredumpctl list= and age-filter) → WARNING
+- Recurring non-noise errors → WARNING
+- Only known noise → INFO
+
+** 16. System Logs (/var/log)
+
+#+begin_src bash
+# Recent pacman errors
+grep -i "error\|failed" /var/log/pacman.log 2>/dev/null | tail -10
+
+# fail2ban anomalies
+tail -50 /var/log/fail2ban.log 2>/dev/null | grep -i "error\|warning"
+#+end_src
+
+- Failed pacman transactions → WARNING
+- fail2ban errors → WARNING
+- Clean → INFO
+
+** 17. Docker/Podman
+
+*Applies when:* =virt= contains =docker= or =podman=
+
+#+begin_src bash
+docker ps -a --filter "status=exited" --format "{{.Names}}: exited {{.Status}}" 2>/dev/null
+docker system df 2>/dev/null
+podman ps -a --filter "status=exited" --format "{{.Names}}: exited {{.Status}}" 2>/dev/null
+podman system df 2>/dev/null
+#+end_src
+
+The WinVM podman container is run on-demand and will often show as exited — this is expected.
+
+*Severity thresholds:*
+- Stopped containers that should be running → WARNING
+- Docker/Podman reclaimable > =services.docker_reclaim_warn_gb= *or* > =services.docker_reclaim_warn_pct= of total (TOML; seeded 5 GB / 50%) → WARNING (propose prune tiers in Phase 2)
+- Docker/Podman reclaimable 1–5 GB → INFO (note for awareness)
+- Expected stopped containers → INFO
+
+*Prune tiers to propose when reclaimable crosses the WARNING threshold:*
+1. =docker container prune= — removes only stopped containers (~tens of MB, unlocks referenced images)
+2. =docker image prune -a= — removes all images not used by any container (biggest bite; requires tier 1 first)
+3. =docker system prune -a --volumes= — full nuke, includes unused volumes and networks. Present as the option when Craig wants to start fresh.
+
+Confirm with Craig before running any tier — especially if work-related containers (e.g. =deepsat-*=) are present.
+
+** 18. libvirt / VMs
+
+*Applies when:* =virt= contains =libvirt=
+
+#+begin_src bash
+sudo virsh list --all 2>/dev/null || echo "libvirt not available"
+#+end_src
+
+VMs (ultmos-15, etc.) are usually shut off unless Craig is actively using them. "shut off" is the normal state.
+
+- VMs in unexpected state → INFO
+- libvirt not running → INFO
+
+** 19. Cron Jobs
+
+#+begin_src bash
+systemctl is-active cronie
+crontab -l 2>/dev/null | grep -v "^#" | grep -v "^$"
+sudo crontab -l 2>/dev/null | grep -v "^#" | grep -v "^$"
+#+end_src
+
+Expected user crontab entries:
+- =0 12 * * *= log-cleanup
+
+Expected root crontab entries:
+- =30 0-1,3-23 * * *= rsyncshot hourly
+- =30 2 * * *= rsyncshot daily
+
+- cronie not running → WARNING
+- Expected crontab entries missing → WARNING
+- Running with expected entries → INFO
+
+** 20. Log Cleanup Cron
+
+#+begin_src bash
+find ~/.local/var/log -type f -name "*.log" -mtime +7 2>/dev/null | head -10
+find ~/.local/var/log -type f 2>/dev/null | wc -l
+#+end_src
+
+The user crontab runs =~/.local/bin/cron/log-cleanup= daily at noon, which should keep logs to ~7 days.
+
+- Log files older than =logs.app_log_warn_days= (TOML; seeded 7) present → WARNING (cleanup cron may not be running)
+- Within the retention window → INFO
+
+** 21. Network
+
+#+begin_src bash
+# DNS resolution (dig may not be available — use ping as fallback)
+ping -c1 -W2 archlinux.org 2>/dev/null && echo "DNS OK" || echo "DNS FAILED"
+
+# NetworkManager status
+nmcli general status 2>/dev/null
+#+end_src
+
+- DNS resolution fails → CRITICAL
+- NetworkManager not running or disconnected → CRITICAL
+- All healthy → INFO
+
+* Report Format
+
+Present findings as a ranked org-mode table, sorted by severity (CRITICAL first, then WARNING, then INFO):
+
+#+begin_example
+| # | Severity | Category    | Finding                        |
+|---+----------+-------------+--------------------------------|
+| 1 | CRITICAL | rsyncshot   | Daily backup failed 2 days ago |
+|---+----------+-------------+--------------------------------|
+| 2 | WARNING  | disk usage  | / at 85% capacity              |
+|---+----------+-------------+--------------------------------|
+| 3 | WARNING  | packages    | 47 orphaned packages           |
+|---+----------+-------------+--------------------------------|
+| 4 | INFO     | uptime      | 12 days                        |
+|---+----------+-------------+--------------------------------|
+| 5 | INFO     | disk health | Both NVMe drives PASSED        |
+|---+----------+-------------+--------------------------------|
+| 6 | INFO     | tailscale   | 3/3 peers online               |
+|---+----------+-------------+--------------------------------|
+#+end_example
+
+After the table, announce the investigation plan:
+
+- If CRITICAL or WARNING issues exist: "Starting with #1 — investigating [category]: [finding]..."
+- If only INFO items: "No issues found. System looks healthy. Here are the details..."
+
+* Known Issues Log
+
+Each entry is scoped to one host (or =any=). When Phase 1 cross-references findings against this log, it matches on both the issue signature AND the host — an entry tagged =:host: mybitch= won't suppress a finding on ratio. For pre-existing entries without an explicit host tag, the hostname is inferred from the entry title where possible (e.g., "mybitch —"). New entries should include =:host: <name>= for clarity.
+
+** 2026-03-24: mybitch — USB autosuspend causing keyboard/numpad disconnects
+:host: mybitch
+- Framework Laptop 16 internal Genesys Logic USB hubs (05e3:0610) had =control=auto= and =delay=0=
+- Caused repeated USB disconnects of keyboard module (1-4.2) and numpad module (1-3.2)
+- Error -71 (EPROTO) on reconnection attempts
+- Fix: udev rule at =/etc/udev/rules.d/99-framework-usb-hub-no-autosuspend.rules=
+- Sets =power/control=on= for matching hubs, disabling autosuspend
+- Added check #4 (Kernel & Hardware Events) to this workflow to catch similar issues in future
+
+** 2026-02-27: Bluetooth journal spam (1,006 errors/boot)
+:host: ratio
+- bluetoothd polling "Craig's Pixel Buds" (B8:7B:D4:19:A6:01) every 60s while out of range
+- Fix: removed stale pairing with =bluetoothctl remove B8:7B:D4:19:A6:01=
+- Note: Craig has recurring Pixel Buds pairing issues with this machine
+
+** 2026-02-27: hyprlock deprecated config options (v0.9.2)
+:host: ratio
+- =general:grace= and =general:no_fade_in= moved to CLI flags (=--grace=, =--no-fade-in=)
+- =input-field:fail_transition= replaced by =animations= block (=inputFieldColors= animation)
+- Fix: removed deprecated options, added =animations= block to hyprlock.conf
+- Config at: =~/code/archsetup/dotfiles/hyprland/.config/hypr/hyprlock.conf=
+
+** 2026-02-27: dunst coredump
+:host: ratio
+- Segfault in libglycin/gdk_pixbuf — malformed notification icon
+- One-off, no action taken. Monitor for recurrence.
+
+** 2026-02-27: Missing udev script for Logitech Brio
+:host: ratio
+- Udev rule references =~/.local/bin/logitech-brio-settings.sh= which is a symlink to archsetup dotfiles
+- Fails at boot because udev runs before home is fully available
+- Script works fine for hotplug after login
+- Decision: leave as-is
+
+** 2026-02-27: waybar LIBDBUSMENU-GLIB warnings (~8,000/session)
+:host: ratio
+- Caused by insync and zoom tray icons with broken dbusmenu implementations
+- Both return empty arrays for menu root item (ID 0)
+- No fix available (insync has no option to disable tray icon)
+- Mitigation: added =grep -v "LIBDBUSMENU-GLIB-WARNING"= filter to waybar log line in hyprland.conf
+- Takes effect on next Hyprland start
+
+** 2026-02-27: Package cleanup
+:host: ratio
+- Removed 51 orphaned packages (build deps, Python dev tools)
+- Swapped =rust= pacman package for =rustup= (toolchain manager)
+- Added note to =~/code/archsetup/inbox/rustup.txt=
+
+** 2026-02-27: System update via topgrade
+:host: ratio
+- 66 packages updated including Mesa 25.3→26.0, Firefox/Thunderbird 148, linux-firmware, Signal 8.0
+- No manual intervention required (checked Arch news)
+- Reboot recommended for firmware and Mesa changes
+
+** 2026-02-27: Resolved .pacnew files
+:host: ratio
+- =/etc/pacman.d/mirrorlist.pacnew= — stock mirrorlist, safe to delete (Reflector manages the active one)
+- =/etc/locale.gen.pacnew= — only added commented =en_SE.UTF-8=, current config correct, deleted
+
+** 2026-04-19: proton.VPN.service transient failure after topgrade
+:host: ratio
+- Appeared as a =failed= unit immediately after =topgrade= bumped iproute2 from 6.19.0 → 7.0.0 mid-session (9-day uptime)
+- Self-healed on reboot; no config change required
+- Pattern: VPN daemon holds routing/ip-rule state referencing old iproute2 ABI, can't reconcile after live package swap
+- Classification: KNOWN-TRANSIENT on iproute2 upgrade → DEFER-TO-POST-REBOOT per Phase 2 rules
+
+** 2026-04-19: Tailscale DNS reachability warning on long uptime
+:host: ratio
+- =tailscale status= footer: "Tailscale can't reach the configured DNS servers. Internet connectivity may be affected."
+- Appeared on 9-day uptime; all core functions (name resolution, netcheck, tailnet routing) tested working
+- Cleared on reboot — footer warning gone, no config change
+- Pattern: tailscaled's internal DNS health check accumulates stale state on long uptime, particularly after a mid-session iproute2 transition
+- Classification: KNOWN-TRANSIENT on long uptime / iproute2 upgrade → DEFER-TO-POST-REBOOT per Phase 2 rules
+
+** 2026-04-19: wpa_supplicant bgscan error spam volume scales with uptime
+:host: ratio
+- Error: =bgscan simple: Failed to enable signal strength monitoring=
+- Volume pre-reboot (9-day uptime): 1,200+ instances in journal
+- Volume post-reboot (fresh boot): 1 instance total
+- Root cause: RSSI monitoring ioctl not implemented in mt7925e driver (Framework Desktop WiFi 7 card) — each retry or roam event re-triggers the failure
+- 1–3 instances per boot is baseline noise; only triage if volume > ~50 AND not explainable by long uptime
+- Classification: KNOWN (mt7925e driver limitation, no upstream fix tracked); annotate low-volume occurrences as =KNOWN — mt7925e baseline=
+
+** 2026-04-19: Orphan package cleanup
+:host: ratio
+- Removed 11 packages (8 orphans + 3 cascading python build-tool deps): cli11, electron34, lua-lpeg, minizip-ng, python-build, python-hatchling, python-installer, yarn + python-editables, python-pyproject-hooks, python-trove-classifiers
+- Freed 280 MiB
+- Future note: electron34 and yarn were previously flagged "ask first" due to possible AUR build-dep relevance — post-removal, no AUR rebuild has failed. Can remove in future without asking unless an active AUR build references them.
+
+** 2026-04-21: velox ZFS — systemd-tmpfiles "Protocol driver not attached"
+:host: velox
+- Symptom: =systemd-tmpfiles-setup.service= (boot) and/or =systemd-tmpfiles-clean.service= (periodic) produce 10-30 =statx(...) failed: Protocol driver not attached= journal errors per run
+- Root cause: on ZFS, statx against another service's =/var/tmp/systemd-private-*/tmp= mount returns errno 132 (ENOTNAM); ext4/btrfs don't surface this as an error. Bare systemd-tmpfiles unit has no =PrivateTmp= set, so it traverses sibling namespaces
+- Fix: drop-in with =PrivateTmp=yes= at =/etc/systemd/system/systemd-tmpfiles-clean.service.d/zfs-private-tmp.conf= AND =.../systemd-tmpfiles-setup.service.d/zfs-private-tmp.conf=
+- Applies to any ZFS-on-root Arch host. Not needed on btrfs hosts.
+
+** 2026-04-19: Docker image bloat cleanup
+:host: ratio
+- Freed 15.3 GB via =docker system prune -a --volumes=
+- Removed 10 images (including nvidia/cuda 12.4 devel, postgis, nginx, python-slim, node-alpine, nerdfonts/patcher) and 4 exited deepsat-* work containers from 9 days prior
+- 130 MB orphan volume survived prune (anonymous, not caught by =--volumes=); ignored
+- Future: Check #17 now promotes reclaimable > 5 GB OR > 50% to WARNING — this will catch bloat earlier without Craig having to notice it
+
+** 2026-05-10: WiFi powersave default caused variable WiFi latency on mybitch + ratio
+:host: mybitch, ratio (velox naturally clean — defensive config added anyway)
+- Symptom: variable LAN/WiFi latency manifesting as "slow-feeling internet". On mybitch (Christine's laptop, the original report): gateway ping avg 67.9 ms, max 173.9 ms, mdev 55.7 ms at 1-second cadence vs avg 14.5 ms / mdev 16.4 ms at 0.2-second cadence (faster cadence kept the card awake, masking the issue). On ratio (Arch desktop): same pattern, mdev 46.7 ms, max 183.5 ms.
+- Root cause: NetworkManager defaults to =wifi.powersave = 3= (enable) when no override is configured. On Mint/Ubuntu, this is shipped explicitly as =/etc/NetworkManager/conf.d/default-wifi-powersave-on.conf=. On Arch, it's the upstream NetworkManager default (no shipped file, but same effective behavior). On bursty traffic (typical browsing), the WiFi card sleeps between bursts and the first packet of each new burst takes 50-150 ms to wake it — pages feel like they "stall" before loading.
+- Fix: drop =/etc/NetworkManager/conf.d/wifi-powersave-off.conf= with =wifi.powersave = 2= (disable). NetworkManager merges conf.d alphabetically; =wifi-...= sorts after =default-...= and =dns.conf= so the override wins. Restart NetworkManager applies it.
+- Verification: post-fix gateway ping (both mybitch and ratio) avg 12 ms, max 21-24 ms, mdev 3 ms — 15-18× improvement in jitter, 8-9× improvement in max latency. Throughput unchanged at the ISP/router ceiling (~85 Mbit/s down, 25 Mbit/s up — same on both clients, so that's the connection limit, not a per-host issue).
+- Velox edge: velox's WiFi card defaults to powersave-off at the driver/kernel level even without an explicit NM config (different chipset behavior). Latency was already healthy. Defensive =wifi-powersave-off.conf= added anyway so the configuration is uniform across the homelab and protected against future NM-default or driver-behavior changes.
+- Tradeoff: ~0.5-1 W more power draw — irrelevant on AC, slight battery hit on battery. All three hosts are mostly on AC.
+- Coverage status as of this entry:
+  - mybitch (Mint): explicit override applied, was active issue → resolved
+  - ratio (Arch): explicit override applied, was active issue → resolved
+  - velox (Arch laptop): explicit override applied defensively, no observable issue
+  - truenas: not on WiFi, N/A
+- Pattern note: any new NM-using host added to the homelab should get this override at provisioning time. The 2026-04-30 mybitch upgrade soak picked up the issue indirectly; future host adds should set this proactively.
+
+** 2026-05-10: mybitch — keyboard soak verified closed (2026-03-24 USB hub fix held over 9-day uptime)
+:host: mybitch
+- Reminder from 2026-04-30 (overdue day 10) was to verify the 2026-03-24 USB-autosuspend fix holds in routine use.
+- Verification (2026-05-10, mybitch uptime 1w 2d 2h): all three Framework 16 internal hubs (1-2, 1-3, 1-4) show =control=on, delay=0= — udev rule active. Total USB error count this boot: *0*. Soak passed.
+- Closes the 2026-04-30 reminder. The 2026-03-27 mybitch BIOS-side fixes (which were gated on this soak result) are now unblocked.
+
+** 2026-05-10: dmesg -T displays future-dated timestamps after suspend cycles
+:host: any (laptops affected; desktops rarely suspend)
+- Symptom: =dmesg -T= shows wall-clock timestamps drifted forward by the cumulative suspend duration since boot. Example: mybitch on 2026-05-10 with 9d 2h monotonic uptime displayed messages dated May 15-17 (~7 days of suspend during the boot session pushed the displayed dates 7 days into the future).
+- Root cause: kernel printk timestamps include suspend time (boottime-style clock). =dmesg -T= computes the boot epoch using =current_realtime − CLOCK_MONOTONIC=, but =CLOCK_MONOTONIC= excludes suspend. The two clocks diverge by however long the system was suspended during the current boot, and =dmesg -T= adds the (boottime-correct) message offset to a (monotonic-derived) boot epoch — producing future-dated displays.
+- Verification: =journalctl -k -b= shows the same kernel messages with correct timestamps because journald stamps =CLOCK_REALTIME= at message-write time. =date=, =timedatectl=, and =/proc/stat btime= are all correct.
+- Workflow fix (applied 2026-05-10): Phase 1 check #4 now uses =journalctl -k -b= instead of =dmesg=. Tradeoff is ~100 ms vs ~10 ms per query — negligible at this cadence. Side benefits: cross-boot queries (=-b -1=, =-b -2=) and longer history retention than the kernel ring buffer.
+- Live cleanup (optional): a reboot resets both clocks and =dmesg -T= comes back accurate until the next long suspend session. Not urgent — the workflow change routes around the bug regardless.
+- Out of scope: the dmesg tool itself isn't going to change behavior here; this is a long-standing util-linux design decision around how to translate ring-buffer timestamps. We don't fight it; we use the right tool.
+
+** 2026-05-10: aardvark-dns "empty response" spam during WinVM podman runs
+:host: ratio
+- Pattern: =aardvark-dns[N]: <port> dns request got empty response= logged at error priority while the podman-rootless aardvark-dns daemon forwards DNS for the WinVM container
+- Volume: ~1,200 lines per WinVM session (Windows guest aggressively retries DNS lookups, every retry that gets back NOERROR/empty triggers one line)
+- Trigger: only logs while a podman container with the rootless aardvark-dns network is running. Daemon stops with the container, so journal noise is bounded by WinVM uptime
+- Active when looked at: aardvark-dns daemon is not running between WinVM sessions; these errors are retrospective journal entries, not a live issue
+- Classification: KNOWN — annotate as =KNOWN — WinVM podman aardvark-dns DNS retry noise= and check the surrounding podman/WinVM lifecycle to confirm correlation. Volume scales with WinVM session length.
+- No fix in scope. Suppression options if it becomes annoying: (a) journald filter rule for unit pattern, (b) switch WinVM to a different DNS path. Neither pursued today.
+
+** 2026-05-10: cameractrls cameraptzmidi.py SEGV during Python 3.14 exit
+:host: ratio
+- =python3 /usr/lib/python3.14/site-packages/CameraCtrls/cameraptzmidi.py -l= prints its output (e.g. "JDS Labs Element IV MIDI 1:32:0") and then SEGVs during interpreter shutdown
+- Stack trace lives entirely inside =libpython3.14.so= on the =Py_Exit= path — no cameractrls frames, no asound frames at the crash point
+- Pattern: ctypes-loaded =libasound= atexit handler vs Python 3.14 finalization order; the script's job completes before the crash so functional behavior is fine
+- Reproduces every invocation. Three coredumps on 2026-05-08 11:49–11:50 were Craig running it three times in a row, not a regression burst
+- Classification: KNOWN (upstream Python 3.14 / cameractrls issue, no local fix). Annotate future cameraptzmidi coredumps as =KNOWN — Py_Exit / libasound finalization=
+- Real fix is upstream: Python 3.14 ctypes-finalization change or cameractrls explicit asound cleanup before exit. Not worth tracking locally.
+
+** 2026-05-11: mybitch — hard freeze caused by amdgpu iGPU MES hang; mitigated with =amdgpu.cwsr_enable=0=
+:host: mybitch
+- Symptom: full system freeze during active GUI use (Christine typing in a browser). Desktop frozen, keyboard + trackpad dead, off the network. Not the s2idle USB-keyboard bug — the machine was awake, not resuming from suspend, and the WiFi (M.2 PCIe, not USB) was dead too, so a USB-hub-only failure is excluded. Required hard power-cycle. =last= records the prior session as ending in "crash".
+- Evidence in =journalctl -b -1=: the journal *stops dead* with no panic / oops / hung-task warning / OOM / GPU-reset trace — classic hard hang (kernel wedged, logging stopped). pstore empty. Preceding the freeze: =amdgpu 0000:c5:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)= + =failed to reg_write_reg_wait= — *38 occurrences* over the 10-day boot, *zero* in every prior boot including the pre-upgrade 6.8.0-110 stretch. Last one ~4.7 min before the hang. =c5:00.0= is the *integrated* GPU (Radeon 780M / Phoenix1, GFX 11.0.x); =03:00.0= is the dGPU (RX 7700S) — the dGPU's =PSP/SMU is resuming= log lines are routine runtime-PM, not errors.
+- Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986.
+- Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot.
+- Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean.
+- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.)
+- *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=.
+- *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug).
+- Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111.
+
+** 2026-05-11: mybitch — power-profiles-daemon was stuck on power-saver (now balanced); benign dGPU power-limit error
+:host: mybitch
+- Symptom: =amdgpu 0000:03:00.0: amdgpu: New power limit (30) is out of range [100,120]= + =amdgpu: Failed to set power limit value= at error priority, once per boot (and on profile changes).
+- Root cause: =power-profiles-daemon= in the =power-saver= profile tried to cap the discrete RX 7700S (=03:00.0=) at 30W, but the GPU's settable power-cap range is [100W, 120W] (current cap 100W). amdgpu rejected the out-of-range request → cosmetic error; the dGPU stayed at its 100W floor regardless. Underlying issue: PPD persists the last-set profile in =/var/lib/power-profiles-daemon/state.ini= and restores it on every boot — someone had set =power-saver= long ago (state.ini mtime Oct 2025) and it had been pinned there ever since. Christine runs mybitch on AC a lot, so this wasn't what she wanted.
+- Resolution (2026-05-11): set the profile to =balanced=. =powerprofilesctl set= over SSH is polkit-denied (=switch-profile= needs an active local session), so done by editing =state.ini= directly under sudo (=Profile=power-saver= → =Profile=balanced=) with =power-profiles-daemon= stopped, then restarting it. Verified =powerprofilesctl get= → =balanced=; persists across reboots; no new power-limit errors. Christine can still flip the profile anytime via the Cinnamon power applet.
+- Classification: if the =New power limit (30) out of range= error reappears, it means the profile drifted back to =power-saver= — annotate as =KNOWN — PPD power-saver dGPU cap rejection= and re-set to =balanced= the same way. Harmless either way; the only reason to fix it is that power-saver throttles the box on AC.
+
+** 2026-05-18: velox — NVMe Error Information Log entries from kernel optional-feature probing
+:host: velox
+- Symptom: =smartctl -A /dev/nvme0n1= shows =Error Information Log Entries: 9,530= and climbing slowly over time, despite =Media and Data Integrity Errors: 0= and =SMART overall-health PASSED=.
+- Detail (=smartctl -l error /dev/nvme0n1=): all 9,530 entries are a single error class — =Status 0x4004 "Invalid Field in Command"= on the Admin Submission Queue (SQId 0). One row in the error info log, accumulated over 20,079 power-on hours (~2.3 years of drive uptime).
+- Root cause: kernel or userspace polls for an optional NVMe feature/log page the drive doesn't implement; the controller rejects with =Status 0x4004=, the host sees ENOTSUP and moves on, but the drive faithfully logs every rejection. Common on NVMe drives where newer kernels probe for features added after the drive's firmware was written.
+- Classification: KNOWN — annotate future findings as =KNOWN — NVMe Invalid-Field-in-Command from kernel optional-feature probing=. The counter going up is not a failure signal; only =Media and Data Integrity Errors=, =Critical Warning=, or =Available Spare= changes indicate actual drive health regression.
+- Out of scope: no kernel-side fix worth chasing. The probe is harmless and would have to be tracked down to a specific kernel subsystem; not worth the time when the only effect is a benign counter.
+
+** 2026-05-25: mybitch — timeshift SIGSEGV on =--check --scripted= cron runs (GObject teardown race)
+:host: mybitch
+- Symptom: =coredumpctl= shows intermittent SIGSEGV of =/usr/bin/timeshift= during the hourly =timeshift --check --scripted= cron job (e.g. 2026-05-23 02:00, 2026-05-24 23:00 — 2 in a 3-day window). Crash is at address =0x30= inside =libgobject-2.0.so.0= =g_object_unref=, with a second thread in =g_file_get_contents= — a multi-threaded GObject teardown race.
+- Timing: the process runs ~29 s (started :01, segfault :30) — it completes its work and dies during cleanup/exit, the same "job done, crash on teardown" shape as the cameractrls Py_Exit case.
+- Backups unaffected: hourly =backup.log= files in =/var/log/timeshift/= are present every hour, =--list= works, 50 snapshots on disk. timeshift rsyncs to a temp dir then promotes, so a crashed run leaves at worst an incomplete dir the next run ignores — not a corrupt snapshot.
+- Long-standing, not a regression: an older coredump from 2026-01-06 has the identical two-thread =g_object_unref= / =g_file_get_contents= signature. Version: =timeshift 25.12.4+zena= (Mint's maintained fork); apt shows 0 pending so no newer build to move to.
+- Classification: KNOWN — annotate future findings as =KNOWN — timeshift zena teardown SIGSEGV on --check=. A real escalation signal would be missing hourly =backup.log= entries or a gap in the snapshot timeline; the coredumps alone are cosmetic. No fix in scope (upstream zena-fork bug); suppression isn't worth it.
+
+** 2026-06-13: ratio — dockerized telega-server SIGSEGVs in musl build
+:host: ratio
+- Symptom: =coredumpctl= shows repeated SIGSEGVs from =telega-server -O 31 -l /home/cjennings/.telega/telega-server.log -v 3= running inside the Telega Docker container. Example stack is entirely in the container's musl loader plus =/usr/bin/telega-server=, not host kernel, GPU, storage, or memory paths.
+- Evidence: =~/.telega/telega-server.log= ends with =Unexpected char 'm' in plist value= followed by =Assertion failed: false (telega-dat.c: tdat_plist_value: 500)=. Surrounding TDLib traffic includes sticker/custom-emoji metadata such as =documentAttributeCustomEmoji= and =PhotoSizeSourceThumbnail[Thumbnail, type = m]=.
+- Prior local triage: =~/.emacs.d/todo.org= recorded the same issue on 2026-06-11 as spontaneous memory-corruption crashes in =zevlg/telega-server:latest='s musl build, with several coredumps occurring without action-verb traffic. Telega package installed at the 2026-06-13 health check was =20260513.509=; MELPA had =20260604.2321= available.
+- Functional status at 2026-06-13 check: no coredumps yet that day; Telegram scans still worked from cached chat state. Treat as an app/server-container crash, not a machine-health fault.
+- Classification: KNOWN — annotate future =telega-server= coredumps on ratio as =KNOWN — dockerized telega-server musl SIGSEGV= if the signature matches =tdat_plist_value= / unexpected plist value or otherwise stays inside the Telega container. Escalate only if crashes become continuous, break Telegram workflows, or appear after moving off the Docker musl build.
+- Deferred remediation options, in order of least disruption: update the Emacs =telega= package, rebuild/pull a newer =telega-server= image, pin a known-good pre-2026-06 image digest, build =telega-server= natively, or report upstream with =coredumpctl= and log evidence.