From 76de64ac803c336252ac225f1f493906a7eacd07 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Thu, 9 Jul 2026 12:58:56 -0500 Subject: docs: move the host inventories into the org-roam knowledge base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four host capability inventories, the TrueNAS hardware specs, and the ratio USB/xHCI record now live as roam nodes under ~/org/roam/hardware/, linked from the Homelab Hardware Inventory index. The copies here are gone. Carried rather than pointed at. A pointer keeps one canonical copy, but a third project that discovers a durable hardware fact would then have to write into this repo to record it, which the cross-project rules forbid — so the fact would land in an inbox instead of on the device's page. Moving keeps one canonical copy without closing the write path. system-health-check.org resolves a host's inventory by its #+HOSTNAME: keyword under ${ROAM_DIR:-$HOME/org/roam}/hardware/, never by filename: a node's timestamp prefix does not survive a rename. A host with no roam clone takes the same NO INVENTORY FILE path it always did. strix-soak-watch.org lands here from home, beside the workflow whose Phase 3 invokes it. It is a workflow, not an inventory page. --- .../2026-01-05-truenas-hardware-specs.org | 229 -------------------- docs/homelab-inventory/mybitch-laptop.org | 238 --------------------- docs/homelab-inventory/ratio-desktop.org | 175 --------------- docs/homelab-inventory/truenas-server.org | 114 ---------- docs/homelab-inventory/velox-laptop.org | 158 -------------- docs/workflows/strix-soak-watch.org | 146 +++++++++++++ docs/workflows/system-health-check.org | 38 ++-- todo.org | 40 ++++ 8 files changed, 207 insertions(+), 931 deletions(-) delete mode 100644 docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org delete mode 100644 docs/homelab-inventory/mybitch-laptop.org delete mode 100644 docs/homelab-inventory/ratio-desktop.org delete mode 100644 docs/homelab-inventory/truenas-server.org delete mode 100644 docs/homelab-inventory/velox-laptop.org create mode 100644 docs/workflows/strix-soak-watch.org diff --git a/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org b/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org deleted file mode 100644 index 42d33ad..0000000 --- a/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org +++ /dev/null @@ -1,229 +0,0 @@ -#+TITLE: TrueNAS Server Hardware Specifications -#+AUTHOR: Auto-generated via SSH -#+DATE: 2026-01-05 - -* Package Management Note - -**IMPORTANT:** Package management tools (apt) are disabled on TrueNAS appliances. Attempting to install packages with apt or methods other than the TrueNAS web interface can result in a nonfunctional system. - -All software installation and updates must be done through the TrueNAS web interface. - -* System Information - -** Hostname: truenas -** IP Address: 192.168.86.5 (static) -** Network Interface: eno1 -** Kernel: Linux 6.12.33-production+truenas -** Build Date: Wed Dec 17 21:17:21 UTC 2025 -** Architecture: x86_64 GNU/Linux - -* CPU Specifications - -** Model: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz -** Architecture: x86_64 (Coffee Lake S) -** Cores: 8 physical cores -** Threads: 16 (2 threads per core) -** Base Clock: 3.60 GHz -** Max Turbo: 5.00 GHz -** Min Clock: 800 MHz - -** Cache: -- L1d cache: 256 KiB (8 instances) -- L1i cache: 256 KiB (8 instances) -- L2 cache: 2 MiB (8 instances) -- L3 cache: 16 MiB (1 instance) - -** Features: -- Virtualization: VT-x -- CPU family: 6 -- Model: 158 -- Stepping: 13 - -* Memory Specifications - -** Total RAM: 64 GiB (62 GiB usable) -** Swap: 0 B (no swap configured) -** Current Usage: ~3.0 GiB used, 59 GiB free - -* Storage Controllers - -** SATA Controllers: -1. Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) -2. ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) - -** NVMe Controllers: -1. Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) - -* Block Devices - -** Hard Drives: -- sda: 10.9 TB (ZFS member) -- sdb: 10.9 TB (ZFS member) -- sdc: 10.9 TB (ZFS member) -- sdd: 10.9 TB (ZFS member) -- sdg: 18.2 TB (ZFS member) -- sdh: 18.2 TB (ZFS member) -- sdi: 18.2 TB (ZFS member) -- sdj: 18.2 TB (ZFS member) - -** System Drives: -- sde: 1.9 TB (Boot pool, ZFS member) - - sde1: 1M partition - - sde2: 512M VFAT partition - - sde3: 1.9T ZFS partition -- sdf: 1.9 TB (ZFS member) - -** NVMe: -- nvme0n1: 1.8 TB - -** Total Raw Capacity: -- 4× 10.9 TB = 43.6 TB -- 4× 18.2 TB = 72.8 TB -- 2× 1.9 TB = 3.8 TB -- 1× 1.8 TB NVMe = 1.8 TB -- **Total: ~122 TB raw storage** - -* Network Interfaces - -** eno1 (Primary - Active): -- Type: Ethernet controller - Intel Corporation Ethernet Connection (7) I219-V (rev 10) -- MAC: 70:85:c2:db:9d:94 -- IP: 192.168.86.5/21 -- Subnet broadcast: 192.168.87.255 -- State: UP -- MTU: 1500 -- Alternative name: enp0s31f6 - -** enp2s0 (Secondary - Down): -- Type: Ethernet controller - Intel Corporation I211 Gigabit Network Connection (rev 03) -- MAC: 70:85:c2:db:9d:94 -- State: DOWN (NO-CARRIER) - -** Wireless: -- Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) -- Status: Not configured - -** Docker Bridge: -- docker0: 172.16.0.1/24 -- State: DOWN (NO-CARRIER) - -* Motherboard/Chipset - -** Chipset: Intel Z390 -- Host Bridge: 8th/9th Gen Core 8-core Desktop Processor (Coffee Lake S) -- ISA Bridge: Z390 Chipset LPC/eSPI Controller (rev 10) -- PCIe Root Ports: Multiple Cannon Lake PCH PCI Express ports - -** Integrated Components: -- Graphics: Intel CoffeeLake-S GT2 [UHD Graphics 630] (rev 02) -- Audio: Intel Cannon Lake PCH cAVS (rev 10) -- USB: Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) -- Thermal: Cannon Lake PCH Thermal Controller (rev 10) -- HECI: Cannon Lake PCH HECI Controller (rev 10) -- SMBus: Cannon Lake PCH SMBus Controller (rev 10) -- SPI: Cannon Lake PCH SPI Controller (rev 10) - -* ZFS Pool Configuration - -** boot-pool (1.86 TB): -- Allocated: 3.06 GB -- Free: 1.86 TB -- Capacity: 0% -- Health: ONLINE - -** sysdata (1.86 TB): -- Allocated: 314 MB -- Free: 1.86 TB -- Capacity: 0% -- Health: ONLINE -- Purpose: System data and applications - -** tank (72.8 TB): -- Allocated: 1.17 MB -- Free: 72.7 TB -- Capacity: 0% -- Health: ONLINE -- Purpose: Large storage pool (empty, ready for expansion) - -** vault (43.6 TB): -- Allocated: 24.5 TB -- Free: 19.1 TB -- Capacity: 56% -- Health: ONLINE -- Purpose: Main media storage - -* Vault Datasets - -** Media (Movies, Music, TV Shows): -- Size: 30 TB (allocated) -- Used: 17 TB -- Available: 14 TB -- Capacity: 54% - -** Lectures: -- Size: 15 TB (allocated) -- Used: 878 GB -- Available: 14 TB -- Capacity: 6% - -** Audiobooks: -- Size: 14 TB (allocated) -- Used: 89 GB -- Available: 14 TB -- Capacity: 1% - -** Magic: -- Size: 15 TB (allocated) -- Used: 543 GB -- Available: 14 TB -- Capacity: 4% - -** Books: -- Size: 14 TB (allocated) -- Used: 177 GB -- Available: 14 TB -- Capacity: 2% - -* Security Mitigations - -** CPU Vulnerabilities (Status): -- Gather data sampling: Mitigated (Microcode) -- Indirect target selection: Mitigated (Aligned branch/return thunks) -- Itlb multihit: KVM Mitigation (Split huge pages) -- L1tf: Not affected -- Mds: Not affected -- Meltdown: Not affected -- Mmio stale data: Mitigated (Clear CPU buffers; SMT vulnerable) -- Reg file data sampling: Not affected -- Retbleed: Mitigated (Enhanced IBRS) -- Spec rstack overflow: Not affected -- Spec store bypass: Mitigated (Disabled via prctl) -- Spectre v1: Mitigated (usercopy/swapgs barriers) -- Spectre v2: Mitigated (Enhanced/Automatic IBRS) -- Srbds: Mitigated (Microcode) -- Tsx async abort: Mitigated (TSX disabled) - -* Full PCI Device List - -#+BEGIN_EXAMPLE -00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d) -00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d) -00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02) -00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) -00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) -00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) -00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) -00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) -00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0) -00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0) -00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0) -00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10) -00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) -00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) -00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) -00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10) -01:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) -02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) -03:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) -04:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) -#+END_EXAMPLE diff --git a/docs/homelab-inventory/mybitch-laptop.org b/docs/homelab-inventory/mybitch-laptop.org deleted file mode 100644 index e522c40..0000000 --- a/docs/homelab-inventory/mybitch-laptop.org +++ /dev/null @@ -1,238 +0,0 @@ -#+TITLE: mybitch - Christine's Laptop -#+DATE: 2026-01-31 -#+HOSTNAME: mybitch - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. - -:PROPERTIES: -:FS: ext4 -:PM: apt -:ORCH: none -:SNAPSHOT: timeshift -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt -:INIT: systemd -:LAST_AUDIT: 2026-06-23 -:END: - -* Overview - -| Field | Value | -|----------+-----------------------------------| -| Hostname | mybitch | -|----------+-----------------------------------| -| Type | Laptop | -|----------+-----------------------------------| -| Owner | Christine | -|----------+-----------------------------------| -| Maker | Framework | -|----------+-----------------------------------| -| Model | Laptop 16 (AMD Ryzen 7040 Series) | -|----------+-----------------------------------| -| OS | Linux Mint 22.3 (Zena) | -|----------+-----------------------------------| -| Kernel | 6.17.0-23-generic | -|----------+-----------------------------------| -| Status | In use | -|----------+-----------------------------------| - -* Hardware Specifications - -** CPU - -| Field | Value | -|--------------+-----------------------------------| -| Model | AMD Ryzen 9 7940HS w/ Radeon 780M | -|--------------+-----------------------------------| -| Architecture | Zen 4 (TSMC 5nm) | -|--------------+-----------------------------------| -| Cores | 8 | -|--------------+-----------------------------------| -| Threads | 16 | -|--------------+-----------------------------------| -| Base Clock | 4.0 GHz | -|--------------+-----------------------------------| -| Boost Clock | 5.25 GHz | -|--------------+-----------------------------------| -| L2 Cache | 8 MiB (8x1024 KiB) | -|--------------+-----------------------------------| -| L3 Cache | 16 MiB | -|--------------+-----------------------------------| - -** Memory - -| Field | Value | -|-----------+------------------------------| -| Total | 32 GB | -|-----------+------------------------------| -| Type | DDR5-5600 | -|-----------+------------------------------| -| Installed | 1x 32GB (A-DATA AD5S560032G) | -|-----------+------------------------------| -| Slots | 2 (1 free) | -|-----------+------------------------------| -| Max | 64 GB | -|-----------+------------------------------| - -** Graphics - -| Device | Model | Driver | Notes | -|------------+---------------------+--------+-----------------| -| Dedicated | AMD Radeon RX 7700S | amdgpu | RDNA 3, Navi 33 | -|------------+---------------------+--------+-----------------| -| Integrated | AMD Radeon 780M | amdgpu | RDNA 3, Phoenix | -|------------+---------------------+--------+-----------------| - -** Storage - -| Device | Model | Size | Interface | -|--------------+---------------------+----------+--------------| -| /dev/nvme0n1 | WD BLACK SN770M 2TB | 1.82 TiB | PCIe Gen4 x4 | -|--------------+---------------------+----------+--------------| -| /dev/nvme1n1 | WD BLACK SN770 2TB | 1.82 TiB | PCIe Gen4 x4 | -|--------------+---------------------+----------+--------------| -| Total | | 3.64 TiB | | -|--------------+---------------------+----------+--------------| - -*** Partitions - -| Partition | Size | Used | Filesystem | Mount | Label | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme1n1p2 | 1.82 TiB | 17.4% | ext4 | / | | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme1n1p1 | 512 MiB | 1.2% | vfat | /boot/efi | | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme0n1p1 | 1.82 TiB | (unmounted) | ext4 | | Backup | -|----------------+----------+-------------+------------+-----------+--------| - -** Display - -| Field | Value | -|------------+--------------------| -| Panel | BOE Display 0x0bc9 | -|------------+--------------------| -| Resolution | 2560x1600 | -|------------+--------------------| -| Size | 16" (345x215mm) | -|------------+--------------------| -| Ratio | 16:10 | -|------------+--------------------| -| DPI | 188 | -|------------+--------------------| - -** Battery - -| Field | Value | -|----------+------------------| -| Model | NVT FRANDBA | -|----------+------------------| -| Capacity | 85.1 Wh (design) | -|----------+------------------| -| Current | 87.5 Wh (102.9%) | -|----------+------------------| -| Cycles | 13 | -|----------+------------------| -| Type | Li-ion | -|----------+------------------| - -** Network - -| Interface | Type | Chipset | -|-----------+--------+--------------------------| -| wlp5s0 | WiFi | MediaTek MT7922 802.11ax | -|-----------+--------+--------------------------| -| Bluetooth | BT 5.2 | MediaTek Wireless_Device | -|-----------+--------+--------------------------| - -** Expansion Cards (Framework) - -| Slot | Card | -|------+---------------------| -| 1 | HDMI Expansion Card | -|------+---------------------| - -** Input Modules (Framework) - -| Module | Notes | -|----------+-------------------------| -| Keyboard | ANSI layout | -|----------+-------------------------| -| Numpad | Laptop 16 Numpad Module | -|----------+-------------------------| - -* Connected Peripherals - -| Device | Connection | -|---------------------------+-------------------------| -| Logitech MX Master 3 | Unifying Receiver (USB) | -|---------------------------+-------------------------| -| Realtek Laptop Camera | Internal USB | -|---------------------------+-------------------------| -| Goodix Fingerprint Reader | Internal USB | -|---------------------------+-------------------------| - -* Network Access - -| Method | Address | -|--------+---------------| -| mDNS | mybitch.local | -|--------+---------------| -| SSH | Yes (OpenSSH) | -|--------+---------------| - -* Notes - -- Framework Laptop 16 with modular expansion card system -- Discrete AMD GPU (RX 7700S) in addition to integrated Radeon 780M -- Second NVMe slot contains 2TB backup drive (unmounted, labeled "Backup") -- Uptime at inventory: 4 days - -* Operational Changes Log - -Deliberate, non-default config changes applied to mybitch (so they can be found and reverted later). Most recent first. - -** 2026-05-12 — amdgpu.mes=0 added (escalation of the MES-hang freeze mitigation) - -The 2026-05-11 =amdgpu.cwsr_enable=0= mitigation didn't reduce the iGPU "MES failed to respond" errors (14 hits in 14.5h after the fix vs. 38 over 10 days before — rate went *up*), so escalated per the documented path: added =amdgpu.mes=0= to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= (now =quiet splash amdgpu.cwsr_enable=0 amdgpu.mes=0=). Backup: =/etc/default/grub.bak-2026-05-12-amdgpu-mes=. =update-grub= run; param verified in all 6 =/boot/grub/grub.cfg= entries. Takes effect on next reboot (planned for after the in-progress first rsyncshot backup). =amdgpu.mes=0= disables the GFX11 hardware MES scheduler and falls back to the legacy KIQ path — sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks); irrelevant for Christine's light desktop workload. =cwsr_enable=0= kept (harmless alongside mes=0). *Revert:* restore =/etc/default/grub.bak-2026-05-12-amdgpu-mes= (or sed out = amdgpu.mes=0=), =update-grub=, reboot. After the reboot, check =journalctl -k -b | grep -c 'MES failed'= → should be 0. See the 2026-05-11 entry in =.ai/project-workflows/system-health-check.org='s Known Issues Log for the full background. - -** 2026-05-12 — rsyncshot backups + travel keep-awake mode - -Set up the same =rsyncshot= → TrueNAS backup mybitch's siblings use, and made mybitch never suspend so backups keep running while Christine travels with it. - -*** rsyncshot install (mirrors ratio's setup) -- =/usr/local/bin/rsyncshot= — the script (source of truth: =~/code/rsyncshot/rsyncshot=). -- =/etc/rsyncshot/config= — =REMOTE_HOST="cjennings@truenas.tailf3bb8c.ts.net"=, =REMOTE_PATH="/mnt/vault/backups"=, =SSH_IDENTITY_FILE="/root/.ssh/id_ed25519"=. (See "destination pinned to Tailscale" below for why the full MagicDNS name.) -- =/etc/rsyncshot/include.txt= — =/home /etc /usr/local/bin=. =/etc/rsyncshot/exclude.txt= — copied verbatim from ratio. -- =/etc/logrotate.d/rsyncshot= — copied verbatim from ratio. =/var/log/rsyncshot.log= — the log. -- Root crontab: hourly at :30 (hours 0-1,3-23, keep 23) + daily at 2:30 (keep 30), each wrapped in =flock -x /tmp/rsyncshot.lock=. Identical to ratio's schedule. -- =/root/.ssh/id_ed25519= (+ =.pub=) — new keypair generated on mybitch, no passphrase, comment =root@mybitch-rsyncshot=. mybitch's =cjennings= account had no outbound key (it's a receive-only account), so the backup runs with a dedicated root key instead. The pubkey was appended to =~cjennings/.ssh/authorized_keys= on TrueNAS (the line tagged =root@mybitch-rsyncshot=). -- =/root/.ssh/config= — =Host truenas truenas.tailf3bb8c.ts.net 100.67.22.65= → =User cjennings=, that IdentityFile, =ServerAliveInterval 60 / ServerAliveCountMax 10 / TCPKeepAlive yes=. Host keys for those names added to =/root/.ssh/known_hosts=. -- TrueNAS side: =/mnt/vault/backups/mybitch/= directory created, owned =cjennings:cjennings= (=vault/backups= is a ZFS dataset; per-host backups are just subdirectories, like =ratio/= and =velox/=). -- *Revert:* =sudo crontab -r= (or remove the rsyncshot lines), =sudo rm -rf /usr/local/bin/rsyncshot /etc/rsyncshot /etc/logrotate.d/rsyncshot /var/log/rsyncshot.log /root/.ssh/id_ed25519* /root/.ssh/config=, restore =/root/.ssh/known_hosts= if desired, remove the =root@mybitch-rsyncshot= line from =~cjennings/.ssh/authorized_keys= on TrueNAS, and =rm -rf /mnt/vault/backups/mybitch= on TrueNAS. - -*** destination pinned to Tailscale -=REMOTE_HOST= uses the full Tailscale MagicDNS name =truenas.tailf3bb8c.ts.net= (not the bare =truenas=) so the backup always routes over the tailnet regardless of what a hotel/away-from-home router's DNS resolves =truenas= to. Tailscale uses a direct LAN path between same-LAN peers, so this is near-zero perf cost when mybitch is on the home network. *Revert:* set =REMOTE_HOST="cjennings@truenas"= in =/etc/rsyncshot/config=. - -*** keep-awake (never suspend) — three layers -1. =systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target= — the hard guarantee; suspend can't happen even if requested. *Revert:* =sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target=. -2. Cinnamon power gsettings for user =cciarm= set to ='nothing'=: =sleep-inactive-ac-type=, =sleep-inactive-battery-type=, =lid-close-ac-action=, =lid-close-battery-action= (schema =org.cinnamon.settings-daemon.plugins.power=). Were ='suspend'/'suspend'/'nothing'/'suspend'=. *Revert:* set them back, or run =gsettings reset= on each key as =cciarm=. -3. =/etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= — =HandleLidSwitch=ignore=, =HandleLidSwitchExternalPower=ignore=, =HandleLidSwitchDocked=ignore=, =IdleAction=ignore=. (Written but not applied live — =systemd-logind= wasn't restarted because sessions were active; takes effect on next reboot. Layers 1-2 cover the gap.) *Revert:* =sudo rm /etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= then reboot or =systemctl restart systemd-logind=. - -When mybitch goes back to being a stationary machine (or this whole arrangement is no longer wanted), reverse layers 1-3. The rsyncshot backup itself is worth keeping regardless. - -*** IPv6 set to link-local-only (fixes the tailscale dropout) -mybitch dropped off the Tailscale tailnet on 2026-05-12 AM. Root cause: the home WiFi ("dupre") has device(s) advertising ULA-only IPv6 RAs (prefixes =fdc4:c4c6:3fb0:16cc::/64= and =fd67:8615:6780:1::/64=) with no default route. NetworkManager (=ipv6.method=auto=) SLAAC'd a "global"-scope ULA address onto wlp5s0 with no v6 internet path, so tailscaled kept trying IPv6 to controlplane/DERP, failed "network is unreachable", flapped, and lost its control connection. mybitch has no working global IPv6 anywhere (no ISP v6), so the fix is to ignore underlay v6: -- =/etc/NetworkManager/conf.d/ipv6-link-local-only.conf= → =[connection] ipv6.method=link-local= (connection default — covers home WiFi, travel WiFi, phone hotspot). -- =nmcli con modify dupre ipv6.method link-local= (explicit on the home connection). -- Effect: wlp5s0 keeps only fe80:: (mDNS still works), no SLAAC, no v6 routes; tailscale0's overlay v6 untouched; tailscaled uses the IPv4 underlay. Activates on reboot or =nmcli dev reapply wlp5s0=. -- *Revert if mybitch ever gets real IPv6:* =sudo rm /etc/NetworkManager/conf.d/ipv6-link-local-only.conf=; =sudo nmcli con modify dupre ipv6.method auto=; reboot or =nmcli dev reapply wlp5s0=. - -*** caveats (relayed to Craig for Christine) -- mybitch must stay plugged into AC for backups to keep running — masking suspend doesn't stop a battery dying with the lid closed. -- mybitch must be on WiFi at the destination and stay on the Tailscale tailnet. diff --git a/docs/homelab-inventory/ratio-desktop.org b/docs/homelab-inventory/ratio-desktop.org deleted file mode 100644 index 234c1a4..0000000 --- a/docs/homelab-inventory/ratio-desktop.org +++ /dev/null @@ -1,175 +0,0 @@ -#+TITLE: ratio - Desktop Workstation -#+DATE: 2026-01-27 -#+HOSTNAME: ratio - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. - -:PROPERTIES: -:FS: btrfs -:PM: pacman -:ORCH: topgrade -:SNAPSHOT: snapper -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt,docker,podman -:INIT: systemd -:LAST_AUDIT: 2026-07-08 -:END: - -* Overview - -| Field | Value | -|-------------+------------------------------------------| -| Hostname | ratio | -|-------------+------------------------------------------| -| Type | Desktop workstation | -|-------------+------------------------------------------| -| Maker | Framework | -|-------------+------------------------------------------| -| Model | Desktop (AMD Ryzen AI Max 300 Series) | -|-------------+------------------------------------------| -| Board | FRANMFCP06 v A6 | -|-------------+------------------------------------------| -| Firmware | INSYDE UEFI v03.03 (2025-09-16) | -|-------------+------------------------------------------| -| OS | Arch Linux | -|-------------+------------------------------------------| -| Kernel | 6.12.67-1-lts | -|-------------+------------------------------------------| -| Desktop | Hyprland (Wayland) | -|-------------+------------------------------------------| -| Previous ID | cogito (repurposed and renamed to ratio) | -|-------------+------------------------------------------| - -* CPU - -| Field | Value | -|---------------+---------------------------------------| -| Model | AMD Ryzen AI MAX+ 395 w/ Radeon 8060S | -|---------------+---------------------------------------| -| Architecture | Zen 5 (TSMC 4nm) | -|---------------+---------------------------------------| -| Cores/Threads | 16 cores / 32 threads | -|---------------+---------------------------------------| -| Base/Boost | 3.0 GHz / 5.15 GHz | -|---------------+---------------------------------------| -| Cache L1 | 1.2 MiB (d-16x48 KiB; i-16x32 KiB) | -|---------------+---------------------------------------| -| Cache L2 | 16 MiB (16x1024 KiB) | -|---------------+---------------------------------------| -| Cache L3 | 64 MiB (2x32 MiB) | -|---------------+---------------------------------------| -| NPU | XDNA 2, 50+ peak AI TOPS | -|---------------+---------------------------------------| -| Peak Perf | 59.4 FP16/BF16 TFLOPS @ 2.9GHz | -|---------------+---------------------------------------| - -* GPU - -| Field | Value | -|--------------+-----------------------------------------| -| Model | AMD Radeon 8060S (Strix Halo) | -|--------------+-----------------------------------------| -| Architecture | RDNA 3.5 | -|--------------+-----------------------------------------| -| CUs | 40 Compute Units | -|--------------+-----------------------------------------| -| Driver | amdgpu (kernel) | -|--------------+-----------------------------------------| -| Max VRAM | 96GB (via AMD Variable Graphics Memory) | -|--------------+-----------------------------------------| -| GPU Arch ID | gfx1151 | -|--------------+-----------------------------------------| -| PCIe | Gen 4, 16 GT/s, 16 lanes | -|--------------+-----------------------------------------| - -* Memory - -| Field | Value | -|--------------+--------------------------| -| Total | 128 GiB unified (LPDDR5) | -|--------------+--------------------------| -| Speed | 8000 MT/s | -|--------------+--------------------------| -| Channels | 8 (8x 16 GiB) | -|--------------+--------------------------| -| Manufacturer | Micron Technology | -|--------------+--------------------------| -| Part Number | MT62F4G32D8DV-023 WT | -|--------------+--------------------------| - -* Storage - -| Device | Model | Size | Interface | -|--------------+------------------------+-----------+-----------| -| /dev/nvme0n1 | WD BLACK SN850X 8000GB | 7.28 TiB | NVMe | -|--------------+------------------------+-----------+-----------| -| /dev/nvme1n1 | WD BLACK SN850X 8000GB | 7.28 TiB | NVMe | -|--------------+------------------------+-----------+-----------| -| *Total* | | 14.55 TiB | | -|--------------+------------------------+-----------+-----------| - -Filesystem: btrfs with subvolumes (@, @home, @snapshots, @log, @pkg) - -* Network - -| Interface | Chipset | Speed | Type | -|-----------+-----------------+---------+----------| -| enp191s0 | Realtek RTL8126 | 5 GbE | Wired | -|-----------+-----------------+---------+----------| -| wlp192s0 | MediaTek MT7925 | Wi-Fi 7 | Wireless | -|-----------+-----------------+---------+----------| - -Also has: docker0, tailscale0, virbr0 (virtual interfaces) - -* Audio - -- AMD Radeon High Definition Audio (HDMI, snd_hda_intel) -- AMD Ryzen HD Audio (3.5mm, snd_hda_intel) -- PipeWire v1.4.10 (pipewire-pulse, wireplumber) - -* PCI Slots - -| Slot | Info | Status | -|------+------------+-----------| -| 1 | M.2 JWLAN | Available | -|------+------------+-----------| -| 2 | M.2 JSSD1 | In use | -|------+------------+-----------| -| 3 | M.2 JSSD2 | In use | -|------+------------+-----------| -| 4 | PCIe Gen 4 | Available | -|------+------------+-----------| - -* Connected Peripherals - -- Monitor: Dell U3419W (HDMI-A-1, 3440x1440@60Hz) -- Keyboard: Das Keyboard 4 (Cherry MX Blue) — replacing with Keychron Q6 Pro -- Mouse: Logitech M650 -- DAC/Amp: JDS Labs Element IV (ordered) - -* AI/LLM Capability - -Primary purpose includes local AI/LLM inference via Ollama. - -| Model | Size | Purpose | -|------------------+-------+----------------------| -| deepseek-r1:70b | 42 GB | Main reasoning model | -|------------------+-------+----------------------| -| qwen3:30b | 18 GB | MoE chat (256K ctx) | -|------------------+-------+----------------------| -| deepseek-r1:14b | 9 GB | Light reasoning | -|------------------+-------+----------------------| -| nomic-embed-text | 274MB | RAG embeddings | -|------------------+-------+----------------------| - -Inference benchmarks (AMD testing, LM Studio 0.3.11): -- Small models (1-3B): ~100+ tok/s -- Medium models (7-8B): ~60-80 tok/s -- Large models (20B): ~58 tok/s -- Very large models (120B): ~38 tok/s diff --git a/docs/homelab-inventory/truenas-server.org b/docs/homelab-inventory/truenas-server.org deleted file mode 100644 index 5ce8613..0000000 --- a/docs/homelab-inventory/truenas-server.org +++ /dev/null @@ -1,114 +0,0 @@ -#+TITLE: truenas - NAS/Storage Server -#+DATE: 2026-01-27 -#+HOSTNAME: truenas - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. -# -# NOTE: TrueNAS SCALE is middleware-managed. No user-level package manager -# or topgrade orchestration. Snapshot and update operations go through the -# TrueNAS UI / API, not pacman/apt. - -:PROPERTIES: -:FS: zfs -:PM: none -:ORCH: none -:SNAPSHOT: zfs-native -:MESH: tailscale -:BACKUP: target -:VIRT: docker -:INIT: systemd -:LAST_AUDIT: 2026-04-21 -:END: - -* Overview - -| Field | Value | -|----------+-----------------------------------------| -| Hostname | truenas (truenas.local / 192.168.86.5) | -|----------+-----------------------------------------| -| Type | NAS / storage server | -|----------+-----------------------------------------| -| OS | TrueNAS | -|----------+-----------------------------------------| -| Purpose | Media storage, backups, Plex, Syncthing | -|----------+-----------------------------------------| - -See [[file:2026-01-05-truenas-hardware-specs.org][TrueNAS Hardware Specs]] for full sysinfo output. - -* CPU - -| Field | Value | -|---------------+----------------------| -| Model | Intel Core i9-9900K | -|---------------+----------------------| -| Architecture | Coffee Lake | -|---------------+----------------------| -| Cores/Threads | 8 cores / 16 threads | -|---------------+----------------------| -| Base/Boost | 3.60 GHz / 5.00 GHz | -|---------------+----------------------| - -* Memory - -| Field | Value | -|-------+-------------| -| Total | 64 GiB DDR4 | -|-------+-------------| - -* Storage - -| Pool | Config | Raw Size | Purpose | -|-----------+-----------------+------------+---------------| -| boot-pool | Mirror | ~1.9 TB x2 | System | -|-----------+-----------------+------------+---------------| -| sysdata | NVMe | ~1.8 TB | System data | -|-----------+-----------------+------------+---------------| -| vault | RAIDZ1 (4 disk) | ~43.6 TB | Primary media | -|-----------+-----------------+------------+---------------| -| tank | RAIDZ1 (4 disk) | ~72.8 TB | Expansion | -|-----------+-----------------+------------+---------------| - -Total raw: ~122 TB - -*** Vault Datasets -| Dataset | Size | -|------------+-------| -| Media | 30 TB | -|------------+-------| -| Lectures | 15 TB | -|------------+-------| -| Audiobooks | 14 TB | -|------------+-------| -| Magic | 15 TB | -|------------+-------| -| Books | 14 TB | -|------------+-------| - -* Network - -| Interface | Chipset | Speed | Status | -|-----------+--------------+--------+--------------| -| eno1 | Intel I219-V | 1 GbE | Current | -|-----------+--------------+--------+--------------| -| enp2s0 | Intel I211 | 1 GbE | Available | -|-----------+--------------+--------+--------------| -| (pending) | Intel I226-V | 2.5GbE | Ordered, NIC | -|-----------+--------------+--------+--------------| - -* Motherboard - -| Field | Value | -|---------+------------| -| Chipset | Intel Z390 | -|---------+------------| - -* Services - -- Plex Media Server -- Syncthing -- Tailscale diff --git a/docs/homelab-inventory/velox-laptop.org b/docs/homelab-inventory/velox-laptop.org deleted file mode 100644 index 6e4f540..0000000 --- a/docs/homelab-inventory/velox-laptop.org +++ /dev/null @@ -1,158 +0,0 @@ -#+TITLE: velox - Laptop -#+DATE: 2026-01-27 -#+HOSTNAME: velox - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. -# -# NOTE: 2026-04 velox was reinstalled Arch-on-ZFS. Values below reflect -# that reinstall but remain TBD where a live probe has not yet confirmed. - -:PROPERTIES: -:FS: zfs -:PM: pacman -:ORCH: topgrade -:SNAPSHOT: sanoid -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt,docker,podman -:INIT: systemd -:LAST_AUDIT: 2026-05-26 -:END: - -* Overview - -| Field | Value | -|----------+---------------------------------| -| Hostname | velox | -|----------+---------------------------------| -| Type | Laptop | -|----------+---------------------------------| -| Maker | Framework | -|----------+---------------------------------| -| Model | Laptop (13th Gen Intel Core) | -|----------+---------------------------------| -| Board | FRANMCCP07 v A7 | -|----------+---------------------------------| -| Firmware | INSYDE UEFI v03.07 (2024-12-26) | -|----------+---------------------------------| -| OS | Arch Linux | -|----------+---------------------------------| -| Kernel | 6.18.6-arch1-1 | -|----------+---------------------------------| -| Desktop | startx (X11) | -|----------+---------------------------------| - -* CPU - -| Field | Value | -|---------------+-----------------------------------------| -| Model | 13th Gen Intel Core i7-1370P | -|---------------+-----------------------------------------| -| Architecture | Raptor Lake (Intel 7 / 10nm) | -|---------------+-----------------------------------------| -| Cores/Threads | 14 cores / 20 threads (6P + 8E, hybrid) | -|---------------+-----------------------------------------| -| Base/Boost | 990 MHz / 5.2 GHz | -|---------------+-----------------------------------------| -| Cache L1 | 1.2 MiB (d-8x32K,6x48K; i-6x32K,8x64K) | -|---------------+-----------------------------------------| -| Cache L2 | 11.5 MiB (6x1.2M, 2x2M) | -|---------------+-----------------------------------------| -| Cache L3 | 24 MiB (1x24M) | -|---------------+-----------------------------------------| -| Socket | BGA1744 (U3E1) | -|---------------+-----------------------------------------| - -* GPU - -| Field | Value | -|--------------+------------------------| -| Model | Intel Iris Xe Graphics | -|--------------+------------------------| -| Architecture | Xe (Intel 7) | -|--------------+------------------------| -| Driver | i915 / modesetting | -|--------------+------------------------| - -* Memory - -| Field | Value | -|----------+------------------------------------| -| Total | 64 GiB (DDR4) | -|----------+------------------------------------| -| Speed | 3200 MT/s | -|----------+------------------------------------| -| Slots | 2 (both populated) | -|----------+------------------------------------| -| Device 1 | 32 GiB Crucial CT32G4SFD832A.C16FB | -|----------+------------------------------------| -| Device 2 | 32 GiB Crucial CT32G4SFD832A.C16FE | -|----------+------------------------------------| - -* Storage - -| Device | Model | Size | Interface | -|--------------+----------------------+----------+-----------| -| /dev/nvme0n1 | Sabrent SB-RKT4P-8TB | 7.28 TiB | NVMe | -|--------------+----------------------+----------+-----------| - -- Filesystem: ZFS on root (zroot pool on nvme0n1p2; reinstalled 2026-04) -- Datasets: zroot/ROOT/default, zroot/home, zroot/var/log, zroot/var/lib/pacman, zroot/var/lib/docker (per-layer), zroot/vms, zroot/media -- Snapshots: sanoid hourly + zfs-scrub-weekly@zroot -- SMART: PASSED (re-verified 2026-04-26); composite 43 °C; 0 media errors -- Power-on history pre-dates reinstall — refresh on next full re-inventory - -* Display - -| Field | Value | -|------------+---------------------------------| -| Panel | BOE Display 0x0bca (built 2022) | -|------------+---------------------------------| -| Resolution | 2256x1504 | -|------------+---------------------------------| -| Size | 13.5" (285x190mm) | -|------------+---------------------------------| -| DPI | 201 | -|------------+---------------------------------| -| Ratio | 3:2 | -|------------+---------------------------------| - -* Network - -| Interface | Chipset | Speed | Type | -|-------------+--------------------------------+---------+----------| -| wlp170s0 | Qualcomm Atheros AR9462 | Wi-Fi | Wireless | -|-------------+--------------------------------+---------+----------| -| enp0s13f0u3 | Realtek USB 10/100/1G/2.5G LAN | 2.5 GbE | USB | -|-------------+--------------------------------+---------+----------| - -Also has: docker0, tailscale0, virbr0 (virtual interfaces) - -* Audio - -- Intel Raptor Lake-P/U/H cAVS (snd_hda_intel) -- PipeWire v1.4.10 (pipewire-pulse, wireplumber) - -* Battery - -| Field | Value | -|-----------+-----------------------| -| Capacity | 55 Wh (design) | -|-----------+-----------------------| -| Condition | 44/55 Wh (80% health) | -|-----------+-----------------------| -| Cycles | 146 | -|-----------+-----------------------| -| Type | Li-ion | -|-----------+-----------------------| - -* Other Hardware - -- Webcam: Realtek Laptop Camera (UVC) -- Fingerprint: Shenzhen Goodix USB Device -- Bluetooth: Foxconn (BT 4.0) diff --git a/docs/workflows/strix-soak-watch.org b/docs/workflows/strix-soak-watch.org new file mode 100644 index 0000000..e2fbf68 --- /dev/null +++ b/docs/workflows/strix-soak-watch.org @@ -0,0 +1,146 @@ +#+TITLE: Strix Halo SMU Deadlock — Kernel Update Watch +#+AUTHOR: Craig Jennings & Claude +#+DATE: 2026-05-10 + +* Purpose + +Companion to =system-health-check.org=. Triggered when linux / linux-lts / linux-firmware land in =checkupdates= on ratio, this addendum checks whether the upstream side of the Strix Halo SMU deadlock has moved — so we know when to retire the locally-patched =linux-lts-strix= kernel and switch back to stock. + +This is a one-time hardware/kernel issue. Delete this file when the strix kernel is retired. + +* Current State (as of 2026-05-10) + +- *Boot kernel:* =linux-lts-strix 6.18.25-1= (custom AUR-style local build), GRUB default pinned +- *SMC firmware:* =100.6.0= (from =/sys/kernel/debug/dri/*/amdgpu_firmware_info= SMC line) +- *Cmdline workaround:* =amdgpu.no_vpe_idle_pg=1= active +- *Soak verdict:* HOLDING since 2026-04-30 — zero SMU/dcn35/VPE freeze signature hits (see =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project)) + +* Background (one-paragraph summary so future sessions don't have to reconstruct it) + +Strix Halo (gfx_v11_5_0, smu_v14_0_2 APU) has an SMU firmware deadlock triggered during VPE (Video Processing Engine) idle power-gating: hard freeze, no journal output, requires power-cycle. Crupi posted a workaround patch on the Framework community forum that disables VPE idle power-gating via =amdgpu.no_vpe_idle_pg=. Real fix has to come from AMD as a PMFW/SMU firmware update (delivered via =linux-firmware=). Until then, ratio runs a custom kernel build that carries the patch + two supporting amdgpu allocation-in-critical-path patches. Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 + +* Local Patch Set + +Located at =~/build/linux-lts-patched/linux-lts/=: + +| File | Strix-relevant? | +|---------------------------------------------------------------------------------+-----------------| +| =0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-CLONE_NEWUSER.patch= | No (security) | +| =0002-drm-amdgpu-avoid-memory-allocation-in-the-critical-code-path-v3.patch= | Yes | +| =0003-drm-amdgpu-use-GFP_ATOMIC-instead-of-NOWAIT-in-the-critical-path.patch= | Yes | +| =0004-drm-amdgpu-add-no_vpe_idle_pg-strix-halo-smu-deadlock-workaround.patch= | Yes (Crupi's) | + +* When This Watch Fires + +Phase 3 of =system-health-check.org= invokes this addendum when *any* of the following appear in =checkupdates=: + +- =linux= (any version bump) +- =linux-lts= (any version bump) +- =linux-firmware= (any version bump — this is the most likely vector for the real fix) +- =mesa= (major version only, e.g. 26.x.y → 27.x.y — display-stack timing can shift the deadlock window) + +* What To Check On Each Fire + +** 1. Stable kernel changelogs (per bumped version) + +Fetch the per-version changelog and search for the keyword set below. The 2026-05-10 baseline pass found *no matches* in 6.18.27, 6.18.28, 7.0.4, 7.0.5 — so anything that does match is news. + +#+begin_src +https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog- +https://cdn.kernel.org/pub/linux/kernel/v7.x/ChangeLog- +#+end_src + +Keyword set (case-insensitive): +- =no_vpe_idle_pg=, =vpe_idle=, =VPE idle=, =VPE power gating= +- =Strix Halo=, =STRIX_HALO=, =strix_halo= +- =gfx_v11_5= (the GFX block id for Strix Halo) +- =smu_v14= (the SMU IP block driver, =drivers/gpu/drm/amd/pm/swsmu/smu14=) +- =dcn351= (Strix Halo display controller; =dcn35= is Phoenix-class — adjacent but not the same) +- =drm/amdgpu/vpe=, =drm/amdgpu/swsmu/smu14=, =amdgpu/pm/swsmu= +- =SMU deadlock=, =critical path= (combined with amdgpu/drm) + +The two adjacent allocation-fix patches (0002, 0003) may land separately from Crupi's patch. They have distinctive subject lines: =avoid memory allocation in the critical code path= and =use GFP_ATOMIC instead of NOWAIT in the critical path=. Search for those too. + +** 2. linux-firmware delta (if linux-firmware is the bumped package) + +The canonical fix arrives as a PMFW/SMU firmware update, NOT a kernel patch. + +#+begin_src bash +# Pre-bump SMC version (from running system, BEFORE topgrade) +sudo cat /sys/kernel/debug/dri/*/amdgpu_firmware_info | grep "SMC feature" + +# After topgrade installs the new linux-firmware, examine the delta. +# Watch for these files in the upgraded package: +# amdgpu/smu_14_0_2.bin — Strix Halo SMU firmware +# amdgpu/psp_14_0_*.bin — PSP/secure-os blobs +# amdgpu/strix* — anything Strix-tagged +pacman -Ql linux-firmware | grep -iE "smu_14_0_2|psp_14_0|strix" + +# Reboot is required to load the new firmware. Re-check SMC version post-reboot. +#+end_src + +A bump from =100.6.0= to anything higher is the canonical Strix-fix signal — that means AMD shipped new SMU firmware. Once that lands, the workaround kernel is no longer needed (in theory; see "Outcomes" below for the validation path). + +** 3. Crupi's forum thread + +#+begin_src +https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 +#+end_src + +Skim recent posts for: +- "Merged upstream" / "Landed in 6.x.y" announcements +- AMD acknowledgements or workaround retirement notes +- Other affected users reporting fix verification on stock kernels + +** 4. amd-pm subsystem mailing list (optional, only if 1-3 are inconclusive) + +#+begin_src +https://lore.kernel.org/amd-gfx/?q=strix+halo+smu+deadlock +https://lore.kernel.org/amd-gfx/?q=no_vpe_idle_pg +#+end_src + +* Outcomes & Actions + +** Nothing Strix-adjacent in the bump (default — also the 2026-05-10 outcome) + +- Continue running =linux-lts-strix 6.18.25-1= +- Do *not* rebuild the strix kernel against the newer base purely for the version bump. Same four patches against a newer base = same workaround behavior + new risk of unrelated base-version regressions. Only rebuild when there's a security-pressing reason. +- Topgrade is safe; the new stock =linux-lts= installs to disk but isn't booted. + +** linux-firmware bumps SMC past =100.6.0= + +This is the most likely real-fix path. + +1. Note the new SMC version (capture pre and post). +2. Schedule a controlled test boot: + - Backup current =/etc/default/grub= GRUB_DEFAULT and =linux-lts-strix= package + - Drop =amdgpu.no_vpe_idle_pg=1= from the cmdline on a test boot of stock =linux-lts= + - Re-run the soak-check script for 1 week (same harness used for the original verdict) +3. If zero signature hits over the soak window → retire =linux-lts-strix=, switch GRUB default back to stock =linux-lts=, remove the custom build dir. +4. If any freeze recurs → restore strix-lts as default and re-engage the workaround. Note the recurrence in this file and on the forum thread. + +** Crupi's patch (or equivalent) lands in 6.18.x stable + +1. Check the merged form: is the =no_vpe_idle_pg= behavior the default-on, or still gated behind a cmdline? +2. If default-on: rebuild =linux-lts-strix= against the new base with patch 0004 dropped (keep 0002, 0003, 0001 as long as they remain non-redundant). +3. If still cmdline-gated: keep the strix kernel as-is until linux-firmware fix arrives — the in-tree path is equivalent to the local one. + +** Patches 0002/0003 land upstream but Crupi's doesn't + +Drop the matching patches from the PKGBUILD on next strix rebuild; keep 0004 until the SMU firmware fix or kernel merge. + +* Reference Artifacts + +- Soak verdict (archived): =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project) +- VPE-patch deployment session: =.ai/sessions/2026-04-29-23-35-linux-lts-strix-vpe-patch-deployed.org= +- Soak fix + housekeeping session: =.ai/sessions/2026-05-03-06-48-soak-fix-and-velox-sync-housekeeping.org= +- Verdict-holds session: =.ai/sessions/2026-05-05-01-48-strix-verdict-holds-mybitch-soak-deferred.org= +- Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 +- Build dir: =~/build/linux-lts-patched/linux-lts/= +- Inventory: the =ratio - Desktop Workstation= node in =~/org/roam/hardware/= (resolve by =#+HOSTNAME: ratio=) + +* Out of Scope + +- Generalizing this to other AMD APUs. The deadlock signature is specific to Strix Halo (gfx_v11_5_0, smu_v14_0_2). Other Phoenix / Hawk Point / RDNA-N APUs are unaffected and shouldn't be checked here. +- Tracking AMD's internal bug tracker. Stick to public signals (kernel.org changelogs, lore.kernel.org, the Framework forum thread). +- This is not a soak-check runner. The soak-check harness (=at=-job + scripts) lives separately under =scripts/= or =inbox/=. This file watches for the *retirement signal* for that harness. diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org index b66a186..98fa17f 100644 --- a/docs/workflows/system-health-check.org +++ b/docs/workflows/system-health-check.org @@ -8,7 +8,7 @@ This workflow performs a comprehensive diagnostic scan of whatever host it runs Run this when Craig asks "how is doing?", "check my system health", or as a periodic maintenance check. -Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=. +Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live in the org-roam knowledge base at =~/org/roam/hardware/= (moved there 2026-07-09). Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3. @@ -23,20 +23,24 @@ Read the TOML at the start of Phase 1 and grade against its values. Where a chec * Hosts & Capabilities -The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift. +The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory nodes in the org-roam knowledge base declare the expected capabilities in a property drawer and act as a cross-check for drift. -Host inventory files (match by =#+HOSTNAME:= keyword inside each): +The inventories live at =~/org/roam/hardware/= (2026-07-09), alongside the rest of the hardware inventory. They moved out of this repo so any project's session can read a host's capabilities — and write back what a diagnosis discovers — without reaching across a project boundary. The index node "Homelab Hardware Inventory" links them all. -| Host | Inventory file | Notes | -|---------+-------------------------------------------+----------------------------------| -| ratio | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]] | Arch, Btrfs RAID1, pacman | -|---------+-------------------------------------------+----------------------------------| -| velox | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]] | Arch, ZFS (re-installed 2026-04) | -|---------+-------------------------------------------+----------------------------------| -| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt | -|---------+-------------------------------------------+----------------------------------| -| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM | -|---------+-------------------------------------------+----------------------------------| +Host inventory nodes (match by =#+HOSTNAME:= keyword inside each): + +| Host | Inventory node | Notes | +|---------+-----------------------------------------------+----------------------------------| +| ratio | =hardware/…-ratio_desktop_workstation.org= | Arch, Btrfs RAID1, pacman | +|---------+-----------------------------------------------+----------------------------------| +| velox | =hardware/…-velox_laptop.org= | Arch, ZFS (re-installed 2026-04) | +|---------+-----------------------------------------------+----------------------------------| +| mybitch | =hardware/…-mybitch_christines_laptop.org= | Mint, ext4, apt | +|---------+-----------------------------------------------+----------------------------------| +| truenas | =hardware/…-truenas_nas_storage_server.org= | TrueNAS SCALE, ZFS, no PM | +|---------+-----------------------------------------------+----------------------------------| + +Never hardcode a node's filename — its timestamp prefix is not stable across a rename. Resolve by the =#+HOSTNAME:= keyword, as Phase 0 does. Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift. @@ -108,10 +112,10 @@ Present the probe result as a one-line capability summary at the top of the repo Locate the inventory file for this host: #+begin_src bash -inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1) +inv=$(grep -l "^#+HOSTNAME: $host\b" "${ROAM_DIR:-$HOME/org/roam}"/hardware/*.org 2>/dev/null | head -1) #+end_src -If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. +If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. A host without the roam clone (mybitch, truenas) takes that path, exactly as it did when the inventories lived in this repo. If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe. @@ -228,7 +232,7 @@ Updates are separate from issue investigation. After all issues are addressed (o 2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware) 3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices 4. Check =/var/log/pacman.log= for recent failed transactions -5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired. +5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at [[file:strix-soak-watch.org][docs/workflows/strix-soak-watch.org]] before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired. 6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=). On maint hosts (ratio, velox) plain =topgrade= resolves to the dotfiles PATH wrapper, which stamps the console's topgrade-freshness metric on success — no extra step. If the run happened outside the wrapper somehow, =maint stamp topgrade= records it by hand. 7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot @@ -996,7 +1000,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi - Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986. - Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot. - Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean. -- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.) +- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in the =mybitch - Christine's Laptop= roam node → Operational Changes Log.) - *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=. - *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug). - Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111. diff --git a/todo.org b/todo.org index 32fcc2a..c83522f 100644 --- a/todo.org +++ b/todo.org @@ -1075,6 +1075,46 @@ Rewrote the bare =if $var= boolean conditionals (=show_status_only=, =fresh_inst *** 2026-05-26 Tue @ 15:27:09 -0500 eval task moot — the line-434 eval is gone, the survivor is deliberate Verified: the only =eval= left in =archsetup= is line 578 in =retry_install=, and it's intentional and documented — it captures =$?= directly from =eval "$cmd"= to dodge the if-compound-swallows-exit-code trap. Replacing it with an array would reintroduce that bug. The line-434 eval this task pointed at no longer exists. Nothing to change. +** TODO [#B] Signal and known-noise counts disagree with the journal :bug:maint:solo: +Logs > SIGNAL reports 10 journal errors and the signal item claims 10, but the group holds one item and the open journal shows it 6 times. KNOWN NOISE reports 19 against 12 visible entries. The counts are the point of the band, so a wrong number undermines the whole diagnostic. Find where the tally diverges from the rendered set — likely counting matched journal lines rather than distinct grouped entries, and counting curated entries against a different window than the one displayed. + +** TODO [#B] Persist the CPU pill setting across reboot :feature:maint: +The power/perf/balance pill sets EPP live, but the value reverts to its original setting after a reboot. Add a way to make a selection permanent. Needs a call on where persistence lives — a systemd unit, a udev rule, or a config file the maint package owns. + +** TODO [#B] Preview the pacman + AUR update queue before upgrade :feature:maint:solo: +Show which pacman and AUR packages are queued before UPDATE or topgrade runs, so the armed live-update guard isn't the first place the package set becomes visible. + +** TODO [#B] Audio PTT state desync across panel, waybar icon, and actual state :bug:audio:solo: +The audio panel's push-to-talk, the waybar PTT icon, and the real PTT state disagree. All three must agree at all times. Cover every state transition with tests, not just the happy path — this is the second PTT bug after the pre-talk mute restore landed 2026-07-05 (dotfiles 1443b9e). + +** DONE [#B] Publish the homelab inventory into the agent knowledge base :chore:docs: +CLOSED: [2026-07-09 Thu] +The four host inventories, the TrueNAS specs asset, and the ratio USB/xHCI record now live as roam nodes in =~/org/roam/hardware/=, linked from the "Homelab Hardware Inventory" index node. =docs/homelab-inventory/= is gone; roam is canonical. + +Carried rather than pointed, per home's argument, which beat mine. Pointing keeps one canonical copy but breaks the write path: a third project that discovers a durable hardware fact would have to write into archsetup's repo to record it, which the cross-project rules forbid, so the fact would sit in an inbox instead of on the device's page. Moving answers the canonical-copy worry without that cost. + +The =* Automated Capabilities= drawer moved with the pages. =system-health-check.org= now resolves a host's node by its =#+HOSTNAME:= keyword under =${ROAM_DIR:-$HOME/org/roam}/hardware/= — never by filename, since the timestamp prefix isn't stable. Verified: all four hosts resolve, an unknown host degrades to "NO INVENTORY FILE", and ratio's capability drawer still parses. A host without the roam clone (mybitch, truenas) takes the same no-inventory path it always did. + +The strix kernel watch stayed at =docs/workflows/strix-soak-watch.org= — it's a workflow, not an inventory page. + +** TODO [#C] Optional ticking sound for the pomodoro :feature:timer: +Timers module. Off by default; needs a sound choice and a volume decision. + +** TODO [#C] Refresh indicator for the top-like items :feature:maint: +Show when the process/memory rows are about to refresh, so a stale-looking board is distinguishable from a frozen one. Shape is a design call — a countdown, a pulse, or a progress hairline. + +** TODO [#C] CPU pill should not reposition on selection :bug:maint:quick:solo: +The perf/balance/power pill moves position depending on which option is selected. The segmented control should hold a fixed layout and mark the active segment in place. + +** TODO [#C] Session identifier needs more width :bug:waybar:quick: +The session section truncates; give it more characters, and keep the window sections that follow from running into it. Screenshot: [[file:/home/cjennings/pictures/screenshots/2026-07-07_145236.png][2026-07-07_145236.png]] + +** TODO [#C] Doctor button for the audio panel :feature:audio: +What happens when pipewire or pulseaudio is broken — would we know? Needs the probe set, the test for each fault, and the remedy tier before anything gets built. Design first. + +** TODO [#C] Telega coredump recurrence tell :bug:maint: +The maintenance console's coredump metric flagged telega-server on ratio (8 coredumps) and velox (18). Root cause was a version skew: the Dockerized =zevlg/telega-server:latest= is frozen at the 2026-06-05 build while the installed elisp lagged at 20260513, so the newer server's plist parser choked on the older elisp's output. .emacs.d fixed it by upgrading telega to 20260706 on both machines (docker kept, =docker pull= is a no-op against the frozen image). Host-coredump pollution should stop. If zevlg later pushes a =:latest= that outruns the installed elisp, the skew and the coredumps recur — the tell is a fresh =tdat_plist_value:500= assertion in =~/.telega/telega-server.log=. The durable escape is a host-native pinned TDLib build, at the cost of an AUR source build. + * Archsetup Resolved ** DONE [#B] Right-click date/time: ntp sync + timezone update :feature:waybar: -- cgit v1.2.3