diff options
| author | Craig Jennings <c@cjennings.net> | 2026-07-09 12:58:56 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2026-07-09 12:58:56 -0500 |
| commit | 76de64ac803c336252ac225f1f493906a7eacd07 (patch) | |
| tree | f5de1a8590e9cb344187fc419825f13fe0bb22dd /docs | |
| parent | 4563af6702bcc44defbca22baeb87ab8261572c6 (diff) | |
| download | archsetup-76de64ac803c336252ac225f1f493906a7eacd07.tar.gz archsetup-76de64ac803c336252ac225f1f493906a7eacd07.zip | |
docs: move the host inventories into the org-roam knowledge base
The four host capability inventories, the TrueNAS hardware specs, and the
ratio USB/xHCI record now live as roam nodes under ~/org/roam/hardware/,
linked from the Homelab Hardware Inventory index. The copies here are gone.
Carried rather than pointed at. A pointer keeps one canonical copy, but a
third project that discovers a durable hardware fact would then have to write
into this repo to record it, which the cross-project rules forbid — so the
fact would land in an inbox instead of on the device's page. Moving keeps one
canonical copy without closing the write path.
system-health-check.org resolves a host's inventory by its #+HOSTNAME: keyword
under ${ROAM_DIR:-$HOME/org/roam}/hardware/, never by filename: a node's
timestamp prefix does not survive a rename. A host with no roam clone takes the
same NO INVENTORY FILE path it always did.
strix-soak-watch.org lands here from home, beside the workflow whose Phase 3
invokes it. It is a workflow, not an inventory page.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org | 229 | ||||
| -rw-r--r-- | docs/homelab-inventory/mybitch-laptop.org | 238 | ||||
| -rw-r--r-- | docs/homelab-inventory/ratio-desktop.org | 175 | ||||
| -rw-r--r-- | docs/homelab-inventory/truenas-server.org | 114 | ||||
| -rw-r--r-- | docs/homelab-inventory/velox-laptop.org | 158 | ||||
| -rw-r--r-- | docs/workflows/strix-soak-watch.org | 146 | ||||
| -rw-r--r-- | docs/workflows/system-health-check.org | 38 |
7 files changed, 167 insertions, 931 deletions
diff --git a/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org b/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org deleted file mode 100644 index 42d33ad..0000000 --- a/docs/homelab-inventory/2026-01-05-truenas-hardware-specs.org +++ /dev/null @@ -1,229 +0,0 @@ -#+TITLE: TrueNAS Server Hardware Specifications -#+AUTHOR: Auto-generated via SSH -#+DATE: 2026-01-05 - -* Package Management Note - -**IMPORTANT:** Package management tools (apt) are disabled on TrueNAS appliances. Attempting to install packages with apt or methods other than the TrueNAS web interface can result in a nonfunctional system. - -All software installation and updates must be done through the TrueNAS web interface. - -* System Information - -** Hostname: truenas -** IP Address: 192.168.86.5 (static) -** Network Interface: eno1 -** Kernel: Linux 6.12.33-production+truenas -** Build Date: Wed Dec 17 21:17:21 UTC 2025 -** Architecture: x86_64 GNU/Linux - -* CPU Specifications - -** Model: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz -** Architecture: x86_64 (Coffee Lake S) -** Cores: 8 physical cores -** Threads: 16 (2 threads per core) -** Base Clock: 3.60 GHz -** Max Turbo: 5.00 GHz -** Min Clock: 800 MHz - -** Cache: -- L1d cache: 256 KiB (8 instances) -- L1i cache: 256 KiB (8 instances) -- L2 cache: 2 MiB (8 instances) -- L3 cache: 16 MiB (1 instance) - -** Features: -- Virtualization: VT-x -- CPU family: 6 -- Model: 158 -- Stepping: 13 - -* Memory Specifications - -** Total RAM: 64 GiB (62 GiB usable) -** Swap: 0 B (no swap configured) -** Current Usage: ~3.0 GiB used, 59 GiB free - -* Storage Controllers - -** SATA Controllers: -1. Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) -2. ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) - -** NVMe Controllers: -1. Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) - -* Block Devices - -** Hard Drives: -- sda: 10.9 TB (ZFS member) -- sdb: 10.9 TB (ZFS member) -- sdc: 10.9 TB (ZFS member) -- sdd: 10.9 TB (ZFS member) -- sdg: 18.2 TB (ZFS member) -- sdh: 18.2 TB (ZFS member) -- sdi: 18.2 TB (ZFS member) -- sdj: 18.2 TB (ZFS member) - -** System Drives: -- sde: 1.9 TB (Boot pool, ZFS member) - - sde1: 1M partition - - sde2: 512M VFAT partition - - sde3: 1.9T ZFS partition -- sdf: 1.9 TB (ZFS member) - -** NVMe: -- nvme0n1: 1.8 TB - -** Total Raw Capacity: -- 4× 10.9 TB = 43.6 TB -- 4× 18.2 TB = 72.8 TB -- 2× 1.9 TB = 3.8 TB -- 1× 1.8 TB NVMe = 1.8 TB -- **Total: ~122 TB raw storage** - -* Network Interfaces - -** eno1 (Primary - Active): -- Type: Ethernet controller - Intel Corporation Ethernet Connection (7) I219-V (rev 10) -- MAC: 70:85:c2:db:9d:94 -- IP: 192.168.86.5/21 -- Subnet broadcast: 192.168.87.255 -- State: UP -- MTU: 1500 -- Alternative name: enp0s31f6 - -** enp2s0 (Secondary - Down): -- Type: Ethernet controller - Intel Corporation I211 Gigabit Network Connection (rev 03) -- MAC: 70:85:c2:db:9d:94 -- State: DOWN (NO-CARRIER) - -** Wireless: -- Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) -- Status: Not configured - -** Docker Bridge: -- docker0: 172.16.0.1/24 -- State: DOWN (NO-CARRIER) - -* Motherboard/Chipset - -** Chipset: Intel Z390 -- Host Bridge: 8th/9th Gen Core 8-core Desktop Processor (Coffee Lake S) -- ISA Bridge: Z390 Chipset LPC/eSPI Controller (rev 10) -- PCIe Root Ports: Multiple Cannon Lake PCH PCI Express ports - -** Integrated Components: -- Graphics: Intel CoffeeLake-S GT2 [UHD Graphics 630] (rev 02) -- Audio: Intel Cannon Lake PCH cAVS (rev 10) -- USB: Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) -- Thermal: Cannon Lake PCH Thermal Controller (rev 10) -- HECI: Cannon Lake PCH HECI Controller (rev 10) -- SMBus: Cannon Lake PCH SMBus Controller (rev 10) -- SPI: Cannon Lake PCH SPI Controller (rev 10) - -* ZFS Pool Configuration - -** boot-pool (1.86 TB): -- Allocated: 3.06 GB -- Free: 1.86 TB -- Capacity: 0% -- Health: ONLINE - -** sysdata (1.86 TB): -- Allocated: 314 MB -- Free: 1.86 TB -- Capacity: 0% -- Health: ONLINE -- Purpose: System data and applications - -** tank (72.8 TB): -- Allocated: 1.17 MB -- Free: 72.7 TB -- Capacity: 0% -- Health: ONLINE -- Purpose: Large storage pool (empty, ready for expansion) - -** vault (43.6 TB): -- Allocated: 24.5 TB -- Free: 19.1 TB -- Capacity: 56% -- Health: ONLINE -- Purpose: Main media storage - -* Vault Datasets - -** Media (Movies, Music, TV Shows): -- Size: 30 TB (allocated) -- Used: 17 TB -- Available: 14 TB -- Capacity: 54% - -** Lectures: -- Size: 15 TB (allocated) -- Used: 878 GB -- Available: 14 TB -- Capacity: 6% - -** Audiobooks: -- Size: 14 TB (allocated) -- Used: 89 GB -- Available: 14 TB -- Capacity: 1% - -** Magic: -- Size: 15 TB (allocated) -- Used: 543 GB -- Available: 14 TB -- Capacity: 4% - -** Books: -- Size: 14 TB (allocated) -- Used: 177 GB -- Available: 14 TB -- Capacity: 2% - -* Security Mitigations - -** CPU Vulnerabilities (Status): -- Gather data sampling: Mitigated (Microcode) -- Indirect target selection: Mitigated (Aligned branch/return thunks) -- Itlb multihit: KVM Mitigation (Split huge pages) -- L1tf: Not affected -- Mds: Not affected -- Meltdown: Not affected -- Mmio stale data: Mitigated (Clear CPU buffers; SMT vulnerable) -- Reg file data sampling: Not affected -- Retbleed: Mitigated (Enhanced IBRS) -- Spec rstack overflow: Not affected -- Spec store bypass: Mitigated (Disabled via prctl) -- Spectre v1: Mitigated (usercopy/swapgs barriers) -- Spectre v2: Mitigated (Enhanced/Automatic IBRS) -- Srbds: Mitigated (Microcode) -- Tsx async abort: Mitigated (TSX disabled) - -* Full PCI Device List - -#+BEGIN_EXAMPLE -00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d) -00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d) -00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02) -00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) -00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) -00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) -00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) -00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) -00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0) -00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0) -00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0) -00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10) -00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) -00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) -00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) -00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10) -01:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) -02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) -03:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) -04:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) -#+END_EXAMPLE diff --git a/docs/homelab-inventory/mybitch-laptop.org b/docs/homelab-inventory/mybitch-laptop.org deleted file mode 100644 index e522c40..0000000 --- a/docs/homelab-inventory/mybitch-laptop.org +++ /dev/null @@ -1,238 +0,0 @@ -#+TITLE: mybitch - Christine's Laptop -#+DATE: 2026-01-31 -#+HOSTNAME: mybitch - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. - -:PROPERTIES: -:FS: ext4 -:PM: apt -:ORCH: none -:SNAPSHOT: timeshift -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt -:INIT: systemd -:LAST_AUDIT: 2026-06-23 -:END: - -* Overview - -| Field | Value | -|----------+-----------------------------------| -| Hostname | mybitch | -|----------+-----------------------------------| -| Type | Laptop | -|----------+-----------------------------------| -| Owner | Christine | -|----------+-----------------------------------| -| Maker | Framework | -|----------+-----------------------------------| -| Model | Laptop 16 (AMD Ryzen 7040 Series) | -|----------+-----------------------------------| -| OS | Linux Mint 22.3 (Zena) | -|----------+-----------------------------------| -| Kernel | 6.17.0-23-generic | -|----------+-----------------------------------| -| Status | In use | -|----------+-----------------------------------| - -* Hardware Specifications - -** CPU - -| Field | Value | -|--------------+-----------------------------------| -| Model | AMD Ryzen 9 7940HS w/ Radeon 780M | -|--------------+-----------------------------------| -| Architecture | Zen 4 (TSMC 5nm) | -|--------------+-----------------------------------| -| Cores | 8 | -|--------------+-----------------------------------| -| Threads | 16 | -|--------------+-----------------------------------| -| Base Clock | 4.0 GHz | -|--------------+-----------------------------------| -| Boost Clock | 5.25 GHz | -|--------------+-----------------------------------| -| L2 Cache | 8 MiB (8x1024 KiB) | -|--------------+-----------------------------------| -| L3 Cache | 16 MiB | -|--------------+-----------------------------------| - -** Memory - -| Field | Value | -|-----------+------------------------------| -| Total | 32 GB | -|-----------+------------------------------| -| Type | DDR5-5600 | -|-----------+------------------------------| -| Installed | 1x 32GB (A-DATA AD5S560032G) | -|-----------+------------------------------| -| Slots | 2 (1 free) | -|-----------+------------------------------| -| Max | 64 GB | -|-----------+------------------------------| - -** Graphics - -| Device | Model | Driver | Notes | -|------------+---------------------+--------+-----------------| -| Dedicated | AMD Radeon RX 7700S | amdgpu | RDNA 3, Navi 33 | -|------------+---------------------+--------+-----------------| -| Integrated | AMD Radeon 780M | amdgpu | RDNA 3, Phoenix | -|------------+---------------------+--------+-----------------| - -** Storage - -| Device | Model | Size | Interface | -|--------------+---------------------+----------+--------------| -| /dev/nvme0n1 | WD BLACK SN770M 2TB | 1.82 TiB | PCIe Gen4 x4 | -|--------------+---------------------+----------+--------------| -| /dev/nvme1n1 | WD BLACK SN770 2TB | 1.82 TiB | PCIe Gen4 x4 | -|--------------+---------------------+----------+--------------| -| Total | | 3.64 TiB | | -|--------------+---------------------+----------+--------------| - -*** Partitions - -| Partition | Size | Used | Filesystem | Mount | Label | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme1n1p2 | 1.82 TiB | 17.4% | ext4 | / | | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme1n1p1 | 512 MiB | 1.2% | vfat | /boot/efi | | -|----------------+----------+-------------+------------+-----------+--------| -| /dev/nvme0n1p1 | 1.82 TiB | (unmounted) | ext4 | | Backup | -|----------------+----------+-------------+------------+-----------+--------| - -** Display - -| Field | Value | -|------------+--------------------| -| Panel | BOE Display 0x0bc9 | -|------------+--------------------| -| Resolution | 2560x1600 | -|------------+--------------------| -| Size | 16" (345x215mm) | -|------------+--------------------| -| Ratio | 16:10 | -|------------+--------------------| -| DPI | 188 | -|------------+--------------------| - -** Battery - -| Field | Value | -|----------+------------------| -| Model | NVT FRANDBA | -|----------+------------------| -| Capacity | 85.1 Wh (design) | -|----------+------------------| -| Current | 87.5 Wh (102.9%) | -|----------+------------------| -| Cycles | 13 | -|----------+------------------| -| Type | Li-ion | -|----------+------------------| - -** Network - -| Interface | Type | Chipset | -|-----------+--------+--------------------------| -| wlp5s0 | WiFi | MediaTek MT7922 802.11ax | -|-----------+--------+--------------------------| -| Bluetooth | BT 5.2 | MediaTek Wireless_Device | -|-----------+--------+--------------------------| - -** Expansion Cards (Framework) - -| Slot | Card | -|------+---------------------| -| 1 | HDMI Expansion Card | -|------+---------------------| - -** Input Modules (Framework) - -| Module | Notes | -|----------+-------------------------| -| Keyboard | ANSI layout | -|----------+-------------------------| -| Numpad | Laptop 16 Numpad Module | -|----------+-------------------------| - -* Connected Peripherals - -| Device | Connection | -|---------------------------+-------------------------| -| Logitech MX Master 3 | Unifying Receiver (USB) | -|---------------------------+-------------------------| -| Realtek Laptop Camera | Internal USB | -|---------------------------+-------------------------| -| Goodix Fingerprint Reader | Internal USB | -|---------------------------+-------------------------| - -* Network Access - -| Method | Address | -|--------+---------------| -| mDNS | mybitch.local | -|--------+---------------| -| SSH | Yes (OpenSSH) | -|--------+---------------| - -* Notes - -- Framework Laptop 16 with modular expansion card system -- Discrete AMD GPU (RX 7700S) in addition to integrated Radeon 780M -- Second NVMe slot contains 2TB backup drive (unmounted, labeled "Backup") -- Uptime at inventory: 4 days - -* Operational Changes Log - -Deliberate, non-default config changes applied to mybitch (so they can be found and reverted later). Most recent first. - -** 2026-05-12 — amdgpu.mes=0 added (escalation of the MES-hang freeze mitigation) - -The 2026-05-11 =amdgpu.cwsr_enable=0= mitigation didn't reduce the iGPU "MES failed to respond" errors (14 hits in 14.5h after the fix vs. 38 over 10 days before — rate went *up*), so escalated per the documented path: added =amdgpu.mes=0= to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= (now =quiet splash amdgpu.cwsr_enable=0 amdgpu.mes=0=). Backup: =/etc/default/grub.bak-2026-05-12-amdgpu-mes=. =update-grub= run; param verified in all 6 =/boot/grub/grub.cfg= entries. Takes effect on next reboot (planned for after the in-progress first rsyncshot backup). =amdgpu.mes=0= disables the GFX11 hardware MES scheduler and falls back to the legacy KIQ path — sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks); irrelevant for Christine's light desktop workload. =cwsr_enable=0= kept (harmless alongside mes=0). *Revert:* restore =/etc/default/grub.bak-2026-05-12-amdgpu-mes= (or sed out = amdgpu.mes=0=), =update-grub=, reboot. After the reboot, check =journalctl -k -b | grep -c 'MES failed'= → should be 0. See the 2026-05-11 entry in =.ai/project-workflows/system-health-check.org='s Known Issues Log for the full background. - -** 2026-05-12 — rsyncshot backups + travel keep-awake mode - -Set up the same =rsyncshot= → TrueNAS backup mybitch's siblings use, and made mybitch never suspend so backups keep running while Christine travels with it. - -*** rsyncshot install (mirrors ratio's setup) -- =/usr/local/bin/rsyncshot= — the script (source of truth: =~/code/rsyncshot/rsyncshot=). -- =/etc/rsyncshot/config= — =REMOTE_HOST="cjennings@truenas.tailf3bb8c.ts.net"=, =REMOTE_PATH="/mnt/vault/backups"=, =SSH_IDENTITY_FILE="/root/.ssh/id_ed25519"=. (See "destination pinned to Tailscale" below for why the full MagicDNS name.) -- =/etc/rsyncshot/include.txt= — =/home /etc /usr/local/bin=. =/etc/rsyncshot/exclude.txt= — copied verbatim from ratio. -- =/etc/logrotate.d/rsyncshot= — copied verbatim from ratio. =/var/log/rsyncshot.log= — the log. -- Root crontab: hourly at :30 (hours 0-1,3-23, keep 23) + daily at 2:30 (keep 30), each wrapped in =flock -x /tmp/rsyncshot.lock=. Identical to ratio's schedule. -- =/root/.ssh/id_ed25519= (+ =.pub=) — new keypair generated on mybitch, no passphrase, comment =root@mybitch-rsyncshot=. mybitch's =cjennings= account had no outbound key (it's a receive-only account), so the backup runs with a dedicated root key instead. The pubkey was appended to =~cjennings/.ssh/authorized_keys= on TrueNAS (the line tagged =root@mybitch-rsyncshot=). -- =/root/.ssh/config= — =Host truenas truenas.tailf3bb8c.ts.net 100.67.22.65= → =User cjennings=, that IdentityFile, =ServerAliveInterval 60 / ServerAliveCountMax 10 / TCPKeepAlive yes=. Host keys for those names added to =/root/.ssh/known_hosts=. -- TrueNAS side: =/mnt/vault/backups/mybitch/= directory created, owned =cjennings:cjennings= (=vault/backups= is a ZFS dataset; per-host backups are just subdirectories, like =ratio/= and =velox/=). -- *Revert:* =sudo crontab -r= (or remove the rsyncshot lines), =sudo rm -rf /usr/local/bin/rsyncshot /etc/rsyncshot /etc/logrotate.d/rsyncshot /var/log/rsyncshot.log /root/.ssh/id_ed25519* /root/.ssh/config=, restore =/root/.ssh/known_hosts= if desired, remove the =root@mybitch-rsyncshot= line from =~cjennings/.ssh/authorized_keys= on TrueNAS, and =rm -rf /mnt/vault/backups/mybitch= on TrueNAS. - -*** destination pinned to Tailscale -=REMOTE_HOST= uses the full Tailscale MagicDNS name =truenas.tailf3bb8c.ts.net= (not the bare =truenas=) so the backup always routes over the tailnet regardless of what a hotel/away-from-home router's DNS resolves =truenas= to. Tailscale uses a direct LAN path between same-LAN peers, so this is near-zero perf cost when mybitch is on the home network. *Revert:* set =REMOTE_HOST="cjennings@truenas"= in =/etc/rsyncshot/config=. - -*** keep-awake (never suspend) — three layers -1. =systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target= — the hard guarantee; suspend can't happen even if requested. *Revert:* =sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target=. -2. Cinnamon power gsettings for user =cciarm= set to ='nothing'=: =sleep-inactive-ac-type=, =sleep-inactive-battery-type=, =lid-close-ac-action=, =lid-close-battery-action= (schema =org.cinnamon.settings-daemon.plugins.power=). Were ='suspend'/'suspend'/'nothing'/'suspend'=. *Revert:* set them back, or run =gsettings reset= on each key as =cciarm=. -3. =/etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= — =HandleLidSwitch=ignore=, =HandleLidSwitchExternalPower=ignore=, =HandleLidSwitchDocked=ignore=, =IdleAction=ignore=. (Written but not applied live — =systemd-logind= wasn't restarted because sessions were active; takes effect on next reboot. Layers 1-2 cover the gap.) *Revert:* =sudo rm /etc/systemd/logind.conf.d/no-suspend-keep-awake.conf= then reboot or =systemctl restart systemd-logind=. - -When mybitch goes back to being a stationary machine (or this whole arrangement is no longer wanted), reverse layers 1-3. The rsyncshot backup itself is worth keeping regardless. - -*** IPv6 set to link-local-only (fixes the tailscale dropout) -mybitch dropped off the Tailscale tailnet on 2026-05-12 AM. Root cause: the home WiFi ("dupre") has device(s) advertising ULA-only IPv6 RAs (prefixes =fdc4:c4c6:3fb0:16cc::/64= and =fd67:8615:6780:1::/64=) with no default route. NetworkManager (=ipv6.method=auto=) SLAAC'd a "global"-scope ULA address onto wlp5s0 with no v6 internet path, so tailscaled kept trying IPv6 to controlplane/DERP, failed "network is unreachable", flapped, and lost its control connection. mybitch has no working global IPv6 anywhere (no ISP v6), so the fix is to ignore underlay v6: -- =/etc/NetworkManager/conf.d/ipv6-link-local-only.conf= → =[connection] ipv6.method=link-local= (connection default — covers home WiFi, travel WiFi, phone hotspot). -- =nmcli con modify dupre ipv6.method link-local= (explicit on the home connection). -- Effect: wlp5s0 keeps only fe80:: (mDNS still works), no SLAAC, no v6 routes; tailscale0's overlay v6 untouched; tailscaled uses the IPv4 underlay. Activates on reboot or =nmcli dev reapply wlp5s0=. -- *Revert if mybitch ever gets real IPv6:* =sudo rm /etc/NetworkManager/conf.d/ipv6-link-local-only.conf=; =sudo nmcli con modify dupre ipv6.method auto=; reboot or =nmcli dev reapply wlp5s0=. - -*** caveats (relayed to Craig for Christine) -- mybitch must stay plugged into AC for backups to keep running — masking suspend doesn't stop a battery dying with the lid closed. -- mybitch must be on WiFi at the destination and stay on the Tailscale tailnet. diff --git a/docs/homelab-inventory/ratio-desktop.org b/docs/homelab-inventory/ratio-desktop.org deleted file mode 100644 index 234c1a4..0000000 --- a/docs/homelab-inventory/ratio-desktop.org +++ /dev/null @@ -1,175 +0,0 @@ -#+TITLE: ratio - Desktop Workstation -#+DATE: 2026-01-27 -#+HOSTNAME: ratio - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. - -:PROPERTIES: -:FS: btrfs -:PM: pacman -:ORCH: topgrade -:SNAPSHOT: snapper -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt,docker,podman -:INIT: systemd -:LAST_AUDIT: 2026-07-08 -:END: - -* Overview - -| Field | Value | -|-------------+------------------------------------------| -| Hostname | ratio | -|-------------+------------------------------------------| -| Type | Desktop workstation | -|-------------+------------------------------------------| -| Maker | Framework | -|-------------+------------------------------------------| -| Model | Desktop (AMD Ryzen AI Max 300 Series) | -|-------------+------------------------------------------| -| Board | FRANMFCP06 v A6 | -|-------------+------------------------------------------| -| Firmware | INSYDE UEFI v03.03 (2025-09-16) | -|-------------+------------------------------------------| -| OS | Arch Linux | -|-------------+------------------------------------------| -| Kernel | 6.12.67-1-lts | -|-------------+------------------------------------------| -| Desktop | Hyprland (Wayland) | -|-------------+------------------------------------------| -| Previous ID | cogito (repurposed and renamed to ratio) | -|-------------+------------------------------------------| - -* CPU - -| Field | Value | -|---------------+---------------------------------------| -| Model | AMD Ryzen AI MAX+ 395 w/ Radeon 8060S | -|---------------+---------------------------------------| -| Architecture | Zen 5 (TSMC 4nm) | -|---------------+---------------------------------------| -| Cores/Threads | 16 cores / 32 threads | -|---------------+---------------------------------------| -| Base/Boost | 3.0 GHz / 5.15 GHz | -|---------------+---------------------------------------| -| Cache L1 | 1.2 MiB (d-16x48 KiB; i-16x32 KiB) | -|---------------+---------------------------------------| -| Cache L2 | 16 MiB (16x1024 KiB) | -|---------------+---------------------------------------| -| Cache L3 | 64 MiB (2x32 MiB) | -|---------------+---------------------------------------| -| NPU | XDNA 2, 50+ peak AI TOPS | -|---------------+---------------------------------------| -| Peak Perf | 59.4 FP16/BF16 TFLOPS @ 2.9GHz | -|---------------+---------------------------------------| - -* GPU - -| Field | Value | -|--------------+-----------------------------------------| -| Model | AMD Radeon 8060S (Strix Halo) | -|--------------+-----------------------------------------| -| Architecture | RDNA 3.5 | -|--------------+-----------------------------------------| -| CUs | 40 Compute Units | -|--------------+-----------------------------------------| -| Driver | amdgpu (kernel) | -|--------------+-----------------------------------------| -| Max VRAM | 96GB (via AMD Variable Graphics Memory) | -|--------------+-----------------------------------------| -| GPU Arch ID | gfx1151 | -|--------------+-----------------------------------------| -| PCIe | Gen 4, 16 GT/s, 16 lanes | -|--------------+-----------------------------------------| - -* Memory - -| Field | Value | -|--------------+--------------------------| -| Total | 128 GiB unified (LPDDR5) | -|--------------+--------------------------| -| Speed | 8000 MT/s | -|--------------+--------------------------| -| Channels | 8 (8x 16 GiB) | -|--------------+--------------------------| -| Manufacturer | Micron Technology | -|--------------+--------------------------| -| Part Number | MT62F4G32D8DV-023 WT | -|--------------+--------------------------| - -* Storage - -| Device | Model | Size | Interface | -|--------------+------------------------+-----------+-----------| -| /dev/nvme0n1 | WD BLACK SN850X 8000GB | 7.28 TiB | NVMe | -|--------------+------------------------+-----------+-----------| -| /dev/nvme1n1 | WD BLACK SN850X 8000GB | 7.28 TiB | NVMe | -|--------------+------------------------+-----------+-----------| -| *Total* | | 14.55 TiB | | -|--------------+------------------------+-----------+-----------| - -Filesystem: btrfs with subvolumes (@, @home, @snapshots, @log, @pkg) - -* Network - -| Interface | Chipset | Speed | Type | -|-----------+-----------------+---------+----------| -| enp191s0 | Realtek RTL8126 | 5 GbE | Wired | -|-----------+-----------------+---------+----------| -| wlp192s0 | MediaTek MT7925 | Wi-Fi 7 | Wireless | -|-----------+-----------------+---------+----------| - -Also has: docker0, tailscale0, virbr0 (virtual interfaces) - -* Audio - -- AMD Radeon High Definition Audio (HDMI, snd_hda_intel) -- AMD Ryzen HD Audio (3.5mm, snd_hda_intel) -- PipeWire v1.4.10 (pipewire-pulse, wireplumber) - -* PCI Slots - -| Slot | Info | Status | -|------+------------+-----------| -| 1 | M.2 JWLAN | Available | -|------+------------+-----------| -| 2 | M.2 JSSD1 | In use | -|------+------------+-----------| -| 3 | M.2 JSSD2 | In use | -|------+------------+-----------| -| 4 | PCIe Gen 4 | Available | -|------+------------+-----------| - -* Connected Peripherals - -- Monitor: Dell U3419W (HDMI-A-1, 3440x1440@60Hz) -- Keyboard: Das Keyboard 4 (Cherry MX Blue) — replacing with Keychron Q6 Pro -- Mouse: Logitech M650 -- DAC/Amp: JDS Labs Element IV (ordered) - -* AI/LLM Capability - -Primary purpose includes local AI/LLM inference via Ollama. - -| Model | Size | Purpose | -|------------------+-------+----------------------| -| deepseek-r1:70b | 42 GB | Main reasoning model | -|------------------+-------+----------------------| -| qwen3:30b | 18 GB | MoE chat (256K ctx) | -|------------------+-------+----------------------| -| deepseek-r1:14b | 9 GB | Light reasoning | -|------------------+-------+----------------------| -| nomic-embed-text | 274MB | RAG embeddings | -|------------------+-------+----------------------| - -Inference benchmarks (AMD testing, LM Studio 0.3.11): -- Small models (1-3B): ~100+ tok/s -- Medium models (7-8B): ~60-80 tok/s -- Large models (20B): ~58 tok/s -- Very large models (120B): ~38 tok/s diff --git a/docs/homelab-inventory/truenas-server.org b/docs/homelab-inventory/truenas-server.org deleted file mode 100644 index 5ce8613..0000000 --- a/docs/homelab-inventory/truenas-server.org +++ /dev/null @@ -1,114 +0,0 @@ -#+TITLE: truenas - NAS/Storage Server -#+DATE: 2026-01-27 -#+HOSTNAME: truenas - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. -# -# NOTE: TrueNAS SCALE is middleware-managed. No user-level package manager -# or topgrade orchestration. Snapshot and update operations go through the -# TrueNAS UI / API, not pacman/apt. - -:PROPERTIES: -:FS: zfs -:PM: none -:ORCH: none -:SNAPSHOT: zfs-native -:MESH: tailscale -:BACKUP: target -:VIRT: docker -:INIT: systemd -:LAST_AUDIT: 2026-04-21 -:END: - -* Overview - -| Field | Value | -|----------+-----------------------------------------| -| Hostname | truenas (truenas.local / 192.168.86.5) | -|----------+-----------------------------------------| -| Type | NAS / storage server | -|----------+-----------------------------------------| -| OS | TrueNAS | -|----------+-----------------------------------------| -| Purpose | Media storage, backups, Plex, Syncthing | -|----------+-----------------------------------------| - -See [[file:2026-01-05-truenas-hardware-specs.org][TrueNAS Hardware Specs]] for full sysinfo output. - -* CPU - -| Field | Value | -|---------------+----------------------| -| Model | Intel Core i9-9900K | -|---------------+----------------------| -| Architecture | Coffee Lake | -|---------------+----------------------| -| Cores/Threads | 8 cores / 16 threads | -|---------------+----------------------| -| Base/Boost | 3.60 GHz / 5.00 GHz | -|---------------+----------------------| - -* Memory - -| Field | Value | -|-------+-------------| -| Total | 64 GiB DDR4 | -|-------+-------------| - -* Storage - -| Pool | Config | Raw Size | Purpose | -|-----------+-----------------+------------+---------------| -| boot-pool | Mirror | ~1.9 TB x2 | System | -|-----------+-----------------+------------+---------------| -| sysdata | NVMe | ~1.8 TB | System data | -|-----------+-----------------+------------+---------------| -| vault | RAIDZ1 (4 disk) | ~43.6 TB | Primary media | -|-----------+-----------------+------------+---------------| -| tank | RAIDZ1 (4 disk) | ~72.8 TB | Expansion | -|-----------+-----------------+------------+---------------| - -Total raw: ~122 TB - -*** Vault Datasets -| Dataset | Size | -|------------+-------| -| Media | 30 TB | -|------------+-------| -| Lectures | 15 TB | -|------------+-------| -| Audiobooks | 14 TB | -|------------+-------| -| Magic | 15 TB | -|------------+-------| -| Books | 14 TB | -|------------+-------| - -* Network - -| Interface | Chipset | Speed | Status | -|-----------+--------------+--------+--------------| -| eno1 | Intel I219-V | 1 GbE | Current | -|-----------+--------------+--------+--------------| -| enp2s0 | Intel I211 | 1 GbE | Available | -|-----------+--------------+--------+--------------| -| (pending) | Intel I226-V | 2.5GbE | Ordered, NIC | -|-----------+--------------+--------+--------------| - -* Motherboard - -| Field | Value | -|---------+------------| -| Chipset | Intel Z390 | -|---------+------------| - -* Services - -- Plex Media Server -- Syncthing -- Tailscale diff --git a/docs/homelab-inventory/velox-laptop.org b/docs/homelab-inventory/velox-laptop.org deleted file mode 100644 index 6e4f540..0000000 --- a/docs/homelab-inventory/velox-laptop.org +++ /dev/null @@ -1,158 +0,0 @@ -#+TITLE: velox - Laptop -#+DATE: 2026-01-27 -#+HOSTNAME: velox - -* Automated Capabilities - -# Maintained by the system-health-check workflow. Manual edits may be -# overwritten when the workflow detects drift between this drawer and the -# live system. Machine-readable capability signals used to dispatch which -# checks run on this host. -# -# NOTE: 2026-04 velox was reinstalled Arch-on-ZFS. Values below reflect -# that reinstall but remain TBD where a live probe has not yet confirmed. - -:PROPERTIES: -:FS: zfs -:PM: pacman -:ORCH: topgrade -:SNAPSHOT: sanoid -:MESH: tailscale -:BACKUP: source -:VIRT: libvirt,docker,podman -:INIT: systemd -:LAST_AUDIT: 2026-05-26 -:END: - -* Overview - -| Field | Value | -|----------+---------------------------------| -| Hostname | velox | -|----------+---------------------------------| -| Type | Laptop | -|----------+---------------------------------| -| Maker | Framework | -|----------+---------------------------------| -| Model | Laptop (13th Gen Intel Core) | -|----------+---------------------------------| -| Board | FRANMCCP07 v A7 | -|----------+---------------------------------| -| Firmware | INSYDE UEFI v03.07 (2024-12-26) | -|----------+---------------------------------| -| OS | Arch Linux | -|----------+---------------------------------| -| Kernel | 6.18.6-arch1-1 | -|----------+---------------------------------| -| Desktop | startx (X11) | -|----------+---------------------------------| - -* CPU - -| Field | Value | -|---------------+-----------------------------------------| -| Model | 13th Gen Intel Core i7-1370P | -|---------------+-----------------------------------------| -| Architecture | Raptor Lake (Intel 7 / 10nm) | -|---------------+-----------------------------------------| -| Cores/Threads | 14 cores / 20 threads (6P + 8E, hybrid) | -|---------------+-----------------------------------------| -| Base/Boost | 990 MHz / 5.2 GHz | -|---------------+-----------------------------------------| -| Cache L1 | 1.2 MiB (d-8x32K,6x48K; i-6x32K,8x64K) | -|---------------+-----------------------------------------| -| Cache L2 | 11.5 MiB (6x1.2M, 2x2M) | -|---------------+-----------------------------------------| -| Cache L3 | 24 MiB (1x24M) | -|---------------+-----------------------------------------| -| Socket | BGA1744 (U3E1) | -|---------------+-----------------------------------------| - -* GPU - -| Field | Value | -|--------------+------------------------| -| Model | Intel Iris Xe Graphics | -|--------------+------------------------| -| Architecture | Xe (Intel 7) | -|--------------+------------------------| -| Driver | i915 / modesetting | -|--------------+------------------------| - -* Memory - -| Field | Value | -|----------+------------------------------------| -| Total | 64 GiB (DDR4) | -|----------+------------------------------------| -| Speed | 3200 MT/s | -|----------+------------------------------------| -| Slots | 2 (both populated) | -|----------+------------------------------------| -| Device 1 | 32 GiB Crucial CT32G4SFD832A.C16FB | -|----------+------------------------------------| -| Device 2 | 32 GiB Crucial CT32G4SFD832A.C16FE | -|----------+------------------------------------| - -* Storage - -| Device | Model | Size | Interface | -|--------------+----------------------+----------+-----------| -| /dev/nvme0n1 | Sabrent SB-RKT4P-8TB | 7.28 TiB | NVMe | -|--------------+----------------------+----------+-----------| - -- Filesystem: ZFS on root (zroot pool on nvme0n1p2; reinstalled 2026-04) -- Datasets: zroot/ROOT/default, zroot/home, zroot/var/log, zroot/var/lib/pacman, zroot/var/lib/docker (per-layer), zroot/vms, zroot/media -- Snapshots: sanoid hourly + zfs-scrub-weekly@zroot -- SMART: PASSED (re-verified 2026-04-26); composite 43 °C; 0 media errors -- Power-on history pre-dates reinstall — refresh on next full re-inventory - -* Display - -| Field | Value | -|------------+---------------------------------| -| Panel | BOE Display 0x0bca (built 2022) | -|------------+---------------------------------| -| Resolution | 2256x1504 | -|------------+---------------------------------| -| Size | 13.5" (285x190mm) | -|------------+---------------------------------| -| DPI | 201 | -|------------+---------------------------------| -| Ratio | 3:2 | -|------------+---------------------------------| - -* Network - -| Interface | Chipset | Speed | Type | -|-------------+--------------------------------+---------+----------| -| wlp170s0 | Qualcomm Atheros AR9462 | Wi-Fi | Wireless | -|-------------+--------------------------------+---------+----------| -| enp0s13f0u3 | Realtek USB 10/100/1G/2.5G LAN | 2.5 GbE | USB | -|-------------+--------------------------------+---------+----------| - -Also has: docker0, tailscale0, virbr0 (virtual interfaces) - -* Audio - -- Intel Raptor Lake-P/U/H cAVS (snd_hda_intel) -- PipeWire v1.4.10 (pipewire-pulse, wireplumber) - -* Battery - -| Field | Value | -|-----------+-----------------------| -| Capacity | 55 Wh (design) | -|-----------+-----------------------| -| Condition | 44/55 Wh (80% health) | -|-----------+-----------------------| -| Cycles | 146 | -|-----------+-----------------------| -| Type | Li-ion | -|-----------+-----------------------| - -* Other Hardware - -- Webcam: Realtek Laptop Camera (UVC) -- Fingerprint: Shenzhen Goodix USB Device -- Bluetooth: Foxconn (BT 4.0) diff --git a/docs/workflows/strix-soak-watch.org b/docs/workflows/strix-soak-watch.org new file mode 100644 index 0000000..e2fbf68 --- /dev/null +++ b/docs/workflows/strix-soak-watch.org @@ -0,0 +1,146 @@ +#+TITLE: Strix Halo SMU Deadlock — Kernel Update Watch +#+AUTHOR: Craig Jennings & Claude +#+DATE: 2026-05-10 + +* Purpose + +Companion to =system-health-check.org=. Triggered when linux / linux-lts / linux-firmware land in =checkupdates= on ratio, this addendum checks whether the upstream side of the Strix Halo SMU deadlock has moved — so we know when to retire the locally-patched =linux-lts-strix= kernel and switch back to stock. + +This is a one-time hardware/kernel issue. Delete this file when the strix kernel is retired. + +* Current State (as of 2026-05-10) + +- *Boot kernel:* =linux-lts-strix 6.18.25-1= (custom AUR-style local build), GRUB default pinned +- *SMC firmware:* =100.6.0= (from =/sys/kernel/debug/dri/*/amdgpu_firmware_info= SMC line) +- *Cmdline workaround:* =amdgpu.no_vpe_idle_pg=1= active +- *Soak verdict:* HOLDING since 2026-04-30 — zero SMU/dcn35/VPE freeze signature hits (see =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project)) + +* Background (one-paragraph summary so future sessions don't have to reconstruct it) + +Strix Halo (gfx_v11_5_0, smu_v14_0_2 APU) has an SMU firmware deadlock triggered during VPE (Video Processing Engine) idle power-gating: hard freeze, no journal output, requires power-cycle. Crupi posted a workaround patch on the Framework community forum that disables VPE idle power-gating via =amdgpu.no_vpe_idle_pg=. Real fix has to come from AMD as a PMFW/SMU firmware update (delivered via =linux-firmware=). Until then, ratio runs a custom kernel build that carries the patch + two supporting amdgpu allocation-in-critical-path patches. Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 + +* Local Patch Set + +Located at =~/build/linux-lts-patched/linux-lts/=: + +| File | Strix-relevant? | +|---------------------------------------------------------------------------------+-----------------| +| =0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-CLONE_NEWUSER.patch= | No (security) | +| =0002-drm-amdgpu-avoid-memory-allocation-in-the-critical-code-path-v3.patch= | Yes | +| =0003-drm-amdgpu-use-GFP_ATOMIC-instead-of-NOWAIT-in-the-critical-path.patch= | Yes | +| =0004-drm-amdgpu-add-no_vpe_idle_pg-strix-halo-smu-deadlock-workaround.patch= | Yes (Crupi's) | + +* When This Watch Fires + +Phase 3 of =system-health-check.org= invokes this addendum when *any* of the following appear in =checkupdates=: + +- =linux= (any version bump) +- =linux-lts= (any version bump) +- =linux-firmware= (any version bump — this is the most likely vector for the real fix) +- =mesa= (major version only, e.g. 26.x.y → 27.x.y — display-stack timing can shift the deadlock window) + +* What To Check On Each Fire + +** 1. Stable kernel changelogs (per bumped version) + +Fetch the per-version changelog and search for the keyword set below. The 2026-05-10 baseline pass found *no matches* in 6.18.27, 6.18.28, 7.0.4, 7.0.5 — so anything that does match is news. + +#+begin_src +https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-<NEW-LTS-VERSION> +https://cdn.kernel.org/pub/linux/kernel/v7.x/ChangeLog-<NEW-MAINLINE-VERSION> +#+end_src + +Keyword set (case-insensitive): +- =no_vpe_idle_pg=, =vpe_idle=, =VPE idle=, =VPE power gating= +- =Strix Halo=, =STRIX_HALO=, =strix_halo= +- =gfx_v11_5= (the GFX block id for Strix Halo) +- =smu_v14= (the SMU IP block driver, =drivers/gpu/drm/amd/pm/swsmu/smu14=) +- =dcn351= (Strix Halo display controller; =dcn35= is Phoenix-class — adjacent but not the same) +- =drm/amdgpu/vpe=, =drm/amdgpu/swsmu/smu14=, =amdgpu/pm/swsmu= +- =SMU deadlock=, =critical path= (combined with amdgpu/drm) + +The two adjacent allocation-fix patches (0002, 0003) may land separately from Crupi's patch. They have distinctive subject lines: =avoid memory allocation in the critical code path= and =use GFP_ATOMIC instead of NOWAIT in the critical path=. Search for those too. + +** 2. linux-firmware delta (if linux-firmware is the bumped package) + +The canonical fix arrives as a PMFW/SMU firmware update, NOT a kernel patch. + +#+begin_src bash +# Pre-bump SMC version (from running system, BEFORE topgrade) +sudo cat /sys/kernel/debug/dri/*/amdgpu_firmware_info | grep "SMC feature" + +# After topgrade installs the new linux-firmware, examine the delta. +# Watch for these files in the upgraded package: +# amdgpu/smu_14_0_2.bin — Strix Halo SMU firmware +# amdgpu/psp_14_0_*.bin — PSP/secure-os blobs +# amdgpu/strix* — anything Strix-tagged +pacman -Ql linux-firmware | grep -iE "smu_14_0_2|psp_14_0|strix" + +# Reboot is required to load the new firmware. Re-check SMC version post-reboot. +#+end_src + +A bump from =100.6.0= to anything higher is the canonical Strix-fix signal — that means AMD shipped new SMU firmware. Once that lands, the workaround kernel is no longer needed (in theory; see "Outcomes" below for the validation path). + +** 3. Crupi's forum thread + +#+begin_src +https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 +#+end_src + +Skim recent posts for: +- "Merged upstream" / "Landed in 6.x.y" announcements +- AMD acknowledgements or workaround retirement notes +- Other affected users reporting fix verification on stock kernels + +** 4. amd-pm subsystem mailing list (optional, only if 1-3 are inconclusive) + +#+begin_src +https://lore.kernel.org/amd-gfx/?q=strix+halo+smu+deadlock +https://lore.kernel.org/amd-gfx/?q=no_vpe_idle_pg +#+end_src + +* Outcomes & Actions + +** Nothing Strix-adjacent in the bump (default — also the 2026-05-10 outcome) + +- Continue running =linux-lts-strix 6.18.25-1= +- Do *not* rebuild the strix kernel against the newer base purely for the version bump. Same four patches against a newer base = same workaround behavior + new risk of unrelated base-version regressions. Only rebuild when there's a security-pressing reason. +- Topgrade is safe; the new stock =linux-lts= installs to disk but isn't booted. + +** linux-firmware bumps SMC past =100.6.0= + +This is the most likely real-fix path. + +1. Note the new SMC version (capture pre and post). +2. Schedule a controlled test boot: + - Backup current =/etc/default/grub= GRUB_DEFAULT and =linux-lts-strix= package + - Drop =amdgpu.no_vpe_idle_pg=1= from the cmdline on a test boot of stock =linux-lts= + - Re-run the soak-check script for 1 week (same harness used for the original verdict) +3. If zero signature hits over the soak window → retire =linux-lts-strix=, switch GRUB default back to stock =linux-lts=, remove the custom build dir. +4. If any freeze recurs → restore strix-lts as default and re-engage the workaround. Note the recurrence in this file and on the forum thread. + +** Crupi's patch (or equivalent) lands in 6.18.x stable + +1. Check the merged form: is the =no_vpe_idle_pg= behavior the default-on, or still gated behind a cmdline? +2. If default-on: rebuild =linux-lts-strix= against the new base with patch 0004 dropped (keep 0002, 0003, 0001 as long as they remain non-redundant). +3. If still cmdline-gated: keep the strix kernel as-is until linux-firmware fix arrives — the in-tree path is equivalent to the local one. + +** Patches 0002/0003 land upstream but Crupi's doesn't + +Drop the matching patches from the PKGBUILD on next strix rebuild; keep 0004 until the SMU firmware fix or kernel merge. + +* Reference Artifacts + +- Soak verdict (archived): =~/projects/home/assets/archive/2026-05-04-strix-soak-check-holding.org= (stays in the home project) +- VPE-patch deployment session: =.ai/sessions/2026-04-29-23-35-linux-lts-strix-vpe-patch-deployed.org= +- Soak fix + housekeeping session: =.ai/sessions/2026-05-03-06-48-soak-fix-and-velox-sync-housekeeping.org= +- Verdict-holds session: =.ai/sessions/2026-05-05-01-48-strix-verdict-holds-mybitch-soak-deferred.org= +- Forum thread: https://community.frame.work/t/smu-deadlock-system-freeze-on-fedora-43/81795 +- Build dir: =~/build/linux-lts-patched/linux-lts/= +- Inventory: the =ratio - Desktop Workstation= node in =~/org/roam/hardware/= (resolve by =#+HOSTNAME: ratio=) + +* Out of Scope + +- Generalizing this to other AMD APUs. The deadlock signature is specific to Strix Halo (gfx_v11_5_0, smu_v14_0_2). Other Phoenix / Hawk Point / RDNA-N APUs are unaffected and shouldn't be checked here. +- Tracking AMD's internal bug tracker. Stick to public signals (kernel.org changelogs, lore.kernel.org, the Framework forum thread). +- This is not a soak-check runner. The soak-check harness (=at=-job + scripts) lives separately under =scripts/= or =inbox/=. This file watches for the *retirement signal* for that harness. diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org index b66a186..98fa17f 100644 --- a/docs/workflows/system-health-check.org +++ b/docs/workflows/system-health-check.org @@ -8,7 +8,7 @@ This workflow performs a comprehensive diagnostic scan of whatever host it runs Run this when Craig asks "how is <host> doing?", "check my system health", or as a periodic maintenance check. -Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=. +Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live in the org-roam knowledge base at =~/org/roam/hardware/= (moved there 2026-07-09). Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3. @@ -23,20 +23,24 @@ Read the TOML at the start of Phase 1 and grade against its values. Where a chec * Hosts & Capabilities -The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift. +The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory nodes in the org-roam knowledge base declare the expected capabilities in a property drawer and act as a cross-check for drift. -Host inventory files (match by =#+HOSTNAME:= keyword inside each): +The inventories live at =~/org/roam/hardware/= (2026-07-09), alongside the rest of the hardware inventory. They moved out of this repo so any project's session can read a host's capabilities — and write back what a diagnosis discovers — without reaching across a project boundary. The index node "Homelab Hardware Inventory" links them all. -| Host | Inventory file | Notes | -|---------+-------------------------------------------+----------------------------------| -| ratio | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]] | Arch, Btrfs RAID1, pacman | -|---------+-------------------------------------------+----------------------------------| -| velox | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]] | Arch, ZFS (re-installed 2026-04) | -|---------+-------------------------------------------+----------------------------------| -| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt | -|---------+-------------------------------------------+----------------------------------| -| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM | -|---------+-------------------------------------------+----------------------------------| +Host inventory nodes (match by =#+HOSTNAME:= keyword inside each): + +| Host | Inventory node | Notes | +|---------+-----------------------------------------------+----------------------------------| +| ratio | =hardware/…-ratio_desktop_workstation.org= | Arch, Btrfs RAID1, pacman | +|---------+-----------------------------------------------+----------------------------------| +| velox | =hardware/…-velox_laptop.org= | Arch, ZFS (re-installed 2026-04) | +|---------+-----------------------------------------------+----------------------------------| +| mybitch | =hardware/…-mybitch_christines_laptop.org= | Mint, ext4, apt | +|---------+-----------------------------------------------+----------------------------------| +| truenas | =hardware/…-truenas_nas_storage_server.org= | TrueNAS SCALE, ZFS, no PM | +|---------+-----------------------------------------------+----------------------------------| + +Never hardcode a node's filename — its timestamp prefix is not stable across a rename. Resolve by the =#+HOSTNAME:= keyword, as Phase 0 does. Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift. @@ -108,10 +112,10 @@ Present the probe result as a one-line capability summary at the top of the repo Locate the inventory file for this host: #+begin_src bash -inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1) +inv=$(grep -l "^#+HOSTNAME: $host\b" "${ROAM_DIR:-$HOME/org/roam}"/hardware/*.org 2>/dev/null | head -1) #+end_src -If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. +If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check. A host without the roam clone (mybitch, truenas) takes that path, exactly as it did when the inventories lived in this repo. If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe. @@ -228,7 +232,7 @@ Updates are separate from issue investigation. After all issues are addressed (o 2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware) 3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices 4. Check =/var/log/pacman.log= for recent failed transactions -5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired. +5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at [[file:strix-soak-watch.org][docs/workflows/strix-soak-watch.org]] before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired. 6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=). On maint hosts (ratio, velox) plain =topgrade= resolves to the dotfiles PATH wrapper, which stamps the console's topgrade-freshness metric on success — no extra step. If the run happened outside the wrapper somehow, =maint stamp topgrade= records it by hand. 7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot @@ -996,7 +1000,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi - Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986. - Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot. - Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean. -- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.) +- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in the =mybitch - Christine's Laptop= roam node → Operational Changes Log.) - *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=. - *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug). - Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111. |
