path: root/docs/workflows/system-health-check.org

#+TITLE: System Health Check Workflow
#+AUTHOR: Craig Jennings & Claude
#+DATE: 2026-02-27

* Overview

This workflow performs a comprehensive diagnostic scan of whatever host it runs on (ratio, velox, mybitch, or truenas via SSH). Unlike status-check.org (which monitors a single long-running job in real-time), this scans the entire system, produces a severity-ranked report, then investigates and proposes fixes one by one.

Run this when Craig asks "how is <host> doing?", "check my system health", or as a periodic maintenance check.

Owned by archsetup (moved from the home project 2026-07-08 — system maintenance is archsetup's domain). Canonical location: =docs/workflows/system-health-check.org= in the archsetup repo; the per-host inventories it cross-references live beside it in =docs/homelab-inventory/=.

Relationship to the =maint= console: on the Arch daily drivers (ratio, velox), =maint status --json= collects most of Phases 0–1 in about a second from the same severity thresholds, and =maint doctor= runs the safe remedies — prefer it for routine checks there. This workflow remains the tool for the non-Arch hosts (mybitch, truenas), for forensic deep dives (Phase 2 investigation, the case files), and for the update/reboot choreography in Phase 3.

* Severity thresholds — the TOML is authoritative

The graded severity thresholds in this workflow are defined in the installed thresholds file, shared with the =maint= console so the two can never drift:

- Installed (read this): =~/.config/archsetup/maintenance-thresholds.toml=
- Canonical seed: =configs/maintenance-thresholds.toml= in the archsetup repo

Read the TOML at the start of Phase 1 and grade against its values. Where a check below names a threshold, it cites the TOML key (e.g. =storage.df_warn_pct=); any inline number is the seeded default kept for readability, and the TOML wins on disagreement. User overrides live in =~/.config/maint/curation.toml= and merge over the shipped layer.

* Hosts & Capabilities

The workflow is capability-dispatched: it probes the live system in Phase 0 and runs only the checks that apply (Btrfs vs ZFS vs ext4 on storage; pacman vs apt vs none on package maintenance; etc.). The per-host inventory files under =docs/homelab-inventory/= declare the expected capabilities in a property drawer and act as a cross-check for drift.

Host inventory files (match by =#+HOSTNAME:= keyword inside each):

| Host    | Inventory file                            | Notes                            |
|---------+-------------------------------------------+----------------------------------|
| ratio   | [[file:../homelab-inventory/ratio-desktop.org][docs/homelab-inventory/ratio-desktop.org]]  | Arch, Btrfs RAID1, pacman        |
|---------+-------------------------------------------+----------------------------------|
| velox   | [[file:../homelab-inventory/velox-laptop.org][docs/homelab-inventory/velox-laptop.org]]   | Arch, ZFS (re-installed 2026-04) |
|---------+-------------------------------------------+----------------------------------|
| mybitch | [[file:../homelab-inventory/mybitch-laptop.org][docs/homelab-inventory/mybitch-laptop.org]] | Mint, ext4, apt                  |
|---------+-------------------------------------------+----------------------------------|
| truenas | [[file:../homelab-inventory/truenas-server.org][docs/homelab-inventory/truenas-server.org]] | TrueNAS SCALE, ZFS, no PM        |
|---------+-------------------------------------------+----------------------------------|

Each inventory file contains an =* Automated Capabilities= section with a property drawer (=:FS:=, =:PM:=, =:ORCH:=, =:SNAPSHOT:=, =:MESH:=, =:BACKUP:=, =:VIRT:=, =:INIT:=, =:LAST_AUDIT:=). Phase 0 parses this drawer, compares it to the live probe, and announces + writes back any drift.

* The Workflow

** Phase 0: Capability Detection

Before running any health checks, probe the live system to determine which checks apply, then cross-reference against the inventory file for this host.

*** Probe the live system

#+begin_src bash
# Hostname — prefer hostnamectl (always present on systemd hosts), fall back to uname
host=$(hostnamectl --static 2>/dev/null || uname -n)

# Filesystem of /
root_fs=$(findmnt -n -o FSTYPE /)

# Any btrfs present?
has_btrfs=$(findmnt -t btrfs -n 2>/dev/null | wc -l)

# Any zfs pool imported?
has_zfs=$(command -v zpool >/dev/null 2>&1 && zpool list -H -o name 2>/dev/null | wc -l || echo 0)

# Package manager
pm=none
command -v pacman >/dev/null && pm=pacman
command -v apt    >/dev/null && pm=apt
command -v dnf    >/dev/null && pm=dnf

# Orchestrator
orch=none
command -v topgrade >/dev/null && orch=topgrade

# Snapshot tool
snapshot=none
command -v snapper >/dev/null && snapshot=snapper
# ZFS-native snapshots present?
[ "$has_zfs" -gt 0 ] && zfs list -t snapshot -H 2>/dev/null | head -1 | grep -q . && snapshot=zfs-native
command -v sanoid >/dev/null && snapshot=sanoid

# Mesh
mesh=none
command -v tailscale >/dev/null && mesh=tailscale

# Backup role
backup=none
[ -f /var/log/rsyncshot.log ] && backup=source
# Target role is declared by inventory, not probed (target hosts are discovered by their role, not a local artifact).

# Virt / containers
virt=""
command -v virsh   >/dev/null && virt="${virt}libvirt,"
command -v docker  >/dev/null && virt="${virt}docker,"
command -v podman  >/dev/null && virt="${virt}podman,"
virt="${virt%,}"
[ -z "$virt" ] && virt=none

# Init
init=$(ps -p 1 -o comm=)

echo "host=$host fs=$root_fs btrfs=$has_btrfs zfs=$has_zfs pm=$pm orch=$orch snapshot=$snapshot mesh=$mesh backup=$backup virt=$virt init=$init"
#+end_src

Present the probe result as a one-line capability summary at the top of the report.

*** Cross-reference against inventory

Locate the inventory file for this host:

#+begin_src bash
inv=$(grep -l "^#+HOSTNAME: $host\b" "${ARCHSETUP_DIR:-$HOME/code/archsetup}"/docs/homelab-inventory/*.org 2>/dev/null | head -1)
#+end_src

If no inventory file matches → print =NO INVENTORY FILE for $host= as a WARNING and skip the drift check.

If found: parse the =:PROPERTIES:= drawer under the =* Automated Capabilities= heading and compare each key to the live probe.

*Canonical parser* (the naive awk range =/^\* Automated Capabilities/,/^\* /= does NOT work — the end pattern matches the start line, truncating the range to one line):

#+begin_src bash
get_inv() {
  local key="$1" inv="$2"
  awk -v key="$key" '
    /^\* Automated Capabilities/ { in_section=1; next }
    /^\* / && in_section { in_section=0 }
    in_section && $0 ~ "^:"key":" {
      sub("^:"key":[[:space:]]*", "")
      sub("[[:space:]]+$", "")
      print; exit
    }
  ' "$inv"
}
#+end_src

Key-to-probed-variable mapping:

| Key          | Probed variable |
|--------------+-----------------|
| =:FS:=       | =root_fs=       |
|--------------+-----------------|
| =:PM:=       | =pm=            |
|--------------+-----------------|
| =:ORCH:=     | =orch=          |
|--------------+-----------------|
| =:SNAPSHOT:= | =snapshot=      |
|--------------+-----------------|
| =:MESH:=     | =mesh=          |
|--------------+-----------------|
| =:BACKUP:=   | =backup=        |
|--------------+-----------------|
| =:VIRT:=     | =virt=          |
|--------------+-----------------|
| =:INIT:=     | =init=          |
|--------------+-----------------|

*** Announce-then-update on drift

For each mismatch (excluding =TBD= on the inventory side, which means "not yet probed"):

1. Print in the Phase 0 summary: =DRIFT: <host> <key>: inventory=X probed=Y — updating=
2. Edit the inventory file: replace the drawer value with the probed value.
3. Update =:LAST_AUDIT:= to today's date.

Skip the write-back if the workflow is running on a host where the repo isn't checked out (e.g., truenas accessed via SSH). In that case, print the drift but flag it as =DRIFT (read-only host): update <file> manually=.

*** Capability summary dictates which checks run

Each check in the Reference section declares =Applies when: <capability>=. Phase 1 runs the check only if the capability is present; otherwise it prints =N/A — <capability> not present= for that check and moves on.

** Phase 1: Scan & Report

Run all checks from the Health Checks Reference below. Collect every finding into a ranked table sorted by severity:

| Severity | Meaning                                                     |
|----------+-------------------------------------------------------------|
| CRITICAL | Service down, disk failing, backups not running             |
|----------+-------------------------------------------------------------|
| WARNING  | Disk space low, stale logs, failed units, overdue maint     |
|----------+-------------------------------------------------------------|
| INFO     | Informational (uptime, versions, counts) — no action needed |
|----------+-------------------------------------------------------------|

*Before presenting findings:* cross-reference each finding against the Known Issues Log at the bottom of this file. If a finding matches a known issue with a decided resolution (won't-fix, accepted-noise, fixed-in-config), annotate it as =KNOWN — <short note>= rather than presenting it as a fresh WARNING. This prevents re-investigating previously-decided items and keeps the report focused on actionable new state.

Present the report as described in Report Format, then proceed to Phase 2.

** Phase 2: Investigate & Fix

After presenting the ranked report:

1. Start with the highest-severity issue
2. Investigate root cause with detailed commands
3. Propose a fix, explain what it does
4. Wait for Craig's approval before applying any changes
5. Verify the fix worked
6. Move to next issue
7. Repeat until all CRITICAL and WARNING items are addressed
8. INFO items are presented for awareness only (no action unless Craig wants to dig in)

Handle issues one at a time — don't batch fixes together. Craig prefers to approve each individually.

*** Defer Transient Warnings Until After Phase 3 Reboot

If the Phase 1 pending-updates list includes any of these packages, =DEFER= Phase 2 deep-dive on network/DNS/VPN/uptime-accumulation warnings — investigate only if they survive the reboot:

| Package (any of)          | Defers these warning categories                               |
|---------------------------+---------------------------------------------------------------|
| kernel (linux, linux-lts) | bgscan/wifi driver spam, firmware-related dmesg noise         |
|---------------------------+---------------------------------------------------------------|
| iproute2                  | VPN daemon failures, ip-rule/routing-related service failures |
|---------------------------+---------------------------------------------------------------|
| systemd                   | resolved/networkd/logind service anomalies                    |
|---------------------------+---------------------------------------------------------------|
| NetworkManager            | DNS reachability health checks, Tailscale DNS warnings        |
|---------------------------+---------------------------------------------------------------|
| wpa_supplicant            | WiFi connection/scan errors                                   |
|---------------------------+---------------------------------------------------------------|

*Rationale:* Warnings on a long-uptime system (>7 days) frequently reflect accumulated mid-transition state from earlier package updates that weren't rebooted into. A fresh reboot resolves these automatically. Investigation pre-reboot is wasted effort.

*Procedure:* Mark the deferred warnings in the Phase 1 report as =DEFER-TO-POST-REBOOT=. Proceed directly from Phase 2 (investigating any non-deferrable warnings) to Phase 3. After reboot (see two-stage pattern in Phase 3), re-evaluate the deferred warnings — most will have vanished. Any that survive get the full Phase 2 treatment.

** Phase 3: System Update

Updates are separate from issue investigation. After all issues are addressed (or deferred per Phase 2's transient-warning rule):

1. Review the pending update list (already gathered in Phase 1)
2. Identify notable packages (major version bumps, GPU drivers, kernel, firmware)
3. Check the [[https://archlinux.org/news/][Arch Linux News]] page for any manual intervention notices
4. Check =/var/log/pacman.log= for recent failed transactions
5. *Host-specific kernel watches.* On ratio: if =linux=, =linux-lts=, =linux-firmware=, or a major =mesa= bump is pending, run the addendum at =strix-soak-watch.org= before topgrade. Retire the addendum (delete the file + this bullet) when the strix-lts custom kernel is retired.
6. Run =topgrade= for the actual update (config at =~/.config/topgrade.toml=)
7. If linux-firmware, kernel, or Mesa were updated, recommend a reboot

*** Two-Stage Reboot Pattern (MANDATORY if Phase 3 installed kernel / iproute2 / systemd / NetworkManager)

When a core networking or kernel package is updated, the workflow formally splits into two sessions:

*Stage 1 — Pause before reboot:*
1. Update =docs/session-context.org= with:
   - What's been completed (Phase 1 findings, Phase 3 summary, packages bumped)
   - What's deferred to post-reboot (the =DEFER-TO-POST-REBOOT= warnings from Phase 2)
   - Remaining work: Phase 4, wrap-up
2. Tell Craig to reboot
3. End session (do NOT delete session-context.org — its presence signals an in-progress health check)

*Stage 2 — Resume in new session post-reboot:*
1. Read =docs/session-context.org= first
2. Run post-reboot verification:
   - =uname -r= → confirm new kernel is running
   - =systemctl --failed= → confirm zero failed services
   - =journalctl -p err -b= → count and compare to pre-reboot (sharp drop = transient warnings were real, accumulated noise)
3. Re-evaluate the deferred warnings — most are gone without action
4. Full Phase 2 treatment for any survivors
5. Complete Phase 4, then wrap up (writes session entry to notes.org covering both halves as one unified session, spanning the reboot)
6. Delete =session-context.org= only after wrap-up is committed

*Rationale:* Trying to cram the whole flow into one session risks either (a) wasting time investigating transient warnings that would have self-healed, or (b) losing context if a crash happens during the reboot gap. The two-stage pattern is explicit about the pause and survives interruptions.

** Phase 4: Resolve .pacnew Files

After updates, check for and resolve .pacnew files:

#+begin_src bash
find /etc -name "*.pacnew" 2>/dev/null
#+end_src

For each .pacnew file:
1. Diff the current config against the .pacnew version
2. Determine if the current config is actively managed (e.g., by Reflector, or manually customized)
3. If the current config is correct and the .pacnew adds nothing useful → delete the .pacnew
4. If the .pacnew has meaningful changes → merge them into the current config, then delete the .pacnew
5. Always explain the diff to Craig before deleting

Common .pacnew files on ratio:
- =/etc/pacman.d/mirrorlist.pacnew= — safe to delete; mirrorlist is managed by Reflector
- =/etc/locale.gen.pacnew= — safe to delete if =en_US.UTF-8= is already uncommented

* Health Checks Reference

Run these checks in order. Each check produces one or more findings for the report.

** 1. System Basics

#+begin_src bash
echo "Date:     $(date)"
echo "Hostname: $(hostnamectl hostname)"
echo "Kernel:   $(uname -r)"
echo "Uptime:   $(uptime -p)"
#+end_src

Severity: INFO (always).

** 2. Failed systemd Units

#+begin_src bash
systemctl --failed
#+end_src

- Any failed units → WARNING (CRITICAL if essential service like NetworkManager, fail2ban, cronie)
- No failed units → INFO "All units healthy"

** 3. Journal Errors (Current Boot)

#+begin_src bash
# Get total error count first
journalctl -p err -b --no-pager | wc -l

# Then separate known-noisy services from real errors
journalctl -p err -b --no-pager | grep -v "Hands-Free Voice gateway" | tail -50
#+end_src

Known noisy patterns to filter when counting:
- bluetoothd "Hands-Free Voice gateway" — paired device out of range, polls every 60s
- pixman "_pixman_log_error" — benign Hyprland rendering debug message
- xkbcomp "Errors from xkbcomp are not fatal" — X compatibility noise

When bluetooth spam is found, identify the device with =bluetoothctl devices Paired= and offer to remove stale pairings.

- Recurring or service-impacting errors → WARNING
- Hardware errors (MCE, disk I/O) → CRITICAL
- Noise-level errors (known benign) → skip, but note the noise source

** 4. Kernel & Hardware Events (journalctl -k)

Check #3 catches error-priority journal messages, but many hardware issues show up as
patterns of lower-severity kernel messages (e.g., repeated USB disconnects, I/O retries).
This check scans the kernel log for those patterns.

*Why =journalctl -k -b= and not =dmesg=:* on systems that suspend (laptops especially), =dmesg -T= computes wall-clock timestamps using =current_realtime − CLOCK_MONOTONIC=, but kernel printk timestamps include suspend time. After a suspend-heavy uptime, dmesg displays timestamps drifted forward by the cumulative suspend duration (e.g., May 17 displayed for messages logged on May 10). =journalctl -k= stamps CLOCK_REALTIME at message-write time, so dates are always correct. Performance difference is ~100 ms vs ~10 ms — negligible at this cadence. (See the 2026-05-10 entry in the Known Issues Log.)

#+begin_src bash
# USB disconnect/reconnect cycles (repeated = hardware or power management issue)
journalctl -k -b --no-pager | grep -c "USB disconnect"

# If disconnects found, show the devices and timestamps
journalctl -k -b --no-pager | grep "USB disconnect" | tail -20

# USB protocol errors (error -71 = EPROTO, error -110 = timeout)
journalctl -k -b --no-pager | grep -iE "usbhid.*error|usb.*error -[0-9]+" | tail -10

# I/O errors on block devices
journalctl -k -b --no-pager | grep -iE "I/O error|blk_update_request|Buffer I/O error" | tail -10

# PCIe errors
journalctl -k -b --no-pager | grep -iE "pcie.*error|AER|corrected error|uncorrected error" | tail -10

# Machine Check Exceptions (CPU hardware errors)
journalctl -k -b --no-pager | grep -iE "mce:|hardware error" | tail -10

# GPU faults or resets
journalctl -k -b --no-pager | grep -iE "amdgpu.*error|amdgpu.*fault|gpu reset|gpu hang" | tail -10

# Thermal throttling
journalctl -k -b --no-pager | grep -iE "thermal.*throttl|cpu.*throttl|prochot" | tail -10

# ACPI errors (power management issues)
journalctl -k -b --no-pager | grep -iE "acpi.*error|acpi.*warning" | tail -10

# USB autosuspend state on internal hubs (Framework 16 specific)
for d in /sys/bus/usb/devices/1-[234]; do
    if [ -d "$d" ]; then
        echo "$d: $(cat $d/product 2>/dev/null) | control=$(cat $d/power/control) | delay=$(cat $d/power/autosuspend_delay_ms)"
    fi
done
#+end_src

What to look for:
- *USB disconnects*: A few at boot or after resume is normal. >5 during a session, or repeated disconnect/reconnect of the same device = problem. On Framework 16, the keyboard and numpad modules are internal USB — disconnects here mean input loss.
- *Error -71 (EPROTO)*: USB protocol error, often caused by aggressive hub autosuspend. Check hub power settings.
- *I/O errors*: Any I/O error on NVMe → immediate CRITICAL.
- *MCE/hardware errors*: Always CRITICAL.
- *GPU faults*: Occasional corrected errors are INFO. Resets or hangs are WARNING.
- *Thermal throttling*: WARNING if active. Check if cooling is obstructed.
- *USB hub autosuspend*: Internal hubs with =control=auto= and =delay=0= → WARNING (causes input disconnects on Framework 16).

- Repeated USB disconnects of input devices → WARNING
- USB protocol errors (EPROTO) → WARNING
- I/O errors, MCE, or hardware errors → CRITICAL
- GPU resets or hangs → WARNING
- Thermal throttling active → WARNING
- Aggressive hub autosuspend on internal devices → WARNING
- Clean dmesg → INFO

** 5. Disk Health (SMART)

#+begin_src bash
sudo smartctl -H /dev/nvme0n1
sudo smartctl -H /dev/nvme1n1
sudo smartctl -A /dev/nvme0n1 | grep -i "critical\|temperature\|wear\|error"
sudo smartctl -A /dev/nvme1n1 | grep -i "critical\|temperature\|wear\|error"
#+end_src

- SMART overall health != PASSED → CRITICAL
- NVMe wear > =storage.smart_wear_warn_pct= or temp > =storage.smart_temp_warn_c= (TOML; seeded 80% / 70°C) → WARNING
- All healthy → INFO

** 6. Disk Usage

Dispatched by filesystem. Run every sub-check for which the capability is present — a host with both Btrfs and ZFS (hypothetical: ratio with a ZFS backup pool) runs 6a AND 6b.

*** 6a. Btrfs Usage

*Applies when:* =has_btrfs > 0=

#+begin_src bash
sudo btrfs filesystem usage /
/usr/bin/df -h / /boot 2>/dev/null
#+end_src

Btrfs reports two numbers that can look confusing:
- *Data allocated %* (e.g., 99.43%) — how full the allocated data chunks are. Btrfs allocates in chunks and can allocate more from the unallocated pool.
- *df %* (e.g., 69%) — actual used space vs total device size. This is what matters.

Check the "Device unallocated" line — as long as there's unallocated space, Btrfs can grow its data chunks.

Severity rules:
- df > =storage.df_crit_pct= (TOML; seeded 90%) → CRITICAL
- df > =storage.df_warn_pct= (seeded 80%) → WARNING
- df under the warn line → INFO (report both df% and unallocated space)

*** 6b. ZFS Pool Health

*Applies when:* =has_zfs > 0=

#+begin_src bash
# Quick health summary across all pools
sudo zpool status -x                  # expect "all pools are healthy"

# Full detail — state, errors, scrub, resilver
sudo zpool status

# Capacity, fragmentation, health per pool
sudo zpool list -o name,size,alloc,free,capacity,health,frag

# Per-dataset usage (top-level only; -d1 depth)
sudo zfs list -o name,used,avail,refer -s used
#+end_src

Severity rules:
- Pool state != ONLINE → CRITICAL
- Any read/write/cksum error > 0 → CRITICAL
- Capacity > =storage.zfs_capacity_crit_pct= (TOML; seeded 90%) → CRITICAL
- Capacity > =storage.zfs_capacity_warn_pct= (seeded 80%) → WARNING (ZFS performance degrades)
- Scrub age > =storage.zfs_scrub_crit_days= (seeded 60) → CRITICAL
- Scrub age > =storage.zfs_scrub_warn_days= (seeded 35) → WARNING
- Resilver in progress → INFO (but prominent in the report — mention drive being replaced)
- Fragmentation > =storage.zfs_frag_info_pct= (seeded 50%) → INFO

Parse scrub age from =zpool status= output: look for the =scan:= line — either =scrub repaired … in … with … errors on <date>= (completed) or =scrub in progress …= (running).

*** 6c. Generic filesystem fallback

*Applies when:* =root_fs= is ext4, xfs, or any other non-Btrfs non-ZFS filesystem

#+begin_src bash
/usr/bin/df -h / /home /boot 2>/dev/null
#+end_src

Severity rules:
- > =storage.df_crit_pct= on any mounted partition → CRITICAL
- > =storage.df_warn_pct= → WARNING
- Normal → INFO

** 7. Snapshots

Dispatched by snapshot tool.

*** 7a. Snapper (Btrfs)

*Applies when:* =snapshot = snapper=

#+begin_src bash
snapper -c home list --columns number,date,description | head -5
snapper -c home list --columns number,date,description | tail -5
snapper -c home list | wc -l
snapper -c home get-config | grep -iE "cleanup|TIMELINE_LIMIT|NUMBER_LIMIT"
# oldest snapshot age (pile-up signal)
oldest=$(sudo ls /home/.snapshots/ 2>/dev/null | grep -E '^[0-9]+$' | sort -n | head -1)
[ -n "$oldest" ] && echo "oldest: $(sudo grep -oP '(?<=<date>).*(?=</date>)' "/home/.snapshots/$oldest/info.xml" 2>/dev/null | head -1)"
#+end_src

Severity rules:
- Zero snapshots → WARNING
- Cleanup disabled → WARNING
- Snapshot count very high, OR oldest snapshot far older than intended retention (e.g. months back) → WARNING (pile-up)
- =TIMELINE_LIMIT_MONTHLY= / =_YEARLY= set high (e.g. 10) on a large, churny subvolume → WARNING: that keeps ~10 months of monthly snapshots, which silently hoard space (this bit /home on 2026-05-26 — monthly snapshots back to Feb were the top space holders). Sane values are the TOML's =[snapshots]= =timeline_*= keys (seeded HOURLY 6, DAILY 7, WEEKLY 2, MONTHLY 2, QUARTERLY 0, YEARLY 0). Manual (=single=) snapshots are NOT touched by timeline cleanup — delete stale ones explicitly.
- Normal range → INFO with count and date range

*Disk-space deep-dive (on demand, only when /home is actually filling).* Per-snapshot reclaimable space is invisible without btrfs quotas. TEMPORARILY enable quota, rank by exclusive space, then DISABLE it again — quotas slow every snapshot deletion, so never leave them on:
#+begin_src bash
sudo btrfs quota enable /home && sudo btrfs quota rescan -w /home
sudo btrfs qgroup show -re --sort=-excl /home | head -20   # top exclusive (reclaimable) holders
sudo btrfs quota disable /home
#+end_src
Reality check: data deleted from the live fs but still in many old snapshots shows ~0 *exclusive* per snapshot (it's shared across them), and only frees once the whole chain holding it is pruned. So reclaiming a large deletion means pruning the old snapshots, not just the newest one.

*** 7b. ZFS Snapshots

*Applies when:* =snapshot = zfs-native= or =sanoid=

#+begin_src bash
# All snapshots, newest first
sudo zfs list -t snapshot -o name,creation,used -s creation | tail -20

# Count per pool
sudo zfs list -t snapshot -H -o name | awk -F'@' '{print $1}' | sort | uniq -c

# Oldest snapshot age per dataset (detect runaway retention)
sudo zfs list -t snapshot -H -o name,creation -s creation | head -20

# Auto-snapshot service status (zfs-auto-snapshot / sanoid / zrepl)
systemctl list-timers --all 2>/dev/null | grep -Ei "zfs|sanoid|zrepl" || echo "no auto-snapshot timer"
#+end_src

Severity rules:
- Zero snapshots on a dataset that should have them → WARNING
- Snapshot count on any dataset > =snapshots.zfs_count_warn= (seeded 1000) → WARNING (runaway retention)
- Snapshot space used > =snapshots.zfs_space_warn_pct= (seeded 20%) of pool capacity → WARNING (heavy divergence)
- Auto-snapshot timer not running → WARNING
- Normal → INFO (counts per pool, oldest snapshot age)

*** 7c. No snapshot system

*Applies when:* =snapshot = none=

Report =INFO: no snapshot system configured on this host= and skip.

** 8. Memory and Swap

#+begin_src bash
free -h
journalctl -b --no-pager | grep -i "out of memory\|oom-kill\|killed process" | tail -10
#+end_src

Ratio has 128GB RAM and no swap configured. No swap is expected — don't flag it.

- OOM kills found → WARNING
- Normal → INFO

** 9. CPU Temperatures

#+begin_src bash
sensors 2>/dev/null || echo "lm_sensors not installed or not configured"
#+end_src

Key sensors on ratio:
- =k10temp= → CPU (Tctl)
- =amdgpu= → GPU edge temp
- =nvme= → NVMe drives (2 drives)
- =cros_ec= → mainboard (power, memory, ambient, CPU)

- CPU temp > =power.cpu_temp_warn_c= or GPU temp > =power.gpu_temp_warn_c= (TOML; seeded 90/95°C) → WARNING, CRITICAL if sustained
- Normal → INFO (report CPU and GPU temps)

** 10. NTP Sync

#+begin_src bash
chronyc tracking
#+end_src

Ratio uses chrony for NTP.

- Clock not synchronized → WARNING
- NTP service inactive → WARNING
- Last offset > =network.ntp_offset_warn_ms= (TOML; seeded 100 ms) → WARNING
- Synchronized → INFO

** 11. rsyncshot Backups

*Applies when:* =backup = source=

#+begin_src bash
tail -30 /var/log/rsyncshot.log 2>/dev/null
#+end_src

rsyncshot runs via root crontab:
- Hourly: =30 0-1,3-23 * * *= (every hour except 2:30)
- Daily: =30 2 * * *= (2:30 AM)

Backups go to TrueNAS via rsync. Look for "rsyncshot completed successfully" and check the timestamp.

- No daily backup in =backups.rsyncshot_daily_crit_hours= (seeded 48) → CRITICAL
- No hourly backup in =backups.rsyncshot_hourly_warn_hours= (seeded 3) → WARNING (only on hosts whose log shows hourly runs)
- Errors in log → WARNING
- Recent and healthy → INFO

** 12. Tailscale

*Applies when:* =mesh = tailscale=

#+begin_src bash
tailscale status
#+end_src

Expected peers: ratio, cjennings, truenas, velox, worker. pixel6 is a phone and is often offline — don't flag it.

- Tailscale not running → WARNING
- Server peers (truenas, velox, worker) offline → WARNING
- All expected peers online → INFO

** 13. fail2ban

#+begin_src bash
sudo fail2ban-client status
sudo fail2ban-client status sshd 2>/dev/null
#+end_src

- fail2ban not running → WARNING
- Running → INFO (note any recent bans for awareness)

** 14. Package Maintenance

Dispatched by package manager.

*** 14a. pacman (Arch)

*Applies when:* =pm = pacman=

#+begin_src bash
# Orphaned packages
pacman -Qtdq 2>/dev/null

# Pending updates
checkupdates 2>/dev/null

# Package cache timer + current size
systemctl list-timers paccache.timer --no-pager
du -sh /var/cache/pacman/pkg 2>/dev/null

# .pacnew files
find /etc -name "*.pacnew" 2>/dev/null
#+end_src

Notes:
- =paccache.timer= runs weekly keeping 3 versions — fine for routine pruning, but it only trims *old versions*. A cache can still balloon from *breadth* (many packages, ≤3 versions each), where =paccache -r= finds nothing to prune. On 2026-05-26 the cache was 17 GB and =paccache -r= pruned 0; the space came from deeper levers:
  - =sudo paccache -ruk0= — remove cache for *uninstalled* packages (safe, re-downloadable).
  - =sudo paccache -rk1= — keep only 1 version of installed packages (frees the most; less downgrade headroom).
  Together those took 17 GB → 5.8 GB.
- When removing orphans, pass package names as arguments (don't pipe through stdin — breaks snap-pac's snapper hook).
- Review orphans before removing — some (like =rust=) warrant discussion (Craig switched to =rustup=).

Severity rules:
- Orphaned packages beyond the curated =curation.kept_orphans= set, in bulk (> ~20) → WARNING
- Pending updates > =updates.pending_warn= (TOML; seeded 50) → WARNING
- Unreviewed .pacnew files → WARNING
- Package cache > =packages.cache_warn_gb= (TOML; seeded 10 GB) → INFO: suggest =paccache -ruk0= (uninstalled) and/or =paccache -rk1= (keep 1) for a deeper reclaim beyond the weekly keep-3.
- Normal counts → INFO

*** 14b. apt (Debian/Ubuntu/Mint)

*Applies when:* =pm = apt=

#+begin_src bash
# Pending updates
apt list --upgradable 2>/dev/null | tail -n +2

# Auto-removable packages
apt -s autoremove 2>&1 | grep -E "^Remv|packages will be REMOVED"

# .dpkg-dist / .dpkg-new files (the apt equivalent of .pacnew)
find /etc -name "*.dpkg-dist" -o -name "*.dpkg-new" 2>/dev/null

# unattended-upgrades timer
systemctl list-timers apt-daily.timer apt-daily-upgrade.timer --no-pager 2>/dev/null
#+end_src

Severity rules:
- Pending security updates > 0 → WARNING
- Auto-removable packages > 30 → WARNING
- Unreviewed .dpkg-dist files → WARNING
- Normal → INFO

*** 14c. none (TrueNAS SCALE, other appliances)

*Applies when:* =pm = none=

Report =N/A — package maintenance handled by middleware= and skip. On TrueNAS SCALE, OS updates go through the TrueNAS UI (System → Update); user-level =apt= exists but is unsupported and shouldn't be used.

*** 14d. Orchestrator (topgrade)

*Applies when:* =orch = topgrade=

If topgrade is present, Phase 3 uses it rather than the raw package manager. Phase 1 still runs 14a/14b/14c above — topgrade is for the update run, not the status scan.

** 15. App Logs (~/.local/var/log)

#+begin_src bash
for log in ~/.local/var/log/*.log; do
    if [ -f "$log" ]; then
        errors=$(grep -ci "error\|critical\|fatal" "$log" 2>/dev/null)
        if [ "$errors" -gt 0 ]; then
            echo "$(basename $log): $errors error lines"
            grep -i "error\|critical\|fatal" "$log" | tail -3
            echo ""
        fi
    fi
done
#+end_src

Known noise patterns:
- *waybar*: LIBDBUSMENU-GLIB-WARNING "Unable to replace properties on 0" — caused by insync and zoom tray icons with broken dbusmenu implementations. Harmless. Filtered in waybar's log output via =grep -v= in hyprland.conf.
- *hyprland*: "_pixman_log_error" and "xkbcomp" — benign
- *gammastep*: "Wayland connection experienced a fatal error" — happens on session restart, self-recovers
- *dunst*: gdk_pixbuf assertion failures — usually a malformed notification icon, one-off

Look for patterns that are NOT in the known noise list.

- Coredumps within =logs.coredump_window_days= (TOML; seeded 14 — =coredumpctl list= and age-filter) → WARNING
- Recurring non-noise errors → WARNING
- Only known noise → INFO

** 16. System Logs (/var/log)

#+begin_src bash
# Recent pacman errors
grep -i "error\|failed" /var/log/pacman.log 2>/dev/null | tail -10

# fail2ban anomalies
tail -50 /var/log/fail2ban.log 2>/dev/null | grep -i "error\|warning"
#+end_src

- Failed pacman transactions → WARNING
- fail2ban errors → WARNING
- Clean → INFO

** 17. Docker/Podman

*Applies when:* =virt= contains =docker= or =podman=

#+begin_src bash
docker ps -a --filter "status=exited" --format "{{.Names}}: exited {{.Status}}" 2>/dev/null
docker system df 2>/dev/null
podman ps -a --filter "status=exited" --format "{{.Names}}: exited {{.Status}}" 2>/dev/null
podman system df 2>/dev/null
#+end_src

The WinVM podman container is run on-demand and will often show as exited — this is expected.

*Severity thresholds:*
- Stopped containers that should be running → WARNING
- Docker/Podman reclaimable > =services.docker_reclaim_warn_gb= *or* > =services.docker_reclaim_warn_pct= of total (TOML; seeded 5 GB / 50%) → WARNING (propose prune tiers in Phase 2)
- Docker/Podman reclaimable 1–5 GB → INFO (note for awareness)
- Expected stopped containers → INFO

*Prune tiers to propose when reclaimable crosses the WARNING threshold:*
1. =docker container prune= — removes only stopped containers (~tens of MB, unlocks referenced images)
2. =docker image prune -a= — removes all images not used by any container (biggest bite; requires tier 1 first)
3. =docker system prune -a --volumes= — full nuke, includes unused volumes and networks. Present as the option when Craig wants to start fresh.

Confirm with Craig before running any tier — especially if work-related containers (e.g. =deepsat-*=) are present.

** 18. libvirt / VMs

*Applies when:* =virt= contains =libvirt=

#+begin_src bash
sudo virsh list --all 2>/dev/null || echo "libvirt not available"
#+end_src

VMs (ultmos-15, etc.) are usually shut off unless Craig is actively using them. "shut off" is the normal state.

- VMs in unexpected state → INFO
- libvirt not running → INFO

** 19. Cron Jobs

#+begin_src bash
systemctl is-active cronie
crontab -l 2>/dev/null | grep -v "^#" | grep -v "^$"
sudo crontab -l 2>/dev/null | grep -v "^#" | grep -v "^$"
#+end_src

Expected user crontab entries:
- =0 12 * * *= log-cleanup

Expected root crontab entries:
- =30 0-1,3-23 * * *= rsyncshot hourly
- =30 2 * * *= rsyncshot daily

- cronie not running → WARNING
- Expected crontab entries missing → WARNING
- Running with expected entries → INFO

** 20. Log Cleanup Cron

#+begin_src bash
find ~/.local/var/log -type f -name "*.log" -mtime +7 2>/dev/null | head -10
find ~/.local/var/log -type f 2>/dev/null | wc -l
#+end_src

The user crontab runs =~/.local/bin/cron/log-cleanup= daily at noon, which should keep logs to ~7 days.

- Log files older than =logs.app_log_warn_days= (TOML; seeded 7) present → WARNING (cleanup cron may not be running)
- Within the retention window → INFO

** 21. Network

#+begin_src bash
# DNS resolution (dig may not be available — use ping as fallback)
ping -c1 -W2 archlinux.org 2>/dev/null && echo "DNS OK" || echo "DNS FAILED"

# NetworkManager status
nmcli general status 2>/dev/null
#+end_src

- DNS resolution fails → CRITICAL
- NetworkManager not running or disconnected → CRITICAL
- All healthy → INFO

* Report Format

Present findings as a ranked org-mode table, sorted by severity (CRITICAL first, then WARNING, then INFO):

#+begin_example
| # | Severity | Category    | Finding                        |
|---+----------+-------------+--------------------------------|
| 1 | CRITICAL | rsyncshot   | Daily backup failed 2 days ago |
|---+----------+-------------+--------------------------------|
| 2 | WARNING  | disk usage  | / at 85% capacity              |
|---+----------+-------------+--------------------------------|
| 3 | WARNING  | packages    | 47 orphaned packages           |
|---+----------+-------------+--------------------------------|
| 4 | INFO     | uptime      | 12 days                        |
|---+----------+-------------+--------------------------------|
| 5 | INFO     | disk health | Both NVMe drives PASSED        |
|---+----------+-------------+--------------------------------|
| 6 | INFO     | tailscale   | 3/3 peers online               |
|---+----------+-------------+--------------------------------|
#+end_example

After the table, announce the investigation plan:

- If CRITICAL or WARNING issues exist: "Starting with #1 — investigating [category]: [finding]..."
- If only INFO items: "No issues found. System looks healthy. Here are the details..."

* Known Issues Log

Each entry is scoped to one host (or =any=). When Phase 1 cross-references findings against this log, it matches on both the issue signature AND the host — an entry tagged =:host: mybitch= won't suppress a finding on ratio. For pre-existing entries without an explicit host tag, the hostname is inferred from the entry title where possible (e.g., "mybitch —"). New entries should include =:host: <name>= for clarity.

** 2026-03-24: mybitch — USB autosuspend causing keyboard/numpad disconnects
:host: mybitch
- Framework Laptop 16 internal Genesys Logic USB hubs (05e3:0610) had =control=auto= and =delay=0=
- Caused repeated USB disconnects of keyboard module (1-4.2) and numpad module (1-3.2)
- Error -71 (EPROTO) on reconnection attempts
- Fix: udev rule at =/etc/udev/rules.d/99-framework-usb-hub-no-autosuspend.rules=
- Sets =power/control=on= for matching hubs, disabling autosuspend
- Added check #4 (Kernel & Hardware Events) to this workflow to catch similar issues in future

** 2026-02-27: Bluetooth journal spam (1,006 errors/boot)
:host: ratio
- bluetoothd polling "Craig's Pixel Buds" (B8:7B:D4:19:A6:01) every 60s while out of range
- Fix: removed stale pairing with =bluetoothctl remove B8:7B:D4:19:A6:01=
- Note: Craig has recurring Pixel Buds pairing issues with this machine

** 2026-02-27: hyprlock deprecated config options (v0.9.2)
:host: ratio
- =general:grace= and =general:no_fade_in= moved to CLI flags (=--grace=, =--no-fade-in=)
- =input-field:fail_transition= replaced by =animations= block (=inputFieldColors= animation)
- Fix: removed deprecated options, added =animations= block to hyprlock.conf
- Config at: =~/code/archsetup/dotfiles/hyprland/.config/hypr/hyprlock.conf=

** 2026-02-27: dunst coredump
:host: ratio
- Segfault in libglycin/gdk_pixbuf — malformed notification icon
- One-off, no action taken. Monitor for recurrence.

** 2026-02-27: Missing udev script for Logitech Brio
:host: ratio
- Udev rule references =~/.local/bin/logitech-brio-settings.sh= which is a symlink to archsetup dotfiles
- Fails at boot because udev runs before home is fully available
- Script works fine for hotplug after login
- Decision: leave as-is

** 2026-02-27: waybar LIBDBUSMENU-GLIB warnings (~8,000/session)
:host: ratio
- Caused by insync and zoom tray icons with broken dbusmenu implementations
- Both return empty arrays for menu root item (ID 0)
- No fix available (insync has no option to disable tray icon)
- Mitigation: added =grep -v "LIBDBUSMENU-GLIB-WARNING"= filter to waybar log line in hyprland.conf
- Takes effect on next Hyprland start

** 2026-02-27: Package cleanup
:host: ratio
- Removed 51 orphaned packages (build deps, Python dev tools)
- Swapped =rust= pacman package for =rustup= (toolchain manager)
- Added note to =~/code/archsetup/inbox/rustup.txt=

** 2026-02-27: System update via topgrade
:host: ratio
- 66 packages updated including Mesa 25.3→26.0, Firefox/Thunderbird 148, linux-firmware, Signal 8.0
- No manual intervention required (checked Arch news)
- Reboot recommended for firmware and Mesa changes

** 2026-02-27: Resolved .pacnew files
:host: ratio
- =/etc/pacman.d/mirrorlist.pacnew= — stock mirrorlist, safe to delete (Reflector manages the active one)
- =/etc/locale.gen.pacnew= — only added commented =en_SE.UTF-8=, current config correct, deleted

** 2026-04-19: proton.VPN.service transient failure after topgrade
:host: ratio
- Appeared as a =failed= unit immediately after =topgrade= bumped iproute2 from 6.19.0 → 7.0.0 mid-session (9-day uptime)
- Self-healed on reboot; no config change required
- Pattern: VPN daemon holds routing/ip-rule state referencing old iproute2 ABI, can't reconcile after live package swap
- Classification: KNOWN-TRANSIENT on iproute2 upgrade → DEFER-TO-POST-REBOOT per Phase 2 rules

** 2026-04-19: Tailscale DNS reachability warning on long uptime
:host: ratio
- =tailscale status= footer: "Tailscale can't reach the configured DNS servers. Internet connectivity may be affected."
- Appeared on 9-day uptime; all core functions (name resolution, netcheck, tailnet routing) tested working
- Cleared on reboot — footer warning gone, no config change
- Pattern: tailscaled's internal DNS health check accumulates stale state on long uptime, particularly after a mid-session iproute2 transition
- Classification: KNOWN-TRANSIENT on long uptime / iproute2 upgrade → DEFER-TO-POST-REBOOT per Phase 2 rules

** 2026-04-19: wpa_supplicant bgscan error spam volume scales with uptime
:host: ratio
- Error: =bgscan simple: Failed to enable signal strength monitoring=
- Volume pre-reboot (9-day uptime): 1,200+ instances in journal
- Volume post-reboot (fresh boot): 1 instance total
- Root cause: RSSI monitoring ioctl not implemented in mt7925e driver (Framework Desktop WiFi 7 card) — each retry or roam event re-triggers the failure
- 1–3 instances per boot is baseline noise; only triage if volume > ~50 AND not explainable by long uptime
- Classification: KNOWN (mt7925e driver limitation, no upstream fix tracked); annotate low-volume occurrences as =KNOWN — mt7925e baseline=

** 2026-04-19: Orphan package cleanup
:host: ratio
- Removed 11 packages (8 orphans + 3 cascading python build-tool deps): cli11, electron34, lua-lpeg, minizip-ng, python-build, python-hatchling, python-installer, yarn + python-editables, python-pyproject-hooks, python-trove-classifiers
- Freed 280 MiB
- Future note: electron34 and yarn were previously flagged "ask first" due to possible AUR build-dep relevance — post-removal, no AUR rebuild has failed. Can remove in future without asking unless an active AUR build references them.

** 2026-04-21: velox ZFS — systemd-tmpfiles "Protocol driver not attached"
:host: velox
- Symptom: =systemd-tmpfiles-setup.service= (boot) and/or =systemd-tmpfiles-clean.service= (periodic) produce 10-30 =statx(...) failed: Protocol driver not attached= journal errors per run
- Root cause: on ZFS, statx against another service's =/var/tmp/systemd-private-*/tmp= mount returns errno 132 (ENOTNAM); ext4/btrfs don't surface this as an error. Bare systemd-tmpfiles unit has no =PrivateTmp= set, so it traverses sibling namespaces
- Fix: drop-in with =PrivateTmp=yes= at =/etc/systemd/system/systemd-tmpfiles-clean.service.d/zfs-private-tmp.conf= AND =.../systemd-tmpfiles-setup.service.d/zfs-private-tmp.conf=
- Applies to any ZFS-on-root Arch host. Not needed on btrfs hosts.

** 2026-04-19: Docker image bloat cleanup
:host: ratio
- Freed 15.3 GB via =docker system prune -a --volumes=
- Removed 10 images (including nvidia/cuda 12.4 devel, postgis, nginx, python-slim, node-alpine, nerdfonts/patcher) and 4 exited deepsat-* work containers from 9 days prior
- 130 MB orphan volume survived prune (anonymous, not caught by =--volumes=); ignored
- Future: Check #17 now promotes reclaimable > 5 GB OR > 50% to WARNING — this will catch bloat earlier without Craig having to notice it

** 2026-05-10: WiFi powersave default caused variable WiFi latency on mybitch + ratio
:host: mybitch, ratio (velox naturally clean — defensive config added anyway)
- Symptom: variable LAN/WiFi latency manifesting as "slow-feeling internet". On mybitch (Christine's laptop, the original report): gateway ping avg 67.9 ms, max 173.9 ms, mdev 55.7 ms at 1-second cadence vs avg 14.5 ms / mdev 16.4 ms at 0.2-second cadence (faster cadence kept the card awake, masking the issue). On ratio (Arch desktop): same pattern, mdev 46.7 ms, max 183.5 ms.
- Root cause: NetworkManager defaults to =wifi.powersave = 3= (enable) when no override is configured. On Mint/Ubuntu, this is shipped explicitly as =/etc/NetworkManager/conf.d/default-wifi-powersave-on.conf=. On Arch, it's the upstream NetworkManager default (no shipped file, but same effective behavior). On bursty traffic (typical browsing), the WiFi card sleeps between bursts and the first packet of each new burst takes 50-150 ms to wake it — pages feel like they "stall" before loading.
- Fix: drop =/etc/NetworkManager/conf.d/wifi-powersave-off.conf= with =wifi.powersave = 2= (disable). NetworkManager merges conf.d alphabetically; =wifi-...= sorts after =default-...= and =dns.conf= so the override wins. Restart NetworkManager applies it.
- Verification: post-fix gateway ping (both mybitch and ratio) avg 12 ms, max 21-24 ms, mdev 3 ms — 15-18× improvement in jitter, 8-9× improvement in max latency. Throughput unchanged at the ISP/router ceiling (~85 Mbit/s down, 25 Mbit/s up — same on both clients, so that's the connection limit, not a per-host issue).
- Velox edge: velox's WiFi card defaults to powersave-off at the driver/kernel level even without an explicit NM config (different chipset behavior). Latency was already healthy. Defensive =wifi-powersave-off.conf= added anyway so the configuration is uniform across the homelab and protected against future NM-default or driver-behavior changes.
- Tradeoff: ~0.5-1 W more power draw — irrelevant on AC, slight battery hit on battery. All three hosts are mostly on AC.
- Coverage status as of this entry:
  - mybitch (Mint): explicit override applied, was active issue → resolved
  - ratio (Arch): explicit override applied, was active issue → resolved
  - velox (Arch laptop): explicit override applied defensively, no observable issue
  - truenas: not on WiFi, N/A
- Pattern note: any new NM-using host added to the homelab should get this override at provisioning time. The 2026-04-30 mybitch upgrade soak picked up the issue indirectly; future host adds should set this proactively.

** 2026-05-10: mybitch — keyboard soak verified closed (2026-03-24 USB hub fix held over 9-day uptime)
:host: mybitch
- Reminder from 2026-04-30 (overdue day 10) was to verify the 2026-03-24 USB-autosuspend fix holds in routine use.
- Verification (2026-05-10, mybitch uptime 1w 2d 2h): all three Framework 16 internal hubs (1-2, 1-3, 1-4) show =control=on, delay=0= — udev rule active. Total USB error count this boot: *0*. Soak passed.
- Closes the 2026-04-30 reminder. The 2026-03-27 mybitch BIOS-side fixes (which were gated on this soak result) are now unblocked.

** 2026-05-10: dmesg -T displays future-dated timestamps after suspend cycles
:host: any (laptops affected; desktops rarely suspend)
- Symptom: =dmesg -T= shows wall-clock timestamps drifted forward by the cumulative suspend duration since boot. Example: mybitch on 2026-05-10 with 9d 2h monotonic uptime displayed messages dated May 15-17 (~7 days of suspend during the boot session pushed the displayed dates 7 days into the future).
- Root cause: kernel printk timestamps include suspend time (boottime-style clock). =dmesg -T= computes the boot epoch using =current_realtime − CLOCK_MONOTONIC=, but =CLOCK_MONOTONIC= excludes suspend. The two clocks diverge by however long the system was suspended during the current boot, and =dmesg -T= adds the (boottime-correct) message offset to a (monotonic-derived) boot epoch — producing future-dated displays.
- Verification: =journalctl -k -b= shows the same kernel messages with correct timestamps because journald stamps =CLOCK_REALTIME= at message-write time. =date=, =timedatectl=, and =/proc/stat btime= are all correct.
- Workflow fix (applied 2026-05-10): Phase 1 check #4 now uses =journalctl -k -b= instead of =dmesg=. Tradeoff is ~100 ms vs ~10 ms per query — negligible at this cadence. Side benefits: cross-boot queries (=-b -1=, =-b -2=) and longer history retention than the kernel ring buffer.
- Live cleanup (optional): a reboot resets both clocks and =dmesg -T= comes back accurate until the next long suspend session. Not urgent — the workflow change routes around the bug regardless.
- Out of scope: the dmesg tool itself isn't going to change behavior here; this is a long-standing util-linux design decision around how to translate ring-buffer timestamps. We don't fight it; we use the right tool.

** 2026-05-10: aardvark-dns "empty response" spam during WinVM podman runs
:host: ratio
- Pattern: =aardvark-dns[N]: <port> dns request got empty response= logged at error priority while the podman-rootless aardvark-dns daemon forwards DNS for the WinVM container
- Volume: ~1,200 lines per WinVM session (Windows guest aggressively retries DNS lookups, every retry that gets back NOERROR/empty triggers one line)
- Trigger: only logs while a podman container with the rootless aardvark-dns network is running. Daemon stops with the container, so journal noise is bounded by WinVM uptime
- Active when looked at: aardvark-dns daemon is not running between WinVM sessions; these errors are retrospective journal entries, not a live issue
- Classification: KNOWN — annotate as =KNOWN — WinVM podman aardvark-dns DNS retry noise= and check the surrounding podman/WinVM lifecycle to confirm correlation. Volume scales with WinVM session length.
- No fix in scope. Suppression options if it becomes annoying: (a) journald filter rule for unit pattern, (b) switch WinVM to a different DNS path. Neither pursued today.

** 2026-05-10: cameractrls cameraptzmidi.py SEGV during Python 3.14 exit
:host: ratio
- =python3 /usr/lib/python3.14/site-packages/CameraCtrls/cameraptzmidi.py -l= prints its output (e.g. "JDS Labs Element IV MIDI 1:32:0") and then SEGVs during interpreter shutdown
- Stack trace lives entirely inside =libpython3.14.so= on the =Py_Exit= path — no cameractrls frames, no asound frames at the crash point
- Pattern: ctypes-loaded =libasound= atexit handler vs Python 3.14 finalization order; the script's job completes before the crash so functional behavior is fine
- Reproduces every invocation. Three coredumps on 2026-05-08 11:49–11:50 were Craig running it three times in a row, not a regression burst
- Classification: KNOWN (upstream Python 3.14 / cameractrls issue, no local fix). Annotate future cameraptzmidi coredumps as =KNOWN — Py_Exit / libasound finalization=
- Real fix is upstream: Python 3.14 ctypes-finalization change or cameractrls explicit asound cleanup before exit. Not worth tracking locally.

** 2026-05-11: mybitch — hard freeze caused by amdgpu iGPU MES hang; mitigated with =amdgpu.cwsr_enable=0=
:host: mybitch
- Symptom: full system freeze during active GUI use (Christine typing in a browser). Desktop frozen, keyboard + trackpad dead, off the network. Not the s2idle USB-keyboard bug — the machine was awake, not resuming from suspend, and the WiFi (M.2 PCIe, not USB) was dead too, so a USB-hub-only failure is excluded. Required hard power-cycle. =last= records the prior session as ending in "crash".
- Evidence in =journalctl -b -1=: the journal *stops dead* with no panic / oops / hung-task warning / OOM / GPU-reset trace — classic hard hang (kernel wedged, logging stopped). pstore empty. Preceding the freeze: =amdgpu 0000:c5:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)= + =failed to reg_write_reg_wait= — *38 occurrences* over the 10-day boot, *zero* in every prior boot including the pre-upgrade 6.8.0-110 stretch. Last one ~4.7 min before the hang. =c5:00.0= is the *integrated* GPU (Radeon 780M / Phoenix1, GFX 11.0.x); =03:00.0= is the dGPU (RX 7700S) — the dGPU's =PSP/SMU is resuming= log lines are routine runtime-PM, not errors.
- Root cause class: amdgpu MES (Micro Engine Scheduler) firmware going unresponsive on AMD GFX11 APUs (Phoenix 780M, Strix Point, Kraken Point) — a known issue across a wide range of recent kernels (6.11 → 6.18+), no clean upstream fix. The MES errors appeared exactly when mybitch moved to kernel 6.17.0-23 (Mint 22.3 HWE-edge, adopted 2026-04-30 to fix the s2idle USB-keyboard bug). Tracked at: Framework Community "AMD GPU MES Timeouts Causing System Hangs", ROCm issues #3207 / #5590 / #5844, drm/amd GitLab #2986.
- Mitigation applied 2026-05-11: =amdgpu.cwsr_enable=0= added to =GRUB_CMDLINE_LINUX_DEFAULT= in =/etc/default/grub= on mybitch (backup: =/etc/default/grub.bak-2026-05-11-amdgpu-cwsr=), =update-grub= run, param verified in all 6 grub.cfg entries. Disables Compute Wave Store and Resume — the feature several reporters found triggers the MES firmware hang. Narrow, low-risk. Takes effect on next reboot.
- Watch: after the next reboot, check =journalctl -k -b | grep -c 'MES failed'= — should stay 0. If a freeze recurs even with cwsr disabled, escalate to =amdgpu.mes=0= (disables hardware MES scheduling entirely, falls back to KIQ; heavier hammer). Last resort = pin back to 6.8.0-111 (still installed), but that reintroduces the s2idle USB-keyboard bug, so not clean.
- *Update 2026-05-12:* =cwsr_enable=0= did NOT reduce the MES errors — 14 "MES failed to respond" hits in the first 14.5h boot after the fix (≈0.97/h) vs. 38 over the prior 10-day boot (≈0.16/h), i.e. *higher* rate; no freeze yet. Escalated proactively (Christine about to travel with the laptop for a week): added =amdgpu.mes=0= to =/etc/default/grub= alongside =cwsr_enable=0= (backup =/etc/default/grub.bak-2026-05-12-amdgpu-mes=, =update-grub= run, in all 6 grub.cfg entries). Takes effect on the next reboot (after the in-progress first rsyncshot backup). =mes=0= disables the GFX11 hardware MES scheduler → kernel falls back to the mature legacy KIQ path → sidesteps the hanging MES firmware entirely. Tradeoff: KIQ-on-GFX11 is a less-traveled config (small chance of cosmetic display/modeset quirks; irrelevant for a light desktop workload). Post-reboot check: =journalctl -k -b | grep -c 'MES failed'= should be 0; watch for any new amdgpu display oddities. (Also tracked in =homelab-inventory/mybitch-laptop.org= → Operational Changes Log.)
- *Update 2026-05-12b (logged retroactively 2026-05-25):* =mes=0= was swapped for =uni_mes=0= the same day (backup =/etc/default/grub.bak-2026-05-12b-uni-mes=). This refinement was applied but never written into this entry until the 2026-05-25 health check found the live cmdline disagreeing with the documented =mes=0= mitigation. Current cmdline: =amdgpu.cwsr_enable=0 amdgpu.uni_mes=0=.
- *Update 2026-05-25 (health check):* Kernel bumped 6.17.0-23 → 6.17.0-29-generic (Mint HWE). MES errors persist: 49 =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0= over a 3d5h boot (≈0.63/h), *no hard freeze this uptime*. Key finding from =modinfo amdgpu= on 6.17.0-29: =mes= now defaults to 0 (disabled) and =uni_mes= defaults to 1 (enabled) — between -23 and -29 AMD moved GFX11 onto the *unified* MES path, so =uni_mes=0= is the current-kernel equivalent of the old =mes=0= mitigation. Setting =amdgpu.mes=0= explicitly on this kernel just restates the default and won't change behavior; the errors are on the unified-MES path, which is already disabled. Decision (Craig, 2026-05-25): leave the cmdline as-is — there is no stronger *documented* knob left to pull beyond =uni_mes=0=, and there's no freeze to chase. The newer kernel appears to recover from each MES timeout rather than wedging. Watch freeze behavior; escalation lever if a hard freeze recurs would be the experimental =amdgpu.mes=0 amdgpu.mes_kiq=0 amdgpu.uni_mes=0= full-legacy-KIQ path (not applied — risky on a traveling laptop) or pinning back to 6.8.0-111 (reintroduces the s2idle USB-keyboard bug).
- Classification: when a future health check on mybitch sees =MES failed to respond= / =reg_write_reg_wait= on =c5:00.0=, annotate as =KNOWN — amdgpu GFX11 iGPU MES hang= and check whether the count is climbing despite the mitigations; a single hard freeze with the dead-journal signature is the same issue recurring. On kernel 6.17.0-29+ the box runs =uni_mes=0= (the unified-MES disable), which is the heaviest documented knob in use — MES errors persisting on it without a freeze is the expected steady state, not an escalation signal. A *hard freeze* with the dead-journal signature is the real escalation trigger; remaining levers at that point are the experimental full-legacy-KIQ cmdline or pinning back to 6.8.0-111.

** 2026-05-11: mybitch — power-profiles-daemon was stuck on power-saver (now balanced); benign dGPU power-limit error
:host: mybitch
- Symptom: =amdgpu 0000:03:00.0: amdgpu: New power limit (30) is out of range [100,120]= + =amdgpu: Failed to set power limit value= at error priority, once per boot (and on profile changes).
- Root cause: =power-profiles-daemon= in the =power-saver= profile tried to cap the discrete RX 7700S (=03:00.0=) at 30W, but the GPU's settable power-cap range is [100W, 120W] (current cap 100W). amdgpu rejected the out-of-range request → cosmetic error; the dGPU stayed at its 100W floor regardless. Underlying issue: PPD persists the last-set profile in =/var/lib/power-profiles-daemon/state.ini= and restores it on every boot — someone had set =power-saver= long ago (state.ini mtime Oct 2025) and it had been pinned there ever since. Christine runs mybitch on AC a lot, so this wasn't what she wanted.
- Resolution (2026-05-11): set the profile to =balanced=. =powerprofilesctl set= over SSH is polkit-denied (=switch-profile= needs an active local session), so done by editing =state.ini= directly under sudo (=Profile=power-saver= → =Profile=balanced=) with =power-profiles-daemon= stopped, then restarting it. Verified =powerprofilesctl get= → =balanced=; persists across reboots; no new power-limit errors. Christine can still flip the profile anytime via the Cinnamon power applet.
- Classification: if the =New power limit (30) out of range= error reappears, it means the profile drifted back to =power-saver= — annotate as =KNOWN — PPD power-saver dGPU cap rejection= and re-set to =balanced= the same way. Harmless either way; the only reason to fix it is that power-saver throttles the box on AC.

** 2026-05-18: velox — NVMe Error Information Log entries from kernel optional-feature probing
:host: velox
- Symptom: =smartctl -A /dev/nvme0n1= shows =Error Information Log Entries: 9,530= and climbing slowly over time, despite =Media and Data Integrity Errors: 0= and =SMART overall-health PASSED=.
- Detail (=smartctl -l error /dev/nvme0n1=): all 9,530 entries are a single error class — =Status 0x4004 "Invalid Field in Command"= on the Admin Submission Queue (SQId 0). One row in the error info log, accumulated over 20,079 power-on hours (~2.3 years of drive uptime).
- Root cause: kernel or userspace polls for an optional NVMe feature/log page the drive doesn't implement; the controller rejects with =Status 0x4004=, the host sees ENOTSUP and moves on, but the drive faithfully logs every rejection. Common on NVMe drives where newer kernels probe for features added after the drive's firmware was written.
- Classification: KNOWN — annotate future findings as =KNOWN — NVMe Invalid-Field-in-Command from kernel optional-feature probing=. The counter going up is not a failure signal; only =Media and Data Integrity Errors=, =Critical Warning=, or =Available Spare= changes indicate actual drive health regression.
- Out of scope: no kernel-side fix worth chasing. The probe is harmless and would have to be tracked down to a specific kernel subsystem; not worth the time when the only effect is a benign counter.

** 2026-05-25: mybitch — timeshift SIGSEGV on =--check --scripted= cron runs (GObject teardown race)
:host: mybitch
- Symptom: =coredumpctl= shows intermittent SIGSEGV of =/usr/bin/timeshift= during the hourly =timeshift --check --scripted= cron job (e.g. 2026-05-23 02:00, 2026-05-24 23:00 — 2 in a 3-day window). Crash is at address =0x30= inside =libgobject-2.0.so.0= =g_object_unref=, with a second thread in =g_file_get_contents= — a multi-threaded GObject teardown race.
- Timing: the process runs ~29 s (started :01, segfault :30) — it completes its work and dies during cleanup/exit, the same "job done, crash on teardown" shape as the cameractrls Py_Exit case.
- Backups unaffected: hourly =backup.log= files in =/var/log/timeshift/= are present every hour, =--list= works, 50 snapshots on disk. timeshift rsyncs to a temp dir then promotes, so a crashed run leaves at worst an incomplete dir the next run ignores — not a corrupt snapshot.
- Long-standing, not a regression: an older coredump from 2026-01-06 has the identical two-thread =g_object_unref= / =g_file_get_contents= signature. Version: =timeshift 25.12.4+zena= (Mint's maintained fork); apt shows 0 pending so no newer build to move to.
- Classification: KNOWN — annotate future findings as =KNOWN — timeshift zena teardown SIGSEGV on --check=. A real escalation signal would be missing hourly =backup.log= entries or a gap in the snapshot timeline; the coredumps alone are cosmetic. No fix in scope (upstream zena-fork bug); suppression isn't worth it.

** 2026-06-13: ratio — dockerized telega-server SIGSEGVs in musl build
:host: ratio
- Symptom: =coredumpctl= shows repeated SIGSEGVs from =telega-server -O 31 -l /home/cjennings/.telega/telega-server.log -v 3= running inside the Telega Docker container. Example stack is entirely in the container's musl loader plus =/usr/bin/telega-server=, not host kernel, GPU, storage, or memory paths.
- Evidence: =~/.telega/telega-server.log= ends with =Unexpected char 'm' in plist value= followed by =Assertion failed: false (telega-dat.c: tdat_plist_value: 500)=. Surrounding TDLib traffic includes sticker/custom-emoji metadata such as =documentAttributeCustomEmoji= and =PhotoSizeSourceThumbnail[Thumbnail, type = m]=.
- Prior local triage: =~/.emacs.d/todo.org= recorded the same issue on 2026-06-11 as spontaneous memory-corruption crashes in =zevlg/telega-server:latest='s musl build, with several coredumps occurring without action-verb traffic. Telega package installed at the 2026-06-13 health check was =20260513.509=; MELPA had =20260604.2321= available.
- Functional status at 2026-06-13 check: no coredumps yet that day; Telegram scans still worked from cached chat state. Treat as an app/server-container crash, not a machine-health fault.
- Classification: KNOWN — annotate future =telega-server= coredumps on ratio as =KNOWN — dockerized telega-server musl SIGSEGV= if the signature matches =tdat_plist_value= / unexpected plist value or otherwise stays inside the Telega container. Escalate only if crashes become continuous, break Telegram workflows, or appear after moving off the Docker musl build.
- Deferred remediation options, in order of least disruption: update the Emacs =telega= package, rebuild/pull a newer =telega-server= image, pin a known-good pre-2026-06 image digest, build =telega-server= natively, or report upstream with =coredumpctl= and log evidence.