diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/2026-06-25-testinfra-validation.org | 8 | ||||
| -rw-r--r-- | docs/design/2026-06-29-waybar-network-module-spec.org | 2094 | ||||
| -rw-r--r-- | docs/design/2026-06-29-waybar-timer-module-spec.org | 217 | ||||
| -rw-r--r-- | docs/design/2026-06-29-zfs-pre-snapshot-installer.org | 106 | ||||
| -rw-r--r-- | docs/design/2026-06-30-captive-portal-login.org | 89 | ||||
| -rw-r--r-- | docs/design/2026-07-02-waybar-expansion-animation-feasibility.org | 53 | ||||
| -rw-r--r-- | docs/design/2026-07-10-audio-failure-taxonomy.org | 444 | ||||
| -rw-r--r-- | docs/design/2026-07-10-net-bt-failure-taxonomy.org | 434 | ||||
| -rw-r--r-- | docs/design/maintenance-console-design-ideas.org | 527 | ||||
| -rw-r--r-- | docs/design/system-monitor-design-ideas.org | 1008 |
10 files changed, 4977 insertions, 3 deletions
diff --git a/docs/design/2026-06-25-testinfra-validation.org b/docs/design/2026-06-25-testinfra-validation.org index 5c82aa2..0f81231 100644 --- a/docs/design/2026-06-25-testinfra-validation.org +++ b/docs/design/2026-06-25-testinfra-validation.org @@ -106,16 +106,18 @@ scripts/testing/tests/ ** Example tests (parity) #+begin_src python +TEST_USER = os.environ.get("ARCHSETUP_TEST_USER", "archie") + def test_ufw_enabled(host): assert host.service("ufw").is_enabled -def test_user_cjennings_exists(host): - u = host.user("cjennings") +def test_primary_user_exists(host): + u = host.user(TEST_USER) assert u.exists assert u.shell == "/usr/bin/zsh" def test_zshrc_stowed_and_readable(host): - f = host.file("/home/cjennings/.zshrc") + f = host.file(f"/home/{TEST_USER}/.zshrc") assert f.is_symlink assert ".dotfiles/" in f.linked_to assert f.exists # not broken diff --git a/docs/design/2026-06-29-waybar-network-module-spec.org b/docs/design/2026-06-29-waybar-network-module-spec.org new file mode 100644 index 0000000..3a1260c --- /dev/null +++ b/docs/design/2026-06-29-waybar-network-module-spec.org @@ -0,0 +1,2094 @@ +#+TITLE: Waybar Network Module — Design Spec +#+AUTHOR: Craig Jennings +#+DATE: 2026-06-29 + +* Status + +*Phases 1-3 SHIPPED* (2026-06-29 → 2026-06-30, dotfiles). The core module is live: +the =net= engine (=status/probe/list/up/down/add/edit/remove/rescan/diagnose/repair/ +doctor/portal/speedtest=), the =waybar-net= indicator (split-cadence cache, redacted +event log, display-only airplane absorption per decision 12), and the GTK4 +layer-shell panel (Connections / Diagnose / Repair / Speed test) with the settled bar +clicks (left = panel, middle = =net portal=, right = =net-fix=; airplane on +Super+Shift+A). 230+ net tests; full dotfiles suite green. Live-verified on velox. + +Built on top since the original spec: +- *Captive-portal login engine* (2026-06-30, dotfiles =a7d7559=) — =net portal= now + runs a native =portal-login= repair tier (drop DoT → recover the portal URL from + the redirect → open a throwaway browser profile → auto-restore DoT once online), + replacing the old shell-out to =captive= for the force-portal flow. =net portal + --restore= is the manual fallback. +- *Portal UX fixes from live testing* (2026-06-30, dotfiles =eef6b0b=) — removed a + polkit-gated =resolvectl flush-caches= that popped an auth dialog (the DoT-drop + restart already clears the cache); added an already-online short-circuit so a + forced run on a working connection opens nothing; suppressed Chrome's first-run + wizard; moved =net portal= off the terminal into the panel status line; hardened + the portal-URL extractor against Firefox's detection page. +- *Panel auto-hide + Close button* (2026-06-30, dotfiles =450b7f0=) — the panel + closes on focus-out (popup behavior, suppressed while a child dialog holds focus) + and carries a Close button bottom-right. + +*V2 redesign in flight* (designed 2026-06-30, not yet built — see todo.org "Network +panel redesign — no terminals, verify-everything, full failure coverage"). It +reverses two earlier choices and widens coverage: +- *No terminals anywhere.* =net-popup= is removed; every action and result renders + in the panel. This depends on a passwordless privileged path — a root-owned helper + plus a narrow NOPASSWD sudoers rule, archsetup-installed — because an in-panel + worker thread has no tty to prompt for a password. Reverses decision 11's + "privileged tiers run in a terminal". +- *New navigation* — top tabs Connections | Diagnostics | Performance. Diagnostics + merges Diagnose + Repair (sub-row Diagnose | Get Me Online | Advanced; a shared + area below shows diagnose items and streams repair progress; Advanced reveals the + individual repair buttons, renamed with tooltips). Speed test lives under + Performance. +- *Verify every action* (each mutating op confirms its effect before reporting + success) and *detect + respond to every failure mode* — the full ~44-mode catalog, + edge cases included, lives in the redesign task and supersedes the table below. + +Phase 4 (docs / rollout) and Phase 5 (VPN) remain. Review incorporated (Codex, +2026-06-30): four review rounds + Craig's cj comments are all dispositioned +([40/40], no open findings) — the fourth round reshaped the V2 panel UX (single nav +target, saved-vs-available groups, join-from-row, the auth matrix, progressive +loading, a findable diagnostics report, and the Waybar visual contract; see "V2 panel +UX"). Phases 1-3's manual live checks are under todo.org "Manual testing and +validation". + +* Goal + +One waybar network component that does the whole job: shows connection state +(including the missing "associated but no internet / captive portal" state), +manages connections from a dropdown (nmcli-backed; secrets stay in +NetworkManager's own store, no separate credential file), and runs the network +diagnostics and remediation off the same place +(captive-portal detection + forcing, bounce/reset, gateway/DNS checks, speed +test). + +It unifies three todo tasks that are really one feature: +- =[#C]= "archsetup Waybar Wi-Fi module should show no-internet state" — the + indicator state plus the 2026-06-22 roam expansion (bounce, diagnostics, speed + test off the component). +- =[#B]= "Network-manager dropdown, nmcli-backed" — the management dropdown. (The + todo task's original "GPG-stored secrets" framing is superseded: secrets stay in + NM's own store, decision 5.) +- The network diagnostics already shipped in =captive= (the hotel/captive-portal + tool, formerly =login-page=) become this module's diagnostics engine rather + than a standalone CLI. + +* Scope + +** In +- *Indicator* — wifi/ethernet icon + signal + SSID, plus an internet sub-state: + online / captive / no-internet / connecting / disconnected / airplane. +- *Absorbs the airplane module* — the airplane state + toggle move into + =custom/net= (airplane is a network concern). Once this ships, the standalone + =custom/airplane= module, the =waybar-airplane= + =airplane-mode= scripts, their + =tests/=, and the css are deleted (listed under Files touched). The + desktop-settings panel (sibling =[#B]=) no longer needs an airplane row. +- *Interface-correct* — targets the wifi (or chosen) device, not the + default-route interface, so an active USB tether or wired link can't mask + wifi state. (Same lesson =captive= fixed; the current =custom/netspeed= keys + off the default route and has the bug.) +- *Connection management (panel)* — list saved connections most-recently-used + first, live signal for in-range wifi, click to switch; add / edit / remove for + open + WPA-PSK; activate any existing saved profile (including enterprise ones + NM already stores); ethernet↔wifi and wifi↔wifi switching even when a link + appears mid-session. +- *Diagnostics (panel)* — read-only Diagnose (captive probe 204-vs-portal with + the extracted portal URL, gateway ping, DNS config) separated from mutating + Repair. Repair has tiers, lightest first: rfkill-unblock, per-connection reset + (fresh MAC), full-stack bounce (=nmcli networking off/on=, then restart + NetworkManager if that fails), and the temporary 1.1.1.1 override test. Each + Repair action confirms and verifies cleanup. +- *Speed test (panel)* — down/up/ping with a progress indicator and last-result + shown, via the already-installed =speedtest-go --json=. +- *Connection secrets* — none of our own. Settings and passwords live where NM + already keeps them: =/etc/NetworkManager/system-connections/*.nmconnection= + (root-only =0600=, the PSK/EAP secret stored inline). We read/write them through + nmcli, which handles the privilege. No separate file, no GPG, no gpg-agent — one + fewer dependency, and NM's store is already the secure-at-rest source of truth. +- *Persistence* — connectivity probe result cached in the runtime dir so the + bar reads it cheaply between probes. +- *Observability* — a redacted JSONL event log so a post-failure session can + diagnose without re-running destructive actions. + +** Out (v1, note for later) +- No replacement of NetworkManager's connection engine. NM stays the thing that + connects; we drive it via nmcli. +- No add/edit *form* for WPA-Enterprise / 802.1X in v1. The reason is effort vs + payoff: 802.1X has many interdependent fields (CA cert, client cert, identity, + anonymous identity, phase-2 auth) where a wrong entry silently fails auth, so a + trustworthy form is a lot of UI for connections Craig rarely adds (open + + WPA-PSK covers home, hotels, and phone hotspots). v1 still *activates* existing + saved enterprise profiles and points editing at =nmtui=/=nmcli=. Settled + (Craig, 2026-06-29): enterprise add/edit is vNext — 24 saved profiles on velox, + 0 enterprise, so the form would be unused UI; if one ever appears nmtui adds it + once and the module activates it thereafter. +- No per-connection captive-portal *auto-login* in v1. (That would mean storing a + portal's login form answers — room number, surname, a checkbox — and replaying + them automatically when a known portal is detected, so the page never appears. + Out for v1 because every portal's form differs and it means storing per-venue + answers; v1 just opens the portal for you.) +- No graphing/history of speed-test results beyond the last run. +- No static-IP / proxy / metered / MAC-randomization editing in v1 (activate + existing, edit elsewhere). +- No VPN / WireGuard management in v1, but it's a planned later phase (Phase 5), + not a permanent exclusion — it folds the existing archsetup wireguard tooling + into the same panel/CLI. +- The desktop-settings dropdown (sibling =[#B]=) is a separate module, but it + shares the GTK4 layer-shell panel shell built here. + +* Architecture + +Three layers. Keep the bar cheap, the panel rich, the logic in one tested place. + +1. *Engine* — a =net= Python package (src-layout, unittest), exposing a CLI. Wraps + every nmcli op and owns the diagnostics. Emits JSON. This is the testable + core (fake =nmcli= / =curl= / =speedtest-go= on PATH, like the existing + =waybar-netspeed= and =waybar-sysmon= test harnesses). Precedent: pocketbook is + Python in the dotfiles repo; =wtimer= is Python for the same testability + reason. +2. *Indicator* — a thin =waybar-net= script that calls =net status --json= and + renders icon + signal + state + tooltip. Replaces =custom/netspeed= + (throughput folds into the tooltip). +3. *Panel* — a GTK4 + gtk4-layer-shell app (mirrors pocketbook's structure) + that imports the engine. Hosts connection management, diagnostics, and the + speed test. + +How the existing pieces map in: +- =captive= (bash, shipped) — its cheap portal-detection logic is mirrored natively + in the engine for the fast status path so the bar never blocks on a subprocess, + and it still exposes a =--probe-json= mode the engine reuses. *As built (2026-06-30): + the force-portal flow is now native too* — =repair.py='s =portal-login= tier does + the DoT drop, portal-URL recovery, clean-browser launch, and auto-restore in + Python, so =net portal= no longer shells out to =captive= for it. =captive= stays a + usable standalone CLI. +- =waybar-netspeed= (sh, shipped) — retired; its throughput sampling moves into + the engine's status output and renders in the indicator tooltip only. +- =nmcli= — the connection backend for every op. + +Language note: the engine is Python; the indicator is a thin Python or sh +wrapper over =net status --json=. The bar path must stay fast (see Performance +budgets), so the indicator does no network I/O itself — it reads link state and +the cached connectivity result. + +Privileged-path model (v2, planned): repairs that need root (rfkill unblock, nmcli +modify/up, networking off/on, =systemctl restart NetworkManager/systemd-resolved=, +resolvectl dns/revert, the DoT toggle) go through a single root-owned helper +installed by archsetup, with a narrow NOPASSWD sudoers rule scoped to that helper +only (never a blanket =mv=/=systemctl= rule). =repair.py= calls =sudo <helper> +<verb>=. This is what lets every action run in-panel with no terminal: a GTK worker +thread has no tty, so without a passwordless path it can't prompt. It also fixes a +latent bug in the shipped portal flow — the detached DoT-restore watcher runs with +no tty and silently fails to restore encrypted DNS when sudo creds aren't cached. + +* Repository + dependencies + +- *Code lives in the dotfiles repo* (=~/.dotfiles=), not archsetup. The =net= + package sits in-tree like pocketbook (src-layout, unittest, Makefile target); + =waybar-net= and the =net= CLI entry live in the hyprland tier + (=hyprland/.local/bin/=). Tests under =tests/net/= and =tests/waybar-net/=. + archsetup owns only the *dependency install*, not the code. +- *archsetup installs the deps* in its Hyprland step: =gtk4-layer-shell=, + =python-gobject=, plus =nmcli=/=curl=/=resolvectl=/=rfkill= (already present via + NetworkManager/curl/systemd/util-linux). Speed test uses =speedtest-go= (AUR + =speedtest-go-bin=, already installed on velox); archsetup adds it to the AUR + list. librespeed-cli is the documented fallback if a self-hosted LibreSpeed + server is ever wanted. No =gpg= dependency (secrets live in NM's own store). +- *Daily-drivers*: a stowed-script + AUR-dep feature, so ratio needs the same + =git pull= + stow + the archsetup-added deps. Note the manual dep step in the + rollout. + +** Makefile targets (console recovery is a first-class path) +=net doctor= and the diagnostics are reachable from a bare TTY when waybar and +the GUI are down — that's the case where you most need them. The dotfiles +Makefile carries targets that wrap the =net= CLI so "get back online" is one make +command from the console: +- =make online= — =net doctor --fix= (diagnose, then apply the lightest repair: + rfkill-unblock → reset → bounce → open portal). The headline recovery target. +- =make net-doctor= — =net doctor= (read-only diagnose + recommendation). +- =make net-status= / =make net-diagnose= / =make net-portal= / =make net-reset= + / =make net-bounce= — the individual ops. +- =make test= — already runs =tests/*=; the =net= package's unittest suites are + collected the same way. +These intentionally need only nmcli/curl/rfkill (no GUI, no waybar, no Python +GTK), so they work from a TTY on a broken graphical session. + +* Connectivity model — split cadence + +The indicator polls every ~2s, but a real internet/captive probe every 2s wastes +battery and can re-trigger a captive portal. So split it: + +- *Fast path (every poll, cheap, no network)* — interface, type, SSID, signal, + IPv4 presence, throughput sample. From nmcli / sysfs only. No network I/O. +- *Slow path (cached, TTL ~45s)* — the actual internet/captive probe (the 204 + check + meta-refresh portal extraction). Result cached at + =$XDG_RUNTIME_DIR/waybar/net-connectivity.json= with a timestamp. + +The indicator reads the cache each poll. When the cache is older than the TTL, +=net status= kicks =net probe= in the background (spawn + detach, never awaited) +and renders the last cached sub-state meanwhile. A user-triggered +diagnose/reconnect refreshes the cache immediately. This keeps the bar +responsive and the portal un-poked. + +** Concurrency, atomicity, staleness +- *Single-flight* — =net probe= takes a lock file at + =$XDG_RUNTIME_DIR/waybar/net-probe.lock= (flock, non-blocking). A second probe + while one runs is a no-op, so a flapping 2s poll can't pile up overlapping + probes. +- *Atomic writes* — the cache is written to a temp file + =os.replace= (atomic + rename), so a reader never sees a half-written cache. Same pattern as =wtimer=. +- *Max probe runtime* — the probe has a hard timeout (≤ 6s total: curl + =--max-time 5= + slack). On timeout it writes an =unknown= result, never hangs. +- *Stale classes* the indicator distinguishes: fresh (< TTL), stale (TTL..3×TTL, + shown with a subdued/aging hint), expired (> 3×TTL → treat as unknown), + unknown (no cache / probe failed). The bar never shows a confident "online" + past the expired threshold. +- *Invalidation* — the cache records the iface + SSID + active-connection UUID it + was taken under; a change in any of them invalidates it immediately (a + reconnect must not show the old network's verdict). +- *Crash cleanup* — a stale lock older than the max runtime is ignored/reclaimed. + +* Performance budgets (hot path) + +The bar exec path (=waybar-net= → =net status=) must stay responsive: +- *Budget*: =net status= returns in < 100ms typical, < 250ms worst case. +- *No sleeping in the bar path.* Throughput is sampled from two reads of + =/sys/class/net/<iface>/statistics/{rx,tx}_bytes= across the *waybar poll + interval itself* (delta since the last cached sample + timestamp), not via an + in-process =sleep= like the old =waybar-netspeed=. The cache holds the prior + counters. +- *Subprocess cap*: at most one =nmcli= invocation on the hot path (a single + =nmcli -t -f ...= multi-field query), plus sysfs reads. Never a per-field + nmcli call. +- *Every subprocess has a timeout* (=nmcli --wait 2=, =subprocess timeout=). On + timeout or error the indicator emits a degraded JSON state (class + =net-degraded=, a neutral glyph) rather than blocking or crashing waybar. +- *Benchmark test*: a fake slow =nmcli= asserts =net status= still returns within + budget by falling back to the degraded state. + +* Engine — =net= CLI surface + +All subcommands take =--json= where a machine reads them. Pure formatting/state +functions under the CLI; IO (nmcli, curl, file) at the edges. Every subcommand +exits non-zero with a JSON error envelope (see JSON schemas) on failure. + +- =net status [--json] [--iface IF]= — fast link state + cached connectivity + sub-state + throughput. The indicator's source. Never does network I/O. +- =net probe [--iface IF]= — run the connectivity/captive probe now, update the + cache (single-flight, atomic), print online | captive (+ portal URL) | + no-internet | unknown. Mirrors =captive='s cheap detection natively. +- =net list [--json]= — saved connections, MRU order, active flag, plus in-range + wifi with signal. +- =net up <uuid>= / =net down [--iface IF]= — switch / disconnect. Operates on + UUID, not name (see nmcli contract). +- =net add= / =net edit <uuid>= / =net remove <uuid>= — manage connections + (open + WPA-PSK) through nmcli; the secret lands in NM's own + =.nmconnection=. Enterprise profiles are activate-only. +- =net rescan [--iface IF]= — wifi rescan. +- =net diagnose [--json]= — read-only report: gateway ping, DNS config, captive + probe. The structured contract below. Doubles as the post-failure snapshot. +- =net repair <action> [--json]= — mutating remediation, lightest first: + =rfkill= (unblock + radio on), =reset= (fresh MAC), =bounce= (full-stack: + =nmcli networking off/on=, escalating to =systemctl restart NetworkManager=), + =dns-test= (temporary 1.1.1.1 override, auto-reverted). Each confirms via the + caller and verifies cleanup. +- =net doctor [--json] [--fix]= — one-shot "get me online" mode for the console: + runs the full diagnose, then applies the lightest repair that fits (unblock + rfkill, reset, bounce, open portal) — read-only without =--fix=, acting with + it. The TTY recovery path when waybar/the GUI is down (see the Makefile + targets). +- =net portal [--restore]= — the native captive-login flow (=repair.py= =portal-login= + tier): short-circuits if already online, else drops DoT to plain DNS, recovers the + portal URL from the redirect, opens it in a throwaway browser profile, and spawns a + detached watcher that restores DoT once online. =--restore= forces the restore now. +- =net speedtest [--json]= — =speedtest-go --json= run; down/up/ping. + +* nmcli contract + +The command wrapper is the reliability boundary; SSIDs and connection names +contain spaces, colons, duplicates, hidden names, and non-ASCII. Rules: + +- *Terse, field-selected output*: =nmcli -t -f <fields> --escape yes ...= and + =nmcli -g <fields> ...= (get-values) for single-value reads. Parse with the + documented escaping (=\:= and =\\=); never naive =cut -d:=. +- *UUID is the handle.* Every saved-profile op (=up=, =down=, =modify=, =delete=) + uses the connection UUID, never the display name — names duplicate and contain + separators. =net list= surfaces UUIDs; the panel maps row → UUID. +- *Wait budgets*: activation/deactivation use =nmcli --wait <n>= with an explicit + budget (hot-path reads =--wait 2=; activation =--wait 30=). No unbounded waits. +- *Connectivity*: NM's own =nmcli networking connectivity= can return + =none/portal/limited/full/unknown=. Use it as a *cheap hint* on the fast path + when present, but the authoritative captive verdict is still our own probe + (NM's portal detection is coarser and config-dependent). +- *Parser tests* (fake nmcli fixtures): escaped colons and backslashes in SSIDs, + embedded newlines, duplicate connection names, hidden SSID (empty name), + non-ASCII SSID, the wired-appears-mid-session case, and the multi-active case + (wifi + tether both up). + +* JSON schemas + +Versioned (="v": 1=) envelopes so tests lock the contract. Sketches (fields +nullable unless noted): + +- =status=: ={v, iface, type: wifi|ethernet|none, ssid, signal, ipv4, + gateway, throughput: {rx_bps, tx_bps}, connectivity: online|captive|no-internet|unknown, + connectivity_age_s, connectivity_class: fresh|stale|expired|unknown, state: + online|captive|no-internet|connecting|disconnected|airplane|wired|degraded}=. +- =probe=: ={v, result: online|captive|no-internet|unknown, portal_url, http_code, + redirect_host, elapsed_ms, ts}=. +- =list=: ={v, connections: [{uuid, name, type, active, last_used, signal, + in_range, security}]}=. +- =diagnose=: ={v, steps: [<diagnostic step, see contract>], overall: + ok|warn|fail}=. +- =speedtest=: ={v, down_mbps, up_mbps, ping_ms, server, elapsed_ms, ts}=. +- error envelope (any command): ={v, error: {code, message, detail, partial: + bool}}= with a non-zero exit. + +* Diagnostics contract + +=net diagnose --json= returns an ordered list of steps. Each step is the unit the +panel renders and the log records: + +- =id= — stable identifier (e.g. =link=, =dhcp=, =gateway-ping=, =dns-config=, + =dns-resolve=, =http-probe=, =portal=). +- =status= — =pending | running | pass | warn | fail | skipped=. +- =title= — short human label. +- =evidence= — redacted detail (the value seen), per the redaction rules. +- =elapsed_ms=. +- =safety= — =read-only= or =mutating= (diagnose steps are all read-only). +- =next_action= — what the user/agent should do on warn/fail (e.g. "open portal", + "reset connection", "switch network"). + +Repair actions (=net repair=) carry the same shape but =safety: mutating=, plus a +=cleanup_verified: bool= field (e.g. the DNS override was reverted) and a +terminal =cleanup-unverified= status when revert can't be confirmed. + +** Diagnose vs Repair (read-only vs mutating) +The panel separates them visually and behaviorally: +- *Diagnose* — probe, gateway ping, DNS config read, captive check. No state + change, no sudo, runnable freely. +- *Repair* — reset (fresh MAC, deletes+recreates the NM profile), DNS override + test (mutates resolver, auto-reverts), portal force. Each needs an explicit + confirm, shows that it's privacy/state-changing, and verifies cleanup. A + Repair whose cleanup can't be verified ends in a visible =cleanup-unverified= + state, never a silent success. + +* Failure states, messages, recovery + +Each row below gives the *exact, final* user-facing string (not a template) with +=<placeholders>= for redacted evidence, plus the evidence field included and the +next action. The string is canonical: every surface renders the same text, so +there's one source of truth. + +Per-surface rendering of the canonical string: +- *Indicator* — the matching glyph + CSS class; the string is the tooltip + (untruncated). +- *Notification* (=notify=) — title = "Networking"; body = the failure label on + its own line, then the canonical string. +- *CLI* — the string on stderr; =--json= puts it in =error.message= with the + evidence in =error.detail= and a stable =error.code=. +- *Panel* — the string as the section banner, with the diagnostic step's evidence + shown beneath. +Evidence is always redacted per the redaction rules (SSID/host shown; PSK/EAP/ +portal tokens never). + +- *associated, no DHCP* — "Connected to <SSID>, no IP (DHCP failed)" → + evidence: SSID, iface → reset / reconnect. +- *no-internet* — "On <SSID>, no internet (gateway reachable, no route out)" → + diagnose / switch network. +- *captive* — "Captive portal at <host> — login required" → Open portal. +- *DNS hijack* — "DNS is being redirected (portal)" → Open portal. +- *DNS broken* — "DNS not resolving (hotel DNS down); 1.1.1.1 works" → use + override / report. +- *HTTP intercepted* — "Traffic is being intercepted before it leaves" → Open + portal. +- *sudo declined* — "Reset needs admin; it was declined — nothing changed" → + retry with auth. +- *command timed out* — "<op> timed out; the system was left unchanged" → retry. +- *partial mutation* — "<op> partially applied: <what>; rolled back to <state>" + → review. +- *missing speedtest-go* — "speedtest-go not installed" → install hint. +- *no wifi hardware* (desktop) — wifi rows hidden; ethernet-only view. +- *wifi rfkill-blocked* — "WiFi is blocked (rfkill)" → unblock. The indicator + detects a soft-blocked radio (=rfkill list= shows the radio off though hardware + is present) and shows this distinct from disconnected. =net repair rfkill= (and + =net doctor --fix= as its first step) runs =rfkill unblock wifi= + =nmcli radio + wifi on= and reconnects. This is the framework-laptop case: an out-of-power + shutdown sometimes leaves wifi soft-blocked at next boot, and yes — the module + recovers it (the rfkill state is the indicator; the rfkill repair / doctor is + the one-step fix). A *hard* block (physical switch) is reported as + not-recoverable-in-software with that message. +- *wifi rfkill hard-blocked* — "WiFi is blocked by the hardware switch" → + evidence: rfkill hard state → flip the physical switch. +- *wrong password / missing secret* — "Saved password for <SSID> was rejected" → + evidence: SSID, NM auth-failure reason → re-enter the password. +- *enterprise auth/cert failure* — "Enterprise login failed for <SSID> (802.1X)" + → evidence: SSID, EAP failure reason → edit the profile in nmtui/nmcli. +- *upstream / AP / provider* — "On <SSID>, link is fine but the network has no + uplink" → evidence: gateway reachable, no route out, not a portal → switch + network or contact the venue. +- *VPN-routed* — "Connected; internet is routed through a VPN (<dev>)" → + evidence: default route on a tun/wg device or non-NM DNS owner → check the VPN, + not WiFi. +- *HTTP interception, no parseable portal URL* — "A portal is intercepting + traffic but didn't give a login link" → evidence: HTTP code, redirect host → + opens neverssl + the gateway page to log in manually. +- *DNS override cleanup unverified* — "Couldn't confirm DNS was restored after the + test" → evidence: iface, attempted revert → revert DNS manually + (=resolvectl revert <iface>=). +- *VPN kill-switch blocking* — "A VPN kill-switch is blocking all traffic, and the + VPN itself is down" → evidence: a block artifact present with no tunnel up → bring + the VPN back, or clear the kill-switch (the exact root command surfaced, not + auto-run). + +*VPN kill-switch detection + correction.* A kill-switch blocks all non-VPN egress when +the tunnel drops, so the link looks up (wifi, IP, gateway) but nothing reaches the +internet. This extends the =deferred-vpn= branch: when a VPN is active and the probe +fails, run a rootless cascade to tell a working tunnel from a kill-switch that's +blocking because the tunnel is down — +- =ip rule= for wg-quick's =not fwmark 0xca6c= + =suppress_prefixlength 0= (and the + PostUp =REJECT ! -o %i= rule that makes it leak-proof); +- =wg show= for an up tunnel interface; +- =nmcli connection show= for Proton's =pvpn-killswitch= / =pvpn-ipv6leak-protection= + (device =pvpnksintrf0=); +- =nft list ruleset= / =iptables -S OUTPUT= for a drop/reject table (=killswitch=, + =protonvpn=, =oifname != "wg0" ... drop=); +- =nmcli -f connection.zone= for a firewalld =drop= zone. +Classify *kill-switch-blocking* only when a block artifact exists AND no tunnel +interface is up — that's what distinguishes it from a healthy VPN. Correction is tiered +by artifact and every option needs root, so surface the exact command rather than +auto-running it: =wg-quick down <iface>=, =nmcli connection delete pvpn-killswitch +pvpn-ipv6leak-protection=, =nft delete table inet killswitch=, or =nmcli connection +modify <con> connection.zone ''=. (Sits alongside the Phase 5 VPN work; detection can +land earlier since =deferred-vpn= already exists.) + +Each message names whether the system was left unchanged, partially changed (with +what), or fully changed, so the user knows the residue. + +* Doctor: escalation, classification, terminal states + +=net doctor= diagnoses, classifies the failure, then (with =--fix=) applies the +*lightest* repair that fits and re-checks — it never loops destructive repairs +against a failure they can't fix. Each failure resolves to one of four outcomes, +and the doctor stops at any terminal one: + +- =fixable= — a local repair should help. Escalate lightest-first: rfkill-unblock + → reset (fresh MAC) → bounce (full stack) → portal, re-probing after each, and + stop as soon as the probe returns online. +- =needs-user-action= (terminal) — no reset/bounce will help; doctor stops and + names the exact next step. Covers: wrong WPA password / missing NM secret + (enter the password), locked keyring or polkit denial (retry with auth), + enterprise 802.1X cert/identity failure (edit the profile in =nmtui=/=nmcli=), + captive portal login-required (open the portal + accept terms). Doctor must not + delete/recreate the profile against these — that loses the saved password and + makes things worse. +- =upstream-not-local= (terminal) — the local link is up but the problem is past + it: AP has no uplink, gateway down/dropping traffic, DHCP server broken, ISP + outage, portal backend failing. =diagnose= proves it (link up + IP + gateway + reachable, but no route out and not a captive redirect), and =doctor --fix= + stops after local repairs are exhausted with "local repairs tried; likely + upstream/AP/provider" + the evidence. Next action: switch networks or contact + the venue. +- =deferred/vpn= (terminal for v1) — an active VPN / policy route / non-NM + resolver owns the default route or DNS, so "no internet" may be the VPN's fault, + not WiFi's. v1 *detects* this (default route on a =tun/wg= device, or DNS owned + by something other than the NM link) and classifies it separately — "link is + fine; internet is VPN-routed" — rather than misclassifying it as a WiFi failure. + v1 does not repair it (VPN management is Phase 5); it names the VPN as the likely + owner and stops. + +** DNS handling in doctor (explicit per class) +- *Captive DNS hijack* — open the portal (the hijack clears on login). No DNS + mutation. +- *Broken resolver, 1.1.1.1 works* — the shipped =dns-test= repair is *diagnostic*: + it sets 1.1.1.1, confirms the venue resolver is the culprit, then auto-reverts + (=cleanup_verified=). Because it reverts, =doctor --fix= does not currently leave + you online in this case — it falls through to =upstream-not-local=, which + misreports a locally-fixable problem. *V2 fix (planned):* on a dns-test *pass* + (public DNS works), set a PERSISTENT resolver override and verify online, with an + offered revert — and classify it as its own outcome rather than upstream. +- *Port-53 / egress blocked* (even 1.1.1.1 fails) — terminal =upstream-not-local=; + doctor stops, since it's not locally fixable. + +* Failure-mode coverage + +*V2 note (2026-06-30):* the authoritative, exhaustive catalog (~44 modes across 10 +connectivity layers, edge cases included, each tagged fix-and-verify or report-text) +now lives in the redesign task (todo.org "Network panel redesign"). The table below is +the v1 baseline; two rows reflect intent the shipped code doesn't yet match, and the +v2 catalog closes them: =gateway unreachable= claims a bounce that doctor never +actually reaches (a no-route failure goes straight to =upstream-not-local=), and +=broken DNS, 1.1.1.1 works= auto-reverts so the user is left offline and misreported +as upstream (the v2 persistent-override fix closes this). + +For each common field failure: does =net diagnose= detect it, can =net doctor +--fix= repair it, and what terminal user action remains when it can't. (The +=needs-user-action= / =upstream-not-local= / =deferred/vpn= outcomes are defined +above.) + +| Failure mode | diagnose detects | doctor --fix | terminal user action | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| rfkill soft block | yes | yes (unblock) | none | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| rfkill hard block | yes | no | flip the physical switch | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| no wifi hardware | yes | n/a | use ethernet | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| associated, no DHCP | yes | yes (reset/bounce) | none, else switch network | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| gateway unreachable | yes | yes (bounce) | switch network if it | +| | | | persists | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| captive DNS hijack | yes | opens portal | log in at the portal | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| broken DNS, 1.1.1.1 works | yes | yes (temp override, | report the venue's DNS | +| | | auto-reverted) | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| HTTP captive portal | yes | opens portal | log in at the portal | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| HTTP interception, no | yes | opens neverssl + gateway | log in manually | +| parseable URL | | | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| upstream / AP outage | yes (link up, no route out) | no (stops after local) | switch network / contact | +| | | | venue | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| wrong WPA password / | yes | no | enter the password | +| missing secret | | | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| enterprise auth / cert | yes | no | edit the profile in | +| failure | | | nmtui/nmcli | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| duplicate SSID / | yes (UUID-keyed) | yes (activate by UUID) | none | +| connection-name | | | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| hidden SSID | yes | yes (connect by name) | enter SSID + password | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| multiple active links | yes | n/a | pick the interface | +| (wifi+tether) | | | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| wedged NetworkManager | yes | yes (bounce → restart NM) | none, else reboot | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| slow / hung command | yes (degraded) | retries within budget | retry | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| stale / corrupt cache | yes | self-heals (atomic + | none | +| | | invalidation) | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| DNS cleanup failure | yes | flags cleanup-unverified | revert DNS manually | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| missing speedtest backend | yes | n/a | install speedtest-go | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| +| VPN / policy-routing | yes (route/DNS ownership) | no (deferred to Phase 5) | check the VPN | +| interference | | | | +|----------------------------+-----------------------------+-----------------------------+-----------------------------| + +* Observability — logging + redaction + +- *Event log*: JSONL at =$XDG_STATE_HOME/net/events.jsonl= (fallback + =~/.local/state/net/events.jsonl=), size-rotated (e.g. 1 MB × 3). Every + mutating op and probe appends an event: =ts, op, argv (redacted), exit_code, + stderr_tail, elapsed_ms, iface, nm_uuid, probe_url_class, http_code, + redirect_host, cache_event=. +- *Redaction (always on)*: PSKs, EAP identities/passwords, NM secrets, and + portal query tokens are never logged. MAC addresses, full IPs, and SSID are + redacted when configured (=redact_mac=, =redact_ip=, =redact_ssid= in config). +- *Post-failure diagnosis*: =net doctor --json= is the snapshot + recommendation + (diagnose plus the suggested repair), =net diagnose --json= the raw report, and + the event log the history. =net doctor= is the console-recoverable entry point + (reachable as =make online= / =make net-doctor=). +- *Secret-leak tests*: assert no PSK/EAP/portal-token ever appears in any JSON + output, log line, or error message. + +** Automatic diagnostic verbose-capture (V2) + +A distinct layer from the event log above: that log records what =net= did; +this captures what the *underlying stack* did at debug verbosity during a run, so a +failed diagnosis leaves real ground-truth instead of relying on memory. Two triggers, +one mechanism: + +- *Automatic — on a failing diagnose.* When =net diagnose= ends =overall: fail=, the + next escalation (or =Get Me Online=) runs inside a verbose-capture session. +- *Manual — a debug on/off toggle in the panel's Advanced section.* "Debug on" + elevates and leaves it elevated (with a visible "debug capturing" indicator) so the + user can reproduce an intermittent problem over time; "Debug off" restores and + writes the bundle. Useful when the failure doesn't reproduce inside one diagnose. + +Mechanism (shared): +1. *Snapshot* the current log levels (=nmcli general logging=, resolved's level, + wpa_supplicant's). +2. *Elevate* the relevant components to debug at runtime, no restarts, scoped to the + domains that matter (NM: =WIFI,DHCP,DNS,CORE=; resolved; wpa_supplicant). +3. *Run* the diagnostics / repair. +4. *Capture the window*: =journalctl= for NetworkManager + systemd-resolved + + wpa_supplicant since the run started, a =dmesg= tail (driver / firmware / rfkill), + and any =curl -v= probe output. +5. *Restore* every level to its snapshot. +6. *Write a redacted support bundle* to =$XDG_STATE_HOME/net/bundles/<ts>/= and + surface it in the panel. + +Hard requirements: +- *Restore is guaranteed and idempotent.* A =try/finally= restores even on error, + and a crash-recovery guard detects "a prior run left NM/resolved/wpa_supplicant + elevated" on the next run and puts it back — the same shape as the DoT-restore + watcher. A crash must never strand the stack at debug verbosity. +- *Redaction before anything leaves.* Raw wpa_supplicant and NM debug logs carry the + PSK and EAP credentials in cleartext. The captured journal is scrubbed before the + bundle is written, shown, or shared; the secret-leak test asserts no passphrase or + EAP secret survives into a bundle. +- *Privilege via the V2 sudo-helper.* The log-level toggles need root, so they become + verbs on the passwordless helper (decision 16) — no extra prompt. + +Bonus — this closes a real detection gap, not just observability: the spec notes live +auth-failure detection is a v1 limit (it leans on a one-shot NM state-120 snapshot). +wpa_supplicant at debug during the run is exactly how a wrong-password or EAP failure +is caught properly, so the capture feeds back into classification. + +* Indicator (task #C — Phase 1, the fast win) + +** States (internet sub-state on top of link state) +- online — associated and the probe returned 204. Normal icon. +- captive — associated, probe hit a portal. Distinct glyph + warning CSS class; + tooltip names the portal host; left-click opens diagnostics with the portal + ready to open (Phase 2+; see interactions for the Phase-1 interim). +- no-internet — associated, probe failed (no portal, no 204). Distinct glyph + + warning class. +- degraded — =net status= couldn't read link state within budget (slow/failed + nmcli). Neutral glyph, =net-degraded= class. Never blocks the bar. +- rfkill-blocked — the radio is soft-blocked (=rfkill=), distinct from + disconnected. Distinct glyph; the fix is =net repair rfkill= / =net doctor=. +- connecting / disconnected / airplane / wired — as today, plus wired shown + correctly even when it appears after session start. (airplane is now this + module's state, absorbed from the retired airplane module.) + +** Glyphs +Nerd-font codepoints, final values verified live before merge (same discipline +as wtimer). Reuse the signal-strength ramp already in =waybar-netspeed=; add a +captive / no-internet / degraded overlay glyph. + +** Tooltip +SSID + signal + IPv4 + gateway + the throughput readout (absorbed from +netspeed) + the last probe result and its age (stale/expired hinted). + +** Interactions (no keyboard-modifier clicks — waybar can't qualify clicks by +modifier, so the rich actions live in the panel, not ctrl/super-click) +Clicks never block the bar: each dispatches a detached background job, single-flight +per action. *As built (settled live with Craig, 2026-06-29):* +- *left* — =net-panel= toggle (pkill-or-launch the GTK panel). +- *middle* — =net portal= (the captive-login flow). +- *right* — =net-fix= (=net doctor= with =--notify=: reports the result when the + outcome is one-way, opens a terminal only when it's fixable; the v2 redesign moves + even that into the panel). +- airplane toggle moved off the bar to Super+Shift+A. + +* Panel (tasks #B + #C diagnostics — Phases 2-3) + +GTK4 + gtk4-layer-shell, pocketbook scaffold (src-layout package, unittest, +Makefile, gtk4-layer-shell anchored dropdown under the bar). One panel shell, +reused by the future desktop-settings panel. + +Sections as built (Phases 1-3, a four-page stack switcher): +1. *Connections* — list, MRU-first, active marked, live signal bars for in-range + wifi; row click switches; buttons for add / edit / remove; a rescan control. +2. *Diagnose* (read-only) — Probe (204/captive, shows portal URL + Open), Gateway + ping, DNS config. Streaming step output (the diagnostics contract). +3. *Repair* (mutating, confirmed) — tiered lightest-first: Unblock rfkill, Reset + (fresh MAC), Bounce (full stack), DNS override test, Force portal. A "Get me + online" button runs =net doctor --fix= (the auto-escalating sequence). +4. *Speed test* — Run button, progress, down/up/ping result + last-run line. +As built, the panel also auto-hides on focus-out (popup behavior, suppressed while a +child dialog holds focus) and carries a Close button bottom-right (2026-06-30). + +*V2 nav (planned):* three top tabs — Connections | Diagnostics | Performance. +Diagnostics merges the Diagnose and Repair pages into one: a sub-row +=Diagnose= | =Get Me Online= | =Advanced= over a shared area that shows diagnose +items and streams repair progress in-panel (no terminal). =Advanced= reveals the +individual repair tiers (renamed, with tooltips) plus a *Debug capture on/off* +toggle (the manual side of the verbose-capture feature; a failing diagnose triggers +it automatically). Speed test moves under Performance. + +** V2 panel UX — the target design + +The shipped four-page stack (Connections / Diagnose / Repair / Speed test) is +*history*, not active design. V2 is the sole current target: one panel opened from +the bar, three top tabs — Connections | Diagnostics | Performance — and the page +model below is the contract for what gets built and what gets deleted, not just for +labels. + +*** Connections — saved vs available, join-from-row +Three labelled groups, never one merged list: +- *Saved* — saved NM profiles, MRU-first, rendered instantly without a scan. +- *Available now* — scan-backed in-range SSIDs with signal + security; may carry a + loading/stale hint; unsaved networks appear here. +- *Wired* — ethernet when a wired device is present. +=net list= already yields this (=connections.py= lists saved MRU-first, merges live +signal/security for in-range saved profiles, then appends unsaved in-range SSIDs with +=uuid: nil=); the panel groups and labels it. *Rescan refreshes only the +availability/signal layer* — it never gates or reloads the Saved list. + +*Progressive loading:* render the Saved group immediately on open, then overlay +availability, signal, and the unsaved Available-now rows when the scan returns. Show a +small scan-in-progress state (elapsed + last-scan age). A slow or bad radio scan must +not make the whole panel feel stuck — this is the direct answer to "why does it take +so long to see my connections?" + +*Join-from-row (no Add page):* selecting an unsaved Available-now row *is* the join +flow — SSID and security come prefilled from the scan, never retyped. Open networks +connect (confirm only if needed); WPA/WPA2/WPA3-Personal ask only for the password. +The standalone Add button + modal are deleted for visible networks. A hidden/manual +SSID join lives behind an Advanced "Join hidden network" affordance. + +*** Supported authentication classes (the join matrix) +From the scanned NM =SECURITY= value, V2 handles: +- *Inline-supported* — open, open-with-captive-portal, WPA/WPA2/WPA3-Personal + (PSK/SAE), and WPA2/WPA3 transition mode. The row shows the security label so the + user knows why a password is or isn't asked. +- *Activate-only* — 802.1X / enterprise: connect if already saved, else "edit in + nmtui/nmcli" (no add form in v1/V2, per decision 9). +- *Hidden / manual* — behind the Advanced "Join hidden network" affordance. +- *Rare / unsupported* — WEP, OWE/enhanced-open, MAC-registration, voucher, or + proxy-required: a clear in-panel explanation ("not supported here yet") plus a + non-terminal next step, never a hand-off to a terminal tool. + +*** Diagnostics owns the diagnostic story +Diagnostics holds the read-only checks, the repair stream, Get Me Online, debug +capture (Advanced), and the doctor report. A *lightweight* latency/throughput probe +runs inline as a Diagnose evidence row when internet is available (skipped offline, on +a metered/hotspot warning, or with no backend), and its result is stored in the doctor +report. The *full* speed test stays under Performance (decision 19) — which is also +the home for future throughput history, so Performance earns its tab rather than being +a lone button. + +*** Forget confirmation — future tense + verified +The destructive copy is future tense and names the scope: "This will remove the saved +NetworkManager profile and its stored password from this machine." After the op, +verify the UUID is gone, refresh the Saved list, and report "Forgot <SSID>" or "Could +not forget <SSID>; nothing changed / partial <evidence>" — the verify-every-action +decision applied to a destructive op. + +*** Findable diagnostics report +Every diagnose, repair, and speed/performance run ends with a "Copy report" / "Open +report" action in Diagnostics. The report carries the step statuses + elapsed, the +final classification, the last speed/latency result when available, scan age, +route/interface owner, the redacted event-log tail, and the bundle path when verbose +capture ran. It states explicitly whether any repair mutated state and whether +cleanup/verification passed. "Logs exist somewhere" isn't enough when the network is +already down — the report is the one artifact the user copies to hand over. + +*** Visual contract — a Waybar-attached popup +The panel reads as part of the bar, not a separate app. Match the live Waybar theme: +the dark rounded capsule (=border-radius: 1rem=), the golden border, compact monospace +text, and the =custom/net= state colors. Avoid square corners next to rounded UI, keep +cards out of cards, and use compact icon+label controls with tooltips for the advanced +repairs. Reuse any existing archsetup-owned GTK/panel conventions. (Non-blocking for +engine work; blocks final V2 UX acceptance.) + +** Panel state, cancellation, permissions +State machines for: connection-list loading, rescan-in-progress, +activation-in-progress, diagnose-running, repair-running, speedtest-running. Plus +the real terminal states on this two-machine fleet: no-wifi-hardware (desktop → +ethernet-only view) and missing speedtest-go. (No GPG-key state — there's no +credential store; secrets live in NM.) ("No NetworkManager" is not a modeled +state — NM is always present +on these machines; if nmcli is somehow absent the panel shows a single hard-error +and exits.) Long operations show elapsed time and are cancellable where the +underlying op allows (rescan, speedtest, probe); clearly non-cancellable ones +(an in-flight activation) show elapsed + a disabled control. Permission-denied +(sudo/polkit declined) is a first-class outcome with the "nothing changed" +message, never a silent failure. + +Interaction-pattern catalog (=~/code/rulesets/patterns/=) principles that apply: +- transient-state-buttons — all the network levers in one place, reachable by + one chord (the bar click), state visible. +- default-most-common-friction-proportional — connections MRU-ordered so the + common pick is first; destructive ops (remove) and privacy-changing ones + (reset, override) get a confirm, switching does not. +- one-prompt-picker-typed-prefix — if the connection picker ever goes + keyboard-driven, kind (wifi/eth/saved/in-range) + name in one typed picker. + +** Panel UX flow (settle before Phase 2) +The concrete interaction defaults, so the GTK build isn't inventing them: +- *Default focus*: the Connections section, current connection's row selected. If + the indicator opened the panel because of a captive/no-internet state, focus + Diagnose instead with the relevant action highlighted. +- *Row content*: glyph (signal bars / wired / active check) + name + a secondary + line (security type, "active"/last-used). The active row is visually pinned at + top of its group. +- *Buttons*: one *primary* per section (Connections: Connect to the selected row; + Diagnose: Run diagnose; Repair: "Get me online"; Speed test: Run). Secondary + actions (add / edit / remove / rescan; individual repair tiers) are smaller and + grouped. +- *Disabled rules*: Connect disabled on the already-active row; Repair tiers + disabled while one runs; Speed test disabled while running; add/edit disabled + for enterprise (with the "edit in nmtui/nmcli" hint). +- *Confirmations* (exact wording): Reset → "Reset <SSID>? This drops the + connection and reconnects with a new MAC."; Bounce → "Restart networking? All + links drop briefly."; DNS override → "Temporarily set DNS to 1.1.1.1 for the + test? It reverts automatically."; Remove → "Forget <SSID>? The saved password is + deleted." +- *"Get me online" reporting*: shows each escalation step live (Unblock rfkill → + Reset → Bounce → Portal) with per-step pass/fail and stops at the first that + restores internet or at a terminal state, naming the next action. +- *After close*: the bar reflects the new state immediately (signal/refresh on + next poll); a running speedtest/diagnose keeps running and notifies on finish + (panel close doesn't cancel it). +- *Keyboard*: Esc closes (wired); arrows move row focus and Enter activates a + row (GTK ListBox defaults — row-activate connects, never disconnects); Tab is + the plain GTK focus chain, widget by widget (inside a list it crawls row by + row — no section jumps); there is NO type-to-filter. Verified live via + targeted-key AT-SPI probe 2026-07-02; the earlier tab-between-sections and + type-to-filter claims were aspirational and are struck. If section-jump Tab + or filtering is ever wanted, it's a new task, not an existing behavior. + +* Connection management (nmcli) + +- Every op via nmcli per the nmcli contract above (terse, escaped, UUID-keyed, + bounded =--wait=). +- MRU ordering from NM's =connection.timestamp= (last activated), descending. +- Ethernet appears in the list whenever a wired device is present, selectable at + any time; switching just brings the chosen connection up. +- *Mutation safety + rollback*: switching keeps the current connection up until + the new one activates successfully (=nmcli --wait 30=); on failure it does not + tear down the working link, surfaces the failure, and leaves the prior + connection active. =net down= notes that NM may auto-reactivate a profile and + reports the post-op active connection so the user isn't surprised. A switch that + needs a password it doesn't have prompts (or fails with "password required"), + never silently strands. The exact NM command sequence (preflight active-state + read → activate target → verify default route → on failure, confirm prior + still up) is pinned in the engine and tested against fake nmcli. +- *Add/edit scope*: open + WPA-PSK only in v1. Existing saved profiles of any + type (including enterprise) can be *activated*; editing an enterprise profile + shows "edit via nmtui/nmcli" rather than a broken partial form. + +* Connection secrets (no separate store) + +Per Craig's call: don't build a parallel credential store. Settings and secrets +live where NetworkManager already keeps them, so there's one source of truth and +no extra dependency (no GPG, no gpg-agent, no =~/.config/net/connections=). + +- *Where secrets live*: =/etc/NetworkManager/system-connections/<name>.nmconnection=, + root-owned =0600=, with the PSK/EAP secret stored inline (the default + =secret-flags=0= "owned by NM"). That's already secure-at-rest (root-only) and + is what =nmcli= reads/writes. +- *How we touch them*: every add/edit/remove goes through =nmcli= (=connection add + / modify / delete=), which writes the =.nmconnection= with the right ownership + and perms. We never read or write =system-connections= files directly (root) and + never copy a secret out of them. +- *No export / import / sync* — there's nothing to sync. A new machine gets its + connections the way it always has (the user joins, or restores NM profiles), + not from a tool-specific vault. +- *config file*: =~/.config/net/config= still exists, but only for non-secret + preferences (speedtest server, redaction flags, probe TTL). It holds no + credentials. +- *No secret leakage*: PSK/EAP never appear in =net=' =--json= output, the event + log, or error text (tested) — even though NM is the store, our surfaces must not + echo a secret =nmcli= happens to return. + +* Speed test + +- Backend: *=speedtest-go=* (=--json=, =--server=, =--no-download/--no-upload=), + already installed on velox (AUR =speedtest-go-bin=). No new dependency for v1. + librespeed-cli is the documented fallback for a self-hosted LibreSpeed server. +- =net speedtest --json= parses speedtest-go's JSON into the =speedtest= schema. +- *Server policy*: auto-select nearest by default; allow a pinned server id in + =~/.config/net/config=. +- *Timeout + cancellation*: a hard run timeout (e.g. 60s); the panel run is + cancellable (kills the child). Offline / rate-limited / no-server errors map to + the failure-message table. +- *Tests*: fixture JSON (success) and fixture stderr (offline, no server, + malformed output) drive =net speedtest= parsing without touching the network. + +* Help + documentation + +In-app help has three layers, each reachable in the situation it's needed: + +- *CLI help (works from a dead-GUI TTY)*: =net --help= lists the subcommands in + one screen; =net <cmd> --help= documents each (flags, what it mutates, the + console-recovery targets). The Makefile targets are self-describing (=make help= + lists =online= / =net-doctor= / etc. with one-line descriptions). This is the + layer that matters most when you're at a console with no network. +- *Panel help (in the GUI)*: a small =?= affordance in the panel header opens an + inline help pane — what each section does, which Repair actions mutate state, + what the indicator glyphs/colors mean. Per-control tooltips on the less-obvious + buttons (rfkill, bounce, DNS override). No external help browser. +- *User guide (the durable doc)*: a README / docs page covering every command, + the indicator states + glyphs, the panel sections, the config file keys, the + recovery make targets, troubleshooting (the failure-message table), and + rollback. Written so a future session — or Craig six months out — can operate + and recover the module from the doc alone. + +The failure-message table above is the single source of truth for the +troubleshooting text; the guide and the panel help both render from it rather +than restating it. + +* Enhancement radar + +Low-cost adjacent affordances, each dispositioned so cheap wins aren't lost and +the v1 panel stays focused. (Several are already in v1 by virtue of other +sections; marked here so the consideration is visible.) + +| Enhancement | Disposition | Reason | +|-------------------------------------+-------------+--------------------------------------------------------| +| Open / copy portal URL | v1 | already in the captive flow; trivial Open + Copy | +|-------------------------------------+-------------+--------------------------------------------------------| +| Forget network | v1 | it's the remove op, already specced | +|-------------------------------------+-------------+--------------------------------------------------------| +| Rescan now | v1 | already a Connections control | +|-------------------------------------+-------------+--------------------------------------------------------| +| Retry with hardware MAC | v1 | captive already has --hardware-mac; expose in Repair | +|-------------------------------------+-------------+--------------------------------------------------------| +| Pin speedtest server | v1 | already a config key | +|-------------------------------------+-------------+--------------------------------------------------------| +| Copy redacted doctor report | v1 | cheap, serves the observability/support goal | +|-------------------------------------+-------------+--------------------------------------------------------| +| Show last good network / result | vNext | needs small history persistence | +|-------------------------------------+-------------+--------------------------------------------------------| +| Watch mode for net doctor | vNext | a --watch loop; handy at a TTY, not v1-critical | +|-------------------------------------+-------------+--------------------------------------------------------| +| Actionable desktop notifications | vNext | dunst supports actions; extra wiring | +|-------------------------------------+-------------+--------------------------------------------------------| +| Keyboard connection picker (fuzzel) | vNext | the typed-prefix pattern; panel covers v1 | +|-------------------------------------+-------------+--------------------------------------------------------| +| QR-code share / import WiFi | rejected | low value for a personal 2-machine setup; phones do QR | +|-------------------------------------+-------------+--------------------------------------------------------| + +* Waybar wiring + +- Replace =custom/netspeed= with =custom/net= in the bar's module list (same + slot). +- Module def: =exec: waybar-net=, =return-type: json=, =interval: 2=, a =signal= + for on-demand refresh (next free signal after wtimer's 14), =on-click=, + =on-click-right=, =on-click-middle= per the phase-aware interactions (each + dispatches a detached job, never blocks). +- Remove the old =on-click: pypr toggle network= scratchpad only once the panel + replaces it (Phase 2); Phase 1 keeps it as the interim manager. + +* Testing plan (TDD) + +- *Engine (normal)* — fake =nmcli= + =curl= + =speedtest-go= on PATH; assert + command sequences and parsed/emitted JSON for status, list, up/down, + add/edit/remove, probe, diagnose, repair, speedtest. Pure state/format + functions tested directly. JSON schemas locked by example. +- *Portal parser* — already covered in =tests/captive= (Normal/Boundary/Error + + the real SONIFI body). The engine's native probe reuses the same cases. +- *nmcli parsing* — escaped colon/backslash/newline in SSID, duplicate names, + hidden SSID, non-ASCII, wired-mid-session, multi-active (wifi+tether). +- *Failure + concurrency (the risky classes)* — slow/hung nmcli/curl/speedtest + (degraded state within budget), concurrent =net status= probe refresh + (single-flight), corrupt cache (recovered), stale cache after SSID change + (invalidated), permission denied / sudo declined, DNS-override cleanup failure + (=cleanup-unverified=), NM partial activation (rollback keeps prior link), + secret redaction, missing speedtest-go, no wifi hardware, rfkill soft/hard + block. +- *Doctor classification* — fixture-driven =net doctor= over fake nmcli/curl + asserting the right terminal classification + that =--fix= stops before + destructive repairs: auth failures (=needs-user-action=), upstream/AP failure + (=upstream-not-local=), VPN-routed failure (=deferred/vpn=), and the DNS classes + (hijack → portal, broken-but-1.1.1.1-works → offered override, egress-blocked → + upstream). Assert the failure-mode coverage table's "detects / repairs / terminal + action" holds for each row. +- *Indicator* — drive =net status --json= through =waybar-net=, assert the JSON + per state (online / captive / no-internet / degraded / wired / disconnected / + rfkill), iface override via env. +- *Panel* — pocketbook-style: backing logic (list ordering, op dispatch, + state-machine transitions), not GTK widgets. +- *NM secrets / no-leak* — add/edit writes the secret into NM via nmcli (asserted + against fake nmcli, never to a tool-owned file); assert no PSK/EAP appears in any + =--json=, log line, or error (there is no credential store to round-trip). +- *Live checklist (gated out of the suite)* — a "Manual testing and validation" + task per phase for the real-network states (captive at a hotel, no-internet, + switch under load, reset, speedtest) that can't be faked. + +** Harness + coverage gate +The concrete contract, matching the repo's existing convention (not pytest — the +dotfiles suites are =unittest=, run by =make test= as =python3 -m unittest= over +=tests/*/test_*.py=; 33 suites today): +- *Framework*: =unittest=. Each suite is =tests/<name>/test_<name>.py= + (=tests/net/=, =tests/waybar-net/=), collected by the existing =make test= loop + — no new runner, no pytest dependency. +- *Fakes on a temp PATH*: =fake-nmcli=, =fake-curl=, =fake-speedtest-go=, + =fake-rfkill=, =fake-resolvectl= live as executable stubs in =tests/<name>/= + (the =tests/layout-navigate/fake-hyprctl= pattern). A fixture file encodes the + command→canned-output map and the stub appends each invocation to a log the test + asserts against. Subprocess timeouts are simulated by a stub that sleeps past the + budget; =net status= must still return the degraded state. +- *Waybar wrappers end-to-end*: =waybar-net= is run as a subprocess with the fake + PATH and the env overrides (iface, cache path), asserting the emitted JSON — same + as =tests/waybar-netspeed=. +- *Coverage*: coverage.py is absent system-wide (and not importable), so coverage + runs in a throwaway venv (=python3 -m venv=, =pip install coverage=, =coverage + run -m unittest=, =coverage report=) — the method the wtimer suite used (95%). + Target: *branch* coverage over =net/= and the wrapper, ≥ 90% on the pure + classifier/parser modules. + +** Coverage as a gap-finder, not a number (per phase) +Line coverage alone misses the branches that matter here, so each phase ends with +a *coverage-gap pass*, not just a percentage: +- After the first green run, read the branch report and map every uncovered branch + to either a new test or a consciously-excluded live-only behavior (with a comment + or a Manual-testing entry naming it). +- *Branch coverage is required* for the pure logic: the doctor classifier (every + outcome — fixable / needs-user-action / upstream-not-local / deferred-vpn), the + cleanup-unverified path, the redaction paths, the degraded hot-path fallback, the + timeout branches, and the portal/nmcli parsers. +- A phase isn't "done" until its coverage-gap pass is recorded — uncovered logic is + either tested or explicitly excused, never silently uncovered. + +* Files touched (planned, all in =~/.dotfiles=) + +- =net/= package (src-layout, like pocketbook) — engine + panel. +- =hyprland/.local/bin/waybar-net= — the indicator (replaces =waybar-netspeed=). +- =hyprland/.local/bin/net= — engine CLI entry (console-script shim). +- =hyprland/.config/waybar/config= — swap =custom/netspeed= → =custom/net=; + remove =custom/airplane=. +- =hyprland/.config/waybar/style.css= — captive / no-internet / degraded / + rfkill classes; remove airplane classes. +- =tests/net/=, =tests/waybar-net/= — suites. +- =captive= — refactor: extract probe + reset into functions callable + non-interactively (a =--json= probe mode) so the engine reuses them. +- =~/.config/net/config= — seed config (probe TTL, speedtest server, redaction + flags). No secrets; not a credential store. +- dotfiles =Makefile= — add the console-recovery targets (=online=, =net-doctor=, + =net-status=, =net-diagnose=, =net-portal=, =net-reset=, =net-bounce=). +- *Deletions once net ships* (the airplane module is absorbed): + =hyprland/.local/bin/waybar-airplane=, =hyprland/.local/bin/airplane-mode=, + =tests/waybar-airplane/=, =tests/airplane-mode/=, and the =custom/airplane= + module + its css. +- archsetup Hyprland step — add =gtk4-layer-shell=, =python-gobject=, + =speedtest-go-bin= to the install lists (the only archsetup change; no =gpg= + added, secrets stay in NM's store). + +* Resolved decisions (Craig's calls + this response) + +1. Panel UI tech → GTK4 + gtk4-layer-shell, shared pocketbook scaffold (one + panel shell, reused by the desktop-settings sibling). +2. Engine language → Python =net= package; shells out to =captive= for the + portal-force flow, native cheap probe for the bar path. +3. Connectivity probe → split cadence (fast link poll every 2s + slow cached + internet/captive probe, TTL ~45s) with single-flight + atomic cache. +4. No keyboard-modifier clicks (waybar can't qualify them) — the panel hosts the + rich actions; bar clicks dispatch detached jobs (phase-aware). +5. No separate credential store (Craig's call, cj). Secrets live in NM's own + =system-connections= (root =0600=, inline), touched via nmcli. No GPG, no + gpg-agent, no =~/.config/net/connections=. Supersedes the earlier GPG-store + design. +6. =custom/netspeed= absorbed into =custom/net=; throughput moves to the tooltip. +7. Speed-test backend → =speedtest-go= (already installed), not a new + librespeed-cli dependency; librespeed-cli is the self-hosted fallback. +8. Code lives in the dotfiles repo; archsetup only installs deps. +9. v1 add/edit scope = open + WPA-PSK; enterprise/802.1X is activate-only, + add/edit is vNext (settled by Craig 2026-06-29 — no enterprise networks in his + history, so the form would be unused UI). +10. =net doctor= is in v1 (Craig's call, cj) — a one-shot diagnose+fix mode, + reachable from a TTY via =make online= / =make net-doctor=. (The earlier + "defer the doctor/bundle command" decision is reversed.) +11. Diagnose (read-only) and Repair (mutating, confirmed) are separated in the + panel and the CLI; Repair is tiered lightest-first (rfkill → reset → bounce). +12. =custom/net= absorbs the airplane module (Craig's call, cj). *As built + (2026-06-29, option 1): display-only.* net shows the airplane state (reads + the airplane-mode state file); the =airplane-mode= low-power toggle is kept + (radios + CPU + brightness + services is not a network concern) and moved to + =custom/net='s right-click + signal 15. Only the redundant display pieces — + =waybar-airplane=, =custom/airplane=, and the retired =waybar-netspeed= — + plus their tests/css were deleted. The earlier "delete airplane-mode" framing + is superseded. +13. Repair includes a full-stack bounce and an rfkill-unblock (Craig's calls, + cj) — the latter recovers the framework-laptop post-power-loss soft-block. +14. VPN / WireGuard is a planned Phase 5 (Craig's call, cj), not a permanent + exclusion. + +V2 redesign decisions (Craig, 2026-06-30): + +15. *No terminals anywhere in the module* — =net-popup= is removed; every action and + result renders in the panel. No terminal is ever used to report information to the + user or to collect input from them: every prompt, confirmation, repair stream, and + result lives in the panel UI (Craig, cj, 2026-06-30). Reverses the part of decision + 11 that ran privileged repairs in a terminal "so sudo/polkit can prompt". (Unrelated + to the doctor's "terminal states" — that word means a final outcome, not a tty. The + one open question is the dead-GUI console-recovery path; see the VERIFY in todo.org.) +16. *Passwordless privileged path* — a root-owned helper + a narrow NOPASSWD sudoers + rule scoped to it, archsetup-installed, run as =sudo <helper> <verb>=. This gates + decision 15 (a worker thread can't prompt). Absorbs the earlier DoT-toggle + follow-up and fixes the detached-restore-watcher bug. +17. *Verify every action* — each mutating op (repair, connect, forget, add, DNS + override) re-checks its effect and surfaces pass/fail in the panel. +18. *Detect + respond to every failure mode, edges included* — the full ~44-mode + catalog (todo.org redesign task) is the contract; auto-fix where safe, else report + the exact in-panel text. Includes IPv6-only awareness and multi-homing, which need + diagnose to stop being IPv4-only and single-iface. +19. *Navigation* — top tabs Connections | Diagnostics | Performance; Diagnostics + merges Diagnose + Repair (Diagnose | Get Me Online | Advanced over a shared + streaming area); Speed test under Performance. +20. *Automatic diagnostic verbose-capture* (Craig, 2026-06-30) — on a failing + diagnose, elevate the underlying stack (NM / resolved / wpa_supplicant) to debug, + capture the journal + dmesg window, restore (guaranteed + crash-guarded), and + write a redacted bundle. Plus a manual Debug on/off toggle in Advanced. Restore + bulletproof, secrets scrubbed before the bundle, log-level toggles via the V2 + helper. See Observability. + +* Implementation phases + +*Phases 1-3 are SHIPPED* (2026-06-29 → 2026-06-30, dotfiles); their acceptance +criteria passed and the work is live on velox. Phase 4 (docs/rollout) and Phase 5 +(VPN) remain. The V2 redesign phases at the end are designed, not yet built. + +- *Phase 1 — Indicator + console recovery (task #C).* =net status= + =net probe= + (native cheap probe, reusing captive's logic) + the =captive= probe refactor + + =waybar-net= + the split-cadence cache (single-flight, atomic, stale classes) + + CSS states (incl. rfkill) + performance budget. Plus the CLI-only recovery path: + =net repair= tiers (rfkill / reset / bounce), =net doctor [--fix]=, and the + Makefile targets (=make online= etc.) — all testable without the GTK panel. + Absorbs the airplane state and removes the standalone airplane module. Interim + left-click keeps the existing scratchpad until the panel lands. + - *Acceptance*: fresh-login waybar smoke test shows correct state on + online/captive/no-internet/wired/rfkill; =net status= stays within budget + under a fake slow nmcli (degraded state); =net doctor --fix= recovers a + soft-blocked radio from a TTY; the live captive checklist passes at a real + portal; the airplane state works and the old airplane module is gone; + reverting = swap =custom/netspeed= + =custom/airplane= back. +- *Phase 2 — Panel shell + connection management (task #B core).* GTK4 + layer-shell scaffold + =net list/up/down/add/edit/remove/rescan= + MRU list + + mutation safety/rollback + panel state machines. + - *Acceptance*: switch wifi↔wifi and ethernet↔wifi without stranding; a failed + switch leaves the prior link up; add/edit open + WPA-PSK writes the secret to + NM; remove confirms; panel states render for loading/rescan/activation. +- *Phase 3 — Diagnostics + speed test in the panel.* Wire =net diagnose= / + =net repair= / =net doctor= / =net portal= / =net speedtest= into the Diagnose + vs Repair sections; the "Get me online" button; portal Open button; speedtest + progress + cancel. + - *Acceptance*: diagnose runs read-only; each repair tier confirms + verifies + cleanup (DNS override reverts, shown); speedtest result parses from + speedtest-go and a fixture-driven failure shows the right message. +- *Phase 4 — Docs + rollout.* In-app help (=net --help= / per-command help, the + panel help affordance), README/user-guide (commands, panel, config, + troubleshooting, the make targets, rollback), and the manual dep step on ratio. + - *Acceptance*: =net --help= and each subcommand's help are complete; the + user-guide covers every command + the recovery targets; ratio rollout + documented. +- *Phase 5 — VPN / WireGuard (future).* Fold the existing archsetup wireguard + tooling into the same panel + CLI (=net vpn ...=). Out of the v1 milestone; + specced separately when picked up. + +V2 redesign phases (designed 2026-06-30, dependency order): +- *V2.1 — Sudo helper + NOPASSWD sudoers (gates everything).* Root-owned helper + dispatching net's fixed privileged verbs, archsetup-installed, narrow sudoers. + Also fixes the detached DoT-restore-watcher bug. + - *Acceptance*: every repair runs passwordless in-panel on a non-NOPASSWD machine; + the sudoers rule is scoped to the helper only. +- *V2.2 — Merged Diagnostics panel + nav restructure.* Connections | Diagnostics | + Performance; the Diagnostics sub-row + shared streaming area; Advanced reveal + + tooltips; delete =net-popup=. + - *Acceptance*: no terminal opens for any action; repair progress streams in the + panel; Speed test lives under Performance. +- *V2.3 — IPv6-aware and multi-homing-aware diagnose.* Stop treating no-IPv4 as a + failure when online over IPv6; identify which interface owns the default route. +- *V2.4 — Close every detect/correct gap in the catalog, with post-action + verification.* Work the redesign-task catalog mode by mode. +- *V2.5 — Automatic diagnostic verbose-capture.* Snapshot/elevate/capture/restore + around a failing diagnose + the Advanced Debug on/off toggle; guaranteed + + crash-guarded restore; redacted support bundle; helper log-level verbs. + - *Acceptance*: a failing diagnose leaves a redacted bundle (NM/resolved/ + wpa_supplicant journal + dmesg) and restores every log level; a crash mid-capture + is detected and restored on the next run; the secret-leak test finds no PSK/EAP in + a bundle; the toggle elevates and restores on demand. + +* Open items / risks + +- gtk4-layer-shell dropdown anchoring under a waybar module needs the same + positioning work pocketbook solved; reuse it. (Phase 2.) +- The =captive= refactor must keep the standalone CLI behavior identical while + exposing a non-interactive =--json= probe; covered by the existing + =tests/captive= suite plus new probe-mode tests. (Phase 1.) +- speedtest-go server selection variance (nearest-server flor) — pin a server in + config if results are noisy. (Phase 3.) +- The background-probe kick from =net status= must be truly non-blocking (spawn + + detach); enforced by the single-flight lock and the performance benchmark test. + +* Rollback + +Each phase is independent. The indicator (Phase 1) is a drop-in replacement for +=custom/netspeed= (and =custom/airplane=); reverting is swapping those modules +back in the config and restoring their scripts. The panel is additive — not +wiring its clicks leaves the bar working as before. No credential store to roll +back (secrets stay in NM throughout). + +* Review findings [40/40] + +** DONE Define the structured diagnostics contract :blocking: +The spec says the engine "emits JSON" and that diagnostics "reuse =captive= +verbatim", but the current =~/.dotfiles/common/.local/bin/captive= flow is a +human-readable bash script that mixes diagnostics, sudo prompts, DNS mutation, +browser launch, and terminal prose. A GTK panel cannot reliably turn that into +clear state, progress, cancellation, or useful error messages. Define the +machine contract before implementation: every diagnostic step should have a +stable id, status (=pending/running/pass/warn/fail/skipped=), redacted evidence, +elapsed time, safety outcome, and next action. Keep =captive= as the interactive +CLI, but either refactor reusable probe/reset functions behind =net diagnose +--json= or make =captive= expose a non-interactive JSON mode. This blocks the +panel and logging work because otherwise the implementer must invent the +boundary. + +Disposition: accept — added the "Diagnostics contract" section (per-step id / +status / evidence / elapsed / safety / next_action) and the =captive= =--json= +probe-mode refactor under Architecture + Files touched. + +** DONE Specify user-facing failure messages and recovery actions :blocking: +The spec names failure states like =no-internet=, =captive=, failed probe, +failed reset, missing DNS, and missing speed-test backend, but it does not define +the messages the user sees or what each message tells them to do next. For this +feature, "error" is not enough: a user needs to know whether WiFi is associated, +whether DHCP succeeded, whether DNS is hijacked/broken, whether HTTP is +intercepted, whether sudo was declined, whether a command timed out, and whether +the system was left unchanged or partially changed. Add a message table for the +indicator, panel, and CLI with: failure class, visible text, evidence included, +redaction rule, and next action. This is blocking because UX quality here is the +product, not an implementation detail. + +Disposition: accept — added the "Failure states, messages, recovery" section +covering each class, the visible message, the "what changed" residue note, and +the next action across indicator/panel/CLI. + +** DONE Define the debug log and redacted support bundle :blocking: +There is no observability section. When this fails in a hotel or cafe, an agent +needs enough evidence to diagnose it without rerunning destructive actions. Add +log location, rotation/retention, JSONL event schema, command argv logging, +exit-code/stderr capture, elapsed time, selected iface, NM active connection +UUID, probe URL class, HTTP code, redirect host, DNS servers, and cache +read/write events. Also define a =net doctor --json= or =net debug-bundle= +command that emits redacted status, recent log events, dependency versions, and +a reproduction command. Redact SSID if configured, MAC addresses, portal query +tokens, PSKs, EAP identities/passwords, IPs when requested, and all GPG/NM +secrets. This blocks implementation readiness because post-failure diagnosis is +currently left to ad hoc terminal spelunking. + +Disposition: modify — accepted the JSONL event log, the schema, and the redaction +rules in full (new "Observability" section). Deferred the dedicated =net +debug-bundle= / =net doctor= command to vNext: for a single-user tool =net +diagnose --json= (the snapshot) plus the event log (the history) cover +post-failure diagnosis; a bundle command is gold-plating for v1. Recorded under +Out + Resolved decision 10. + +** DONE Pin the nmcli parsing and timeout contract :blocking: +The spec lists nmcli operations but not the exact fields, output modes, escaping +rules, ID semantics, or timeouts. This is risky because SSIDs and connection +names can contain spaces, colons, duplicates, hidden names, and non-ASCII; the +current =waybar-netspeed= already had an SSID parsing bug. The nmcli manual +documents =--terse=, =--get-values=, =--escape=, =--wait=, ID/UUID/path +selection, =passwd-file=, and built-in connectivity states +(=none/portal/limited/full/unknown=) at +https://man.archlinux.org/man/nmcli.1.en. The spec should require UUIDs for +saved-profile operations, explicit =--wait= budgets, parser tests for escaped +colons/backslashes/newlines/duplicate names/hidden SSIDs, and a decision on when +to use or ignore =nmcli networking connectivity [check]=. This is blocking +because the command wrapper is the core reliability boundary. + +Disposition: accept — added the "nmcli contract" section: terse + =--escape= + +=--get-values=, UUID-keyed ops, explicit =--wait= budgets, NM connectivity as a +cheap hint (our probe authoritative), and the parser test matrix. + +** DONE Define cache concurrency, atomicity, and stale-state behavior :blocking: +=net status= may spawn =net probe= whenever the cache is stale, but the spec +does not define locking, process coalescing, atomic writes, crash cleanup, or +what happens when the probe hangs. With a 2s Waybar interval, a bad network could +start overlapping probes, corrupt the runtime cache, or keep showing stale +"online" while the link is gone. Add a single-flight lock under +=$XDG_RUNTIME_DIR/waybar=, atomic write+rename for cache updates, max probe +runtime, stale age classes (fresh/stale/expired/unknown), cache invalidation on +iface/SSID/connection UUID change, and tests for concurrent =net status= calls. +This blocks the fast-path design because it is the main performance and +correctness risk. + +Disposition: accept — added "Concurrency, atomicity, staleness" under the +Connectivity model: flock single-flight, temp+rename atomic write, ≤6s probe +timeout, fresh/stale/expired/unknown classes, iface/SSID/UUID invalidation, stale +lock reclaim, plus concurrency tests in the test plan. + +** DONE Bound hot-path performance with measured budgets :blocking: +The spec says the cheap poll should be sub-100ms, but the proposed fast path +still may call multiple =nmcli= commands every two seconds, read sysfs, parse +throughput, and maybe spawn a background probe. The existing =waybar-netspeed= +had a deliberate sleep for throughput sampling; replacing it must define how +throughput is sampled without sleeping in the bar path. Add a per-command budget +for =waybar-net= and =net status=, a maximum number of subprocesses on the hot +path, a timeout for every subprocess, benchmark tests with fake slow =nmcli=, +and a rule that the indicator emits a degraded JSON state rather than blocking. +This is blocking because Waybar custom modules can visibly freeze or lag when +their exec path stalls. + +Disposition: accept — added the "Performance budgets" section: <100ms typical / +<250ms worst, throughput sampled across the poll interval (no in-process sleep), +one nmcli call max on the hot path, timeouts on every subprocess, the degraded +state, and a fake-slow-nmcli benchmark test. + +** DONE Make click actions non-blocking and visible :blocking: +Waybar right-click runs =net reset= and middle-click runs =net portal= directly. +Those operations can require sudo, open browsers, mutate DNS, delete/recreate NM +profiles, or hang on network commands, but Waybar click handlers provide no +panel, terminal, progress, or cancellation surface by default. Define whether +right/middle click instead opens the panel focused on the action, dispatches a +background job with notifications, or is removed from v1. If kept, specify +single-flight behavior, how sudo/polkit prompts surface, how success/failure is +reported, and how the user can inspect logs. This blocks UX readiness because +the fastest remediation path is currently the easiest place to hide failure. + +Disposition: modify — accepted the concern; made the interactions phase-aware and +non-blocking. Every click dispatches a detached, single-flight background job and +reports via =notify=; sudo surfaces through polkit/the normal prompt; failures go +to the notify + the event log. In Phase 1 (no panel) left-click runs probe + +notify and keeps the scratchpad; from Phase 2 left-click opens the panel focused +on the action. Recorded in the Indicator "Interactions" subsection. + +** DONE Specify connection mutation safety and rollback :blocking: +The spec says row click switches connections and remove gets a confirm, but it +does not define what happens when a switch partially succeeds, disconnects the +current working link, needs a password, loses the default route, or triggers +auto-activation. The nmcli manual warns that =connection down= does not prevent +future auto-activation and may internally block a profile until user action. +Define preflight, the exact NM command sequence, whether the old active +connection is kept until the new one proves usable, when rollback is attempted, +how long activation waits, and what the panel says when rollback fails. This is +blocking because the module can strand the user offline. + +Disposition: accept — added "Mutation safety + rollback" under Connection +management: keep the prior link up until the target activates (=--wait 30=), no +teardown on failure, password-required surfaced not stranded, =net down= reports +post-op active state + the auto-reactivation caveat, and the pinned NM command +sequence is tested against fake nmcli. + +** DONE Define the credential-store security model :blocking: +The GPG store is described as optional and default-unencrypted, but the spec does +not define file modes, schema, secret-source rules, import/export prompts, +recipient verification, stale secret handling, or what is logged. It also says +NM remains source of truth while the user-owned store contains PSK/EAP secrets, +which creates two truth sources for sensitive data. Add a precise schema, +=0600= file creation with parent-dir permissions, encrypted-recipient checks, +plaintext warning text, explicit opt-in flow, redaction requirements, behavior +when NM has a secret not in the store, behavior when the store has a secret NM +rejects, and tests for no secret leakage in JSON/logs/errors. This blocks Phase +4 and the full spec because otherwise the implementer must make security +decisions mid-code. + +Disposition: accept — rewrote "Credential storage" with the versioned schema, +=0600= file / =0700= dir, recipient verification on opt-in, the plaintext +warning, secret-source rule (entered/exported, never harvested from root store), +the two-source reconciliation policy (NM wins live, store wins for what NM +lacks, stale-secret flagging), and the no-leak tests. + +** DONE Define EAP, enterprise WiFi, and unsupported connection behavior :blocking: +The store says "PSK/EAP" and connection management says add/edit, but there is +no v1 contract for WPA-Enterprise fields, certificates, identity vs anonymous +identity, hidden networks, static IP, proxy settings, metered flags, MAC +randomization, or 802.1X prompt behavior. Either scope v1 to open/WPA-PSK plus +existing saved-profile activation, or define the minimum EAP form and the +unsupported-state messages. This blocks add/edit/import because enterprise WiFi +is too sensitive to hand-wave. + +Disposition: modify (scope) — scoped v1 to open + WPA-PSK add/edit, with +*activation* of any existing saved profile (including enterprise). Enterprise / +802.1X add/edit, static-IP, proxy, metered, and MAC-randomization editing are +vNext, shown as "edit via nmtui/nmcli". Recorded in Scope/Out, Connection +management, and Resolved decision 9. + +** DONE Split read-only diagnostics from mutating remediation :blocking: +The panel's diagnostics section includes probe, bounce/reset, gateway ping, and +DNS override test in one area, while =captive= currently performs resets and +temporary DNS changes as part of its flow. Users need to know which buttons are +read-only and which mutate NM profiles, MAC mode, DNS, or browser state. Add +separate "Diagnose" and "Repair" actions, confirmations for destructive or +privacy-changing operations, explicit cleanup verification for DNS override, and +a terminal state when cleanup is unverified. This blocks readiness because +network repair must not surprise the user or leave hidden residue. + +Disposition: accept — split the panel into a read-only Diagnose section and a +confirmed, mutating Repair section (and split the CLI into =net diagnose= vs =net +repair=). Added =cleanup_verified= + a terminal =cleanup-unverified= state to the +diagnostics contract. + +** DONE Define panel state, cancellation, and permissions UX :blocking: +The panel sections list buttons and a streaming output area, but not loading +states, disabled states, empty states, keyboard/focus behavior, cancellation, or +permission-denied handling. Add panel state machines for connection list loading, +rescan in progress, activation in progress, diagnostics running, speedtest +running, and no NetworkManager/no WiFi/no permissions/no GPG key/no +librespeed-cli. Each long operation should be cancellable where possible or +clearly non-cancellable with an elapsed-time display. This blocks the GTK work +because without it the implementer must invent the user flow. + +Disposition: modify — accepted the state-machine requirement (added "Panel state, +cancellation, permissions"), but scoped the state set to what can actually occur +on the two-machine fleet: dropped "no NetworkManager" as a modeled state (NM is +always present; a missing nmcli is a single hard-error exit) and kept +no-wifi-hardware, missing speedtest-go, no-GPG-key, plus the in-progress states +with elapsed-time + cancellation where the op allows. + +** DONE Verify speed-test dependency, server choice, and failure contract :blocking: +The spec chooses =librespeed-cli= and notes availability/default-server research +as an open risk, but Phase 3 still depends on parsing its JSON and showing +progress. I checked the upstream project page +(https://github.com/librespeed/speedtest-cli) and the AUR URL named by search is +not sufficient as a verified package/install contract in this spec. Add the +exact package name/source to install, command version expected, JSON shape, +server-selection policy, timeout, cancellation behavior, offline/rate-limited +messages, and tests with fixture JSON and fixture stderr. This blocks Phase 3 +because speed-test failure modes are otherwise undefined. + +Disposition: modify — verified live and changed the backend: =speedtest-go= (AUR +=speedtest-go-bin=, 1.x) is already installed on velox and supports =--json=, +=--server=, =--no-download/--no-upload=, so v1 needs no new dependency. +librespeed-cli (AUR =librespeed-cli= / =-bin=) is the documented self-hosted +fallback. Added the "Speed test" section with server policy, timeout, +cancellation, the failure-message mapping, and fixture-JSON/stderr tests. + +** DONE Define dependency installation and repo boundaries :blocking: +The files touched section alternates between archsetup paths and the external +dotfiles repo, while pocketbook has been folded into this repo and its previous +archsetup provisioning was intentionally removed. The spec should state where +the =net= package actually lives, which repository owns the scripts/tests, +whether =gtk4-layer-shell=, =python-gobject=, =librespeed-cli=, =gpg=, =nmcli=, +=curl=, and =resolvectl= are installed by archsetup or assumed present, and the +Makefile targets for test/lint/install. This blocks implementation because the +current path plan can produce code that is not installed on a fresh machine. + +Disposition: accept — added the "Repository + dependencies" section: all code in +=~/.dotfiles= (=net/= package in-tree like pocketbook, scripts in the hyprland +tier, tests under =tests/=), archsetup owns only the dep install +(=gtk4-layer-shell=, =python-gobject=, =speedtest-go-bin=; nmcli/curl/resolvectl +already present), Makefile =make test= collects the package suite, and a +daily-drivers note for ratio. Rewrote Files touched to match. + +** DONE Expand the test plan for failure, concurrency, and live verification :blocking: +The testing plan covers normal parsing and fake command sequences, but it misses +the riskiest behaviors: slow/hung =nmcli=/=curl=/=librespeed=, concurrent +=net status= cache refresh, corrupt cache, stale cache after SSID change, +permission denied, sudo declined, DNS override cleanup failure, NM partial +activation, duplicate connection names, secret redaction, missing optional +dependencies, no WiFi hardware, wired+tether+WiFi ambiguity, portal redirect +tokens, and Waybar click handlers. Add unit/fixture tests for each class plus a +manual/live checklist gated out of the normal suite. This is blocking because +the current plan would leave the exact "things that can go wrong here" mostly +untested. + +Disposition: accept — rewrote the Testing plan with the "Failure + concurrency" +class (slow/hung commands, single-flight, corrupt/stale cache, perm-denied, +cleanup-failure, partial activation, redaction, missing deps, no-wifi, +multi-active) and a per-phase live checklist gated out of the suite. + +** DONE Define status JSON schemas and compatibility rules +The spec says all subcommands take =--json= but does not define schemas. Add +versioned JSON examples for =status=, =probe=, =list=, =diagnose=, =speedtest=, +and error envelopes, including nullable fields and unknown/degraded states. This +is non-blocking for product direction but should be fixed before code so tests +can lock the CLI contract. + +Disposition: accept — added the "JSON schemas" section with versioned (=v:1=) +envelopes for status / probe / list / diagnose / speedtest and a shared error +envelope, including the degraded/unknown states. + +** DONE Rename or alias the phasing section for workflow compatibility +The spec has a usable =Phasing= section, but the spec-review workflow expects an +=Implementation phases= section that can be lifted into =todo.org=. Rename it or +add an alias heading during response. This is non-blocking because the existing +phase decomposition is understandable, but aligning the heading prevents future +workflow friction. + +Disposition: accept — renamed =Phasing= → =Implementation phases= and added +per-phase acceptance criteria. + +** DONE Add documentation and rollout acceptance checks +Rollback is described, but docs and rollout are thin. Add README/user-guide +updates for commands, panel behavior, config file, GPG opt-in, troubleshooting, +and rollback; add acceptance checks for each phase, including a fresh-login +Waybar smoke test and restoring =custom/netspeed=. This is non-blocking but +important for handing the feature to a future session without re-discovery. + +Disposition: accept — added per-phase acceptance criteria under Implementation +phases (incl. the fresh-login waybar smoke test and the =custom/netspeed= +restore), a Phase 4 "Docs + rollout", and (answering Craig's cj follow-up) a +dedicated "Help + documentation" section with the three help layers (CLI help, +panel help affordance, user guide). + +** DONE Add a failure-mode coverage table :blocking: +The spec now names many individual network failures, but it still does not carry +one compact coverage matrix that says, for each common failure mode, whether +=net diagnose= detects it, whether =net doctor --fix= can repair it, and what +terminal user action remains when it cannot. Add a table covering at least: +rfkill soft block, rfkill hard block, no WiFi hardware, associated/no DHCP, +gateway unreachable, captive DNS hijack, broken DNS where 1.1.1.1 works, HTTP +portal, HTTP interception without a parseable portal URL, upstream/AP outage, +wrong WPA password or missing secret, enterprise auth/cert failure, duplicate +SSID/connection-name ambiguity, hidden SSID, multiple active links, wedged +NetworkManager, slow/hung command, stale/corrupt cache, DNS cleanup failure, +missing speedtest backend, and VPN/routing interference. This blocks because +Craig asked for confidence that the diagnostics and doctor cover the real field +failures, and prose scattered across sections is too easy to misread. + +Disposition: accept — added the "Failure-mode coverage" section: a 22-row table +(every mode the finding named) with detect / doctor-fix / terminal-action +columns, conformed to the org-table standard (rules under every row, ≤120). + +** DONE Pin DNS repair semantics in doctor :blocking: +The spec diagnoses DNS hijack, broken hotel DNS, and the temporary 1.1.1.1 +override test, but =net doctor --fix= does not say whether it merely recommends +the override, applies a temporary override during recovery, or leaves DNS alone +after diagnosis. Define the exact behavior for each DNS class: captive hijack +should open the portal, broken DNS where 1.1.1.1 works should either offer an +explicit temporary repair with cleanup verification or recommend the command, +and port-53/egress blocking should stop as upstream/not locally fixable. This is +blocking because DNS is one of the most common "connected but unusable" failures +and the current doctor contract is ambiguous. + +Disposition: accept — added "DNS handling in doctor (explicit per class)" under +the new Doctor section: hijack → open portal (no DNS mutation); broken-but-1.1.1.1 +→ explicit temporary override with cleanup verification under =--fix=, recommend +otherwise; egress-blocked → terminal =upstream-not-local=. + +** DONE Make auth failures terminal user-action states :blocking: +Wrong WPA password, missing NM secret, locked keyring/polkit denial, enterprise +802.1X certificate/identity failure, and portal login-required are not fixed by +resetting or bouncing NetworkManager. The doctor sequence should classify these +as =needs-user-action= terminal states, stop before looping through destructive +repairs, and tell the user the exact next action (enter password, edit profile in +=nmtui=/=nmcli=, accept portal terms, provide cert/identity, or retry with +admin auth). This blocks because repeated reset/bounce against auth failures is +slow, noisy, and can make the network state worse without helping. + +Disposition: accept — added the =needs-user-action= terminal outcome to the +Doctor section: wrong password / missing secret / keyring-or-polkit denial / +802.1X cert-or-identity failure / portal-login-required all stop the doctor before +any destructive repair and name the exact next step. + +** DONE Define upstream/AP/provider failure terminal states :blocking: +Some failures are not client-repairable: AP has no uplink, hotel gateway is +down, DHCP server is broken, gateway drops traffic, ISP outage, or captive +portal backend is failing. The spec should define how =diagnose= proves "local +link is up but upstream is broken" and how =doctor --fix= stops after local +repairs are exhausted with a clear message like "local repairs tried; likely +upstream/AP/provider" plus the evidence. This blocks because users need to know +when to stop poking the laptop and switch networks or contact the venue. + +Disposition: accept — added the =upstream-not-local= terminal outcome: diagnose +proves link-up + IP + gateway-reachable but no route out and no captive redirect; +=doctor --fix= stops after local repairs with "local repairs tried; likely +upstream/AP/provider" + evidence → switch network / contact venue. + +** DONE Decide how VPN and policy routing affect v1 diagnosis +VPN/WireGuard management is Phase 5, but active VPNs, policy routes, DNS +overrides, and firewall killswitches can break apparent internet access in v1. +The current spec does not say whether v1 detects active VPN/policy routing and +classifies "network is fine, VPN route/DNS is broken" separately from WiFi +failure. Add either a v1 diagnostic check for active VPN/default-route/DNS +ownership with a "deferred repair" outcome, or explicitly state that VPN-routed +failures are out of scope and may be misclassified. This is blocking if Craig +expects the module to diagnose normal daily-driver network failures while VPN +tooling remains separate. + +Disposition: accept (chose the detect-and-classify option) — v1 detects an active +VPN / non-NM default route / non-NM DNS owner and classifies =deferred/vpn= ("link +is fine; internet is VPN-routed"), distinct from a WiFi failure. v1 does not +repair it (VPN management is Phase 5); it names the VPN as the likely owner and +stops. Added to the Doctor section + the coverage table + a doctor-classification +test. + +** DONE Remove stale GPG-store references from the resolved spec +The spec now decides "no separate credential store; secrets live in +NetworkManager", but the Testing plan still mentions =gpg round-trip= and =GPG +store= tests, and the panel-state list still mentions a no-GPG-key state. Remove +those stale references and replace them with NM-secret/no-secret-leak tests. +This is non-blocking for product behavior but blocking for implementation +clarity: otherwise tests will be written for a credential store that no longer +exists. + +Disposition: accept — replaced the Testing-plan =gpg round-trip= / =GPG store= +bullets with an "NM secrets / no-leak" test (add/edit writes the secret via nmcli; +assert no PSK/EAP in any JSON/log/error; no store to round-trip) and dropped the +=no-GPG-key= panel state. Residue from the cj-comment pass that dropped the store. + +** DONE Reconcile status, goal, and task text before implementation :blocking: +The spec status says "Implementation-ready with caveats" and "Phase 1 ready to +build", but the body still has an unresolved enterprise add/edit VERIFY, the +Goal still says "optional GPG-encrypted secret store", and the unified task title +still names "GPG-stored secrets" even though the accepted design removed the +store. Before implementation, make the top-level status, goal, scope, task +mapping, and resolved decisions agree with the current design. This blocks +readiness because a developer starting from the top of the file would still build +or plan around abandoned GPG-store behavior. + +Disposition: accept — fixed the Goal ("secrets stay in NM's own store"), the +=[#B]= task-mapping line (notes the "GPG-stored secrets" framing is superseded by +decision 5), the enterprise VERIFY (now resolved → Status updated), and corrected +the stale =pytest= mentions to =unittest= (the repo's actual harness). Top-of-file +status/goal/scope/decisions now agree with the design. + +** DONE Resolve enterprise add/edit scope or make the caveat explicit :blocking: +The spec still says "One open question for Craig: pull enterprise add/edit into +v1?" and points to a VERIFY in =todo.org=. That is a real product-scope decision: +if enterprise add/edit is in v1, panel forms, nmcli command sequences, tests, +error messages, and docs change materially; if it is out, the UI must consistently +show activate-only with "edit in nmtui/nmcli". Decide it in the spec before +implementation, or downgrade the status to =Ready with caveats= with this exact +accepted caveat. As written, the spec cannot be plain =Ready=. + +Disposition: accept — Craig decided (2026-06-29): enterprise add/edit is vNext, +activate-only in v1. Settled in the Status line, the Scope/Out bullet, decision 9, +and the VERIFY (now DONE in todo.org). The UI shows activate-only with "edit in +nmtui/nmcli" consistently. Evidence: 24 saved profiles, 0 enterprise. + +** DONE Define the concrete test harness and coverage gate :blocking: +The spec says TDD, fake binaries on PATH, and benchmark tests, but it does not +define the actual harness contract: pytest vs unittest for the =net= package, +where fake =nmcli=/=curl=/=speedtest-go=/=rfkill=/=resolvectl= live, how test +fixtures encode command histories, how subprocess timeouts are simulated, how +Waybar scripts are executed end-to-end, and how coverage is run. Add the exact +Makefile targets (=test=, =test-unit= or package-local =pytest=), pytest config, +coverage command (e.g. branch coverage over =net/= and =waybar-net= wrappers), +minimum threshold, and the rule for reading the coverage report to add missing +tests before declaring a phase done. This blocks readiness because "what is the +test harness?" is still answerable only by analogy to older suites. + +Disposition: accept — added the "Harness + coverage gate" section. Corrected the +premise: the repo is =unittest= (=make test= → =python3 -m unittest=, 33 suites), +not pytest. Pinned the fake-binary stub convention (=tests/<name>/fake-*= on a +temp PATH), the fixture command→output map, timeout simulation, the end-to-end +=waybar-net= subprocess run, and coverage via a throwaway venv (coverage.py is +absent system-wide) with a ≥90% branch target on the pure modules. + +** DONE Use coverage to find missing behavior, not just report a percentage :blocking: +The spec does not say how coverage findings affect implementation. For this +feature, line coverage alone can miss the important holes: doctor classification +branches, cleanup-unverified paths, redaction paths, degraded hot-path fallbacks, +timeout branches, and auth/upstream/VPN terminal states. Define coverage review +criteria per phase: branch coverage for pure classifiers and parsers, named +untested branches allowed only with comments or manual-check entries, and a +required "coverage gap pass" after the first green test run that maps uncovered +logic back to tests or consciously excluded live-only behavior. This blocks +readiness because the current test plan is broad but does not force the suite to +expose missing edge tests. + +Disposition: accept — added the "Coverage as a gap-finder, not a number (per +phase)" subsection: branch coverage required for the doctor classifier (every +outcome), cleanup-unverified, redaction, degraded-fallback, timeout, and the +parsers; a mandatory coverage-gap pass after the first green run mapping each +uncovered branch to a test or a named live-only exclusion; a phase isn't done +until that pass is recorded. + +** DONE Convert error classes into exact user-facing strings and evidence fields :blocking: +The failure table and doctor outcomes classify errors well, but many messages +are still templates or descriptions rather than final text. Add exact strings +for indicator tooltip, notification, CLI stderr, JSON =error.message=, and panel +banner/step text for every failure-mode row, including cases doctor cannot fix: +wrong password, missing secret, enterprise cert failure, upstream/AP/provider +failure, VPN-routed failure, hard rfkill block, DNS cleanup failure, speedtest +missing, and HTTP interception without parseable URL. For each string, specify +the redacted evidence included and the next action. This blocks UX readiness +because "useful error" is only testable once the actual text and evidence are +defined. + +Disposition: accept — rewrote the Failure states section: each row now carries the +exact final string (with =<placeholder>= evidence), the evidence field, and the +next action, plus a per-surface rendering rule (indicator tooltip / notify / +CLI+JSON error.message+detail+code / panel banner all render the one canonical +string). Added the missing doctor-unfixable rows: hard rfkill, wrong password / +missing secret, enterprise cert failure, upstream/AP/provider, VPN-routed, HTTP +interception without a parseable URL, and DNS cleanup-unverified. + +** DONE Add an enhancement disposition table +The spec captures several good enhancements (doctor, Makefile recovery, rfkill, +airplane absorption, VPN phase), but it does not show that low-cost adjacent +enhancements were considered and accepted/deferred/rejected. Add a small radar +table for likely affordances: copy redacted doctor report, open/copy portal URL, +retry with hardware MAC, forget network, rescan now, pin speedtest server, show +last good network/result, watch mode for =net doctor=, desktop notification +actions, QR-code/share WiFi import/export, and keyboard picker. Mark each +=v1=, =vNext=, or =rejected= with a one-line reason. This is non-blocking, but it +prevents accidental loss of cheap UX wins and keeps the v1 panel focused. + +Disposition: accept — added the "Enhancement radar" table dispositioning all the +named affordances: open/copy portal URL, forget network, rescan, hardware-MAC +retry, pin speedtest server, copy redacted doctor report = v1; last-good +network/result, doctor watch mode, actionable notifications, keyboard picker = +vNext; QR-share = rejected (low value for a 2-machine personal setup). + +** DONE Tighten the panel UX flow before Phase 2 +The panel has sections and state machines, but not a concrete interaction flow: +default focused section, row content, primary/secondary buttons, disabled-state +rules, confirmation wording for reset/bounce/DNS override, how "Get me online" +reports each escalation, what stays visible after the panel closes, and keyboard +navigation. Add a short UX flow spec or wire-level outline before Phase 2. This +is non-blocking for Phase 1, but it blocks Phase 2 implementation because a GTK +panel can easily become noisy or surprising if these defaults are invented while +coding. + +Disposition: accept — added the "Panel UX flow (settle before Phase 2)" +subsection: default focus (Connections, or Diagnose when opened from a captive +state), row content, one primary button per section, disabled-state rules, exact +confirmation wording for reset/bounce/DNS-override/remove, the live "Get me +online" escalation reporting, what survives panel close, and keyboard nav. + +** DONE Reconcile the panel navigation source of truth :blocking: +Disposition: accept — folded into "V2 panel UX". V2 (Connections | Diagnostics | +Performance) is the sole current target; the shipped four-page stack is marked history, +not active design. +The spec now names at least three navigation shapes: the shipped four-page stack +(Connections / Diagnose / Repair / Speed test), the V2 three-tab plan +(Connections / Diagnostics / Performance), and the redesign task's Diagnostics +sub-row (Diagnose / Get Me Online / Advanced). That leaves an implementer free +to keep extra pages and buttons even though Craig is explicitly asking for the +opposite. Make V2 the sole current target: one panel opened from the bar, top +tabs =Connections | Diagnostics | Performance=, with Diagnostics owning the +read-only checks, repair stream, debug capture, doctor report, and related +diagnostic evidence. Mark the old four-page stack as shipped history only, not +active design. This blocks the redesign because the page model determines what +code is deleted, not just labels. + +** DONE Fold speed tests into the diagnostic story :blocking: +Disposition: modify — Craig pre-decided Speed test lives under Performance (decision +19), and Performance carries future throughput history, which meets this finding's own +"keep the tab only if it carries ongoing throughput" condition. Accepted the rest: +Diagnostics runs a lightweight inline latency/throughput probe as a Diagnose evidence +row (with skip conditions for offline / metered / no-backend), and the full speed +result is stored in the doctor report. Folded into "V2 panel UX → Diagnostics owns the +diagnostic story". +Speed test is currently isolated under =Performance=, while the Goal and user +mental model treat speed, latency, and packet loss as part of "diagnostics." +That split risks another top-level button/page whose only job is a diagnostic +measurement. Keep the top-level =Performance= tab only if it carries ongoing +throughput/history later; for V2, specify that Diagnostics can run a lightweight +performance check from the same Diagnose/Get Me Online flow when internet is +available, and that the full speed test is presented as a diagnostic evidence +row or secondary action rather than a separate repair-adjacent workflow. Define +when it is skipped (offline, metered/hotspot warning, missing backend) and how +the result is stored in the doctor report. This is blocking because otherwise +the implementation preserves avoidable navigation and misses a useful failure +signal. + +** DONE Define saved-list vs available-scan semantics :blocking: +Disposition: accept — folded into "V2 panel UX → Connections". Saved / Available now / +Wired groups; Rescan refreshes only the availability/signal layer, never the Saved +list. +=net list= merges saved profiles with in-range scanned networks, while the panel +copy calls the page "Connections" and the control "Rescan." It is not clear to a +user whether they are looking at saved connections, currently available +networks, or both. The current implementation confirms the ambiguity: +=connections.py= lists saved profiles MRU-first, merges live signal/security for +saved profiles that are in range, then appends unsaved in-range SSIDs with +=uuid: nil=. Rename and specify the groups: e.g. =Saved= (instant, does not +require scan), =Available now= (scan-backed, may still be loading/stale), and +=Wired=. =Scan= should refresh only the availability/signal layer, not gate the +saved profile list. This blocks readiness because it affects loading behavior, +button enablement, and whether unsaved rows can be selected. + +** DONE Replace the Add page with join-from-row behavior :blocking: +Disposition: accept — folded into "V2 panel UX → Connections". Selecting an unsaved +Available-now row is the join flow (SSID/security prefilled); the standalone Add modal +is deleted for visible networks; hidden/manual join lives behind Advanced. +The current Add dialog asks for an SSID as free text even though a scan usually +already found the SSID and security type. That is redundant UI and a common +network-manager mistake: it turns "join this visible network" into "copy a name +from the list and type it again." V2 should remove the standalone Add button and +modal for normal visible networks. Selecting an unsaved available row should +become the join flow: the SSID/security are prefilled from the row, open +networks connect with a confirmation only if needed, WPA/WPA2/WPA3-Personal ask +only for the password, and hidden/manual SSID is tucked behind an Advanced +"Join hidden network" affordance. Keep edit/create for enterprise profiles out +of v1/V2 unless explicitly added later. This blocks the redesign because it +changes the primary connection workflow and deletes a whole page/control. + +** DONE Pin the supported authentication types in the join flow :blocking: +Disposition: accept — folded into "V2 panel UX → Supported authentication classes". +The spec says "open + WPA-PSK" and "enterprise activate-only," but cafe/hotel +networks also commonly appear as open captive portals, WPA/WPA2/WPA3-Personal +(PSK/SAE), and sometimes transition-mode networks; less commonly they use +enterprise/802.1X, WEP, OWE/enhanced-open, MAC registration, voucher portals, or +proxy-required networks. Define the V2 join matrix from the scanned NM +=SECURITY= value: supported inline (open, captive/open, WPA/WPA2/WPA3 Personal), +activate-only if already saved (802.1X/enterprise), hidden-manual behind +Advanced, and unsupported/rare types with a clear in-panel explanation plus a +non-terminal next step. If an auth type is common enough to support, support it +in the panel; if it is too rare for V2, say "not supported here yet" and keep +the user in the same UI rather than sending them to a terminal tool. Also define +what security label appears in the row so the user knows why a password is or is +not requested. This blocks because the Add/Join deletion above cannot be +implemented safely without knowing which auth classes the simplified flow covers. + +** DONE Fix destructive confirmation tense and verification +Disposition: accept — folded into "V2 panel UX → Forget confirmation". +The Forget confirmation says "The saved password is deleted" before the user has +clicked Forget. That reads as if the destructive action already happened. Change +the copy to future tense and name the scope, e.g. "This will remove the saved +NetworkManager profile and its stored password from this machine." After the +operation, verify the UUID is gone, refresh the Saved list, and report either +"Forgot <SSID>" or "Could not forget <SSID>; nothing changed / partial state +<evidence>." This is non-blocking because the existing confirm prevents an +accidental click, but the wording is misleading and the V2 "verify every action" +decision should cover it. + +** DONE Make connection loading progressive and observable :blocking: +Disposition: accept — folded into "V2 panel UX → Connections (progressive loading)". +Opening the panel currently says "Loading connections..." while =net list= +collects both saved profiles and the WiFi scan. Saved profiles do not require a +network scan, so a slow scan should not delay the saved list. Split loading into +two phases: render saved NM profiles immediately, then overlay availability, +signal, and unsaved in-range rows when the scan completes. Show a small +scan-in-progress state with elapsed time and stale-last-scan age, and make +Rescan update only the scan-backed fields. This blocks because it is the direct +answer to "why does it take so long to see the list of connections?" and keeps a +bad radio scan from making the whole panel feel broken. + +** DONE Define the visual contract with Waybar and existing Archsetup UI +Disposition: accept — folded into "V2 panel UX → Visual contract". +The panel is a layer-shell popup anchored under Waybar, but the spec does not +state the visual contract. The live Waybar theme uses a dark rounded capsule +(=border-radius: 1rem=), golden border, compact monospace text, and state colors +for =custom/net=; the GTK panel currently has a generic title, stack switcher, +default GTK controls, and square-ish/default widget corners. Add a short style +section: panel should read as a Waybar-attached popup, not a separate app; match +Waybar's palette, border/radius, spacing density, and state colors; avoid square +corners where surrounding UI is rounded; keep cards out of cards; use compact +icon+label controls with tooltips for advanced repairs. Also cite any existing +Archsetup-owned GTK/panel conventions that should be reused. This is +non-blocking for engine work but should block final V2 UX acceptance. + +** DONE Add a diagnostics report affordance that users can actually find +Disposition: accept — folded into "V2 panel UX → Findable diagnostics report". +The observability design has a JSONL event log, =diagnose --json=, automatic +verbose capture, and redacted bundles, but the panel flow does not yet define +the user affordance that turns those into an inspectable diagnosis. Add a +Diagnostics-side "Copy report" / "Open report" action after every diagnose, +repair, and speed/performance run. The report should include the current step +statuses, elapsed time, final classification, last speed/latency result when +available, scan age, route/interface owner, relevant redacted event-log tail, +and bundle path when verbose capture ran. It must explicitly say whether any +repair mutated state and whether cleanup/verification passed. This is blocking +for observability because "logs exist somewhere" is not enough when the network +is already failing. + +* Review and iteration history + +** 2026-06-29 Mon @ 17:00:39 -0400 — Codex — reviewer + +- *What changed or was recommended:* Rubric: =Not ready=. Applied the + spec-review workflow and added blocking findings for diagnostics structure, + user-facing errors, observability, nmcli contracts, cache concurrency, + hot-path performance, Waybar click actions, mutation rollback, credential + security, unsupported WiFi types, panel states, speed-test dependency + verification, install boundaries, and test coverage. +- *Why:* The spec has the right broad shape, but this feature combines a status + indicator, NetworkManager mutation, sudo-backed remediation, captive-portal + probing, secret storage, and GTK UI. Too many failure modes were still left for + implementers to invent during coding. +- *Artifacts:* Findings recorded in =Review findings=. Local code read included + =~/.dotfiles/hyprland/.local/bin/waybar-netspeed=, + =~/.dotfiles/common/.local/bin/captive=, + =~/.dotfiles/tests/waybar-netspeed/test_waybar_netspeed.py=, + =~/.dotfiles/tests/captive/test_captive.py=, pocketbook's GTK layer-shell + scaffold, current Waybar config, and the linked =todo.org= tasks. External + checks included the Arch nmcli man page and upstream LibreSpeed CLI page. + +** 2026-06-29 Mon @ 17:35:00 -0400 — Claude Code (archsetup) — responder + +- *What changed:* Processed all 18 findings — 13 accepted, 5 modified, 0 + rejected. Accepts added new sections: Repository + dependencies, Performance + budgets, nmcli contract, JSON schemas, Diagnostics contract (+ Diagnose vs + Repair), Failure states/messages/recovery, Observability, cache + concurrency/atomicity/staleness, mutation safety + rollback, the rewritten + credential security model, the expanded test plan, JSON schemas, the renamed + Implementation phases with per-phase acceptance criteria, and a Status line. + Modifies: deferred the =net debug-bundle= command (log + =diagnose --json= + cover it), made click actions phase-aware + non-blocking, scoped enterprise + WiFi to activate-only (add/edit vNext), trimmed the panel state set to states + that occur on the two-machine fleet, and switched the speed-test backend to the + already-installed =speedtest-go= (verified live) instead of a new + librespeed-cli dependency. +- *Why:* Codex's review was strong and mostly right; the modifies are scope calls + for a single-user, two-machine tool (avoid gold-plating, use installed deps, + don't model impossible states) and one phasing reconciliation (the panel is + Phase 2, so Phase-1 clicks can't open it). +- *Artifacts:* All 18 findings completed =DONE= in place with dispositions. Live + check: =pacman -Qq | grep speed= (speedtest-go-bin installed; =speedtest-go + --json= confirmed). Status now "review incorporated (Codex, 2026-06-29)", + implementation-ready with the named Phase-2/3 caveats; Phase 1 ready to build. + +** 2026-06-29 Mon @ 17:37:58 -0400 — Claude Code (archsetup) — responder (cj comments) + +- *What changed:* Folded in Craig's 10 cj comments on the spec. Design changes: + (1) dropped the separate credential store entirely — secrets stay in NM's own + =system-connections= (root =0600=, inline), touched via nmcli; no GPG, no + gpg-agent (rewrote the secrets section, decision 5, dropped Phase 4 + the gpg + dep). (2) Added =net doctor [--fix]= + Makefile console-recovery targets + (=make online= etc.) as a first-class TTY path; reversed the earlier + defer-the-doctor call (decision 10). (3) Added a full-stack =bounce= repair and + an =rfkill= unblock repair + indicator state — the rfkill one recovers the + framework-laptop post-power-loss soft-block Craig hit. (4) =custom/net= absorbs + the airplane module; the standalone airplane scripts/tests/module are deleted on + ship. (5) Moved VPN/WireGuard from "out" to a planned Phase 5. (6) Added a + "Help + documentation" section (CLI help / panel help / user guide). Answered + the enterprise-defer rationale and the captive-auto-login explanation inline. +- *Why:* Craig's comments simplify (NM-only secrets, fewer deps) and harden the + recovery story (doctor + make targets + rfkill/bounce reachable from a dead-GUI + console — his stated need). Enterprise add/edit in v1 is the one open call, + raised as a VERIFY. +- *Artifacts:* All 10 cj blocks removed. Live checks: airplane scripts/tests + present (confirmed deletion targets); =/etc/NetworkManager/system-connections/= + is root =0600= with inline secrets (confirms the NM-only secret model); + =rfkill= present. VERIFY filed in =todo.org= for the enterprise-scope decision. + +** 2026-06-29 Mon @ 17:49:24 -0400 — Codex — reviewer + +- *What changed or was recommended:* Added a second feedback batch from the + common network failure-mode review: require a diagnose/doctor coverage table, + pin DNS repair semantics, make auth failures terminal user-action states, + define upstream/AP/provider terminal states, decide how active VPN/policy + routing affects v1 diagnosis, and remove stale GPG-store references. +- *Why:* The design now has a solid recovery shape, but the field question is + sharper: for each common failure, does =diagnose= catch it, does =doctor --fix= + fix it, or does the user need to switch networks/provide credentials/contact + the venue? That mapping needs to live in the spec, not in chat. +- *Artifacts:* Findings appended under =Review findings= after reviewing the + current spec's Diagnostics, Failure states, Doctor, Testing plan, and + responder history. + +** 2026-06-29 Mon @ 18:06:22 -0400 — Codex — reviewer + +- *What changed or was recommended:* Added a readiness pass focused on + development-time questions, missed enhancements, UX cleanliness, error wording, + and test/coverage mechanics. New findings require reconciling stale status/goal + text, resolving or explicitly caveating enterprise add/edit, defining the + concrete harness and coverage gate, making coverage actionable, turning failure + classes into exact user-facing strings, adding an enhancement disposition + table, and tightening the Phase-2 panel UX flow. +- *Why:* The network failure model is now strong, but implementation would still + force developers to invent test infrastructure, coverage expectations, final + error copy, and several UX details. The spec should decide those up front or + explicitly mark them as caveats. +- *Artifacts:* Findings appended under =Review findings= after reading the + updated spec, current archsetup Makefile test targets, pocketbook's pyproject, + and the linked =todo.org= enterprise VERIFY. + +** 2026-06-29 Mon @ 17:55:08 -0400 — Claude Code (archsetup) — responder (2nd review) + +- *What changed:* Processed all 6 second-batch findings — 6 accepted, 0 modified, + 0 rejected. Added a "Doctor: escalation, classification, terminal states" + section with four outcomes (=fixable= escalating lightest-first; =needs-user- + action=, =upstream-not-local=, =deferred/vpn= terminal) and explicit per-class + DNS handling. Added a "Failure-mode coverage" table (22 rows: detect / doctor-fix + / terminal action, org-table-standard conformant, ≤120). Cleaned the last + GPG-store residue (Testing plan → NM-secret/no-leak test; dropped the no-GPG-key + panel state). Added a doctor-classification test class. +- *Why:* The findings sharpen the doctor from "escalate repairs" to "classify, + then stop at the right terminal state" — the safety property that keeps + =doctor --fix= from looping destructive repairs against a wrong password or an + upstream outage. The VPN finding got the detect-and-classify option (not repair; + that's Phase 5) so v1 doesn't misread a VPN-routed failure as WiFi down. The + coverage table is the single artifact Craig asked for to trust the field + coverage. +- *Artifacts:* All 6 findings =DONE= in place; cookie =[24/24]=. Table conformed + via =wrap-org-table.el= (max line 120). No new findings introduced; no scope + expansion past what the findings asked. Spec remains implementation-ready, the + enterprise add/edit VERIFY still the one open decision. + +** 2026-06-29 Mon @ 18:24:51 -0400 — Claude Code (archsetup) — responder (3rd review) + +- *What changed:* Processed the third review (7 findings). The enterprise-scope + finding closed first on Craig's call (vNext, activate-only). The other 6, all + accepted: (1) reconciled the top-of-file text — fixed the Goal's GPG-store + wording, the =[#B]= task-mapping line, the resolved enterprise VERIFY, and + corrected the stale =pytest= mentions to =unittest= (the repo's real harness). + (2) Added a "Harness + coverage gate" section (unittest, fake-binary stubs on a + temp PATH, venv coverage, ≥90% branch on pure modules). (3) Added a per-phase + "coverage as a gap-finder" pass. (4) Rewrote the Failure states section to exact + final strings + evidence fields + a per-surface rendering rule, and added the + missing doctor-unfixable rows. (5) Added the "Enhancement radar" table + (v1/vNext/rejected). (6) Added the "Panel UX flow" subsection. +- *Why:* The findings close the gap between "design decided" and "a developer can + start": the harness/coverage contract, the exact UX strings, and the panel flow + are the things otherwise invented mid-code. The =pytest=→=unittest= correction + was a real defect — the spec contradicted the repo's actual test convention. +- *Artifacts:* All 31 findings =DONE=; cookie =[31/31]=. Both new tables conformed + via =wrap-org-table.el= (coverage 120, radar 110). Harness verified against the + live repo (33 unittest suites, =make test=, coverage.py absent → venv). Status + raised to "Ready for Phase 1; Ready-with-caveats overall" — no open decisions + remain. + +** 2026-06-30 Tue @ 16:32:07 -0400 — Claude Code (archsetup) — responder (build + V2 redesign) + +- *What changed:* Brought the spec current with what shipped and what got decided + across 2026-06-29/30. Recorded Phases 1-3 as SHIPPED (engine, indicator, GTK4 + panel, bar clicks). Added the native captive-login engine (=portal-login= repair + tier replacing the =captive= shell-out), the live-testing portal UX fixes (removed + the polkit-gated flush, already-online short-circuit, Chrome first-run suppression, + in-panel portal, extractor hardening), and the panel auto-hide + Close button. + Then folded in the V2 redesign Craig directed: no terminals anywhere, a passwordless + root-helper + NOPASSWD sudoers as the enabler, verify-every-action, the full + ~44-mode failure catalog (now the authoritative coverage, in todo.org), and the + Connections | Diagnostics | Performance nav. Added decisions 15-19, V2 phases + 2.1-2.4, and corrected the two failure-table rows the shipped code never matched + (gateway-bounce, venue-DNS auto-revert). +- *Why:* The spec had drifted behind the code (still said "Phase 1 shipped, terminal + for privileged ops") and behind Craig's redesign calls. The doc is the contract for + the V2 build, so it has to state the no-terminal reversal, the sudo-helper + dependency, and where coverage now lives. +- *Artifacts:* Updated Status, Architecture, Engine CLI surface, Doctor DNS handling, + Failure-mode coverage, Interactions, Panel, Resolved decisions, Implementation + phases. Shipped commits referenced: dotfiles =a7d7559=, =eef6b0b=, =450b7f0=. Full + catalog + redesign children in todo.org "Network panel redesign". + +** 2026-06-30 Tue @ 17:29:29 -0400 — Codex — reviewer + +- *What changed or was recommended:* Rubric: =Not ready= for the V2 redesign until + the new blocking findings are dispositioned. Added a fourth review batch focused on + diagnostics/observability and the simple UX questions Craig raised: reconcile the + active panel nav, fold performance/speed testing into the diagnostics story, make + saved-vs-available list semantics explicit, replace the Add page with join-from-row, + pin supported WiFi authentication classes, fix Forget confirmation tense, make + connection loading progressive, define the Waybar/Archsetup visual contract, and add + a findable redacted diagnostics report affordance. +- *Why:* The spec has strong engine and failure-mode coverage, but the current doc and + shipped implementation still replicate common network-manager UX mistakes: ambiguous + mixed lists, scan-gated saved data, redundant SSID entry, misleading destructive + copy, diagnostics split across too many pages, and observability that exists in logs + more than in the user's workflow. +- *Artifacts:* Findings appended under =Review findings= after reading the + spec-review workflow, the target spec, todo.org's redesign task, and current + dotfiles implementation files =net/src/net/connections.py=, + =net/src/net/gui.py=, =net/src/net/panel.py=, and Waybar =style.css=. + +** 2026-06-30 Tue @ 17:36:40 -0400 — Claude Code (archsetup) — responder (4th review) + +- *What changed:* Dispositioned all nine fourth-review findings (8 accept, 1 modify, + 0 reject) and wove them into a new "V2 panel UX — the target design" section. + Accepts: V2's three-tab nav is the sole target (the four-page stack is history); + Connections splits into Saved / Available now / Wired with Rescan touching only the + scan layer; selecting an unsaved row is the join flow (the Add modal is deleted); + a join matrix pins which NM SECURITY classes are inline-supported / activate-only / + hidden / unsupported; loading renders Saved first and overlays the scan; the Forget + copy goes future-tense + verified; every run ends with a Copy/Open redacted report; + and a Waybar visual contract (rounded capsule, golden border, state colors). Modify: + the speed-test finding kept Craig's decision-19 placement (full speed test under + Performance, which carries future history) while accepting a lightweight inline + latency probe as Diagnose evidence stored in the doctor report. Cookie [40/40]. +- *Why:* Codex read the live implementation and caught the UX places where the module + still replicated common network-manager mistakes — mixed lists, scan-gated saved + data, redundant SSID entry, misleading destructive copy, diagnostics scattered + across pages, observability that lived in logs more than the workflow. +- *Artifacts:* Findings 32-40 completed in place with dispositions; the modify reason + on the speed-test finding. New "V2 panel UX" section under Panel. todo.org redesign + task updated to point the V2 build at the dispositioned design. + +** 2026-07-01 Wed @ 10:43:18 -0400 — Claude Code (archsetup) — responder (cj comments) + +- *What changed:* Folded in Craig's three cj comments from his review. (1) Notification + rendering: title = "Networking", body = the failure label on its own line then the + canonical string. (2) VPN kill-switch: added a "kill-switch blocking" failure state + plus a detection-and-correction strategy off the =deferred-vpn= branch (rootless + cascade over =ip rule= fwmark 0xca6c / =wg show= / Proton =pvpn-*= NM connections / + =nft=/=iptables= drop tables / firewalld =drop= zone; classify blocking only when a + block artifact exists AND no tunnel is up; correction surfaces the exact root command + per artifact). (3) Terminals: strengthened decision 15 to "no terminal ever reports + to or collects input from the user", disambiguated from the doctor's "terminal + states" wording. +- *Why:* Craig's review annotations. The kill-switch closes a real gap in the + VPN-routed classification; the terminal directive makes the no-terminal rule + absolute for the module UX. +- *Artifacts:* Three cj blocks removed. VPN research subagent cited wg-quick man page, + Pro Custodibus, System76/Proton killswitch docs, and local =doctor.py:42= / + =classify.py:60= / =USNY.conf:15=. One open tension filed as a VERIFY in todo.org: + the dead-GUI console-recovery path (=make online= from a TTY) vs the no-terminal + directive. diff --git a/docs/design/2026-06-29-waybar-timer-module-spec.org b/docs/design/2026-06-29-waybar-timer-module-spec.org new file mode 100644 index 0000000..e8eecbe --- /dev/null +++ b/docs/design/2026-06-29-waybar-timer-module-spec.org @@ -0,0 +1,217 @@ +#+TITLE: Waybar Timer Module (wtimer) — Design Spec +#+AUTHOR: Craig Jennings +#+DATE: 2026-06-29 + +* Goal + +One always-visible waybar module that keeps time four ways — countdown timer, +wall-clock alarm, count-up stopwatch, and pomodoro — with several items running +at once. The bar shows the most urgent item with a per-type glyph; the tooltip +lists them all. Backed by a single =wtimer= script over a small JSON state file. +notify fires on completion. fuzzel drives creation. No GTK app. + +Source task: archsetup =todo.org= "Waybar timer module" (=:waybar:=), including +the folded roam-capture scope expansion (mode-selectable single panel, +stopwatch, multiple simultaneous, per-mode hover text). + +* Scope + +** In +- *Timer* — count down a duration, notify on elapse, then remove. +- *Alarm* — fire at a wall-clock time, notify, then remove. +- *Stopwatch* — count up from start; pause/resume; manual stop. +- *Pomodoro* — work/break cycles (25/5, long break 15 after 4 works), auto-advance with a notify at each phase change, runs until cancelled. +- *Multiple simultaneous* — N items of any mix held in state. Bar shows one primary item plus a =+N= badge; tooltip lists every item with its remaining/elapsed and label. +- *Pause / resume* per item; *cancel* one or all. +- *Interactions* — click to create (fuzzel), middle-click pause/resume primary, right-click cancel (fuzzel pick), scroll to cycle which item is primary. +- *Per-type glyph + CSS state classes* (running / paused / urgent / break). +- *Persistence across waybar restarts* (state file in the runtime dir). + +** Out (v1, note for later) +- No GTK panel — waybar module + tooltip + fuzzel only. +- No persistence across *reboot* (runtime-dir state clears). Alarms set before a reboot won't survive. Acceptable v1; revisit with =~/.local/state= + a catch-up-on-boot pass if wanted. +- No sound selection per item (uses notify's type sound). +- No history/stats of completed pomodoros beyond the current run's cycle count. + +* Architecture + +- =wtimer= — a single executable Python script in =hyprland/.local/bin/=. Chosen over POSIX sh (the other waybar backings) deliberately: the multi-item state machine, time arithmetic, pomodoro FSM, and JSON I/O are cleaner in Python, and it gives real line/branch *coverage numbers* (Craig asked for them). Precedent: pocketbook is Python in this repo. +- *Pure core + thin IO shell.* All logic is pure functions taking =now= as a parameter (dependency-injected clock — satisfies testing.md: no recursion, no scope-shadowing, production reads =time.time()=, tests pass an explicit instant). The CLI layer does the IO: read state, call pure fns, write state, emit JSON, shell out to notify/fuzzel. +- *State file*: =$XDG_RUNTIME_DIR/waybar/wtimer.json= (env override =WTIMER_STATE= for tests). Same runtime-dir convention as =sysmon-metric=. +- *Heartbeat*: waybar calls =wtimer render= every 1s. =render= runs the tick logic first (detect elapsed items, fire notify, advance pomodoro, drop finished timers/alarms), then prints the waybar JSON. One entry point waybar polls; no separate daemon. +- *Concurrency (BLOCKER from review).* The 1s =render= and the click/scroll handlers (=add=, =toggle=, =cancel=, =cycle=) are separate processes doing read-modify-write on the same state file. Without serialization, last-writer-wins drops a click's =add=, or clobbers render's "item removed/advanced" write so the same item ticks and notifies again next second. So every read-modify-write takes an exclusive =flock= on the state file for the whole cycle, and writes go through a temp file + =os.replace= (atomic), so a concurrent render never reads a half-written file. This is what actually makes "notify fires once" true — the mutation is only authoritative under the lock. +- *State dir*: =render= and the mutating commands =mkdir -p= the state dir first (=$XDG_RUNTIME_DIR/waybar/= may not exist on a fresh boot). +- *Clock injection everywhere*: =now= comes from =WTIMER_NOW= (epoch) if set, else =time.time()=. Pure fns take =now= as a parameter; the CLI seeds it from the env. This lets the CLI integration tests hit boundary instants (exactly-at-target), not just the pure-fn tests. +- *Instant refresh*: after any mutating command, send waybar =SIGRTMIN+14= (the module's signal) so the bar updates immediately instead of lagging up to 1s. Faked in tests (=WTIMER_REFRESH= override, default =pkill -RTMIN+14 waybar=). + +* State model + +#+begin_src json +{ + "items": [ + {"id": "1", "type": "timer", "label": "tea", "target": 1751240400, "duration": 300, "paused_left": null}, + {"id": "2", "type": "alarm", "label": "", "target": 1751251200, "paused_left": null}, + {"id": "3", "type": "stopwatch", "label": "", "start": 1751240000, "paused_elapsed": null}, + {"id": "4", "type": "pomodoro", "label": "", "target": 1751241900, "phase": "work", + "cycle": 1, "work": 1500, "short": 300, "long": 900, "interval": 4, "paused_left": null} + ], + "primary": "1", + "seq": 4 +} +#+end_src + +- =seq= is the monotonic id source (string ids). +- *Paused* timer/pomodoro: =paused_left= holds seconds remaining; =target= ignored while paused; resume sets =target = now + paused_left=, =paused_left = null=. +- *Paused* stopwatch: =paused_elapsed= holds elapsed seconds; resume sets =start = now - paused_elapsed=. +- =primary= is the id the bar shows; =null= or stale → auto-select (below). + +* Display logic + +** Primary selection (bar text) +1. If =primary= names a live item, show it. +2. Else the running countdown (timer/alarm/pomodoro) with the smallest remaining. +3. Else the first running stopwatch. +4. Else idle (no items). + +** Bar text +- =<glyph> <time>= for the primary, plus = +N= when N other items exist. +- Idle: a dim timer glyph alone (or empty — decide at render; lean dim glyph so the module has a stable click target). +- =time= formatting: =M:SS= under 1h, =H:MM:SS= at/over 1h. Stopwatch counts up; timer/alarm/pomodoro count down to target. +- Paused item: prefix a pause glyph or rely on the =paused= class (CSS dims it). + +** Glyphs (nerd font; final codepoints verified live before merge) +- timer , alarm , stopwatch , pomodoro-work , pomodoro-break (coffee), paused , idle (dim). +- One glyph table at the top of the script so a live-render tweak is one edit. + +** Tooltip (all items) +One line per item: =<glyph> <label-or-type> <remaining/elapsed> (<state>)=. Pomodoro line shows phase + cycle (e.g. =work 2/4=). Header line summarizes count. Empty state: "No timers". + +** CSS classes (the =alt=/=class= field) +=timer= / =alarm= / =stopwatch= / =pomodoro-work= / =pomodoro-break=, plus =paused= and =urgent= (remaining < 60s). Drives color in style.css + both themes. + +* Commands (CLI) + +| Command | Effect | +|---------------------------------+---------------------------------------------------------------------| +| =wtimer render= | tick + emit waybar JSON (the heartbeat) | +| =wtimer add timer <dur> [label]=| add a countdown (=dur= like =25m=, =90s=, =1h30m=, =5= → minutes) | +| =wtimer add alarm <HH:MM> [lbl]=| add a wall-clock alarm (next occurrence of that time) | +| =wtimer add stopwatch [label]= | start a count-up | +| =wtimer add pomodoro [label]= | start a pomodoro at work phase | +| =wtimer new= | fuzzel: pick type, prompt value, dispatch to =add= (thin wrapper) | +| =wtimer toggle [id]= | pause/resume the item (default: primary) | +| =wtimer cancel <id>= | remove one item | +| =wtimer pick-cancel= | fuzzel: choose an item to cancel (right-click handler) | +| =wtimer cancel-all= | clear all | +| =wtimer cycle [next|prev]= | move the primary pointer across all items (incl. paused), state-list order, wrapping | + +Duration parse: =Nh=, =Nm=, =Ns= combos, or a bare integer = minutes. Reject +unparseable input (exit non-zero, notify nothing). Alarm parse: =HH:MM= 24h; if +that time today already passed, target tomorrow. + +* Notifications + +- Timer elapse: =notify alarm "Timer" "<label or duration> done" --persist=. +- Alarm fire: =notify alarm "Alarm" "<HH:MM><, label>" --persist=. +- Pomodoro phase change: =notify info "Pomodoro" "Work → short break (3/4)"= (no =--persist=; phase nudges shouldn't pile up), long-break and work-resume worded accordingly. +- notify is faked on PATH in tests; assert type + that it fired once per event. + +* Pomodoro semantics + +- Defaults: work 25m, short 5m, long 15m, interval 4 (long break after every 4th work). +- FSM: work → short → work → short → work → short → work → long → work … +- =cycle= counts completed works in the current set (1..interval); resets after a long break. +- Each phase elapse advances =phase=, recomputes =target=, fires the phase notify. Pomodoro never auto-removes; cancel ends it. + +* Waybar wiring + +** Module def (config) — signal 14 (next free; 8–13 used) +#+begin_src json +"custom/timer": { + "exec": "wtimer render", + "return-type": "json", + "interval": 1, + "signal": 14, + "on-click": "wtimer new", + "on-click-middle": "wtimer toggle", + "on-click-right": "wtimer pick-cancel", + "on-scroll-up": "wtimer cycle next", + "on-scroll-down": "wtimer cycle prev" +} +#+end_src + +** Position — right of the sysmon (battery/resource) module +Insert =custom/timer= into =modules-right= immediately after =custom/sysmon= +(between =custom/sysmon= and =custom/netspeed=). On screen that places it just +right of the battery/resource readout. + +** Not collapsible — survives the right-side collapse +The module *definition* lives in the canonical config object, and =waybar-collapse= +only swaps the =modules-right= *array* in the runtime copy (which it seeds from +canonical, so the def is always present). So making the timer non-collapsible is +purely an array-membership change: add =custom/timer= to the =waybar-collapse= +right *base set* so it stays listed when the right side collapses: +- laptop: =["custom/arrow-right","custom/sysmon","custom/timer","tray","custom/date","custom/worldclock"]= +- desktop: =["custom/arrow-right","custom/timer","tray","custom/date","custom/worldclock"]= +Update the =tests/waybar-collapse= base-set expectations to match (TDD the change). + +* CSS + +Add =#custom-timer= plus the state classes to all three stylesheets. Keep the +*selectors and structure* parallel across the three (what the theme-drift test +checks); the actual color *values* are per-theme (dupre vs hudson) and differ by +design, so this is structural parity, not byte-identity. Confirm against the real +CSS files what the drift test compares before editing. +- =hyprland/.config/waybar/style.css= +- =hyprland/.config/themes/dupre/...= waybar css +- =hyprland/.config/themes/hudson/...= waybar css +Colors: normal = foreground; =urgent= = a warning hue (reuse the sysmon +warn/crit palette); =paused= = dimmed; =pomodoro-break= = a calmer accent. + +* Testing plan (TDD) + +- Suite: =tests/wtimer/test_wtimer.py= (auto-discovered by =make test='s =tests/*/test_*.py= glob — no enumeration gap). +- *Pure-function tests* (fast, the bulk), explicit injected =now=: + - =parse_duration=: =25m=, =90s=, =1h30m=, =5= (→min), =0=, negative, garbage, empty (Normal/Boundary/Error). + - =parse_alarm=: future today, already-passed-today → tomorrow, =00:00=, =23:59=, =24:00=/=12:60= invalid, non-=HH:MM=. + - =format_time=: 0, 59s, 60s, 3599s, 3600s, multi-hour, negative clamps to 0. + - =add_item= for each type; =seq= increments; ids unique. + - =tick=: timer not-yet-elapsed (no change), exactly-at-target, past-target (fires once, removed); alarm same; pomodoro work→short→…→long→work advance + cycle counting + the 4th-work→long boundary; paused items never tick; multiple items in one tick. + - =select_primary=: explicit primary, stale primary falls back, soonest-remaining rule, stopwatch-only, empty. + - =render_payload=: text/tooltip/class for each type + paused + urgent + =+N= badge + idle. + - =toggle= pause then resume round-trips remaining/elapsed exactly; =cycle= wraps; =cancel= / =cancel-all=. +- *CLI integration tests* (subprocess, fakes on PATH, =WTIMER_NOW= to hit boundaries): =add= then =render= round-trip; =render= fires the faked =notify= once on an elapsed item and drops it; state file created if absent; *missing parent dir* created (fresh-boot case); corrupt/empty state file → treated as empty, no crash; mutating command sends the faked refresh signal. +- *Concurrency test*: spawn overlapping =render= + a mutating command against one state file; assert no lost update (the added item survives) and exactly-once notify (no double-fire from a clobbered tick). This is the regression guard for the flock/atomic-write fix. +- *Mocking boundary*: fake =notify=, =fuzzel=, =killall= on PATH (record calls); never mock the wtimer logic. Clock injected as a parameter. +- *Coverage*: measure with =coverage.py= if present (target 90%+ on the logic per testing.md business-logic bar); report the actual number. If =coverage= is absent, report per-command/per-branch case coverage explicitly and flag the tool gap (verification.md). +- =tests/waybar-collapse= base-set expectations updated for the new module. +- =tests/= theme-drift check stays green (CSS parity). + +* Files touched + +dotfiles branch =waybar-timer-module=: +- =hyprland/.local/bin/wtimer= (new, executable) +- =tests/wtimer/test_wtimer.py= (new) +- =hyprland/.config/waybar/config= (module def + modules-right position) +- =hyprland/.local/bin/waybar-collapse= (base-set) + =tests/waybar-collapse/...= (expectations) +- =hyprland/.config/waybar/style.css= + dupre + hudson waybar css (CSS) + +archsetup (main, at the end): +- this spec +- =todo.org= task closure + +* Resolved decisions (no approvals — my calls) + +- Python, not sh — testability + coverage; pocketbook precedent. +- One =render= heartbeat (no daemon) — simplest, waybar already polls. +- notify fires from =render='s tick, mutation guarantees once-only. +- Primary = user-cycled, else soonest-remaining; =+N= badge for the rest. +- Multiple simultaneous via tooltip list + badge (not a GTK panel) — keeps it "cool yet simple". +- Pomodoro is one self-advancing item, not four chained timers. +- Runtime-dir state (waybar-restart durable, not reboot durable) — v1. + +* Rollback + +All code on the dotfiles =waybar-timer-module= branch off =09815f3=. Squash-merge +at the end; =git switch main && git branch -D waybar-timer-module= reverts cleanly +if it goes sideways. diff --git a/docs/design/2026-06-29-zfs-pre-snapshot-installer.org b/docs/design/2026-06-29-zfs-pre-snapshot-installer.org new file mode 100644 index 0000000..e5a339e --- /dev/null +++ b/docs/design/2026-06-29-zfs-pre-snapshot-installer.org @@ -0,0 +1,106 @@ +#+TITLE: ZFS pre-pacman snapshot installer step (durable retention) +#+DATE: 2026-06-29 +#+SOURCE: handoff from the home project, 2026-06-29 + +* Problem + +A pacman =PreTransaction= hook snapshots =zroot/ROOT/default@pre-pacman_<ts>= +before every transaction, but nothing prunes them. Sanoid doesn't manage them +(they aren't =autosnap_= names), so they accumulated to 53 on velox between +April and the 2026-06-29 health check. Unbounded, they fill the pool over time. + +* What's actually on velox vs. archsetup + +The live =/usr/local/bin/zfs-pre-snapshot= is *not* authored by archsetup — +=git grep= for its content (=MIN_INTERVAL=, the pre-pacman =LOCKFILE= logic) +finds nothing tracked. The =PreTransaction= hooks in the archsetup monolith +(~lines 910, 1907, 1942) are the live-update guard, a different hook. The +script appears hand-placed on velox. + +The 2026-01-17 security doc line "ZFS pre-pacman snapshots (already in +install-archzfs)" is therefore *out of date* — archsetup does not install this. +Incorporating the fix is a NET-NEW installer step, not a patch to an existing +one. Correct that stale doc line as part of the work. + +velox was patched live (pruned to 10, script replaced with the self-pruning +version below); live backup at =/usr/local/bin/zfs-pre-snapshot.bak-2026-06-29=. + +* Proposed installer step + +In the archzfs / ZFS-on-root install path, gated to ZFS-root installs (velox is +the only ZFS daily driver; ratio is btrfs), install: + +1. =/etc/pacman.d/hooks/zfs-snapshot.hook= — the =PreTransaction= hook that + runs the script. *Not included in the handoff* — source it from velox + (=/etc/pacman.d/hooks/zfs-snapshot.hook=) or write it. +2. =/usr/local/bin/zfs-pre-snapshot= — the =KEEP=10= self-pruning version + below. + +Tests live in archsetup, so this wants an archsetup session and a ZFS-root VM +test (=make test FS_PROFILE=zfs=), not a cross-project edit from home. + +* The script (KEEP=10 self-pruning version) + +#+begin_src bash +#!/bin/bash +POOL="zroot" +DATASET="$POOL/ROOT/default" +LOCKFILE="/tmp/.zfs-pre-snapshot.lock" +MIN_INTERVAL=60 +KEEP=10 # how many pre-pacman snapshots to retain (rollback safety for recent transactions) + +# Skip if a snapshot was created within the last 60 seconds +if [ -f "$LOCKFILE" ]; then + last=$(stat -c %Y "$LOCKFILE" 2>/dev/null || echo 0) + now=$(date +%s) + if (( now - last < MIN_INTERVAL )); then + exit 0 + fi +fi + +TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S) +SNAPSHOT_NAME="pre-pacman_$TIMESTAMP" + +if zfs snapshot "$DATASET@$SNAPSHOT_NAME"; then + echo "Created snapshot: $DATASET@$SNAPSHOT_NAME" + touch "$LOCKFILE" + + # Retention: keep only the most recent $KEEP pre-pacman snapshots, destroy older ones. + # Sanoid does not manage these (they aren't autosnap_), so prune them here at creation time. + zfs list -H -o name -t snapshot -s creation "$DATASET" 2>/dev/null \ + | grep '@pre-pacman_' \ + | head -n -"$KEEP" \ + | while read -r old; do + zfs destroy "$old" && echo "Pruned old snapshot: $old" + done +else + echo "Warning: Failed to create snapshot" >&2 +fi +#+end_src + +* Implementation (2026-06-30) + +- Hook sourced from velox (=/etc/pacman.d/hooks/zfs-snapshot.hook=) and embedded + as a heredoc in =configure_pre_pacman_snapshots()=. +- Insertion point: a new =configure_pre_pacman_snapshots()= gated on + =is_zfs_root=, called from =boot_ux= (the last step) so the hook doesn't fire + during the install's own package operations — the first pre-pacman snapshot is + the fresh system. The script ships as =scripts/zfs-pre-snapshot= (the + =zfs-replicate= pattern), made =ZFS_PRE_*=-env-overridable for testability. +- Tests: =tests/zfs-pre-snapshot/= unit-tests the pruning logic against a fake + =zfs= (creates, prunes oldest-past-KEEP, ignores non-=pre-pacman_= snapshots, + honors the lockfile, warns on snapshot failure); =test_boot.py= asserts the + hook + script land on a ZFS install; the orchestrator test pins the new + =boot_ux= substep. + +* Note on the "stale security doc" + +The 2026-01-17 line "ZFS pre-pacman snapshots (already in install-archzfs)" is +*not* stale: that file is an archive generated by install-archzfs (see its +header and footer), and the claim is accurate for install-archzfs. The real gap +was that archsetup took sanoid from install-archzfs but never ported the +pre-pacman hook. This change ports it. The archive is left untouched. + +* Remaining + +- ZFS-root VM verification (=make test FS_PROFILE=zfs=) before the task closes. diff --git a/docs/design/2026-06-30-captive-portal-login.org b/docs/design/2026-06-30-captive-portal-login.org new file mode 100644 index 0000000..1739689 --- /dev/null +++ b/docs/design/2026-06-30-captive-portal-login.org @@ -0,0 +1,89 @@ +#+TITLE: Captive-portal login — learnings + baking it into the net panel +#+DATE: 2026-06-30 +#+SOURCE: the 2026-06-30 Hyatt wifi saga (velox) + +* Why this exists + +On a locked-down-DNS laptop, captive portals never show their login page, even +though phones get on fine. We spent hours on a Hyatt portal before finding the +mechanism; this captures it so the fix becomes a panel feature instead of a +one-off script. + +* The mechanism (what actually blocks the login) + +A redirect portal works by *DNS hijack*: you query a name, the hotel's resolver +hands back the portal, you get the login page. Two things on velox stop that: + +- *System resolver forces DNS-over-TLS.* =/etc/systemd/resolved.conf.d/dns-over-tls.conf= + hardcodes =DNS=1.1.1.1#... 9.9.9.9#...= with =DNSOverTLS=yes=. The system never + queries the hotel's resolver at all. The hotel blocks 853 (DoT) and external + 53, so system DNS is simply dead on the portal — only 443 (DoH) gets out. +- *Browser DoH.* Chrome "secure DNS" on bypasses the hotel DNS too, so the + browser never gets redirected either. + +A phone works because it uses *plain DNS* from the hotel plus a built-in +captive-portal popper. The laptop has neither. + +Confirmed facts from the saga: +- Front desk: it's a normal redirect-to-login portal. Phone: connects fine. +- No DHCP option 114 (RFC 8910) — the portal doesn't advertise its URL. But the + URL is recoverable from the HTTP 302 once you're on plain DNS. +- The walled garden whitelists OS captive-detection endpoints + (=captive.apple.com= returns "Success") — a *misleading* signal, not real + internet. Don't trust it. +- 443/DoH egress works broadly on the portal; only port-53 DNS is held. So + "system DNS fails" never means "no internet" here. + +* The working fix (=~/.local/bin/hotel-wifi=, to be folded in) + +Temporarily disable DoT → plain hotel DNS → discover the portal URL from the +redirect → open it in a clean browser profile (no DoH, no stale HSTS/cookies) → +click the button → restore DoT. Reversible; tested to restore cleanly. + +#+begin_src sh +#!/bin/sh +# hotel-wifi disable DoT -> find the portal login URL -> open it +# hotel-wifi off restore normal encrypted DNS (run once online) +conf=/etc/systemd/resolved.conf.d/dns-over-tls.conf +if [ "${1:-on}" = "off" ]; then + [ -f "$conf.captive-disabled" ] && sudo mv "$conf.captive-disabled" "$conf" + sudo systemctl restart systemd-resolved + echo "Encrypted DNS (DoT) restored."; exit 0 +fi +[ -f "$conf" ] && sudo mv "$conf" "$conf.captive-disabled" +sudo systemctl restart systemd-resolved; sleep 1 +resolvectl flush-caches 2>/dev/null || true +portal="" +for t in http://captive.apple.com/hotspot-detect.html http://neverssl.com \ + http://detectportal.firefox.com/canonical.html; do + loc=$(curl -sS -m 6 -o /dev/null -w '%{redirect_url}' "$t" 2>/dev/null) + [ -n "$loc" ] && { portal="$loc"; break; } + url=$(curl -sS -m 6 "$t" 2>/dev/null | grep -ioE 'https?://[^"'"'"' >]+' \ + | grep -ivE 'apple\.com|neverssl|firefox|w3\.org|gstatic' | head -1) + [ -n "$url" ] && { portal="$url"; break; } +done +prof=$(mktemp -d) +setsid -f google-chrome-stable --user-data-dir="$prof" "${portal:-http://neverssl.com}" >/dev/null 2>&1 +echo "Click the login button. When online: hotel-wifi off" +#+end_src + +* Baking it into the net panel (the task) + +- The net engine already diagnoses captive / no-internet. When it sees a held + portal, the panel should offer a first-class *"Log in to this network"* + action that runs the plain-DNS + clean-browser flow above, reversibly, and + auto-restores DoT when connectivity returns (or on a timeout). +- Reconcile with the existing =net portal= command and the =captive= helper — + they assumed a DNS-hijack-to-gateway model that did NOT match this portal + (gateway served no web; DNS was held, not hijacked-to-portal). The plain-DNS + approach is the one that worked; make it the engine's portal path. +- The DoT toggle must be safe and reversible (the =off= step). Consider a + per-connection or time-boxed DoT-off that can't strand encrypted DNS. +- Surface the misleading-"Success" lesson: a whitelisted captive-check passing + is not "online" — gate on a real, non-whitelisted fetch. + +* Related fix that unblocked the panel (already shipped) + +The panel could never switch networks because =net up= placed =--wait= after the +nmcli subcommand (it's a global option). Fixed in dotfiles 2432311; fake-nmcli +now rejects the misplaced flag so it can't regress. diff --git a/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org b/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org new file mode 100644 index 0000000..cb195c6 --- /dev/null +++ b/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org @@ -0,0 +1,53 @@ +#+TITLE: Waybar Expansion Animation — Feasibility Assessment +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-02 + +* Question + +The todo.org task "Smooth waybar expansion animation" [#C]: the collapse/expand +jump is abrupt, and a few systray icons pop in one-by-one afterward. Can the +expansion animate smoothly? + +* How the collapse actually works + +=waybar-collapse= rewrites the module arrays in a runtime copy of the config +(=$XDG_RUNTIME_DIR/waybar/config=) and sends waybar SIGUSR2. Waybar reloads: +it tears down every module widget and rebuilds the bar from the new config. + +* Findings + +1. *No widget survives the reload, so nothing can transition.* GTK3 CSS + transitions animate property changes on live widgets. The collapse + mechanism replaces the whole widget tree; there is no widget on both sides + of the change to interpolate. The jump is structural, not stylistic. +2. *GTK3 doesn't animate widget add/remove anyway.* Smooth insert/remove needs + =GtkRevealer= wrapping, which waybar does not use for modules. Making it do + so is an upstream waybar patch, not a config or CSS matter. +3. *The layer surface resize isn't animatable either.* Hyprland layerrules can + animate map/unmap (slide/fade), but the bar stays mapped through a collapse + — the same surface changes width. No compositor-side hook exists for that. +4. *A CSS-only fake covers custom modules at best.* Custom modules could emit + a "collapsed" class and transition font-size/padding toward zero (GTK3 CSS + can animate those). But the collapsed set includes built-ins — tray, + pulseaudio, workspaces — which take no script-driven classes. The result + would be half the modules gliding and half popping: worse than the clean + jump. +5. *Tray icons popping in one-by-one is separate and unfixable here.* That's + asynchronous StatusNotifier re-registration after the reload; each app + answers on its own schedule. Only keeping the tray alive across the change + (i.e. not reloading) avoids it. + +* Conclusion + +Not feasible with the current collapse mechanism, and no acceptable partial +measure exists. A real animation requires waybar itself to support dynamic +module sets with Revealer-style transitions (an upstream feature), or +replacing the collapse-by-reload design entirely. + +* Recommendation + +Close the task as infeasible-for-now (or park at [#D] with a pointer here). +Revisit only if waybar upstream gains dynamic module visibility (worth a +check at major waybar releases) or if the bar ever migrates to a custom +GTK4 shell — the Blueprint pipeline from the net panel would make Revealer +transitions natural there. diff --git a/docs/design/2026-07-10-audio-failure-taxonomy.org b/docs/design/2026-07-10-audio-failure-taxonomy.org new file mode 100644 index 0000000..efb6e0a --- /dev/null +++ b/docs/design/2026-07-10-audio-failure-taxonomy.org @@ -0,0 +1,444 @@ +#+TITLE: Linux Audio Failure Taxonomy — for the audio doctor +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-10 + +* Purpose + +A catalogue of real, user-reported ways Linux audio input (microphone) and output (speakers/headphones) fail, grounded in forum and issue-tracker reports. Built to feed the audio doctor: each entry will later be triaged into build (an auto/confirm remedy the doctor applies), guide (a diagnosis plus the exact fix the user runs), or out (beyond the doctor's reach). Triage is a later pass — for now the goal is breadth. + +Target: 50 input failure modes and 50 output failure modes, or exhaustion. Each entry names the layer, the symptom as a user reports it, the root cause, the known resolution, and a source. + +** Saturation test (2026-07-10) + +To check whether the nine-cluster-per-direction taxonomy was representative or an artifact of how we first sampled, a blind resample: ten agents drew 108 fresh failure reports through five different doors per direction (specific hardware lines, external devices, non-Arch distros, broad community Q&A, and call/gaming apps), weighted toward 2024-2026 and non-Arch sources, none of them shown the clusters or the existing entries. Every result was then classified against the existing 18 clusters. + +Result: zero of 108 needed a new cluster. Roughly 70 re-found entries already in the catalogue (many hitting the exact same bug and source URL we had cited independently), which is entry-level saturation. The other ~34 were genuinely new distinct root causes, and all of them slotted into existing clusters. The new results spread across every cluster and concentrated where the original catalogue did (cluster 1 firmware/driver, and the app cluster). The app cluster grew densest, which confirms the Chrome incident that started this was representative, not a fluke. + +Conclusion: the clusters — what the doctor's probe ladder and verdict structure rest on — are complete and stable. Entries will keep growing (hardware × kernel × app is unbounded), but the design can rest on the taxonomy. The ~34 new entries the sweep surfaced are folded in below, under a "Saturation resample additions" heading in each direction, each carrying its cluster and remedy class inline. + +* Input (microphone) failure modes + +Built 2026-07-10 from a parallel research sweep across eight layers (kernel/driver/firmware, ALSA, PipeWire/WirePlumber, Bluetooth, USB, app/portal, hardware, config/misc). 64 raw entries deduped to 58 distinct root causes. Each names the layer, the user-reported symptom, the cause, the known fix (with the concrete command where one exists), whether the fix needs root or a reboot, and a source. The =sudo/reboot= field is the first triage signal: a no-sudo, no-reboot fix is a candidate for an auto or guide remedy; a both/physical fix can only ever be a guide. + +** Kernel / driver / firmware + +- Missing SOF firmware for a DSP-attached mic (both). Internal mic not recognised on a modern Intel laptop; speakers fine. Digital mic hangs off the Intel DSP and needs Sound Open Firmware. Fix: =pacman -S sof-firmware alsa-ucm-conf=, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=287361][arch 287361]] +- Stale =dmic_detect=0= / =dsp_driver=1= modprobe override (both). Mic dead after an old "fix"; no DMIC input. A leftover =/etc/modprobe.d= option forces the legacy HDA path and bypasses the SOF driver that owns the digital mic. Fix: delete the override, install sof-firmware, rebuild initramfs, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=260827][arch 260827]] +- Missing codec model / ADC quirk, ALC236/ALC256 (both). Capture controls visible and unmuted, but =arecord= records silence. =snd_hda_intel= lacks the board-specific pin/ADC-gain quirk, so the ADC is never routed. Fix: =options snd-hda-intel model=...= workaround; real fix is a kernel patch_realtek entry. [[https://bugs.launchpad.net/ubuntu/+source/alsa-driver/+bug/2142994][launchpad 2142994]] +- Kernel-version regression broke a working mic (both). Internal mic recorded fine until a kernel update, then stopped. A regression between builds broke ALC256 capture routing. Fix: boot the last-good kernel or install linux-lts until a fixed build lands. [[https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2117316][launchpad 2117316]] +- AMD ACP6x DMIC needs a DMI quirk, not firmware (both). Ryzen laptop internal array mic absent or silent; sof-firmware changes nothing. The acp6x mic array needs the laptop's DMI entry in the =SND_SOC_AMD_YC_MACH= quirk table. Fix: newer kernel with the board's DMI quirk, or add the entry and rebuild. [[https://bbs.archlinux.org/viewtopic.php?id=311093][arch 311093]] +- Missing NHLT ACPI table in firmware (both). No DMIC; dmesg shows "NHLT table not found". The platform firmware omits the NHLT table, so SOF can't discover the digital-mic count. Fix: flash firmware that supplies NHLT, or override the dmic count via SOF modprobe params. [[https://github.com/MrChromebox/firmware/issues/920][mrchromebox 920]] +- Brand-new AMD ACP7.0 (Strix Halo) has no mic driver/UCM yet (both). Internal DMIC non-functional on very recent AMD hardware. The kernel ACP7.0 SOF driver and UCM profile weren't upstreamed yet. Fix: wait for the kernel + alsa-ucm-conf that add ACP7.0 support. [[https://github.com/alsa-project/alsa-ucm-conf/issues/745][ucm-conf 745]] + +** ALSA (profiles, UCM, mixer, pins, state) + +- Wrong card profile, no input route (no). Speakers work, no capture device appears, only a ".monitor". The card sits on an output-only profile so ALSA never exposes the capture PCM. Fix: =pactl set-card-profile <card> output:analog-stereo+input:analog-stereo=. [[https://bbs.archlinux.org/viewtopic.php?id=269173][arch 269173]] +- ACP vs UCM detection mismatch (no). alsa-card-profiles fixes the mic but PipeWire breaks it again. WirePlumber's ACP probing (or an auto-selected UCM) mis-detects the capture ports. Fix: set =api.alsa.use-acp = false= (or =use-ucm = true=) in WirePlumber, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=291807][arch 291807]] +- Capture / Internal Mic switch muted in mixer (no). Mic recognized but records silence. The "Capture" or "Internal Mic" mixer switch is toggled off in ALSA state. Fix: =amixer set Capture cap= (or Space on the item in alsamixer). [[https://wiki.archlinux.org/title/Advanced_Linux_Sound_Architecture/Troubleshooting][archwiki alsa]] +- Input Source enum on the wrong pin (no). Plugged-in mic silent; the wrong jack is live. The "Input Source" control selects a different pin (Rear vs Front Mic). Fix: =amixer set 'Input Source' 'Front Mic'=. [[https://bbs.archlinux.org/viewtopic.php?id=145531][arch 145531]] +- alsactl restores a muted capture at boot (yes). Mic works after manual unmute, muted again every reboot. alsa-restore replays a saved state with capture muted. Fix: unmute, then =sudo alsactl store=. [[https://bbs.archlinux.org/viewtopic.php?id=183211][arch 183211]] +- Internal Mic Boost at zero (no). Mic works but far too quiet to use. The "Mic Boost" gain sits at 0. Fix: =amixer set 'Internal Mic Boost' 100%=. [[https://forum.manjaro.org/t/microphone-array-input-too-quiet-alsamixer-mic-boost-not-changing-anything/90904][manjaro 90904]] +- Auto-Mute Mode disabling capture (no). Internal mic cuts out, often tied to headphone-jack state. The codec's "Auto-Mute Mode" mutes internal capture when a jack is sensed. Fix: =amixer set 'Auto-Mute Mode' Disabled=. [[https://wiki.archlinux.org/title/Advanced_Linux_Sound_Architecture/Troubleshooting][archwiki alsa]] +- Wrong default capture in .asoundrc (no). Apps with no picker record from the wrong card. =pcm.!default= points at the wrong capture card. Fix: set =pcm.!default= to the right =plughw= in =~/.asoundrc=; find it with =arecord -l=. [[https://bbs.archlinux.org/viewtopic.php?id=274238][arch 274238]] +- Missing UCM capture profile for the codec (no). No input profile offered at all for the built-in mic. alsa-ucm-conf lacks a UCM definition for the card. Fix: fall back to ACP (=use-ucm = false=) or install/patch the card's UCM config. [[https://github.com/alsa-project/alsa-ucm-conf/issues/594][ucm-conf 594]] + +** PipeWire / WirePlumber + +- WirePlumber fails to start on old/empty config (no; sudo only if under /etc). No audio devices at all after an update; wpctl lists nothing. WirePlumber won't parse a leftover 0.4-era config or empty drop-in and dies. Fix: remove the offending config, =systemctl --user restart wireplumber=. [[https://bbs.archlinux.org/viewtopic.php?id=294649][arch 294649]] +- Capture source muted on startup, stale WP state (no). Mic dead every boot until you mute then unmute it. Stale state in =~/.local/state/wireplumber= restores the source muted. Fix: stop wireplumber, =rm -r ~/.local/state/wireplumber=, start; or =wpctl set-mute <id> 0=. [[https://bbs.archlinux.org/viewtopic.php?id=297727][arch 297727]] +- Default source is a monitor / wrong device, and not persisted (no). Apps pick up silence or loopback; the "mic" is a sink Monitor, or the default resets to the wrong card each reboot. WirePlumber's auto-selection picks the wrong node and doesn't persist a manual default. Fix: =wpctl set-default <mic-id>=, and pin it with a node.name rule to survive reboot. [[https://bbs.archlinux.org/viewtopic.php?id=285076][arch 285076]] +- pipewire-pulse dead or wrong socket (no). pactl apps report "connection refused"; pw-cat may still work. pipewire-pulse isn't running, or XDG_RUNTIME_DIR is wrong so =/run/user/1000/pulse/native= is missing. Fix: correct XDG_RUNTIME_DIR, =systemctl --user restart pipewire pipewire-pulse wireplumber=. [[https://bbs.archlinux.org/viewtopic.php?id=281062][arch 281062]] +- Node suspended, never resumes (no). Mic works once, then silent when idle; drop when an app reopens it. =session.suspend-timeout-seconds= parks the source and it fails to wake on some 0.5.x builds. Fix: drop-in setting =session.suspend-timeout-seconds = 0=, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=309630][arch 309630]] +- A filter/virtual source shadows the real mic (no). Only a virtual "echo-cancel" or EasyEffects source shows; apps capture nothing, or apps get the raw unfiltered mic instead of the processed one. module-echo-cancel / filter-chain / EasyEffects creates a virtual source with the wrong node.target or default. Fix: fix node.target, or =wpctl set-default <right-id>=. [[https://docs.pipewire.org/page_module_echo_cancel.html][pw echo-cancel]] +- Broken ~/.config/wireplumber override disables the node (no). Internal mic stops defaulting after a config edit; system configs seem ignored. A malformed or disabling user drop-in shadows the system defaults. Fix: remove/repair the drop-in, restart wireplumber. [[https://discussion.fedoraproject.org/t/wireplumber-not-connecting-microphone/63354][fedora 63354]] +- Forced or mismatched sample rate locks the source (no). Mic stuck at 48 kHz, or silent, or apps fail to open it; also seen on USB interfaces that only do 48k. A global =default.clock.rate= / =node.force-rate= pins the graph to a rate the device or app can't negotiate, with no allowed-rates fallback. Fix: set =default.clock.allowed-rates= to a list rather than forcing one, restart pipewire; runtime =pw-metadata -n settings 0 clock.force-rate 0=. [[https://bbs.archlinux.org/viewtopic.php?id=288932][arch 288932]] + +** Bluetooth + +- Headset stuck in A2DP, no mic input (no). Headphones play great but the mic never appears. A2DP is sink-only, no microphone source. Fix: =pactl set-card-profile <bt-card> headset-head-unit=. [[https://bbs.archlinux.org/viewtopic.php?id=280718][arch 280718]] +- oFono installed but unconfigured, HFP profile absent (yes, to remove pkg). No HFP/HSP profile listed after moving to PipeWire. oFono grabs the HFP backend but is never configured. Fix: remove oFono (=pacman -R ofono=) or set =bluez5.hfphsp-backend = "native"=, restart pipewire. [[https://bbs.archlinux.org/viewtopic.php?id=269195][arch 269195]] +- mSBC codec not enabled / unavailable (no). Headset won't offer mSBC; mic is muffled CVSD or missing. mSBC role/codec not enabled in WirePlumber. Fix: set =bluez5.codecs = [ sbc sbc_xq msbc ]= and enable =hfp_hf=, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=266252][arch 266252]] +- Auto-switch to headset profile never triggers (no). Open a call app and the mic stays dead; nothing flips to HFP. WirePlumber's autoswitch isn't firing. Fix: confirm autoswitch is on, or switch manually with set-card-profile. [[https://forum.endeavouros.com/t/my-bluetooth-headphone-mic-not-reconisied-after-switching-to-pipewire/17586][endeavour 17586]] +- Codec negotiation silently drops to baseline (no). Mic works but quality is terrible; preferred codec never applies. Negotiation fails and PipeWire quietly falls back. Fix: pin codec order in a WirePlumber =bluez5.codecs= rule, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=300230][arch 300230]] +- Adapter/headset rejects hsp_ag and hfp_ag together (no). Enabling both roles breaks the profile on some headsets (Sony WH-1000XM3). Fix: restrict roles to =hfp_hf=/=hfp_ag= (drop =hsp_ag=), restart wireplumber. [[https://pipewire.pages.freedesktop.org/wireplumber/daemon/configuration/bluetooth.html][wp bluetooth]] +- BT mic invisible to arecord / /proc/asound (no). ALSA tools show no Bluetooth mic, so the user thinks it's undetected. The bluez5 mic is a PipeWire node, not an ALSA card, by design. Fix: list it with =wpctl status=, record with =pw-record=. [[https://wiki.archlinux.org/title/PipeWire][archwiki pipewire]] +- Switching to the mic profile tanks playback quality (no; expected). Music turns to tinny phone-call audio the moment the mic works. HFP shares narrow bandwidth for duplex. Fix: expected tradeoff; switch back to A2DP off-call, or enable mSBC to soften it. [[https://webcodr.io/2024/10/fixing-no-a2dp-with-bluetooth-headsets-on-linux/][webcodr]] + +** USB microphones + +- USB autosuspend powers the mic down while idle (both). Mic works at first, then silent or vanishes after sitting unused. Kernel autosuspend puts the idle device to sleep and it never wakes cleanly. Fix: a udev rule pinning =power/control=on= for the device, or =usbcore.autosuspend=-1=. [[https://bbs.archlinux.org/viewtopic.php?id=199504][arch 199504]] +- Focusrite Scarlett needs a quirk / device_setup (both). XLR interface enumerates but capture is broken or wrong-channel. The device mis-reports config; snd-usb-audio needs a quirk (or the scarlett2 driver, built-in from 6.7). Fix: =options snd_usb_audio ... device_setup=1= on older kernels, or run kernel >= 6.7. [[https://blog.rtrace.io/posts/fedora-support-focusrite-scarlett/][rtrace scarlett]] +- Intermittent disconnect with URB resets (no). Mic drops mid-use; dmesg shows disconnect and "cannot submit urb". A flaky bus/hub/cable link causing URB failures. Fix: use a direct motherboard port or powered hub, swap the cable. [[https://linux-usb.vger.kernel.narkive.com/MdlWpDZG/cannot-submit-urb-0-error-22-internal-error-followed-by-usb-hung-tasks][linux-usb list]] +- Bus/hub can't supply enough power (no). Interface fails to power up or drops out, often through an unpowered hub. A bus-powered interface exceeds what a USB2 port or passive hub delivers. Fix: powered hub or the interface's PSU; a direct rear port. [[https://bbs.archlinux.org/viewtopic.php?id=267926][arch 267926]] +- Mic dead after boot until physically replugged (both). USB mic gives no input after cold boot; replug fixes it every time. Ports stay powered through shutdown, so the device never power-cycles to enumerate at boot. Fix: enable ErP / "power off USB on shutdown" in BIOS. [[https://bbs.archlinux.org/viewtopic.php?id=280206][arch 280206]] +- WirePlumber cached a no-input card profile for the USB device (no). Card present but no capture source appears. WirePlumber remembers a profile with input off, or a corrupt state cache. Fix: =pactl set-card-profile <card> input:...=; if it won't stick, clear =~/.local/state/wireplumber=. [[https://bbs.archlinux.org/viewtopic.php?id=294444][arch 294444]] + +** Application / browser / portal / sandbox + +- Wayland browser returns an empty mic list to WebRTC (no). Google Meet shows no microphone, no permission prompt (Firefox/Chrome/Zen on Wayland). The WebRTC PipeWire capturer isn't active so enumerateDevices() yields nothing. Fix: launch with =--enable-features=WebRTCPipeWireCapturer --ozone-platform=wayland= (Chromium) or the portal prefs (Firefox). [[https://github.com/zen-browser/desktop/issues/9745][zen 9745]] +- Flatpak app lacks device / pulse permission (no). Discord (Flatpak) can't hear the mic. The sandbox withholds the mic; no =--device= or pulse access. Fix: =flatpak override --user --device=all <app>= (or Flatseal). [[https://lionir.ca/flathub-discord-permissions-explained/][flathub discord]] +- Flatpak app has no audio socket exposed (no). Zoom/Viber Flatpak sees no mic at all. The sandbox lacks the audio socket (=--socket=pulseaudio= / =xdg-run/pipewire-0=). Fix: =flatpak override --user --socket=pulseaudio <app>=. [[https://forums.linuxmint.com/viewtopic.php?t=456895][mint 456895]] +- Snap audio-record interface not connected, or AppArmor mediation (yes). Mumble/OBS snap gets no mic; or pipewire-pulse breaks on a confined client's connect. The =audio-record= interface is off by default and doesn't auto-connect; the AppArmor/LSM path can block or crash. Fix: =sudo snap connect <snap>:audio-record=, or the pipewire-pulse SystemCallFilter drop-in. [[https://forum.snapcraft.io/t/i-am-unable-to-record-audio-from-several-snaps/18703][snapcraft 18703]] +- xdg-desktop-portal not started under Hyprland (no; relogin). Screen/mic capture silently fails, no portal prompt on Hyprland. The portal needs =graphical-session.target=, which a bare Hyprland exec never raises. Fix: import the environment and raise the session target in the Hyprland startup. [[https://github.com/hyprwm/Hyprland/discussions/10841][hyprland 10841]] +- Permission-store caches a stale "deny" (no). An app was denied once and never re-prompts. The portal permission store persists the earlier "no". Fix: =flatpak permission-reset <app-id>= (or =permission-remove devices microphone=). [[https://man.archlinux.org/man/flatpak-permission-reset.1.en][flatpak permission-reset]] +- Browser per-site mic permission set to Block (no). A site says mic blocked though the OS mic works. Chrome stores a per-origin "Not allowed" rule. Fix: Site settings, reset Microphone to Ask/Allow. [[https://support.google.com/chrome/answer/2693767][chrome mic]] +- An app grabs the mic exclusively, starving others (no). While OBS/Discord runs, other apps get no mic; or arecord reports the device busy. The app opens the node (or raw ALSA hw) in exclusive mode. Fix: point the app at shared pipewire capture instead of the raw device source. [[https://obsproject.com/forum/threads/mic-echos-into-all-outputs-arch-pipewire.150207/][obs 150207]] + +** Hardware / physical / BIOS + +- Hardware mic-mute / privacy kill-switch (physical). All ALSA levels up and unmuted, still no signal. A physical slider or EC-level mute holds capture off below software. Fix: toggle the physical switch; some models need a BIOS re-enable. [[https://bbs.archlinux.org/viewtopic.php?id=182671][arch 182671]] +- Laptop mic-mute key not honored, e.g. ThinkPad Fn+F4 (no). The LED toggles but the key never mutes/unmutes, and pactl is out of sync. The key maps to F20 and the EC mute isn't wired to the audio stack. Fix: bind keycode 190 (F20) to a pactl source-mute toggle. [[https://bbs.archlinux.org/viewtopic.php?id=268269][arch 268269]] +- Combo jack keeps the internal mic selected (yes, if model option). Headset plugged in but the internal mic stays active. The shared combo jack doesn't expose "Headset Microphone"; an auto-mute/model quirk. Fix: pick "Headset Microphone" in pavucontrol, or =model=headset-mode= / disable Auto-Mute. [[https://bbs.archlinux.org/viewtopic.php?id=252594][arch 252594]] +- Jack retasked to the wrong function (both). A mic port never registers a device, or is seen as line-out. A buggy BIOS pin-config assigns the jack the wrong default function. Fix: =hdajackretask=, override the pin to Microphone, install boot override, reboot. [[https://fossies.org/linux/alsa-tools/hdajackretask/README][hdajackretask]] +- TRRS wiring mismatch, CTIA vs OMTP (physical). Audio plays but the headset mic is dead. The headset is wired to the opposite standard (mic/ground swapped). Fix: an OMTP-to-CTIA adapter, or a USB dongle. [[https://forums.linuxmint.com/viewtopic.php?t=336989][mint 336989]] +- Broken / loose 3.5mm jack (physical). Mic cuts in/out when the cable is flexed, or nothing detected. Bent contacts or a cracked solder joint. Fix: confirm on another device; use a USB dongle if the jack is dead. [[https://forums.linuxmint.com/viewtopic.php?t=452191][mint 452191]] +- A webcam's built-in mic is a separate USB device (yes, for udev rule). The system switches to low-quality audio from the webcam the user didn't know had a mic. The webcam enumerates as its own USB Audio Class interface. Fix: select the intended mic as default, or blacklist the webcam's audio interface with a udev rule. [[https://www.mjt.me.uk/posts/blacklisting-certain-microphones-linux/][mjt blacklist]] +- BIOS/UEFI disables the internal mic or audio controller (both). No capture device present at all in Linux. A firmware setting turns off the mic or the audio controller. Fix: re-enable it in BIOS setup. [[https://www.dell.com/community/en/conversations/inspiron/bios-setting-to-disable-onboard-microphone/7331179][dell inspiron]] + +** Config / permissions / misc + +- PulseAudio and pipewire-pulse both installed (yes). Mic worked on PulseAudio, took no input after switching; both seem to run. Leftover pulseaudio auto-spawns and grabs the device, fighting pipewire-pulse. Fix: remove pulseaudio, set =autospawn = no=, enable =pipewire-pulse.socket=. [[https://bbs.archlinux.org/viewtopic.php?id=283345][arch 283345]] +- User lacks realtime privileges / RTKit denied (both). Mic crackles or drops; logs show "could not make thread realtime using RTKit". No polkit session and user not in the realtime group. Fix: =pacman -S realtime-privileges=, add user to =realtime=, re-login. [[https://forum.manjaro.org/t/pulseeffects-cant-get-realtime-priority/42584][manjaro 42584]] +- XDG_RUNTIME_DIR unset in ssh/cron/non-login session (no). Recording works in the desktop terminal but fails from ssh or cron. Non-login sessions skip pam_systemd, so the socket in =/run/user/<uid>= isn't found. Fix: run as the seat user with =XDG_RUNTIME_DIR= set. [[https://bbs.archlinux.org/viewtopic.php?id=286176][arch 286176]] +- Auto-gain (AGC) pulling the mic level toward zero (no). The mic gets progressively quieter; something keeps lowering the input volume. An app or WirePlumber is allowed to auto-adjust the source gain and only ratchets down. Fix: a WirePlumber access rule blocking the client from changing gain. [[https://www.lumeh.org/wiki/audio/stop-adjusting-my-microphone/][lumeh]] + +** Saturation resample additions (2026-07-10) + +Genuinely-new input root causes from the blind resample, each tagged with the cluster it fits and its remedy class (so this doubles as their triage). Zero forced a new cluster. + +- Mic boost too high, records static/distortion (cluster 3 · AUTO). The inverse of boost-at-zero. Fix: lower Mic Boost. [[https://bbs.archlinux.org/viewtopic.php?id=211318][arch 211318]] +- App auto-gain zeroes the mic, Zoom/Discord (cluster 7 · GUIDE). The app's own auto-input-adjust drives the level to 0. Fix: disable "automatically adjust microphone volume" / input-sensitivity. [[https://community.zoom.com/t5/Meetings/Zoom-for-Linux-not-working-well-with-newer-distributions/m-p/22805][zoom community]] +- App selects the wrong input, "monitor of output", Teams (cluster 7 · GUIDE). Fix: set the device before launch, avoid the in-call chooser. [[https://learn.microsoft.com/en-us/answers/questions/183216/in-ms-teams-for-linux-microphone-stopped-working][ms teams]] +- App requires PulseAudio, fails on bare ALSA, Signal/Webex (cluster 7 · GUIDE). WebRTC won't enumerate devices without the Pulse layer. Fix: install pipewire-pulse. [[https://github.com/signalapp/Signal-Desktop/issues/5099][signal 5099]] +- Browser WebRTC captures one channel only, Jitsi/Firefox (cluster 7 · GUIDE). Firefox-specific channel handling. Fix: Chrome as workaround. [[https://github.com/jitsi/jitsi-meet/issues/16828][jitsi 16828]] +- Mic classified as an output device, Pop COSMIC (cluster 2 · GUIDE). A desktop-environment device-role bug. Fix: none confirmed; tracked upstream. [[https://github.com/pop-os/cosmic-epoch/issues/1368][cosmic 1368]] +- SOF topology-file mismatch, DMIC visible but silent (cluster 1 · PRIV, REBOOT-TAIL). The firmware is present but the wrong .tplg loads. Fix: swap the topology file in firmware-sof. [[https://bugzilla.redhat.com/show_bug.cgi?id=1772498][rh 1772498]] +- Webcam mic needs a USB init-delay quirk, Logitech C922 (cluster 1 · REBOOT-TAIL). Fix: kernel with USB_QUIRK_DELAY_INIT for the device. [[https://bugs.launchpad.net/bugs/1885259][lp 1885259]] +- USB interface implicit-feedback / clock-source stutter, Behringer UMC (cluster 4 · REBOOT-TAIL). Fix: =implicit_fb= modprobe, or kernel 5.19+. [[https://nandakumar.org/blog/2023/01/upgrade-linux-for-umc202hd.html][nandakumar]] +- USB dual-channel device won't do input and output at once until a profile toggle, Elgato Wave (cluster 4 · AUTO). Fix: toggle the Pro Audio vs Duplex profile. [[https://forum.level1techs.com/t/elgato-wave-3-fails-to-work-as-microphone/221807][level1techs]] +- USB mic not reliably enumerated, a port/controller issue or a PipeWire add-race (cluster 4 · AUTO). Fix: restart PipeWire, or a different port. [[https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/1534][pw 1534]] +- PipeWire buffer/scheduling failure, source active but no samples (cluster 5 · GUIDE). "port_use_buffers" / "scheduling stopped node" in the log. Fix: unresolved upstream. [[https://github.com/NixOS/nixpkgs/issues/271847][nixpkgs 271847]] +- Sudo-run audio tools corrupt the WirePlumber state (cluster 9 · AUTO). Fix: =rm -r ~/.local/state/wireplumber=; never run audio tools under sudo. [[https://www.kubuntuforums.net/forum/currently-supported-releases/kubuntu-24-04-nitpick-noble-lts/hardware-support-bg/681510-kubuntu-24-04-microphone-volume-at-100-per-cent-on-every-reboot][kubuntu]] +- Audio-group membership blocks per-session access, older udev/ConsoleKit distros (cluster 9 · PRIV). Fix: remove the user from the =audio= group. [[https://wiki.ubuntu.com/Audio/TheAudioGroup][ubuntu audio group]] +- Windows fast-startup wedges audio on a dual-boot (cluster 9 · GUIDE). Fix: disable Windows fast boot, or reload the sof module. [[https://bbs.archlinux.org/viewtopic.php?id=279824][arch 279824]] +- Dead mic transducer / faulty mic module (cluster 8 · GUIDE). Fix: hardware defect, replaced under warranty. [[https://community.frame.work/t/fedora-41-13th-gen-intel-core-microphone-not-working/64010][framework 64010]] + +* Input triage + +The 58 input failure modes sorted into nine symptom clusters, each owned by a probe the doctor runs (or would run). Every entry carries a remedy class: + +- *AUTO* — user-scope, reversible, runs anywhere and on any machine. The existing Auto/Confirm tiers apply. +- *PRIV* — needs sudo. Runs silently where passwordless sudo exists (every archsetup install), prompts on a CLI with a tty, degrades to Guide only in a GUI with neither. Defaults to Confirm/Arm, never silent Auto. +- *REBOOT-TAIL* — the applicable part runs (often PRIV), then the doctor instructs the reboot it cannot complete or verify past. +- *GUIDE* — nothing to run: a physical action, a BIOS setting, an app-permission the doctor can't grant, or a wait-for-upstream fix. Diagnose and print the instruction. + +Rough distribution: ~26 AUTO, ~6 PRIV, ~6 REBOOT-TAIL, ~20 GUIDE (a handful are AUTO now plus PRIV to persist). The buildable share (AUTO plus the runnable part of PRIV/REBOOT-TAIL) is a little over half. Triage is a proposal — correct the class on any line. + +** Cluster 1 — no capture device at all +Probe: kernel =/proc/asound= plus the unit tier. New work: dmesg-pattern hints would let the doctor name the specific cause (firmware-load-fail, NHLT-missing) rather than a generic "no hardware". + +- Missing SOF firmware :: REBOOT-TAIL — install =sof-firmware= (PRIV), reboot to finish, verify after. +- Stale dmic_detect / dsp_driver modprobe override :: REBOOT-TAIL — remove the override, rebuild initramfs (PRIV), reboot. +- Missing codec model / ADC quirk :: REBOOT-TAIL — the =model=...= modprobe workaround (PRIV) plus reboot; GUIDE the real kernel-quirk fix. +- Kernel regression broke the mic :: GUIDE — boot the previous kernel or install linux-lts; the doctor can't choose your boot kernel. +- AMD ACP6x DMIC DMI quirk :: GUIDE — needs a newer kernel carrying the board quirk; nothing to run. +- Missing NHLT ACPI table :: GUIDE — flash firmware that supplies NHLT; out of reach. +- New AMD ACP7.0 unsupported :: GUIDE — wait for the kernel plus UCM support. +- Focusrite Scarlett needs a quirk :: REBOOT-TAIL — =device_setup= modprobe (PRIV) plus reboot on old kernels; GUIDE "run kernel >= 6.7" otherwise. + +** Cluster 2 — device present but records silence +Probe: semantic tier (source mute, volume, pin, default). This is the input mirror of the output side's silenced and stale-default rules. + +- Wrong card profile, no input route :: AUTO — =set-card-profile= to a duplex/input profile. +- Capture / Internal Mic switch muted :: AUTO — unmute via wpctl/pactl (and the ALSA capture switch via amixer). +- Input Source enum on the wrong pin :: AUTO — set Input Source to the connected pin. +- alsactl restores a muted capture at boot :: AUTO + PRIV — unmute now (AUTO); persist with =sudo alsactl store= (PRIV). +- Auto-Mute Mode disabling capture :: AUTO — set Auto-Mute Disabled. +- Wrong default capture in .asoundrc :: AUTO — set the right default source (the PipeWire-era fix, not the legacy file). +- Default source is a monitor / wrong device :: AUTO — =set-default= to the real mic, pin it. +- Capture source muted on startup (stale WP state) :: AUTO — unmute; clear the stale WirePlumber state. +- Filter / virtual source shadows the real mic :: AUTO — =set-default= to the real source, or fix node.target. + +** Cluster 3 — works but too quiet +Probe: semantic volume/gain. + +- Internal Mic Boost at zero :: AUTO — raise Mic Boost. +- Auto-gain (AGC) pulling the level to zero :: AUTO — a WirePlumber rule blocking the client from changing gain. + +** Cluster 4 — works then dies or drops out +Probe: node state over time / suspend. New work: a re-probe-after-idle check to catch the suspend cases. + +- Node suspended, never resumes :: AUTO — =session.suspend-timeout-seconds = 0= drop-in plus restart. +- Forced or mismatched sample rate :: AUTO — =allowed-rates= config plus restart. +- USB autosuspend powers the mic down :: PRIV — a udev =power/control=on= rule (or =usbcore.autosuspend=-1=, REBOOT-TAIL). +- Intermittent USB disconnect (URB resets) :: GUIDE — swap the cable or use a direct port; physical. +- USB bus/hub can't supply power :: GUIDE — a powered hub or the interface PSU; physical. +- Mic dead after boot until replug :: GUIDE — enable ErP in BIOS so the port power-cycles; physical/BIOS. + +** Cluster 5 — the sound server lost a device the hardware has +Probe: kernel-vs-graph (the mic-unrecognized rule) plus WirePlumber liveness. This is where the spec's coarse set-emptiness rule lives. + +- ACP vs UCM detection mismatch :: AUTO — set =use-acp=/=use-ucm= plus restart wireplumber. +- Missing UCM capture profile :: AUTO — fall back to ACP; GUIDE the UCM patch/install. +- WirePlumber fails to start on old config :: AUTO — remove the user drop-in plus restart (PRIV if the file is under =/etc=). +- pipewire-pulse dead or wrong socket :: AUTO — restart the user services; check XDG_RUNTIME_DIR. +- Broken ~/.config/wireplumber override :: AUTO — remove/repair the drop-in plus restart. +- WirePlumber cached a no-input profile (USB) :: AUTO — =set-card-profile input:...=; clear stale state if it won't stick. + +** Cluster 6 — Bluetooth mic +Probe: bluez card profile plus codec. Needs the graph-first gate from the review so a BT mic never reads as no-mic-hardware. + +- Headset stuck in A2DP, no mic :: AUTO — =set-card-profile headset-head-unit=. +- oFono installed but unconfigured :: PRIV — remove oFono (or set the native backend in config, AUTO). +- mSBC codec not enabled :: AUTO — codecs config plus restart wireplumber. +- Auto-switch to headset profile never triggers :: AUTO — set-card-profile / autoswitch config. +- Codec negotiation drops to baseline :: AUTO — pin codec order. +- Adapter rejects hsp_ag plus hfp_ag together :: AUTO — restrict roles in config. +- BT mic invisible to arecord :: GUIDE — not a fault; it's a PipeWire node. The graph-first gate prevents a false no-mic-hardware here. +- Switching to the mic profile tanks playback :: GUIDE — expected A2DP/HFP tradeoff. + +** Cluster 7 — stack healthy, an app can't hear me +Probe: none in-stack. This is the tail on a healthy verdict: name the likely app/portal cause and print the fix. The Chrome incident lives here. + +- Wayland browser returns an empty mic list :: GUIDE — the WebRTC/PipeWire launch flags or browser prefs. +- Flatpak app lacks device permission :: GUIDE — print =flatpak override --user --device=all <app>=. +- Flatpak app has no audio socket :: GUIDE — print =flatpak override --user --socket=pulseaudio <app>=. +- Snap audio-record interface not connected :: GUIDE — print =sudo snap connect <snap>:audio-record=. +- xdg-desktop-portal not started under Hyprland :: GUIDE — fix the Hyprland startup env / session target. +- Permission-store cached a deny :: GUIDE — print =flatpak permission-reset <app>=. +- Browser per-site mic block :: GUIDE — the browser's site settings. +- An app grabs the mic exclusively :: GUIDE — change the app to shared capture. + +** Cluster 8 — hardware / physical +Probe: kernel tier plus a signature test (issue an unmute, re-probe; a mute that doesn't clear is a hardware switch). + +- Hardware mic-mute / privacy kill-switch :: GUIDE — flip the physical switch; the doctor detects the unmute-doesn't-stick signature. +- Laptop mic-mute key not honored (F20) :: GUIDE — bind the key to a pactl toggle (a config change). +- Combo jack keeps the internal mic selected :: AUTO + PRIV — select Headset Mic (AUTO); the =model== option to persist (PRIV, reboot). +- Jack retasked to the wrong function :: REBOOT-TAIL — hdajackretask override plus install plus reboot. +- TRRS wiring mismatch (CTIA/OMTP) :: GUIDE — a wiring adapter; physical. +- Broken / loose 3.5mm jack :: GUIDE — physical/hardware. +- Webcam's built-in mic is a separate device :: AUTO + PRIV — =set-default= away from the webcam (AUTO); a udev blacklist to persist (PRIV). +- BIOS/UEFI disables the internal mic :: GUIDE — re-enable it in BIOS; physical/firmware. + +** Cluster 9 — environment / config +Probe: process plus environment checks (a leftover pulseaudio process, XDG_RUNTIME_DIR, group membership). + +- PulseAudio and pipewire-pulse both installed :: PRIV — remove pulseaudio, set =autospawn = no=. +- User lacks realtime privileges / RTKit :: PRIV — =limits.d= plus the realtime group (relogin/reboot). +- XDG_RUNTIME_DIR unset in ssh/cron :: GUIDE — set it in the non-login context; environment-specific. + +* Output (speakers / headphones) failure modes + +Built 2026-07-10 from a parallel sweep across eight output-specific layers (kernel/driver/firmware, ALSA mixer/pin, PipeWire routing, Bluetooth, HDMI/DisplayPort, USB DAC + hardware/BIOS, app/browser, and audio quality: crackle/xrun/latency). 64 raw entries deduped to 59 distinct root causes. Same fields and =sudo/reboot= triage signal as the input section. + +** Kernel / driver / firmware + +- SOF DSP firmware not installed (both). Fresh install shows only "Dummy Output"; nothing in =/proc/asound/cards=; dmesg "failed to load DSP firmware". The Intel DSP needs Sound Open Firmware to boot. Fix: =pacman -S sof-firmware=, reboot. [[https://github.com/basecamp/omarchy/issues/6110][omarchy 6110]] +- Cirrus CS35L41 smart amp, missing ACPI _DSD (both). Card present, speakers dead on 2023 ASUS/HP; dmesg "ACPI _DSD Properties are missing for HID CSC3551". BIOS omits the props the =cs35l41-hda= driver needs, and the amp firmware is required. Fix: install =firmware-cirrus=, apply the _DSD override, or kernel >= 6.9 / BIOS update. [[https://asus-linux.org/guides/cirrus-amps/][asus-linux cirrus]] +- TI TAS2781 smart amp, driver/firmware absent (both). Lenovo Legion speakers silent; "Direct firmware load for TIAS2781RCA2.bin failed". The =SND_HDA_SCODEC_TAS2781= driver isn't built and the RCA firmware is undistributable. Fix: kernel with the option enabled + TI firmware in =/lib/firmware=. [[https://bbs.archlinux.org/viewtopic.php?id=289379][arch 289379]] +- Wrong HDA model autodetection, speaker dead (both). ALSA works but the internal speaker is silent; only headphone/HDMI plays. =snd_hda_intel= picked the wrong codec model so the speaker pin/amp isn't enabled. Fix: =options snd-hda-intel model=<quirk>= in =/etc/modprobe.d/=. [[https://bbs.archlinux.org/viewtopic.php?pid=1109929][arch 1109929]] +- snd_soc_skl grabs the card, only Dummy Output (both). After a kernel update, sound gone; only "Dummy Output". The SST driver claims the controller and the DMIC probe fails, leaving no playback path. Fix: =options snd-hda-intel dmic_detect=0= + =blacklist snd_soc_skl=. [[https://www.linuxuprising.com/2018/06/fix-no-sound-dummy-output-issue-in.html][linuxuprising]] +- AMD Ryzen HDA codec probe timeout (both). No internal speakers; =/proc/asound= shows only the HDMI codec. The analog codec probe fails ("no codecs initialized") on Renoir/Cezanne. Fix: newer kernel/firmware; =probe_mask=1= / =dmic_detect=0= workaround. [[https://forums.linuxmint.com/viewtopic.php?t=448906][mint 448906]] +- Kernel regression killed HDA output (both). Speakers worked, went silent after a kernel bump. An HDA/codec regression in the new kernel. Fix: boot the previous kernel or linux-lts until a fix lands. [[https://bbs.archlinux.org/viewtopic.php?id=278370][arch 278370]] + +** ALSA (mixer, pins, jack, state) + +- Master / PCM channel muted, MM in alsamixer (no). No sound from any output. The mixer channel is left muted. Fix: =amixer sset Master unmute; amixer sset PCM unmute=. [[https://bbs.archlinux.org/viewtopic.php?id=200955][arch 200955]] +- Auto-Mute Mode silences speakers (no). Speakers muted when headphones are (un)plugged, or stay muted with nothing plugged. The codec's Auto-Mute ties speaker output to jack-sense. Fix: =amixer sset 'Auto-Mute Mode' Disabled=. [[https://bbs.archlinux.org/viewtopic.php?id=185098][arch 185098]] +- alsactl restores a muted master at boot (yes). Sound works after manual unmute, muted every reboot. alsa-restore replays a muted state, or runs before the card enumerates. Fix: unmute, =sudo alsactl store=; add a delay to alsa-restore if it races. [[https://bugs.archlinux.org/task/35438][arch bug 35438]] +- Front/Surround/Center/LFE channel muted (no). Only some speakers play. Per-channel mixer controls muted by default on multi-channel codecs. Fix: =amixer set Surround/Center/LFE 100% unmute=; verify =speaker-test -c 6=. [[https://bbs.archlinux.org/viewtopic.php?id=172793][arch 172793]] +- Dead headphone pin needs hdajackretask (both). Speakers work, headphone jack silent and undetected. A BIOS pin-config leaves the headphone pin unconnected. Fix: =hdajackretask=, override the pin to Headphone, install boot override, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=229296][arch 229296]] +- Phantom jack-sense keeps speakers muted (yes). Speakers stay muted because the codec reports headphones plugged when nothing is. A spurious/stuck jack-detect state. Fix: force re-detect via hdajackretask reconfigure, or disable jack-detect for the speaker path. [[https://bbs.archlinux.org/viewtopic.php?id=265363][arch 265363]] +- Wrong codec model quirk, no headphone or speaker (both). The auto-parser misdetects the board and picks the wrong pin routing. Fix: =options snd-hda-intel model=<quirk>= in =/etc/modprobe.d/=, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=252295][arch 252295]] +- Headphone path maps volume to the wrong control (yes). Headphone slider does nothing / headphones silent. The card-profile mixer path sets the Speaker element =volume = off=, breaking routing. Fix: edit =analog-output-headphones.conf=, change =volume = off= to =merge=, restart audio. [[https://bbs.archlinux.org/viewtopic.php?id=237456][arch 237456]] + +** PipeWire / WirePlumber routing + +- Default sink lands on HDMI instead of speakers (no). Boot with silence; playback goes to HDMI while speakers idle. WirePlumber auto-selects the HDMI node as default. Fix: =wpctl set-default <speaker-id>=. [[https://bbs.archlinux.org/viewtopic.php?id=309588][arch 309588]] +- Default keeps reverting to the wrong sink (no). set-default holds until reboot, then flips back every session. Each sink's =priority.session= re-picks the wrong default. Fix: set =priority.session= per node in a WirePlumber rule, restart wireplumber. [[https://gist.github.com/DasBen/df9c2f6c18fff54a0ad6d19304915429][dasben gist]] +- Sink suspended on idle, doesn't resume, and pops on start/stop (no). After a few seconds of silence the sink stays suspended; a loud pop/click and a delay when it wakes. =session.suspend-timeout-seconds= parks the node and the codec power-cycles. Fix: drop-in =session.suspend-timeout-seconds = 0=, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=264337][arch 264337]] +- WirePlumber fails to start or a stale config jams routing (no; sudo if under /etc). Total silence; wpctl reports no sinks. The service crashes on a bad/incompatible config (user drop-in or a leftover =/etc/wireplumber/wireplumber.conf=). Fix: =journalctl --user -u wireplumber=, remove/move the offending config, restart. [[https://bbs.archlinux.org/viewtopic.php?id=294649][arch 294649]] +- Corrupted WirePlumber restore state (no). No output after an update; sink present but never usable, or wrong routing persists. A corrupt =~/.local/state/wireplumber/= wedges default-node selection. Fix: =rm -rf ~/.local/state/wireplumber/=, restart pipewire pipewire-pulse wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=290825][arch 290825]] +- pipewire-pulse dead, Pulse apps silent (no). Some apps play, but pactl/Pulse clients fail "Connection refused". The pipewire-pulse socket/service isn't running. Fix: =systemctl --user enable --now pipewire-pulse.socket pipewire-pulse.service=. [[https://bbs.archlinux.org/viewtopic.php?id=281062][arch 281062]] +- Sink present but silent until a mute/unmute toggle (no). Correct default, un-suspended, but no sound until volume is touched or mute toggled. WirePlumber restores a muted / 0-volume state, or the node never opens until a mute change forces activation. Fix: =wpctl set-mute @DEFAULT_AUDIO_SINK@ 0= and =set-volume ... 1.0=. [[https://bbs.archlinux.org/viewtopic.php?id=300427][arch 300427]] + +** Bluetooth output + +- A2DP Sink profile deselects itself / unavailable (yes). A2DP switches back off when picked; only HSP/HFP shows. Stale pairing state in =/var/lib/bluetooth= from a PulseAudio-era config confuses bluez negotiation. Fix: stop bluetooth, remove/rename =/var/lib/bluetooth=, restart, re-pair. [[https://bbs.archlinux.org/viewtopic.php?id=280717][arch 280717]] +- LDAC/aptX never negotiates, falls to SBC (no). Headphones connect but sound bad, stuck on SBC. Decoder libs absent, or =bluez5.codecs= order drops to baseline. Fix: install the codec packages, set =bluez5.codecs = [ ldac aptx_hd aptx aac sbc ]=, re-pair. [[https://bbs.archlinux.org/viewtopic.php?id=290413][arch 290413]] +- Playback drops to HSP/HFP mid-song (no). Music goes mono/tinny when an app opens the mic. WirePlumber autoswitch-to-headset-profile fires on any input stream. Fix: =wpctl settings --save bluetooth.autoswitch-to-headset-profile false=; force =a2dp-sink=. [[https://wiki.archlinux.org/title/Bluetooth_headset][archwiki bt]] +- BT sink not made default on connect (no). Headphones connect but audio keeps playing from laptop speakers. No switch-on-connect; the new sink's priority doesn't beat the built-in. Fix: =wpctl set-default <bt-sink-id>=, or a priority.session rule, or module-switch-on-connect. [[https://bbs.archlinux.org/viewtopic.php?id=304269][arch 304269]] +- Silent sink after suspend/resume despite reconnect (no). After waking, headphones reconnect but running apps play no sound. The stream isn't re-initialized on the resumed sink; a seat-monitoring race. Fix: =monitor.bluez.seat-monitoring = disabled= drop-in; workaround =systemctl --user restart wireplumber pipewire=. [[https://forums.linuxmint.com/viewtopic.php?p=2781321][mint 2781321]] +- Volume slider does nothing, absolute-volume conflict (no). OS volume has no effect on the headphones. PipeWire's absolute-volume passthrough hands volume to the device. Fix: =bluez5.enable-hw-volume = false= in a WirePlumber rule to restore software volume. [[https://wiki.debian.org/BluetoothUser/a2dp][debian a2dp]] +- Adapter/dongle never exposes an audio sink (both). Dongle pairs but no A2DP sink; dmesg shows a firmware load failure. Controller firmware missing/outdated (e.g. MediaTek MT7921). Fix: install/update linux-firmware, reload btusb or reboot. [[https://bbs.archlinux.org/viewtopic.php?id=296026][arch 296026]] + +** HDMI / DisplayPort + +- GPU HDMI codec never initializes (both). No HDMI sink appears; aplay shows no HDMI device. The DSP/legacy driver-path selection is wrong so the HDMI codec is skipped. Fix: =options snd_intel_dspcfg dsp_driver=3=, rebuild initramfs. [[https://bbs.archlinux.org/viewtopic.php?id=290824][arch 290824]] +- NVIDIA GPU audio removed by a runtime-PM udev rule (both). The NVIDIA HDMI card is missing or "inactive," only profile "off". An old NVIDIA runtime-PM udev rule powers down the audio function. Fix: comment out the audio-removal lines in the udev rule, reload/reboot. [[https://forums.developer.nvidia.com/t/no-audio-over-hdmi-linux-only/236341][nvidia 236341]] +- NVIDIA HDMI audio needs DRM modesetting (both). Video works over HDMI, no audio sink. Without nvidia-drm modeset the HDMI audio function isn't exposed. Fix: =options nvidia_drm modeset=1=, rebuild initramfs. [[https://bbs.archlinux.org/viewtopic.php?id=264504][arch 264504]] +- HDMI audio dies after suspend/resume, codec state (yes). Sound before suspend, silent after; dmesg "spurious response". The HD-Audio codec state isn't restored across resume. Fix: =sudo alsa force-reload= after resume, or =alsactl store=/=restore=. [[https://forums.developer.nvidia.com/t/nvidia-hdmi-audio-errors-after-resume-snd-hda-intel-spurious-response-last-cmd/149003][nvidia 149003]] +- Empty ELD after resume marks the port disconnected (yes). HDMI port shows unavailable post-resume; WirePlumber logs "ELD info empty". A kernel regression fails to re-read the ELD on resume. Fix: boot linux-lts until fixed; replug the cable to force an ELD re-read. [[https://bbs.archlinux.org/viewtopic.php?id=306343][arch 306343]] +- Wrong monitor is the audio sink, multi-monitor (no). Two displays on one GPU; audio goes to the wrong one. Only one HDMI/DP profile is active and the default picks the wrong port. Fix: =pactl set-card-profile <card> output:hdmi-stereo-extraN= for the right port. [[https://bbs.archlinux.org/viewtopic.php?id=295404][arch 295404]] +- HDMI sink present but silent until daemon restart (no). HDMI profile selectable but no sound until PipeWire restarts. WirePlumber didn't pick up the sink on hotplug. Fix: =systemctl --user restart pipewire pipewire-pulse wireplumber=. [[https://bbs.archlinux.org/viewtopic.php?id=280352][arch 280352]] +- HDMI sample-rate mismatch, display rejects the stream (no). HDMI audio crackles or drops on some content; the TV/monitor rejects non-48k rates. PipeWire negotiates a rate the display's ELD doesn't accept. Fix: pin the rate, =pw-metadata -n settings 0 clock.force-rate 48000= or set allowed-rates. [[https://bbs.archlinux.org/viewtopic.php?id=288932][arch 288932]] + +** USB DAC and hardware / BIOS + +- USB autosuspend powers the DAC down when idle (both). DAC plays, then silent after inactivity; sometimes needs a replug. The kernel autosuspends the port and severs the stream. Fix: =usbcore.autosuspend=-1=, or a udev rule setting =power/control=on=, or a silent keep-alive stream. [[https://destinmoulton.com/notes/howto/linux-usb-audio-keep-alive-service/][destinmoulton]] +- USB DAC not auto-selected as default output (no). Sound keeps coming from onboard after the DAC is plugged in. WirePlumber gives the DAC an equal/lower node priority. Fix: a WirePlumber rule raising =priority.session=/=priority.driver= for the device, restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=290341][arch 290341]] +- Kernel USB-audio regression, DAC has only an "Off" profile (both). After an upgrade the DAC (MOTU M4) enumerates but the only profile is "Off". A snd-usb-audio regression breaks stream setup for some devices. Fix: boot linux-lts or wait for the fixed kernel; downgrade as a stopgap. [[https://bbs.archlinux.org/viewtopic.php?id=279889][arch 279889]] +- DAC needs an implicit-feedback quirk to sync (both). Playback stutters or drops to silence and won't recover; the device has no explicit feedback endpoint. snd-usb-audio doesn't apply generic implicit-feedback sync. Fix: =options snd_usb_audio implicit_fb=1= (or the per-device quirk_flags bit) in =/etc/modprobe.d/=. [[https://lkml.iu.edu/hypermail/linux/kernel/2206.0/08494.html][lkml usb-audio]] +- Headphone jack not detected, speakers keep playing (both). Plugging headphones doesn't mute the speakers; no auto-switch on insert. The HDA auto-parser picks the wrong model so jack events aren't wired to the switch. Fix: =options snd-hda-intel model=<board>= in =/etc/modprobe.d/=, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=265363][arch 265363]] +- Front-panel headphone jack not wired by BIOS pin config (both). The front jack produces nothing and never appears; rear jacks work. The BIOS marks that codec pin "not connected". Fix: =hdajackretask= override the pin to Headphone/Line-out, install the boot override, reboot. [[https://github.com/alsa-project/alsa-tools/blob/master/hdajackretask/README][hdajackretask]] +- BIOS-disabled onboard audio, or a GPU HDA controller hides it (both). No card detected (empty =aplay -l=), or only the GPU/HDMI device appears. Onboard audio is disabled in BIOS, or the GPU's HDA controller takes over ordering. Fix: re-enable onboard audio in BIOS; =options snd-hda-intel enable=1,0= to disable the GPU's audio function. [[https://bbs.archlinux.org/viewtopic.php?id=221296][arch 221296]] + +** Application / browser / sandbox + +- Flatpak app has no PulseAudio socket (no). A Flatpak app is silent while native apps play. The manifest was built without =--socket=pulseaudio=. Fix: =flatpak override --user --socket=pulseaudio <app>=. [[https://github.com/flatpak/flatpak/issues/878][flatpak 878]] +- Flatpak silent on a PipeWire-only system (no). App with pulse permission still has no sound on pure PipeWire. Flatpak has no native PipeWire socket permission. Fix: =flatpak override --user --filesystem=xdg-run/pipewire-0 <app>=. [[https://github.com/flathub/com.valvesoftware.Steam/issues/1451][flathub steam 1451]] +- Snap app can't play, audio-playback interface unconnected (no). A snap (often =--dangerous=) produces no sound. Locally-installed snaps skip the auto-connect assertion. Fix: =snap connect <snap>:audio-playback=. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1685284][mozilla 1685284]] +- Browser per-tab mute zeroes the app stream (no). Firefox/Chrome plays nothing; the app stream shows muted, device fine. A tab was muted, muting the per-stream sink-input. Fix: Unmute Tab, or un-mute the app's stream in pavucontrol. [[https://bbs.archlinux.org/viewtopic.php?id=275694][arch 275694]] +- Wine/Proton game silent, wrong audio driver (no; sudo for 32-bit libs). A Wine/Proton game has no output. The Wine driver is set wrong, or 32-bit audio libs are missing. Fix: winecfg output device to System default; install =lib32-libpulse= / =lib32-pipewire=. [[https://bbs.archlinux.org/viewtopic.php?id=135032][arch 135032]] +- App hard-coded to raw hw:0, device busy (no). MPD/shairport errors "Device or resource busy" and stays silent, blocking others. The app points at raw =hw:0=, grabbing the card exclusively. Fix: change the app's device to =default= or =plughw=; =fuser -v /dev/snd/*= to find the holder. [[https://community.volumio.com/t/fix-mpd-failed-to-open-alsa-device-hw-0-0-device-busy/686][volumio mpd]] +- Chromium doesn't follow the default-sink change (no). After switching the default output, Chrome keeps playing to the old sink. Chromium latches its stream to the start-time sink. Fix: reassign the Chromium stream in pavucontrol; PipeWire remembers it. [[https://bbs.archlinux.org/viewtopic.php?id=116888][arch 116888]] +- Electron app routed to the wrong sink (no). Discord is silent while YouTube works; its stream went elsewhere. Electron apps share a generic stream name so a prior routing rule misdirects it. Fix: move the app's stream to the right sink in pavucontrol while it plays. [[https://bbs.archlinux.org/viewtopic.php?id=277447][arch 277447]] + +** Audio quality (crackle, xruns, latency) + +- Quantum drops too low under load (no). Crackling/popping that worsens under CPU load. PipeWire's dynamic quantum negotiates a buffer that underruns the sink. Fix: =default.clock.min-quantum = 1024= (raise as needed) in a pipewire.conf.d drop-in, restart pipewire. [[https://bbs.archlinux.org/viewtopic.php?id=303024][arch 303024]] +- Realtime scheduling denied, RTKit (yes). Xruns/dropouts uncorrelated with CPU load; the audio thread misses deadlines. PipeWire can't get SCHED_FIFO (RTKit unavailable or capped). Fix: rtkit-daemon running, user in =realtime=, =/etc/security/limits.d/= granting rtprio + memlock; re-login. [[https://bbs.archlinux.org/viewtopic.php?id=283334][arch 283334]] +- Timer-based scheduling on a fussy USB DAC (no). Periodic crackle on a USB interface. tsched mis-estimates the batch USB device's buffer position. Fix: a WirePlumber ALSA rule with =api.alsa.disable-tsched = true= and a modest headroom, or the Pro Audio profile. [[https://bbs.archlinux.org/viewtopic.php?id=284888][arch 284888]] +- Forced global sample rate causes resample artifacts (no). Distortion on 44.1k content, or a stream stuck upsampling; also seen on USB DACs that only accept a limited rate set. A hard =default.clock.rate= forces everything through the resampler. Fix: set =default.clock.allowed-rates= to a list so the device follows the source, restart pipewire. [[https://bbs.archlinux.org/viewtopic.php?id=288932][arch 288932]] +- Small quantum fine for music, too tight for browser/games (no). Audio crackles only in Chrome/Firefox or games, clean elsewhere. The heavy client can't keep the small global quantum fed. Fix: launch with =PIPEWIRE_LATENCY=2048/48000=, or a WirePlumber stream rule setting =node.latency= for that app. [[https://forum.endeavouros.com/t/pipewire-guide-audio-crackling-popping-and-latency/69602][endeavour 69602]] +- HDA codec power-save clicks and BT-latency hits (yes). Loud click/pop on first play after idle on internal HDA; also wrecks Bluetooth latency. =snd_hda_intel power_save= cuts codec power and clicks on wake. Fix: =options snd_hda_intel power_save=0 power_save_controller=N= in =/etc/modprobe.d/=. [[https://forums.linuxmint.com/viewtopic.php?t=186531][mint 186531]] +- Insufficient ALSA headroom on an emulated/quirky device (no). Constant fine crackle from a sink whose hardware pointer PipeWire can't track (VM audio, cheap onboard). The default small =api.alsa.headroom= leaves no slack. Fix: a WirePlumber ALSA rule adding =api.alsa.headroom = 2048=. [[https://pipewire.pages.freedesktop.org/wireplumber/daemon/configuration/alsa.html][wp alsa config]] + +* Output triage + +The 59 output failure modes sorted into nine symptom clusters, same remedy classes as the input triage (AUTO / PRIV / REBOOT-TAIL / GUIDE — see that section's legend). The clusters shift for output: no mic-specific groups, but HDMI/DisplayPort and degraded-audio (crackle/xrun) each earn a dedicated cluster, and speaker-vs-headphone jack routing replaces the input pin cluster. + +Rough distribution: ~24 AUTO, ~8 PRIV, ~11 REBOOT-TAIL, ~16 GUIDE. Output leans more REBOOT-TAIL than input, because its hardware tail (smart-amp firmware, HDA model quirks, hdajackretask, GPU modeset) is heavier. Triage is a proposal — correct the class on any line. + +** Cluster 1 — no sound at all / dummy output +Probe: units + kernel =/proc/asound= + =dsp_driver=. The output mirror of input cluster 1. + +- SOF DSP firmware not installed :: REBOOT-TAIL — install =sof-firmware= (PRIV), reboot. +- Cirrus CS35L41 smart amp, missing _DSD :: REBOOT-TAIL — =firmware-cirrus= + the _DSD override (PRIV), reboot; GUIDE the BIOS-update path. +- TI TAS2781 smart amp absent :: GUIDE — needs a kernel built with the driver + TI firmware; nothing to run on a stock stack. +- Wrong HDA model autodetection :: REBOOT-TAIL — =model== modprobe (PRIV) + reboot. +- snd_soc_skl grabs the card, only Dummy Output :: REBOOT-TAIL — =dmic_detect=0= + blacklist =snd_soc_skl= (PRIV) + reboot. +- AMD Ryzen HDA codec probe timeout :: REBOOT-TAIL — the =probe_mask=/=dmic_detect= workaround (PRIV) + reboot; GUIDE a newer kernel. +- Kernel regression killed HDA output :: GUIDE — boot the previous kernel / linux-lts. +- Kernel USB-audio regression, DAC "Off" profile :: GUIDE — boot linux-lts / wait for the fix. +- BIOS-disabled onboard, or a GPU HDA controller steals ordering :: GUIDE + PRIV — re-enable in BIOS (GUIDE); =enable=1,0= modprobe to demote the GPU's audio (PRIV, reboot). + +** Cluster 2 — sound plays but from the wrong device +Probe: default sink + priority. The output side of routing. + +- Default sink lands on HDMI instead of speakers :: AUTO — =set-default= to the speakers. +- Default keeps reverting to the wrong sink :: AUTO — a =priority.session= rule. +- WirePlumber fails to start / stale config jams routing :: AUTO — remove the drop-in + restart (PRIV if under =/etc=). +- Corrupted WirePlumber restore state :: AUTO — =rm= the state + restart. +- pipewire-pulse dead, Pulse apps silent :: AUTO — restart the user services. +- USB DAC not auto-selected as default :: AUTO — a =priority.session= rule. + +** Cluster 3 — right default but silent (muted or zero) +Probe: semantic sink mute/volume. The output mirror of input cluster 2. + +- Master / PCM channel muted :: AUTO — unmute. +- Auto-Mute Mode silences speakers :: AUTO — disable Auto-Mute. +- alsactl restores a muted master at boot :: AUTO + PRIV — unmute now (AUTO); =sudo alsactl store= to persist (PRIV). +- Front/Surround/Center/LFE channel muted :: AUTO — unmute the channels. +- Sink present but silent until a mute/unmute toggle :: AUTO — =set-mute 0= / =set-volume=. + +** Cluster 4 — speakers work but headphones don't (or the reverse) +Probe: jack-sense + pin config. Mostly hdajackretask/model-quirk territory, so this cluster leans REBOOT-TAIL. + +- Dead headphone pin needs hdajackretask :: REBOOT-TAIL — hdajackretask override + install + reboot. +- Phantom jack-sense keeps speakers muted :: REBOOT-TAIL — hdajackretask reconfigure; GUIDE if it needs a codec quirk. +- Wrong codec model quirk, no headphone or speaker :: REBOOT-TAIL — =model== modprobe + reboot. +- Headphone path maps volume to the wrong control :: PRIV — edit the alsa-card-profile mixer path (a system file), restart audio. +- Headphone jack not detected, speakers keep playing :: REBOOT-TAIL — =model== modprobe + reboot. +- Front-panel headphone jack not wired by BIOS :: REBOOT-TAIL — hdajackretask override + install + reboot. + +** Cluster 5 — works then dies or pops +Probe: node state / suspend + re-probe-after-idle. + +- Sink suspended on idle, pops on start/stop :: AUTO — =session.suspend-timeout-seconds = 0= drop-in + restart. +- USB autosuspend powers the DAC down :: PRIV — a udev =power/control=on= rule (or =usbcore.autosuspend=-1=, REBOOT-TAIL). +- DAC needs an implicit-feedback quirk :: REBOOT-TAIL — =implicit_fb= modprobe (PRIV) + reboot. + +** Cluster 6 — Bluetooth output +Probe: bluez profile + codec. + +- A2DP Sink profile deselects / unavailable :: PRIV — clear =/var/lib/bluetooth= + re-pair (or a config fix, AUTO). +- LDAC/aptX never negotiates, falls to SBC :: PRIV — install the codec packages; AUTO for the codecs config. +- Playback drops to HSP/HFP mid-song :: AUTO — autoswitch off; force =a2dp-sink=. +- BT sink not made default on connect :: AUTO — =set-default=; a switch-on-connect rule. +- Silent sink after suspend/resume :: AUTO — a seat-monitoring drop-in; restart as the workaround. +- Volume slider does nothing (absolute volume) :: AUTO — =enable-hw-volume = false=. +- Adapter never exposes a sink (firmware) :: REBOOT-TAIL — update linux-firmware, reload/reboot. + +** Cluster 7 — HDMI / DisplayPort +Probe: GPU audio codec + ELD. Output-only cluster. + +- GPU HDMI codec never initializes :: REBOOT-TAIL — =dsp_driver= modprobe + initramfs + reboot. +- NVIDIA GPU audio removed by a PM udev rule :: PRIV — edit the udev rule (reload/reboot). +- NVIDIA HDMI audio needs DRM modesetting :: REBOOT-TAIL — =modeset=1= modprobe + initramfs + reboot. +- HDMI audio dies after suspend/resume :: PRIV — =sudo alsa force-reload= (or alsactl store/restore). +- Empty ELD after resume (kernel regression) :: GUIDE — boot linux-lts; replug to force an ELD re-read. +- Wrong monitor is the audio sink :: AUTO — =set-card-profile= to the right hdmi-extra port. +- HDMI sink present but silent until daemon restart :: AUTO — restart the user services. +- HDMI sample-rate mismatch, display rejects :: AUTO — pin the rate. + +** Cluster 8 — degraded audio (crackle, xrun, latency) +Probe: xrun counters (pw-top) + config. Output-only cluster; sound plays but is degraded. + +- Quantum drops too low under load :: AUTO — a =min-quantum= drop-in + restart. +- Realtime scheduling denied (RTKit) :: PRIV — =limits.d= + the realtime group (relogin). +- Timer-based scheduling on a fussy USB DAC :: AUTO — a =disable-tsched= rule. +- Forced global sample rate causes resample artifacts :: AUTO — =allowed-rates= config. +- Small quantum too tight for browser/games :: AUTO — per-app =PIPEWIRE_LATENCY= / a stream rule. +- HDA power-save clicks (and BT-latency hits) :: PRIV — =power_save=0= modprobe. +- Insufficient ALSA headroom :: AUTO — a =headroom= rule. + +** Cluster 9 — stack healthy, an app is silent +Probe: none in-stack; the tail on a healthy verdict. The output mirror of input cluster 7. + +- Flatpak app has no PulseAudio socket :: GUIDE — print =flatpak override --user --socket=pulseaudio <app>=. +- Flatpak silent on a PipeWire-only system :: GUIDE — print =flatpak override --user --filesystem=xdg-run/pipewire-0 <app>=. +- Snap audio-playback interface unconnected :: GUIDE — print =snap connect <snap>:audio-playback=. +- Browser per-tab mute zeroes the stream :: GUIDE — unmute the tab, or the app's stream. +- Wine/Proton game silent, wrong audio driver :: GUIDE — winecfg output device; install the 32-bit audio libs. +- App hard-coded to raw hw:0, device busy :: GUIDE — change the app's device to =default= / =plughw=. +- Chromium doesn't follow the default-sink change :: GUIDE — reassign the stream in pavucontrol. +- Electron app routed to the wrong sink :: GUIDE — move the stream in pavucontrol. + +** Saturation resample additions (2026-07-10) + +Genuinely-new output root causes from the blind resample, each tagged with the cluster it fits and its remedy class. Zero forced a new cluster. + +- App picks the wrong audio backend, Spotify (cluster 9 · GUIDE). The app's built-in backend mis-routes. Fix: force the pulseaudio backend in the launch flags (=--audio-api=pulseaudio=). [[https://bbs.archlinux.org/viewtopic.php?id=285730][arch 285730]] +- A PipeWire regression breaks game audio, Proton XAudio2 titles (cluster 9 · GUIDE). Fix: downgrade PipeWire, or update to the fixed release. [[https://github.com/ValveSoftware/Proton/issues/8730][proton 8730]] +- App muted in the desktop per-app mixer, KDE Chrome (cluster 9 · AUTO). The Flatpak permission is fine; the app's stream is muted in the DE volume applet. Fix: unmute the app there. [[https://github.com/flathub/com.google.Chrome/issues/297][chrome 297]] +- App connection stale after an audio-stack update, Firefox (cluster 9 · AUTO). Fix: restart the app or the audio daemon. [[https://forum.endeavouros.com/t/solved-no-audio-in-firefox-from-pipewire-or-pulseaudio-update/26032][endeavour 26032]] +- Output defaults to the wrong digital out, needs alsactl init, Framework 16 (cluster 2 · AUTO). Fix: =alsactl init=, then select the analog output. [[https://community.frame.work/t/solved-framework-16-ai-300-series-ubuntu-24-04-speakers-work-but-not-headphones/81363][framework 81363]] +- WirePlumber disabled in config, no enumeration, NixOS (cluster 2 · AUTO). Fix: enable wireplumber in the config, rebuild. [[https://github.com/NixOS/nixpkgs/issues/402999][nixpkgs 402999]] +- A distro update drops an audio package, openSUSE =wireplumber-audio= (cluster 1 · PRIV). Fix: reinstall and lock the package. [[https://forums.opensuse.org/t/no-output-or-input-devices-after-zypper-dup-only-dummy-output-in-pipewire-on-tumbleweed/185154][opensuse 185154]] +- USB audio module not auto-loaded after a kernel update (cluster 1 · PRIV). Fix: add =snd_usb_audio= to =modules-load.d=. [[https://forums.linuxmint.com/viewtopic.php?t=443590][mint 443590]] +- Stray =model=generic= modprobe forces the wrong codec (cluster 1 · REBOOT-TAIL). Left by OEM/installer tooling. Fix: delete the leftover =/etc/modprobe.d= line. [[https://bugs.launchpad.net/bugs/2092728][lp 2092728]] +- SOF driver aborts when the codec is hda_generic, Acer Swift (cluster 1 · REBOOT-TAIL). Fix: kernel 5.8+; =dmic_detect=0= workaround. [[https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1877757][lp 1877757]] +- Cirrus CS35L56 smart-amp firmware not applied, ROG G14 2024 (cluster 1 · REBOOT-TAIL). A newer amp than the CS35L41 already catalogued. Fix: newer kernel + firmware. [[https://www.mail-archive.com/debian-kernel@lists.debian.org/msg139811.html][debian-kernel]] +- Creative CA0132 card needs a kernel quirk, AE-5 Plus (cluster 1 · GUIDE). Reuses an existing PCI id with a different config. Fix: no userspace fix; kernel patch. [[https://forum.manjaro.org/t/soundblaster-creative-ae-5-plus-no-sound/60516][manjaro 60516]] +- ALC4080 enumerates as USB not HDA, front-panel node not exposed (cluster 4 · AUTO). Fix: an explicit ALSA sink in =pipewire.conf= pointing at the FP device. [[https://discussion.fedoraproject.org/t/headphone-audio-not-working-asus-rog-strix-z590-f-alc4080/76380][fedora 76380]] +- GPU HDMI audio KAE regression, Intel Arc DG2 (cluster 7 · GUIDE). Fix: kernel fix disables KAE for DG2; update kernel. [[https://www.phoronix.com/news/Intel-DG2-Audio-Fix-Linux-KAE][phoronix]] +- USB DAC silent at a specific sample rate, FiiO 44.1k (cluster 8 · AUTO). Fix: set =allowed-rates=. [[https://bbs.archlinux.org/viewtopic.php?id=180561][arch 180561]] +- BIOS firmware bug, crackle or dead speakers fixed by a BIOS update, MSI GF63 (cluster 1 · GUIDE). Broke on Windows too. Fix: flash the vendor BIOS update. [[https://forums.linuxmint.com/viewtopic.php?t=424474][mint 424474]] +- USB sound device not reliably enumerated after boot, an init race, Creative/Schiit (cluster 5 · GUIDE). Fix: replug; pin the profile. [[https://forum.zorin.com/t/usb-soundcard-creative-sound-blaster-hd-not-always-recognized-after-boot/66177][zorin 66177]] +- Audio-group membership blocks access, older udev/ConsoleKit distros (cluster 9 · PRIV). The same cross-cutting trap the input side hit. Fix: remove the user from the =audio= group. [[https://wiki.ubuntu.com/Audio/TheAudioGroup][ubuntu audio group]] + +* Sources + +Every entry carries its source inline as an org link. Reports span the Arch/Manjaro/EndeavourOS/Fedora/Ubuntu/Mint/openSUSE forums, askubuntu, the PipeWire and WirePlumber docs, ALSA and Flatpak issue trackers, kernel bug trackers, and vendor forums (NVIDIA, Dell). Collected 2026-07-10. diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org new file mode 100644 index 0000000..74790c6 --- /dev/null +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -0,0 +1,434 @@ +#+TITLE: Linux Network and Bluetooth Failure Taxonomy — for the net and bt doctors +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-10 + +* Purpose + +A catalogue of real, user-reported ways Linux networking (WiFi/ethernet/DNS/egress) and Bluetooth fail, grounded in forum and issue-tracker reports. Built to feed the net-panel and bt-panel doctors the same way the audio taxonomy fed the audio doctor: each entry is triaged into a remedy class — AUTO (the doctor applies a user-scope reversible fix), PRIV (needs root; runs where passwordless sudo exists), REBOOT-TAIL (a privileged fix that only takes hold after a reboot), or GUIDE (nothing to run — a physical action, a config decision, or an upstream/venue problem). This mirrors [[file:2026-07-10-audio-failure-taxonomy.org][the audio taxonomy]] and reuses its four-class privilege model. + +The net and bt doctors already exist (=~/.dotfiles= =net/= and =bluetooth/=). This taxonomy grounds their *expansion* — widening the diagnosis space and hardening the verdict/remedy structure — the way the audio input-side spec grew from the audio taxonomy. It fed two DRAFT specs: [[file:../specs/2026-07-11-net-doctor-expansion-spec.org][the net doctor expansion]] and [[file:../specs/2026-07-11-bt-doctor-expansion-spec.org][the bt doctor expansion]]. + +** Method + +Built 2026-07-10 from a blind, by-layer parallel research fan-out — the same first-sweep method as the audio taxonomy. Eleven agents (six network, five bluetooth), each scoped to one layer and blind to any cluster structure, drew real failure reports from Arch BBS/wiki, kernel bugzilla/launchpad/gitlab, distro forums, and upstream issue trackers, weighted toward 2023-2026 and the Arch / NetworkManager / systemd-resolved / bluez stack. Each entry names the layer, the user-reported symptom, the root cause, the known fix (with the concrete command where one exists), the privilege/reboot signal, and a source. + +The =(privilege signal)= after each title is the first triage cue: =no= = user-scope, no reboot; =yes= = needs root; =both= = root AND a reboot; =physical= = a hardware/physical/BIOS action. A =no= is an AUTO candidate; =both= or =physical= can only ever be REBOOT-TAIL or GUIDE. + +A saturation resample (the audio taxonomy's blind-different-axis check) is a later pass, noted at the end. + +* Network failure modes + +Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gateway/route → DNS → egress) plus the NetworkManager control plane that sits beside all of them. ~98 raw entries from the sweep; the obvious cross-layer duplicates (VPN route capture, PMTUD blackhole, broken-IPv6/happy-eyeballs, unmanaged device, dhcpcd-vs-NM, wifi powersave, connectivity-check false negative, MAC randomization, iwd/wpa_supplicant backend) are merged inline and cross-noted, leaving ~74 distinct root causes. + +** Link / radio / driver / firmware / rfkill + +- Missing iwlwifi firmware for a very recent Intel card (both). No wifi interface in =ip link=; dmesg "Direct firmware load for iwlwifi-… failed with error -2". The installed linux-firmware predates the card's blob. Fix: update/install linux-firmware (or linux-firmware-git), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=193570][arch 193570]] +- iwlwifi firmware-load regression after a kernel update (both). Wifi vanished right after an upgrade; dmesg shows iwlwifi failing to load firmware it loaded before. A kernel-side change broke firmware negotiation for that module. Fix: boot linux-lts or update to the fixed kernel, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=304848][arch 304848]] +- Realtek rtw89 in-tree module shadowed by an AUR DKMS driver (both). RTL8852BE wifi disappeared after an update; two conflicting modules present. A stale 8852be/rtw89-dkms-git overrides the in-tree rtw89_8852be. Fix: remove the AUR DKMS package, ensure linux-firmware is current, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=286109][arch 286109]] +- MediaTek mt7921e broken by a linux-firmware update (both). Wifi worked, then a pacman upgrade killed it; firmware load timeout / "hardware init failed" for WIFI_RAM_CODE_MT7961. A regression in the packaged blob fails to init the chip. Fix: downgrade/pin linux-firmware-mediatek to the last-good version, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=308360][arch 308360]] +- Broadcom BCM4360 needs the proprietary wl driver (both). "No wifi adapter"; lspci shows BCM4360 but no wlan interface. The open brcm/b43/bcma drivers claim it but can't drive it. Fix: install broadcom-wl-dkms, blacklist b43/bcma/brcmsmac/brcmfmac/ssb, rebuild initramfs, reboot. [[https://wiki.archlinux.org/title/Broadcom_wireless][archwiki broadcom]] +- broadcom-wl (non-DKMS) breaks after a kernel upgrade (both). Wifi gone after a kernel bump; wl won't load against the new kernel. The prebuilt module isn't rebuilt automatically. Fix: switch to broadcom-wl-dkms, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=305252][arch 305252]] +- USB wifi dongle needs an out-of-tree DKMS driver (both). An RTL8811AU/8812AU/8821AU dongle enumerates in lsusb but no wlan interface appears. No in-tree driver binds it. Fix: install the matching DKMS package with kernel headers, reboot. [[https://aur.archlinux.org/packages/rtl8812au-dkms-git][aur rtl8812au]] +- Wifi soft-blocked by rfkill (no). =rfkill list= reports "Soft blocked: yes"; the radio is off. Software (a prior toggle, a hotkey, NM state) set the soft block. Fix: =rfkill unblock wifi= (or =nmcli radio wifi on=). [[https://bbs.archlinux.org/viewtopic.php?id=236715][arch 236715]] +- Wifi hard-blocked by a physical switch / airplane / BIOS (physical). "Hard blocked: yes"; no software clears it. A kill-switch, Fn toggle, or BIOS setting physically disables the radio. Fix: flip the physical/Fn switch or enable the radio in BIOS. [[https://bbs.archlinux.org/viewtopic.php?id=207376][arch 207376]] +- rfkill stuck hard-blocked after suspend from competing WMI modules (both). Wifi comes back hard-blocked after resume and unblock won't clear it. Two vendor WMI modules (acer_wmi + hp_wmi) both claim the switch and wedge it. Fix: blacklist the wrong WMI module, rebuild initramfs, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=236657][arch 236657]] +- iwlwifi crashloops on resume, needs a module reload (yes). Wifi dead after suspend; dmesg shows microcode errors / firmware crashloop until the driver reloads. Firmware state isn't restored across resume. Fix: =modprobe -r iwlmvm iwlwifi && modprobe iwlwifi= (or a system-sleep hook). [[https://bbs.archlinux.org/viewtopic.php?id=293404][arch 293404]] +- ath11k (QCA6390/QCNFA765) firmware crash, dead until reload (yes). Wifi drops after resume or at random; dmesg "ath11k_pci … firmware crashed: MHI_CB_SYS_ERROR". The card's firmware faults and doesn't recover. Fix: =modprobe -r ath11k_pci && modprobe ath11k_pci=. [[https://bbs.archlinux.org/viewtopic.php?id=310363][arch 310363]] +- WiFi powersave drops the connection (yes; merges the auth-layer and NM-layer duplicates). Wifi connects then stalls/disconnects, worse on battery. Aggressive power management (driver =power_save= and/or NM =wifi.powersave=3=) breaks association with some APs. Fix: =iw dev wlan0 set power_save off= now; persist via NM =[connection] wifi.powersave=2= drop-in and/or =options iwlwifi power_save=0= in modprobe.d. [[https://bbs.archlinux.org/viewtopic.php?id=171342][arch 171342]] +- USB wifi dongle killed by USB autosuspend (yes). A working dongle drops out after idle and returns on replug; =power/control= reads "auto". Autosuspend powers the device down mid-use. Fix: a udev rule pinning =power/control=on= (or disable USB autosuspend in TLP). [[https://bbs.archlinux.org/viewtopic.php?id=184823][arch 184823]] +- RTL8125 ethernet flips from r8125 to in-tree r8169 after a kernel update (both). Wired network dies after an upgrade; the NIC binds r8169 and shows NO-CARRIER. A kernel update overrode the AUR r8125 module. Fix: blacklist r8169, install/force r8125-dkms, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=262120][arch 262120]] +- RTL8125 link downshifts / won't negotiate full speed (yes). The 2.5GbE NIC links but only at 1Gbps/100Mbps; ethtool shows the reduced speed. EEE / Giga-Lite negotiation downshifts the PHY. Fix: =ethtool --set-eee <if> eee off= (or force speed via =ethtool -s=), or swap cable/port. [[https://bbs.archlinux.org/viewtopic.php?id=264742][arch 264742]] +- Ethernet NO-CARRIER from an unplugged/faulty cable or dead switch port (physical). Interface UP but NO-CARRIER, no IP ever assigned. No link partner: cable unseated/broken or switch port down. Fix: reseat/replace the cable, try another port — no software repair until carrier returns. [[https://bbs.archlinux.org/viewtopic.php?id=268745][arch 268745]] +- Adapter present but left "unmanaged" by NetworkManager (no/yes; merges the NM-layer duplicate). =nmcli device= lists the device "unmanaged". Another daemon (iwd, systemd-networkd) owns it, or an =unmanaged-devices= rule / =NM_MANAGED=no= excludes it. Fix: =nmcli device set <dev> managed yes= (AUTO); remove the config rule or disable the rival manager to persist (PRIV). [[https://bbs.archlinux.org/viewtopic.php?id=291445][arch 291445]] + +** WiFi association / authentication + +- Wrong PSK, NM state 120 / secrets required (no; TERMINAL). "Secrets were required, but not provided"; journal "4-Way Handshake failed — pre-shared key may be incorrect". NM stored the wrong key — no repair fixes it; the user must re-enter. Fix: =nmcli con modify <name> wifi-sec.psk <key>= or re-enter in the picker. [[https://bbs.archlinux.org/viewtopic.php?id=306757][arch 306757]] +- Pure WPA3-SAE won't associate (no). "No PSK available for association" / "SME: Failed to set WPA key management" on a WPA3-only AP. The profile defaults to WPA-PSK key-mgmt with no PMF, which a pure-SAE AP rejects. Fix: =nmcli con modify <name> wifi-sec.key-mgmt sae 802-11-wireless-security.pmf 3=. [[https://bbs.archlinux.org/viewtopic.php?id=256573][arch 256573]] +- PMF mismatch on a WPA2/WPA3 transition AP (no). SAE assoc rejected on mixed-mode; connects to WPA2 but not the SAE path. Transition mode requires PMF-capable and rejects SAE when 802.11w isn't negotiated. Fix: match the AP's PMF — =nmcli con modify <name> 802-11-wireless-security.pmf 2=. [[https://hostap.shmoo.narkive.com/38Iowl2x/ieee-802-11w-support-in-wpa-supplicant-management-frame-protection-required-but-client-did-not][hostap 802.11w]] +- SAE H2E-only AP vs hunt-and-peck client (yes). WPA3 assoc fails "unspecified failure" on a Hash-to-Element-only AP. wpa_supplicant defaults to hunt-and-peck; the PWE method mismatches. Fix: =sae_pwe=2= in wpa_supplicant global config. [[https://wiki.archlinux.org/title/Wpa_supplicant][archwiki wpa_supplicant]] +- eduroam PEAP cert validation failed (no). "Certificate verification failed"; EAP drops. Missing/wrong CA cert, or NM ignoring system CA certs. Fix: set the correct CA cert / server-name in 802-1x (temporarily =system-ca-certs=false= to test). [[https://debianforum.de/forum/viewtopic.php?t=169560][debianforum 169560]] +- iwd eduroam bad_certificate (no). iwd refuses eduroam "bad_certificate" though it works elsewhere. iwd's EAP needs the server CA cert and a matching domain. Fix: add =EAP-PEAP-CACert=<ca.pem>= (and Domain/ServerDomainMask) to the iwd profile. [[https://bbs.archlinux.org/viewtopic.php?id=291921][arch 291921]] +- 802.1X won't save/verify the CA cert (no). Enterprise wifi only connects with "No CA certificate required" ticked. NM wants a PEM (not DER) bundle and mishandles the ca-cert setting. Fix: =nmcli con modify <name> 802-1x.ca-cert /etc/ssl/certs/ca-certificates.crt=. [[https://bugzilla.redhat.com/show_bug.cgi?id=1455833][redhat 1455833]] +- Wrong eduroam anonymous/outer identity (no). Auth rejected at the RADIUS server. The outer identity has the wrong realm, or the anonymous ID is reused as the inner identity. Fix: set =anonymous_identity="anonymous@realm"= and =identity="user@realm"= to the home realm. [[https://community.jisc.ac.uk/library/network-and-technology-service-docs/2021-04-advisory-android-11-configuration-issues][jisc eduroam]] +- Hidden SSID never connects (no). A saved non-broadcast SSID doesn't auto-connect; "No network with SSID found". NM won't send a directed probe unless the profile is flagged hidden. Fix: =nmcli con modify <name> wifi.hidden yes=. [[https://bbs.archlinux.org/viewtopic.php?id=39839][arch 39839]] +- Randomized MAC breaks captive portal / MAC allow-list (no; merges the NM-layer duplicate). Connects but the portal re-prompts every time, or a MAC-filtered router refuses the device. Per-connection random MAC changes each connect. Fix: =nmcli con modify <name> wifi.cloned-mac-address stable= (or =permanent=). [[https://fedoramagazine.org/randomize-mac-address-nm/][fedora NM MAC]] +- scan-rand-mac-address breaks scan/assoc on some drivers (yes). After an NM upgrade no networks appear, or assoc fails "Authentication timeout" (ath9k/ath9k_htc, some Realtek/Atheros). The driver can't handle the randomized scan MAC. Fix: =[device] wifi.scan-rand-mac-address=no= drop-in. [[https://github.com/qca/open-ath9k-htc-firmware/issues/132][qca ath9k-htc 132]] +- Wrong regulatory domain hides/blocks channels (yes). A 5GHz AP is invisible / "no suitable AP"; the card won't use the AP's channels. Regulatory domain unset or wrong, so the kernel forbids those frequencies. Fix: =iw reg set <CC>=, persist via =WIRELESS_REGDOM= (install wireless-regdb). [[https://bbs.archlinux.org/viewtopic.php?id=249979][arch 249979]] +- Won't roam, stuck on a weak AP (no). The client clings to a weak BSSID even with a stronger same-SSID AP in range. NM+wpa_supplicant roaming is conservative. Fix: bounce the connection, tune bgscan / lock a bssid, or try the iwd backend. [[https://bbs.archlinux.org/viewtopic.php?id=291466][arch 291466]] +- 4-way-handshake timeout / deauth loop, reason 15 (both). "4way_handshake_timeout" or repeated deauth (Reason 15) on iwlwifi/some drivers. An 802.11n handshake bug or AP port-security kicks the client mid-handshake. Fix: =options iwlwifi 11n_disable=1= (reboot), or try the other backend. [[https://bbs.archlinux.org/viewtopic.php?id=264378][arch 264378]] +- iwd associates but NM(wpa_supplicant) doesn't (yes; see also the NM backend entries). Standalone iwd connects but NM's wpa_supplicant path fails to associate on the same AP. Backend-specific supplicant behavior differs. Fix: =[device] wifi.backend=iwd= drop-in, disable standalone iwd.service, restart NM. [[https://bbs.archlinux.org/viewtopic.php?id=292603][arch 292603]] + +** IP / DHCP / gateway / routing / VPN policy + +- dhcpcd competing with NM's internal DHCP client (yes; merges the two DHCP-conflict duplicates). Interface never gets an IPv4 address; NM stuck "getting IP configuration". A standalone =dhcpcd.service= and NM's internal client fight over the interface. Fix: =systemctl disable --now dhcpcd= and let NM own DHCP. [[https://bbs.archlinux.org/viewtopic.php?id=165793][arch 165793]] +- dhcpcd 9 backend breaks NM IPv4 (yes). No IPv4 after a dhcpcd upgrade; IPv6 may still work. NM's dhcpcd backend was incompatible with dhcpcd 9's changed behavior. Fix: =dhcp=internal= in NetworkManager.conf (or NM ≥1.30.4-3), restart NM. [[https://bugs.archlinux.org/task/66231][arch fs66231]] +- APIPA / link-local only, no DHCP answer (no). The interface self-assigns 169.254.x.x; only local-link hosts reachable. No DHCP server answered in the timeout. Fix: re-trigger DHCP — =nmcli device reapply <dev>=; a recurring case is a server-side fault. [[https://oneuptime.com/blog/post/2026-03-20-fix-apipa-169-254-address/view][apipa 169.254]] +- Default route never installed by the connection (yes). Has an IP and pings the LAN, but no default route. The profile has =ipv4.never-default yes= ("use only for resources on its network"). Fix: =nmcli con modify <name> ipv4.never-default no=, up. [[https://bbs.archlinux.org/viewtopic.php?id=84453][arch 84453]] +- DHCP gateway in a different subnet than the lease (yes). IP assigned but no default route; "SIOCADDRT: No such process". The DHCP-offered gateway is off the assigned prefix, so the on-link route is refused. Fix: add the on-link route then default (=ip route add <gw> dev <iface>; ip route add default via <gw>=), or fix the DHCP scope. [[https://bbs.archlinux.org/viewtopic.php?id=72685][arch 72685]] +- Duplicate IPv4 on the LAN, address skipped by ACD (yes). Connection comes up but no usable address; journal logs the conflicting MAC. NM's RFC 5227 conflict detection found the IP already in use. Fix: change the offending host / DHCP reservation, or set a distinct static IP. [[https://bbs.archlinux.org/viewtopic.php?id=287424][arch 287424]] +- IPv6 RA not received, no SLAAC address or route (yes). No IPv6 address/route; interface hangs "configuring". =accept_ra=0=, forwarding enabled without =accept_ra=2=, or an upstream RA-guard drops ICMPv6 type 134. Fix: =sysctl net.ipv6.conf.<iface>.accept_ra=2= (or unblock RA at the router). [[https://oneuptime.com/blog/post/2026-03-20-ipv6-ra-not-received/view][ipv6 ra]] +- DHCPv6-only network, address but no route (yes). An IPv6 address via stateful DHCPv6, but no default route. Routing lives in the RA, not DHCPv6. Fix: ensure the router sends an RA with the M flag and let NM process both (=ipv6.method auto=). [[https://bbs.archlinux.org/viewtopic.php?id=273336][arch 273336]] +- Happy-eyeballs stall on broken IPv6 (no; the routing angle — see also the DNS AAAA and egress v6 entries). Pages load with a visible hang; IPv6 is present but black-holed. Dual-stack apps race IPv6 first and wait to fall back. Fix: =nmcli con modify <name> ipv6.method disabled= (or fix the upstream v6 path). [[https://oneuptime.com/blog/post/2026-03-20-ipv6-happy-eyeballs-issues/view][happy-eyeballs]] +- WireGuard AllowedIPs=0.0.0.0/0 strands the host when the tunnel drops (no). Everything offline after the VPN drops; ping and DNS fail. Full-tunnel installed a default route through wg0 with no live peer. Fix: bring the tunnel down (=wg-quick down <iface>= / =nmcli con down <vpn>=) to restore the physical route. [[https://discourse.coreelec.org/t/wireguard-client-not-connected-when-allowedips-0-0-0-0-0/18187][wg allowedips]] +- WireGuard split-tunnel misconfig captures the LAN (yes). VPN connects but LAN devices/printers become unreachable. =AllowedIPs=0.0.0.0/0= used where a split tunnel was intended. Fix: set AllowedIPs to only the intended subnets, re-up. [[https://oneuptime.com/blog/post/2026-03-20-wireguard-allowedips-ipv4-split-tunneling/view][wg split-tunnel]] +- OpenVPN redirect-gateway route left behind after an abnormal disconnect (yes). No internet/DNS after the tunnel dies; can't even reconnect. redirect-gateway rewrote the default route and a hard link-down never restored it. Fix: =ip route del default dev tun0; systemctl restart openvpn-client@<name>= (use =def1= going forward). [[https://forums.openvpn.net/viewtopic.php?t=22572][openvpn redirect-gw]] +- Tailscale exit node captures the default route (no). Internet dies after selecting an exit node; tailnet peers still reachable. The exit-node route conflicts with the working one, or policy lacks =autogroup:internet=. Fix: =tailscale set --exit-node== to drop back to the local default route. [[https://github.com/tailscale/tailscale/issues/2654][tailscale exitnode]] +- Docker bridge subnet collides with a real external subnet (yes). A specific external host (172.17.x.x) is unreachable while the rest of the internet works. docker0 claimed 172.17.0.0/16. Fix: set ="bip": "172.26.0.1/16"= in daemon.json, prune, restart docker. [[https://github.com/moby/moby/issues/36619][moby 36619]] +- libvirt virbr0 subnet collides with the real network (yes). Host/VMs can only reach the bridge/host, not the WAN. virbr0 came up on a range overlapping the LAN. Fix: re-address the virtual net via =virsh net-edit default=, destroy/start it. [[https://bugzilla.redhat.com/show_bug.cgi?id=811967][rhbz 811967]] +- Wifi + ethernet up with conflicting metrics, traffic exits the dead link (yes). Intermittent/no connectivity with two links up; egress uses the wrong interface. Both installed a default route and the lower metric wins even when it can't reach the internet. Fix: raise the bad link's metric — =nmcli con modify <name> ipv4.route-metric <higher>=. [[https://www.baeldung.com/linux/change-network-routing-metric][route-metric]] + +** DNS / resolver / DoT / DNSSEC + +- Dangling resolv.conf symlink after resolved is disabled/not started (yes). Can ping 1.1.1.1 but nothing resolves; =/etc/resolv.conf= symlinks to a stub file that doesn't exist. The stub only exists while systemd-resolved runs. Fix: =systemctl enable --now systemd-resolved= (or repoint the link at a real file). [[https://github.com/systemd/systemd/issues/5395][systemd 5395]] +- resolv.conf not pointing at the stub while resolved runs (yes). Names fail though resolved is up; resolv.conf is a stale static file. An installer/upgrade replaced it. Fix: =ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf=, verify with =resolvectl status=. [[https://wiki.archlinux.org/title/Systemd-resolved][archwiki resolved]] +- systemd-resolved stub 127.0.0.53 unreachable / port 53 conflict (yes). resolv.conf says 127.0.0.53 but every query is connection-refused. A second resolver (dnsmasq, bind) holds port 53, so the stub never binds. Fix: stop the conflicting resolver or set DNSStubListener, restart resolved. [[https://github.com/systemd/systemd/issues/10298][systemd 10298]] +- NetworkManager not integrated with resolved (yes). resolved is up but knows no upstream servers; NM writes its own resolv.conf. =dns=systemd-resolved= not set, so DHCP servers never reach resolved. Fix: add =dns=systemd-resolved= under [main] in NetworkManager.conf, restart NM. [[https://wiki.archlinux.org/title/Systemd-resolved][archwiki resolved]] +- DNSSEC=yes rejects a non-compliant venue resolver (yes). Pages won't load on hotel/ISP wifi though ping works; "DNSSEC validation failed". resolved rejects the venue resolver that mangles DNSSEC records. Fix: =DNSSEC=allow-downgrade= (or no) in resolved.conf, restart resolved. [[https://bbs.archlinux.org/viewtopic.php?id=240427][arch 240427]] +- DNSSEC "no-signature" verdict sticks after a network change (yes). Resolution dies sporadically after reconnecting; resolved reuses a cached bogus/downgrade verdict from the previous network. Fix: =resolvectl reset-server-features= (and flush), or DNSSEC=allow-downgrade. [[https://github.com/systemd/systemd/issues/12388][systemd 12388]] +- DoT strict mode with port 853 blocked (yes). Everything stalls; DNSOverTLS=yes forbids fallback and the network drops 853. Fix: =DNSOverTLS=opportunistic= (or no) in resolved.conf, restart resolved. [[https://wiki.archlinux.org/title/Systemd-resolved][archwiki resolved]] +- DoT strict to an unreachable / cert-mismatched server (yes). "Failed to invoke gnutls_handshake: Error in the certificate verification"; the pinned DoT server is down or MITM'd. Strict mode plus IP#hostname pinning rejects the cert and won't fall back. Fix: correct the server/#hostname, switch provider, or relax to opportunistic, restart resolved. [[https://github.com/systemd/systemd/issues/16531][systemd 16531]] +- DHCP handed a broken resolver (no). Nothing resolves on one network though egress works; the DHCP DNS server is dead/filtering. Fix: =nmcli con mod <con> ipv4.ignore-auto-dns yes ipv4.dns "1.1.1.1 9.9.9.9"=, up. [[http://adam.younglogic.com/2019/05/using-nmcli-to-set-nameservers/][nmcli dns]] +- Manually-pinned 8.8.8.8 blocked by the venue (no). A hard-set public DNS resolves nothing though 1.1.1.1 pings; the network blocks external port 53. Fix: drop the manual pin and use DHCP DNS (=ipv4.ignore-auto-dns no=, clear =ipv4.dns=), reconnect. [[https://forum.netgate.com/topic/187060/port-53-dns][netgate port 53]] +- Stale poisoned cache after captive-portal login (yes). First lookups after the portal keep returning the portal/blocked address. Pre-login hijacked answers were cached. Fix: =resolvectl flush-caches=. [[https://oneuptime.com/blog/post/2026-03-20-flush-dns-cache-linux-macos-windows/view][flush-caches]] +- VPN split-DNS not applied / leaks to the ISP resolver (no). Internal/corp names fail (or a censored ISP resolver still answers) after connecting the VPN. NM/resolved didn't install the VPN link's domain as a routing domain. Fix: =resolvectl domain <vpn-link> '~corp.example'= and =resolvectl default-route <vpn-link> false=. [[https://blogs.gnome.org/mcatanzaro/2020/12/17/understanding-systemd-resolved-split-dns-and-vpn-configuration/][catanzaro split-dns]] +- IPv6 AAAA lookups stall resolution (yes; the DNS angle of the broken-v6 family). Every name takes ~5s then works; A and AAAA go in parallel and AAAA is never answered. Broken/half-configured IPv6 swallows the AAAA query. Fix: fix or disable IPv6 on the link, or =options single-request timeout:1=. [[https://bbs.archlinux.org/viewtopic.php?id=262926][arch 262926]] +- Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] +- nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] +- Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] + +** Egress / captive portal / MTU / proxy / clock / upstream + +- Captive portal held state, no auto-popup (no). Wifi associates and gets a lease but every site fails; no login window appears. Without a DE NetworkManager integration or a dispatcher script, NM detects the PORTAL state but nothing opens the login page. Fix: open a plain-HTTP probe (=xdg-open http://neverssl.com=) or the URL NM reports; add an nm-dispatcher script to auto-launch. [[https://wiki.archlinux.org/title/NetworkManager#Captive_portals][archwiki captive portals]] +- Connectivity check over HTTPS never triggers the portal redirect (no). Behind a portal the login popup never fires. NM's connectivity URI was probed over HTTPS, which the portal can't transparently redirect. Fix: point NM's check at an HTTP URI (=20-connectivity.conf= =uri=http://…=) and open a plain-HTTP page. [[https://bugzilla.redhat.com/show_bug.cgi?id=1252839][redhat 1252839]] +- Custom DNS / DoT / DNSSEC hides the captive portal (yes). No portal page ever resolves and the machine stays cut off. resolved pinned to 1.1.1.1 with DoT/DNSSEC bypasses the portal's DNS hijack, so lookups just fail. Fix: drop DoT/DNSSEC for the captive network (=resolvectl dnsovertls <iface> no; resolvectl dnssec <iface> no=), log in, restore. [[https://github.com/systemd/systemd/issues/27485][systemd 27485]] +- HSTS blocks the portal redirect in the browser (no). The login page won't load; the browser shows a cert/HSTS error instead of the form. The first site was HSTS-pinned, so the browser refuses the portal's HTTP redirect. Fix: browse to a non-HSTS plain-HTTP address (http://neverssl.com, http://captive.apple.com). [[https://community.ui.com/questions/Solution-for-HSTS-issues-with-captive-portal/17b033e7-3dfe-4830-af8f-bf6ead23d8b0][ui hsts portal]] +- PMTUD blackhole from dropped ICMP (no; merges the IP-layer duplicate). Small requests and DNS work, pages start then stall, large downloads/SSH hang. A middlebox drops ICMP "fragmentation needed", so PMTUD never learns the real MTU and oversized packets vanish. Fix: lower the interface MTU (=nmcli con modify <con> 802-11-wireless.mtu 1400=), or MSS-clamp. [[https://ipfyi.com/scenario/path-mtu-blackhole/][pmtud blackhole]] +- PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] +- Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] +- Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] +- VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] +- IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify <con> ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] +- False "limited connectivity" from a local DNS/VPN service (no; merges the NM-layer connectivity-check duplicate). The applet shows "limited/no internet" and apps that gate on NM refuse to work, yet browsing functions. NM's check to its connectivity URI fails — a local Pi-hole/VPN blocks the probe host, a false negative. Fix: point =20-connectivity.conf= at a reachable 204 endpoint, or =[connectivity] enabled=false=, reload NM. [[https://bbs.archlinux.org/viewtopic.php?id=272292][arch nm limited]] +- ISP outage or modem stuck in a bad state — not locally fixable (physical). Everything downstream is "connected, no internet"; the router's WAN IP is 0.0.0.0. The upstream link is down or the modem latched into a bad state. The doctor must recognize this and STOP. Fix: power-cycle modem then router; check ISP status / contact the provider. [[https://whizz-tech.com/support/wifi-router-fixes/wifi-stops-working-after-power-outage/][modem bad state]] +- Specific outbound port blocked while web works (no). Browsing is fine but email won't send or a service times out. The network/ISP blocks that port (SMTP 25, some VPN ports). A partial block, not a full outage. Fix: use an allowed alternate (submission 587, VPN over 443/TCP), or switch networks — the policy isn't host-fixable. [[https://help.dreamhost.com/hc/en-us/articles/217071167-Port-25-Blocking][port 25 blocking]] + +** NetworkManager control plane / backends / config + +- NetworkManager service masked or failed to start (yes). No networking at all; "Unit NetworkManager.service is masked" or the unit dies immediately. The unit was masked, or an invalid unit file / malformed NetworkManager.conf blocks startup. Fix: =systemctl unmask NetworkManager && systemctl start NetworkManager=; reinstall if the unit is corrupt, validate the conf. [[https://bbs.archlinux.org/viewtopic.php?id=193036][arch 193036]] +- NetworkManager segfaults and restart-loops (yes). Connectivity drops intermittently; journal "Failed with result 'core-dump'" and repeated respawns. A bug in the running NM build crashes the daemon. Fix: =coredumpctl info NetworkManager=, then update/downgrade the package, restart NM. [[https://bbs.archlinux.org/viewtopic.php?id=236562][arch 236562]] +- iwd and wpa_supplicant both active, backends fight (both). Wifi scans but never associates, or associates then drops; both daemons drive one radio. Fix: pick one backend, =systemctl disable --now= the other, set =[device] wifi.backend==, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=292603][arch 292603]] +- wifi.backend=iwd configured but iwd not installed/running (both). No wifi device appears at all; NM acts as if there's no radio. NM is told to use iwd but the package isn't installed or its service can't start. Fix: =pacman -S iwd= (or remove the drop-in to fall back), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=305442][arch 305442]] +- systemd-networkd and NetworkManager both managing the link (both). The interface gets an IP then loses it, or flaps; two managers own one NIC. Fix: =systemctl disable --now systemd-networkd systemd-networkd.socket= (mask if dbus-activated), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=184164][arch 184164]] +- Keyfile permissions void a saved connection (yes). A saved network shows in nmcli but never activates; journal marks it insecure and skips it. The =.nmconnection= isn't 600 root-owned. Fix: =chmod 600= and =chown root:root= the file in system-connections, =nmcli connection reload=. [[https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/assembly_networkmanager-connection-profiles-in-keyfile-format_configuring-and-managing-networking][rhel keyfile]] +- Autoconnect disabled on the profile (no). The network works when activated by hand but never comes up on boot/resume. =connection.autoconnect=no=. Fix: =nmcli connection modify <name> connection.autoconnect yes= (user-scope). [[https://bbs.archlinux.org/viewtopic.php?id=281611][arch 281611]] +- Duplicate/conflicting profiles for one SSID (no). Two entries for the same network; the wrong one activates. Leftover =.nmconnection= files created a second profile with conflicting settings. Fix: =nmcli connection delete <uuid>= the stale one, or raise the good one's =autoconnect-priority=. [[https://bbs.archlinux.org/viewtopic.php?id=274552][arch 274552]] +- A dispatcher script errors and blocks connectivity (yes). Network events don't complete, or a connectivity hook silently doesn't run. A script in dispatcher.d fails on a permission/ownership problem or loops. Fix: make it root-owned and non-group-writable or remove it; debug via =journalctl -u NetworkManager-dispatcher=. [[https://man.archlinux.org/man/NetworkManager-dispatcher.8.en][arch dispatcher]] +- Polkit won't authorize a non-root user to control networking (no). =nmcli= up/modify fails "Not authorized to control networking". No polkit agent, or the login has no active seat session. Fix: start a polkit agent and ensure a proper loginctl session (relogin via the display manager), or add a polkit rule. [[https://bbs.archlinux.org/viewtopic.php?id=141465][arch 141465]] +- No reconnect after suspend/resume (yes). After waking, wifi/wired stays down until the service is kicked. On resume NM doesn't re-establish the link. Fix: =systemctl restart NetworkManager= (the doctor's nm-restart); durable fix is a system-sleep hook. [[https://bbs.archlinux.org/viewtopic.php?id=234725][arch 234725]] +- NetworkManager-wait-online hangs boot (both). Boot stalls ~90s on NetworkManager-wait-online; the desktop is late. The service waits for connectivity that never arrives (unplugged NIC, non-autoconnect port). Fix: =systemctl disable NetworkManager-wait-online.service= (or drop the unused profile); effective next boot. [[https://bbs.archlinux.org/viewtopic.php?id=298817][arch 298817]] + +* Bluetooth failure modes + +Five layers, mirroring the bt doctor's chain (adapter present → rfkill → bluetooth.service → adapter powered → per-device → audio profile). ~70 raw entries from the sweep; hardware-block duplicates (a Bluetooth hard-block appears in every one of the first three layers) and the pairing/connection overlaps are merged inline, leaving ~55 distinct root causes. + +** Adapter / hardware / driver / firmware + +- btusb module not auto-loaded (yes). lsusb shows the adapter but no hci0, =/sys/class/bluetooth= empty, "No default controller available". btusb didn't load at boot (missing modules-load entry or a blacklist). Fix: =modprobe btusb=, persist via =/etc/modules-load.d/bluetooth.conf=. [[https://bbs.archlinux.org/viewtopic.php?id=267428][arch 267428]] +- Intel ibt-*.sfi firmware missing/mismatched (both). No controller; dmesg "Direct firmware load for intel/ibt-NN-N.sfi failed with error -2". linux-firmware lacks the exact .sfi/.ddc the controller requests. Fix: update linux-firmware (or symlink the requested name), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=302314][arch 302314]] +- Intel "Failed to send firmware data" during load (both). Adapter present then dies; "hci0: Failed to send firmware data (-38)/(-110)". Firmware transfer aborts (USB/init timing, often kernel-specific). Fix: =options btusb reset=1= and reload, or downgrade the kernel, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=248784][arch 248784]] +- Intel AX200/AX210 controller lost after a kernel update (both). WiFi fine, BT gone, "No default controller available" right after an upgrade. A kernel/firmware regression stops the BT half enumerating. Fix: boot linux-lts or downgrade until fixed, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=282611][arch 282611]] +- Intel broken-initial-NCMD init hang (both). "hci0: command tx timeout" during setup; the controller never comes up. Some Intel controllers need the BTUSB_INTEL_BROKEN_INITIAL_NCMD quirk absent on older kernels. Fix: run a kernel new enough to carry the quirk, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=226312][arch 226312]] +- MediaTek MT7921/MT7961 firmware filename mismatch (both). No adapter; "Direct firmware load for mediatek/BT_RAM_CODE_MT7961_1a_2_hdr.bin failed with error -2". linux-firmware ships the blob compressed or under a different name. Fix: decompress/rename to the requested file, reload btusb, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=296026][arch 296026]] +- MediaTek MT7922 btmtk regression after a kernel bump (both). BT stopped initializing after a specific kernel; WiFi unaffected. A regression in the mt7921e/btmtk path. Fix: downgrade to the last working kernel (or linux-lts), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=313561][arch 313561]] +- MediaTek MT7922 stuck in a residual power state (physical). "No Bluetooth adapters found" persists across warm reboots; only a full power-off clears it. The controller latched in a bad state a soft reboot doesn't reset. Fix: shut down, pull power ~10s, cold boot. [[https://bbs.archlinux.org/viewtopic.php?id=308169][arch 308169]] +- MediaTek MT7920/MT7922 bogus firmware in linux-firmware (both). WiFi works, BT half fails "Failed to set up firmware" after a linux-firmware update. A shipped BT blob is broken in the current package. Fix: downgrade linux-firmware-mediatek to the prior version, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=311544][arch 311544]] +- MediaTek MT7925 killed by aggressive USB autosuspend (both). BT flaky/absent on a recent kernel; "Opcode 0x0c03 failed: -16". Kernel power-saving suspends the controller. Fix: =usbcore.autosuspend=-1 btusb.enable_autosuspend=n= on the kernel cmdline, reboot. [[https://bbs.archlinux.org/viewtopic.php?id=306366][arch 306366]] +- Realtek RTL8761B(U) firmware name mismatch (yes). USB dongle dead; driver requests rtl_bt/rtl8761bu_fw.bin but firmware ships rtl8761b_fw.bin. Fix: symlink to the bu names in /lib/firmware/rtl_bt (or update linux-firmware), replug — no reboot. [[https://bbs.archlinux.org/viewtopic.php?id=309671][arch 309671]] +- Realtek RTL8852BU/newer dongle needs an out-of-tree driver (both). A recent combo dongle isn't recognized at all; no in-kernel support. Fix: build the vendor DKMS driver (lwfinger/rtl8852bu), reboot. [[https://github.com/lwfinger/rtl8852bu][rtl8852bu]] +- Fake/clone CSR dongle resets and times out (yes). A cheap 0a12:0001 "CSR" dongle works briefly then "command tx timeout"; "Unbranded CSR clone detected". A counterfeit controller mishandles CSR reset. Fix: rely on the kernel clone quirks, or =options btusb reset=1 enable_autosuspend=0=, reload btusb. [[https://bbs.archlinux.org/viewtopic.php?id=272230][arch 272230]] +- Qualcomm WCN6855/QCA6390 version read fails (both). No BT; "Reading QCA version information failed (-110)" via ath11k. Missing/mismatched QCA firmware or a btqca init-timing bug. Fix: update linux-firmware and run a newer kernel (or reload ath11k_pci/btusb), reboot. [[https://bbs.archlinux.org/viewtopic.php?id=284506][arch 284506]] +- Broadcom BCM .hcd patch file missing (both). Adapter enumerates then stalls; "hci0: BCM: Patch brcm/BCMxxxxx-….hcd not found". The .hcd firmware isn't in linux-firmware. Fix: install broadcom-bt-firmware (AUR) or drop the correct .hcd into /lib/firmware/brcm, reboot. [[https://github.com/winterheart/broadcom-bt-firmware/issues/30][broadcom-bt-fw 30]] +- USB autosuspend powers the adapter off after suspend/idle (both; the hardware angle — see also the resume entry in the rfkill layer). BT works at boot, then hci0 vanishes after a suspend/resume or idle; only a full shutdown restores it. btusb autosuspend cuts power and the controller doesn't recover. Fix: =btusb.enable_autosuspend=0= on the kernel cmdline (or exclude the device in TLP), reboot. [[https://forum.manjaro.org/t/bluetooth-adapter-disappears-after-suspend-only-shutdown-can-fix/157919][manjaro 157919]] +- Bluetooth hard-blocked by a BIOS/hardware switch (physical; the canonical hard-block — appears identically across the rfkill layer). No adapter and =rfkill list= shows Bluetooth "Hard blocked: yes" (or hci0 absent) after a wireless switch/Fn toggle or a BIOS setting. Fix: flip the physical switch / Fn key, or enable Bluetooth in BIOS/UEFI — no software can clear a hard block. [[https://github.com/systemd/systemd/issues/4117][systemd 4117]] + +** rfkill / power / service + +- Soft-blocked adapter needs manual unblock (no). =rfkill list= shows Bluetooth "Soft blocked: yes"; bluetoothctl "Failed to set power on: org.bluez.Error.Failed". A prior software toggle left the soft-block set and bluetoothd won't power a blocked adapter. Fix: =rfkill unblock bluetooth= (then =bluetoothctl power on=). [[https://bbs.archlinux.org/viewtopic.php?id=169530][arch 169530]] +- AutoEnable off leaves BT powered down every boot (yes). Adapter present and service running, but "Powered: no" at every login. bluez main.conf has AutoEnable=false (also the static default when main.conf is absent), so bluetoothd never powers the controller. Fix: =AutoEnable=true= under [Policy] in main.conf, restart bluetooth. [[https://github.com/bluez/bluez/issues/886][bluez 886]] +- bluetooth.service masked (yes). =systemctl start bluetooth= returns "Unit bluetooth.service is masked". The unit was masked by an earlier disable or a tuning guide. Fix: =systemctl unmask bluetooth.service && systemctl enable --now bluetooth=. [[https://bbs.archlinux.org/viewtopic.php?id=220081][arch 220081]] +- bluetooth.service not enabled at boot (yes). BT works after a manual start but is "inactive (dead)" on every fresh boot. On Arch the service ships disabled, so nothing starts bluetoothd at boot. Fix: =systemctl enable --now bluetooth.service=. [[https://bbs.archlinux.org/viewtopic.php?id=287428][arch 287428]] +- D-Bus policy blocks bluetoothd from owning org.bluez (yes). "D-Bus setup failed: Connection is not allowed to own the service 'org.bluez'"; bluetoothctl can't reach bluez. A missing/broken =/usr/share/dbus-1/system.d/bluetooth.conf= policy (partial install/upgrade). Fix: reinstall bluez to restore the policy, restart dbus and bluetooth. [[https://bbs.archlinux.org/viewtopic.php?id=283337][arch 283337]] +- bluez upgrade leaves a bad unit-file setting (yes). After a bluez update "Unit bluetooth.service has a bad unit file setting". A changed ExecStart path or a removed option in the shipped unit breaks parsing until daemon-reload. Fix: reconcile the .pacnew, correct ExecStart to =/usr/lib/bluetooth/bluetoothd=, =daemon-reload && restart bluetooth=. [[https://github.com/bluez/bluez/issues/736][bluez 736]] +- Airplane mode / physical switch hard-blocks BT (physical; the rfkill-layer view of the shared hard-block). =rfkill list= "Hard blocked: yes"; software toggles and unblock do nothing. A physical/firmware kill switch, Fn key, or BIOS airplane state holds the block. Fix: turn airplane mode off / flip the switch, or enable BT in BIOS. [[https://bugzilla.redhat.com/show_bug.cgi?id=713006][rh 713006]] +- WiFi rfkill toggle also kills BT on a combo chip (no). Turning WiFi off drops Bluetooth too, and both come up soft-blocked at boot. A shared WiFi/BT combo exposes a coupled rfkill so the wlan block cascades. Fix: =rfkill unblock bluetooth= after the toggle; unblock all at boot. [[https://forum.artixlinux.org/index.php/topic,5062.0.html][artix 5062]] +- TLP disables Bluetooth on startup (yes). BT is soft-blocked at every boot on a TLP laptop though the service is fine. TLP's =DEVICES_TO_DISABLE_ON_STARTUP= includes bluetooth (or RESTORE_DEVICE_STATE restores off). Fix: remove bluetooth from that list in =/etc/tlp.conf=; =rfkill unblock bluetooth= now. [[https://dev.to/sharafat/disable-bluetooth-at-startup-in-tlp-2k8d][tlp startup]] +- systemd-rfkill restores a stale soft-block at boot (yes). BT was on before shutdown but boots soft-blocked, or airplane-off never sticks. systemd-rfkill saved a blocked state to =/var/lib/systemd/rfkill= and reapplies it (a hotplug race can overwrite the good state). Fix: =rfkill unblock bluetooth=; clear the saved state (or mask systemd-rfkill if TLP owns radio state). [[https://github.com/systemd/systemd/issues/42148][systemd 42148]] +- rfkill block puts bluetoothd into a 100% CPU loop (yes). After =rfkill block bluetooth= (an airplane toggle) bluetoothd spins at 100% CPU and the adapter is unresponsive even after unblock. A bluez rfkill-handling regression busy-loops. Fix: =systemctl restart bluetooth= after unblocking; upgrade/downgrade bluez past the affected version. [[https://github.com/bluez/bluez/issues/785][bluez 785]] +- Adapter powers back off after a desktop-driven power-off (no). =bluetoothctl power on= fails specifically after BT was switched off in GNOME/desktop settings; only the desktop toggle restores it. The desktop's saved off-state re-blocks the adapter under the CLI. Fix: re-enable from the desktop toggle, or =rfkill unblock bluetooth= and power on. [[https://discourse.gnome.org/t/cannot-power-on-bluetooth-from-bluetoothctl-if-it-was-powered-off-from-the-gnome-settings/24782][gnome 24782]] +- BT soft-blocked/unresponsive on resume, needs a module reload (yes; distinct from the hardware-autosuspend case — here power state is stuck, not cut). After suspend/resume the adapter is lost or soft-blocked; toggling "shows" enabled but does nothing and restarting the service doesn't help. btusb fails to re-init on resume, leaving a stuck rfkill/power state. Fix: =rmmod btusb && modprobe btusb= (then restart bluetooth); automate via a post-resume sleep hook. [[https://forums.opensuse.org/t/btusb-broken-after-resuming-from-sleep/133499][opensuse 133499]] +- Wrong/missing default controller with multiple adapters (no; see also the multi-controller pairing case). "No default controller available" or commands hit the wrong hciN when a dongle and an onboard radio are both present. bluez picked a different default than the user wants. Fix: =bluetoothctl select <MAC>= then power on; pin it if needed. [[https://bbs.archlinux.org/viewtopic.php?id=286766][arch 286766]] + +** Pairing / bonding + +- Generic AuthenticationFailed on pair (no). "Failed to pair: org.bluez.Error.AuthenticationFailed" on an audio device. A half-written or contested bond from a prior attempt makes the device reject the new authentication. Fix: bluetoothctl disconnect, untrust, remove <mac>, restart bluetooth, pair from a clean scan; factory-reset the device if it persists. [[https://bbs.archlinux.org/viewtopic.php?id=267381][arch 267381]] +- Paired but not trusted, no auto-reconnect (no). Device pairs but never reconnects on its own after a reboot/power cycle; =info= shows "Trusted: no". bluez won't auto-accept a connection from an untrusted bond. Fix: =bluetoothctl trust <mac>=. [[https://techwiser.com/fix-bluetooth-device-doesnt-auto-connect-in-linux/][techwiser autoconnect]] +- Stale link key after a dual-boot re-pair (yes). A device paired in Windows no longer connects in Linux with AuthenticationFailed. A device stores one link key per host; re-pairing under the other OS overwrote it. Fix: =bluetoothctl remove <mac>= and pair again, or sync the keys between OSes (bt-dualboot / edit the LinkKey under /var/lib/bluetooth). [[https://armujahid.me/blog/dual-boot-bluetooth-pairing/][armujahid dualboot]] +- No default pairing agent registered (no). Pairing a PIN/keyboard device silently fails or the passkey prompt never appears (esp. headless). bluetoothctl's built-in agent is unregistered once it exits, so nothing answers the request. Fix: =agent on= then =default-agent= (or a persistent bt-agent / NoInputNoOutput agent for unattended Just Works). [[https://technotes.kynetics.com/2018/pairing-agents-bluez/][kynetics agents]] +- BLE device never appears in scan (no). A BLE peripheral shows in other OSes but never in =bluetoothctl scan=, while BR/EDR devices scan fine. The scan defaults to BR/EDR-only or LE isn't enabled. Fix: =menu scan / transport le= (or =btmgmt le on=); last resort =ControllerMode = le= in main.conf. [[https://bbs.archlinux.org/viewtopic.php?id=281064][arch 281064]] +- Corrupt device cache survives remove (yes). Repeated pairing failures for one device; =remove= reports success but the cache stays and the bad state returns. bluez doesn't fully purge =/var/lib/bluetooth/<adapter>/<mac>=. Fix: stop bluetooth, =rm -rf= the stale device dir, start bluetooth, pair fresh. [[https://github.com/bluez/bluez/issues/191][bluez 191]] +- ConnectionAttemptFailed / br-connection-page-timeout (yes). Pairing or first connect fails "Page Timeout" / "br-connection-page-timeout", often after a bluez upgrade. The daemon's baseband page times out; a version regression triggered it. Fix: =systemctl restart bluetooth= and retry; downgrade bluez if it recurs after an update. [[https://github.com/bluez/bluez/issues/230][bluez 230]] +- Re-pair loop: pairs then instantly disconnects (yes). Device pairs, drops within seconds, and re-pairing only works until the next attempt. A corrupt in-memory bond/device DB wedges the daemon. Fix: =rm -rf /var/lib/bluetooth/*= (or the device dir) and =systemctl restart bluetooth=, then pair once cleanly. [[https://bbs.archlinux.org/viewtopic.php?id=267233][arch 267233]] +- Headset un-pairs itself on disconnect (yes). A headset loses its bond every time it disconnects. The device sends an HID virtual_cable_unplug on disconnect and bluez treats it as an un-pair. Fix: pair again; no clean user setting — avoid fully powering the device off; restart bluetooth if the bond is gone. [[https://github.com/bluez/bluez/issues/2048][bluez 2048]] +- BLE address rotates every session (physical). A BLE device pairs once but can't reconnect; its address differs each power cycle. It uses a random/resolvable-private address, so a bond keyed to the old address won't match unless IRK resolution works. Fix: pair so the IRK is exchanged (bond, not just connect), or disable address privacy on the device; re-pair to the current identity. [[https://github.com/hbldh/bleak/issues/363][bleak 363]] +- Legacy keyboard needs the PIN typed on the device (physical). A legacy BT keyboard fails with an authentication timeout during pairing. It uses legacy PIN/passkey-entry: bluez shows a passkey that must be typed on the keyboard itself. Fix: register a KeyboardDisplay agent, type the displayed passkey on the keyboard, press Enter within the timeout. [[https://bbs.archlinux.org/viewtopic.php?id=174239][arch 174239]] +- Too many bonded devices, keys lost (yes). With many devices paired, after a reboot or dongle re-insert none connect. Past ~14 bonds bluez fails to restore the LTKs. Fix: =bluetoothctl remove= enough devices to drop below the limit, restart bluetooth. [[https://github.com/bluez/bluez/issues/201][bluez 201]] +- Dual-mode device pairs on the wrong transport (yes). A dual-mode device (AirPods) refuses to pair/connect because bluez tries BR/EDR when the device needs LE, with no per-device override. Fix: =ControllerMode = bredr= (or le) in main.conf [General], restart, pair, then restore =dual=. [[https://github.com/bluez/bluez/issues/514][bluez 514]] +- Adapter Pairable/Discoverable off (no). Scan finds nothing or the controller rejects pairing. The adapter has =Pairable: no= / =Discoverable: no=. Fix: =power on=, =pairable on=, =scan on= (or set the main.conf defaults). [[https://wiki.archlinux.org/title/Bluetooth][archwiki bt]] +- Target device not in pairing mode (physical). A peripheral never appears in scan though the adapter works. The device is powered but not advertising. Fix: hold the pairing button until the LED signals, re-run =scan on=. [[https://bbs.archlinux.org/viewtopic.php?id=261540][arch 261540]] +- Paired without a bond via blueman (no). A gamepad/mouse paired through blueman connects but is unusable, while bluetoothctl works. blueman completed pairing without establishing bonding, so no usable link key was stored. Fix: remove the device and =bluetoothctl pair <mac>= (which bonds), then trust and connect. [[https://github.com/blueman-project/blueman/issues/2219][blueman 2219]] + +** Connection stability + +- Xbox/PS controller drops seconds after connecting unless ERTM is off (both). A controller pairs and connects, then disconnects within seconds, repeatedly. bluez's Enhanced Retransmission Mode is incompatible with these controllers' L2CAP handling. Fix: =echo 1 > /sys/module/bluetooth/parameters/disable_ertm=, persist =options bluetooth disable_ertm=1= in modprobe.d, reboot. [[https://github.com/atar-axis/xpadneo/issues/295][xpadneo 295]] +- Paired+trusted device never reconnects on boot/resume (no). A trusted HID/audio device stays disconnected after boot or resume and needs a manual connect each time. AutoEnable only powers the adapter; it doesn't initiate outbound connections to peripherals that don't self-reconnect. Fix: install and enable bluetooth-autoconnect (a systemd service that connects trusted devices when the adapter powers on, including post-resume). [[https://github.com/jrouleau/bluetooth-autoconnect][bt-autoconnect]] +- ReconnectIntervals/AutoEnable policy leaves the link dead after a drop (yes). After a device drops out of range and returns, bluetoothd doesn't re-establish the link. The [Policy] AutoEnable and [GATT] ReconnectIntervals settings are unset or too sparse. Fix: set =AutoEnable=true= and =ReconnectIntervals=1,2,4,8,16,32,64= in main.conf, restart bluetooth. [[https://wiki.archlinux.org/title/Bluetooth][archwiki bt]] +- USB 3.0 port radio noise degrades BT on the same side of the chassis (physical). A BT mouse/headset stutters or drops, worst under USB load or with an external SSD in an adjacent USB3 port; SuperSpeed signaling adds ~20 dB of broadband noise across 2.4-2.5 GHz. Fix: move the dongle/device off the USB3 port (USB2 port or an extension cable to distance the receiver). [[https://www.usb.org/sites/default/files/327216.pdf][usb-if 327216]] +- WiFi/BT coexistence on a 2.4GHz combo chip causes stutter and drops (yes). On an AX200 (and similar combos) BT audio/HID stutters and 2.4GHz WiFi collapses when both radios are active, sharing one antenna path. Fix: move WiFi to 5GHz, or tune coexistence via =/etc/modprobe.d/iwlwifi.conf=, reload iwlwifi. [[https://community.intel.com/t5/Wireless/Wi-Fi-6-AX200-Bluetooth-causing-WiFi-to-disconnect-randomly/td-p/1437797][intel ax200]] +- btusb USB autosuspend causes input lag/dropout after idle (both; the connection-health view of the adapter-layer autosuspend cause). A BT mouse/keyboard lags or briefly disconnects after no input, then wakes sluggishly; the kernel autosuspends the btusb controller. Fix: =btusb.enable_autosuspend=0= on the kernel cmdline (or a udev =power/control=on= rule), reboot. [[https://community.frame.work/t/solved-bluetooth-mouse-lag-linux-autosuspend/26763][framework autosuspend]] +- Adapter dead / devices won't reconnect after suspend-resume (yes; the same resume-init failure the rfkill layer's module-reload entry names, from the device-connection side). After resume the adapter is gone or devices refuse to reconnect (dmesg shows btusb resume errors). The controller doesn't survive suspend/resume cleanly on some kernels. Fix: a systemd sleep hook that runs =rmmod btusb; modprobe btusb= (or restarts bluetooth) post-resume. [[https://bugzilla.redhat.com/show_bug.cgi?id=2100761][rh 2100761]] +- BLE device disconnects with reason 0x08 from a too-aggressive supervision timeout (yes). A BLE peripheral connects then drops after a few seconds with reason 0x08 (Timeout); the negotiated supervision timeout is too short for the interval and latency. Fix: raise it via =/sys/kernel/debug/bluetooth/hci0/supervision_timeout=, persist with a startup service (debugfs resets on reboot). [[https://bbs.archlinux.org/viewtopic.php?id=279872][arch 279872]] +- Realtek RTL8761B wrong-firmware reconnect-loop (yes; same root cause as the adapter-layer name-mismatch entry, manifesting as a fast reconnect loop). An RTL8761B/BU dongle disconnects and reconnects several times a second; autodetect requests the wrong firmware name (or a bad blob). Fix: symlink the correct blob in /lib/firmware/rtl_bt, reload btusb. [[https://bbs.archlinux.org/viewtopic.php?id=309671][arch 309671]] +- Disconnected/loose internal antenna gives a few-feet range then drops (physical). BT works only within a couple of feet and cuts out beyond; a U.FL antenna lead popped off the card or none is attached. Fix: reseat the U.FL coax, or use an external USB adapter; keep the device closer as a stopgap. [[https://community.intel.com/t5/Wireless/How-can-I-deal-with-the-extremely-limited-range-of-Bluetooth-in/td-p/692325][intel range]] +- "Software caused connection abort" on connect-after-scan (no). A device with a fine bond aborts with "Software caused connection abort" (err 103) when a scan was stopped just before connecting; stopping discovery races the connection attempt in bluez. Fix: keep discovery running across the connect, or retry without a preceding scan-stop. [[https://github.com/hbldh/bleak/issues/631][bleak 631]] +- A2DP audio stutters whenever a BT mouse is also connected (no). Headset audio skips only while a BT mouse/HID is connected on the same adapter; the two profiles contend for the single controller's scheduling. Fix: switch the audio codec (SBC → SBC-XQ/LDAC) to lower airtime, or move the mouse to a separate dongle; disconnecting the mouse clears it. [[https://bbs.archlinux.org/viewtopic.php?id=279749][arch 279749]] +- BT mouse stutters at ~20 Hz unless a scan is active (no). Mouse motion is choppy but becomes smooth the instant a scan starts; without background inquiry the controller's scheduling starves the HID link. Fix: keep discovery on as a workaround, or apply the connection-parameter fix below. [[https://bbs.archlinux.org/viewtopic.php?id=261632][arch 261632]] +- BLE mouse lag from a too-high negotiated connection interval (yes). A BLE mouse feels laggy because it negotiated a power-saving (long) connection interval and high slave latency. Fix: set =[ConnectionParameters]= (MinInterval=6, MaxInterval=9, Latency=44, Timeout=216) in =/var/lib/bluetooth/<adapter>/<device>/info=, restart bluetooth. [[https://wiki.archlinux.org/title/Bluetooth_mouse][archwiki bt mouse]] + +** Bluetooth audio profiles / device classes + +Bluetooth-specific audio failures the bt doctor names at its last chain link. The generic PipeWire/WirePlumber internals are owned by [[file:2026-07-10-audio-failure-taxonomy.org][the audio taxonomy]]; these are the ones a bt-panel probe would surface for a connected BT audio device. + +- Stale bluez pairing state hides A2DP (yes). Headset connects but only HSP/HFP exist, no A2DP sink. Corrupt/partial pairing records in /var/lib/bluetooth leave A2DP unadvertised. Fix: =rm -rf /var/lib/bluetooth/<adapter>/<dev>=, restart bluetooth, re-pair. [[https://bbs.archlinux.org/viewtopic.php?id=280717][arch 280717]] +- Leftover pulseaudio-bluetooth module steals the endpoint (yes). After moving to PipeWire the device connects but no sink shows. pulseaudio-bluetooth still installed grabs the bluez D-Bus endpoint so pipewire-pulse never creates the card. Fix: =pacman -Rns pulseaudio-bluetooth pulseaudio=, install pipewire-pulse, restart the session. [[https://bbs.archlinux.org/viewtopic.php?id=267690][arch 267690]] +- bluez SPA plugin not installed, no card at all (yes). Device connects at the protocol level but pw-dump shows no bluez node; "api.bluez5.enum.dbus could not be loaded". The libspa bluetooth plugin is missing, so WirePlumber's bluez monitor can't enumerate it. Fix: install the PipeWire bluetooth SPA plugin (libspa-0.2-bluetooth on split distros), restart wireplumber. [[https://bbs.archlinux.org/viewtopic.php?id=304688][arch 304688]] +- WirePlumber seat-monitoring drops the node (no). A connected BT audio device intermittently has no card/node on a non-logind or nested seat. WirePlumber gates node creation on an active logind seat and misfires. Fix: =monitor.bluez.seat-monitoring = disabled= drop-in, restart wireplumber. [[https://pipewire.pages.freedesktop.org/wireplumber/daemon/configuration/bluetooth.html][wp bt config]] +- Autoswitch-to-headset fires on any input stream (no). Music drops to mono phone-call quality whenever any app opens the mic. WirePlumber's =bluetooth.autoswitch-to-headset-profile= flips to HSP/HFP on any capture stream and doesn't switch back. Fix: =wpctl settings --save bluetooth.autoswitch-to-headset-profile false=, or force =a2dp-sink= via set-card-profile. [[https://bbs.archlinux.org/viewtopic.php?pid=1973004][arch 1973004]] +- Codec falls back to SBC, wanted codec never negotiates (no). aptX/LDAC-capable headphones connect but only SBC is active. Default codec order drops to the SBC baseline, or the wanted codec isn't in the allowed set. Fix: =bluez5.codecs = [ ldac aptx_hd aptx aac sbc ]= in a wireplumber drop-in, reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300230][arch 300230]] +- AAC unavailable because PipeWire lacks fdk-aac (yes). An AAC-only headset never offers AAC and lands on SBC. PipeWire was built without the fdk-aac module (non-free, omitted on some builds). Fix: install libfdk-aac and a PipeWire build with =bluez5-codec-aac= enabled. [[https://bbs.archlinux.org/viewtopic.php?id=289177][arch 289177]] +- LDAC negotiates but adaptive bitrate stays low (no). LDAC is active yet sounds no better than SBC; bitrate never reaches 990k. The default =ldac.quality = auto= collapses to the low tier on a busy 2.4GHz link. Fix: pin =bluez5.a2dp.ldac.quality = "hq"= in a monitor.bluez.rules drop-in, reconnect. [[https://www.guyrutenberg.com/2025/01/09/configuring-ldac-quality-in-pipewire/][guyrutenberg ldac]] +- BT sink not made default, audio stays on laptop speakers (no). Headphones connect but sound keeps coming from the built-in speakers until switched by hand. WirePlumber doesn't promote a freshly-connected BT sink, and the default isn't restored across reconnect. Fix: =wpctl set-default <bt-sink-id>= now; persist with a priority.session bump or default-target restore. [[https://bbs.archlinux.org/viewtopic.php?id=300751][arch 300751]] +- Absolute-volume passthrough makes the slider a no-op (both). The OS volume slider does nothing or only toggles mute/max. AVRCP absolute-volume hands volume to device hardware that only honors 0/100. Fix: =bluez5.enable-hw-volume = false= in a wireplumber drop-in, restart pipewire+wireplumber. [[https://wiki.archlinux.org/title/Bluetooth_headset][archwiki bt headset]] +- Silent sink after suspend/resume despite reconnect (no; the BT-transport view of the audio taxonomy's resume-silence family). After wake the device reconnects but the playing stream is silent; audio returns only on a new track. The resumed transport isn't re-initialized for the existing stream. Fix: =wpctl set-default= to bounce the stream, or disable suspend-on-idle for the node; scripted disconnect/reconnect on resume as a workaround. [[https://forums.linuxmint.com/viewtopic.php?p=2781321][mint 2781321]] +- oFono present blocks native HFP / mSBC (yes; overlaps the audio taxonomy's BT-mic cluster — kept here for the bt-doctor probe). No HFP profile or only CVSD narrowband; mSBC never appears. An installed-but-unconfigured oFono claims the HFP backend so the native backend (which offers mSBC) stays inactive. Fix: remove ofono (or set the native HFP backend), restart pipewire, reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=266252][arch 266252]] +- mSBC wideband unsupported by adapter/kernel (physical). HFP mic works only in muffled narrowband; mSBC never negotiates even on the native backend. mSBC needs adapter+kernel support for USB ALT6 / the transparent SCO path, which some controllers lack. Fix: none in software — use a different adapter, or accept CVSD narrowband. [[https://www.collabora.com/news-and-blog/news-and-events/pipewire-bluetooth-support-status-update.html][collabora bt]] +- HFP profile output dead, mic works (no). In HSP/HFP the mic captures fine but there's no playback through the headset. The native HFP backend's SCO output path fails to route on certain PipeWire versions/devices. Fix: force a2dp-sink for playback and only enter HFP for the call; try switching the HFP backend or a PipeWire version without the regression. [[https://discussion.fedoraproject.org/t/no-audio-output-with-bluetooth-headset-in-hsp-hfp-profile-mic-works-output-doesn-t/155135][fedora hfp-output]] +- Device seen as LE/GATT-only, no audio class (yes). A headset connects but is treated as a data/LE device with no A2DP sink. bluez brings it up over LE (GATT only) instead of Classic/BR-EDR. Fix: =DisableLE= (or =ControllerMode = bredr=) in main.conf, restart bluetooth, remove and re-pair. [[https://bbs.archlinux.org/viewtopic.php?id=303288][arch 303288]] +- Battery percent never reported (yes). =bluetoothctl info= shows no Battery line and the tray shows no BT battery. bluez's battery provider is gated behind experimental mode. Fix: =Experimental = true= under [General] in main.conf, restart bluetooth, reconnect (the device must expose the Battery Service). [[https://github.com/bluez/bluez/issues/362][bluez 362]] +- Multi-device: first-connected BT audio wins, wrong default (no). Two BT audio devices paired; audio always goes to whichever connected first and the other can't be made default cleanly. WirePlumber keeps the earlier sink as default. Fix: =wpctl set-default <desired-sink-id>=; give the preferred device a higher priority.session via a monitor.bluez.rules match. [[https://0110.be/posts/Connecting_two_Bluetooth_headsets_to_your_Linux_system:_audio_routing_in_PipeWire][0110.be dual bt]] +- HFP narrowband mic during calls is an expected tradeoff (no; GUIDE). Opening the mic (calls/meetings) tanks playback to tinny mono. Inherent to HFP/HSP sharing one low-bandwidth SCO link for bidirectional audio; A2DP has no mic path. Fix: none to repair — keep A2DP for listening, accept HFP for calls, or use a duplex codec (FastStream / aptX-LL) that carries the mic without leaving A2DP. [[https://webcodr.io/2024/10/fixing-no-a2dp-with-bluetooth-headsets-on-linux/][webcodr a2dp]] + +* Network triage + +The ~74 network failure modes sorted into eight symptom clusters, each owned by a tier the net doctor already probes (link → IP → route → DNS → egress), plus a control-plane cluster for NetworkManager itself and a drops cluster for intermittent faults. Every entry carries a remedy class, using the audio taxonomy's four: + +- *AUTO* — user-scope, reversible, no root. For net this is the nmcli surface the doctor already drives (bounce/reset, =con modify= for dns/mtu/ipv6/mac/autoconnect, tunnel-down, portal-open) plus user-service restarts. Polkit on an active session (or passwordless sudo on an archsetup install) covers these. +- *PRIV* — needs root: =systemctl restart NetworkManager/systemd-resolved=, editing =/etc/=, installing a firmware/driver package, a modprobe option. Runs silently where passwordless sudo exists, else Confirm/Arm. +- *REBOOT-TAIL* — the runnable part (often PRIV: firmware/driver install, initramfs rebuild, kernel change, a modprobe option) then the reboot the doctor instructs but can't verify past. +- *GUIDE* — nothing to run: the user must supply a secret/cert, flip a physical switch, change a BIOS setting, make a config trade-off, or wait out a venue/ISP outage. + +Rough distribution: ~30 AUTO, ~14 PRIV, ~12 REBOOT-TAIL, ~24 GUIDE (several are AUTO-now-plus-PRIV-to-persist, or AUTO-plus-GUIDE where the fix is a decision). The buildable share (AUTO plus the runnable part of PRIV/REBOOT-TAIL) is a little over half — the same shape as audio. Triage is a proposal; correct any class. + +Two structural notes carried from the audio design. First, the clusters map onto the probe ladder, so the fix/guide boundary falls along tiers the doctor already walks — cluster N is "the fault the Nth probe step catches". Second, the AUTO core is a handful of primitives (bounce/reset, an nmcli-setter, tunnel-down, a user-service restart, portal-open); the breadth is in the *diagnosis*, not the remedies — exactly as the audio doctor's ~50 diagnoses collapsed to five remedy primitives. + +** Cluster 1 — no link (radio/carrier down) +Probe: the link step (=_step_link=: rfkill, has-wifi-device, carrier). New work: a dmesg firmware-load hint would let the doctor name the specific blob (iwlwifi/rtw89/mt7921) rather than a generic "no adapter". + +- Wifi soft-blocked by rfkill :: AUTO — =rfkill unblock wifi= / =nmcli radio wifi on= (the doctor's rfkill repair). +- Wifi hard-blocked by a physical switch / BIOS :: GUIDE — flip the switch or enable in BIOS. +- rfkill stuck hard-blocked after suspend (WMI conflict) :: REBOOT-TAIL — blacklist the wrong WMI module, rebuild initramfs, reboot. +- Missing iwlwifi/mt7921/rtw89 firmware for a new card :: REBOOT-TAIL — install/update linux-firmware (PRIV), reboot. +- Firmware-load regression after a kernel/firmware update :: REBOOT-TAIL — boot linux-lts or pin the last-good firmware, reboot. +- Realtek rtw89 shadowed by an AUR DKMS driver :: REBOOT-TAIL — remove the DKMS package, reboot. +- Broadcom BCM4360 needs the wl driver / wl breaks after a kernel upgrade :: REBOOT-TAIL — install broadcom-wl-dkms, blacklist the open drivers, reboot. +- USB wifi dongle needs an out-of-tree DKMS driver :: REBOOT-TAIL — install the DKMS package, reboot. +- Ethernet NO-CARRIER (cable/switch) :: GUIDE — reseat/replace the cable, try another port. +- RTL8125 flips to r8169 after a kernel update :: REBOOT-TAIL — blacklist r8169, install r8125-dkms, reboot. +- RTL8125 downshifts / won't negotiate speed :: PRIV — =ethtool --set-eee eee off= (GUIDE the cable/switch swap). +- Adapter left "unmanaged" by NetworkManager :: AUTO + PRIV — =nmcli device set managed yes= (AUTO); remove the config rule / disable the rival manager to persist (PRIV). Also lives in cluster 7. + +** Cluster 2 — associated but no IP +Probe: the DHCP step (=_step_dhcp=). New work: distinguish "no DHCP answer" (server-side) from "conflict/ACD refused" (a duplicate on the LAN) so the message points the right way. + +- dhcpcd competing with NM's internal client :: PRIV — =systemctl disable --now dhcpcd=. Also cluster 7. +- dhcpcd 9 backend breaks NM IPv4 :: PRIV — =dhcp=internal= in NetworkManager.conf. +- APIPA / no DHCP answer :: AUTO — re-trigger DHCP (=nmcli device reapply=); a recurrence is server-side (GUIDE). +- Duplicate IPv4, address skipped by ACD :: GUIDE — change the conflicting host/reservation (a network decision). +- IPv6 RA not received, no SLAAC :: PRIV — =sysctl accept_ra=2= (or GUIDE the router RA-guard). +- DHCPv6-only network, address but no route :: GUIDE — the router must send an RA with the M flag; =ipv6.method auto= is AUTO but the fix is router-side. + +** Cluster 3 — have IP, no route to the internet +Probe: gateway + route + VPN-policy steps (=_step_gateway=, =_step_route=, =_vpn_policy_signal=). This is where the doctor's tunnel-down repair and the "VPN owns the route" terminal case live. + +- Default route never installed (=never-default=) :: AUTO — =nmcli con modify never-default no=, up. +- DHCP gateway off the lease's subnet :: PRIV — add the on-link route then default (GUIDE the DHCP-scope fix). +- Wifi+ethernet conflicting metrics, exits the dead link :: AUTO — raise the bad link's =ipv4.route-metric=. +- WireGuard AllowedIPs=0.0.0.0/0 strands the host :: AUTO — bring the tunnel down (the doctor's tunnel-down repair). +- WireGuard split-tunnel captures the LAN :: GUIDE — fix AllowedIPs (a config decision). +- OpenVPN redirect-gateway route left behind :: PRIV — flush the stale route, restart the client. +- Tailscale exit node captures the default route :: AUTO — =tailscale set --exit-node==. +- Docker bridge / libvirt virbr0 subnet collision :: PRIV — re-address the bridge (a config decision, GUIDE-adjacent). + +** Cluster 4 — authentication rejected (mostly terminal) +Probe: NM state 120 + GENERAL.REASON + the journal tail (the doctor's =_recent_auth_failure=). This is the cluster the classifier already marks TERMINAL and refuses to loop repairs on — the fix is almost always the user supplying a correct secret or cert. + +- Wrong PSK, NM state 120 :: GUIDE (TERMINAL) — the user re-enters the key; no repair fixes it. +- Pure WPA3-SAE won't associate :: AUTO — set =key-mgmt sae= + PMF on the profile. +- PMF mismatch on a transition AP :: AUTO — match the AP's =pmf=. +- SAE H2E-only vs hunt-and-peck :: PRIV — =sae_pwe=2= in wpa_supplicant config. +- eduroam/enterprise cert validation failed (PEAP/iwd/802.1X/anon-identity) :: GUIDE — supply the correct CA cert / server-name / realm; the doctor can't invent institutional certs. +- Hidden SSID never connects :: AUTO — =wifi.hidden yes= on the profile. +- Randomized MAC breaks a portal / allow-list :: AUTO — =cloned-mac-address stable= (or =permanent=). +- scan-rand-mac-address breaks scan/assoc :: PRIV — a =wifi.scan-rand-mac-address=no= drop-in. +- Wrong regulatory domain hides channels :: PRIV — =iw reg set=, persist via wireless-regdom. +- Won't roam, stuck on a weak AP :: AUTO — bounce the connection (GUIDE the bgscan/backend tuning). +- 4-way-handshake timeout / deauth loop :: REBOOT-TAIL — =11n_disable=1= modprobe option, reboot. + +** Cluster 5 — route works, names don't +Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which proves venue-resolver-broken vs egress-dead) and its dns-override/revert repair. + +- Dangling resolv.conf symlink / resolv.conf not at the stub :: PRIV — enable resolved or repoint the symlink. +- resolved stub unreachable / port-53 conflict :: PRIV — stop the conflicting resolver, restart resolved. +- NM not integrated with resolved :: PRIV — =dns=systemd-resolved= in NetworkManager.conf. +- DNSSEC rejects a venue resolver / sticky no-signature verdict :: PRIV — =DNSSEC=allow-downgrade= (=resolvectl reset-server-features= is AUTO for the sticky case). GUIDE the trade-off decision. +- DoT strict with 853 blocked / unreachable server :: PRIV — =DNSOverTLS=opportunistic=. GUIDE the trade-off. +- DHCP handed a broken resolver / pinned public DNS blocked :: AUTO — the doctor's dns-override (switch to working DNS) / drop the pin. +- Stale poisoned cache after a portal login :: AUTO — =resolvectl flush-caches=. +- VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. +- IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. +- Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. + +** Cluster 6 — names resolve, egress blocked +Probe: the http/egress-edges steps + the terminal upstream outcome. This is the cluster where the doctor most often must STOP and say "not locally fixable" rather than loop repairs. + +- Captive portal held state / no auto-popup / HTTPS-check miss / HSTS block :: AUTO — the doctor's portal-open (load a plain-HTTP probe); PRIV to fix NM's connectivity URI to http. +- Custom DNS/DoT hides the captive portal :: AUTO + PRIV — drop DoT/DNSSEC for the network, log in, restore. +- PMTUD blackhole / PPPoE-VPN MTU not clamped :: AUTO — the doctor's mtu-test then mtu-override (lower the interface MTU). +- Stale http_proxy env / unreachable PAC :: AUTO — clear the proxy env / set system proxy to None. +- Clock skew breaks TLS :: PRIV — the doctor's clock-sync (=timedatectl set-ntp true=); GUIDE the CMOS-battery replacement. +- Firewall default-deny / VPN kill-switch leftover rule :: PRIV — allow egress / flush the stale rule. +- IPv6 egress broken while IPv4 works :: AUTO — disable IPv6 on the connection. Also cluster 8. +- False "limited connectivity" from a local DNS/VPN :: PRIV — fix NM's connectivity URI (AUTO to disable the check). Also cluster 7. +- ISP outage / modem in a bad state :: GUIDE (TERMINAL) — power-cycle the modem, contact the ISP; the doctor recognizes and stops. +- Specific outbound port blocked :: GUIDE — use an allowed alternate or switch networks; not host-fixable. + +** Cluster 7 — control plane broken (NetworkManager itself) +Probe: process/service state + config checks. New work: a "two managers fighting" detector (dhcpcd / systemd-networkd / iwd co-active with NM) and a keyfile-permission check, since these fail silently with a healthy-looking radio. + +- NM masked / failed to start :: PRIV — =systemctl unmask && start=; reinstall if corrupt. +- NM segfaults / restart-loops :: PRIV — coredump triage, update/downgrade the package. +- iwd + wpa_supplicant both active / backend mismatch / iwd configured-not-installed :: REBOOT-TAIL — pick one backend, disable the other, reboot. +- systemd-networkd + NM both managing the link :: REBOOT-TAIL — disable networkd, reboot. +- Keyfile permissions void a saved connection :: PRIV — =chmod 600=, =chown root=, reload. +- Autoconnect disabled :: AUTO — =connection.autoconnect yes=. +- Duplicate/conflicting profiles for one SSID :: AUTO — delete the stale profile / raise the good one's priority. +- Dispatcher script errors and blocks connectivity :: PRIV — fix ownership or remove the script. +- Polkit won't authorize a non-root user :: GUIDE — start a polkit agent / relogin for an active seat session. +- NetworkManager-wait-online hangs boot :: PRIV — disable the wait-online service. + +** Cluster 8 — flaky / drops / powersave +Probe: over-time / event-log signals (the panel's eventlog) plus a re-probe-after-idle check — the least-developed tier, the net analogue of the audio "works then dies" cluster. New work: correlate drop events with powersave/roam/suspend transitions. + +- WiFi powersave drops the connection :: AUTO — =iw set power_save off= now; PRIV to persist the NM/modprobe setting. +- USB wifi dongle killed by autosuspend :: PRIV — a udev =power/control=on= rule. +- iwlwifi/ath11k firmware crashloop on resume :: PRIV — reload the module (a sleep hook to persist). +- No reconnect after suspend/resume :: AUTO — the doctor's nm-restart; PRIV for a durable sleep hook. +- IPv6 happy-eyeballs stalls :: AUTO — disable IPv6 on the link. Shared with clusters 5/6 — same broken-v6 family, three probe points (route absent, egress black-holed, AAAA unanswered). + +* Bluetooth triage + +The ~55 bluetooth failure modes sorted into five symptom clusters, each owned by a link in the bt doctor's existing chain (adapter → rfkill/service/powered → per-device → audio). Same four remedy classes. The bt doctor's design already encodes a key constraint that shapes this triage: *connecting and pairing are user intents, not health repairs* — so the doctor never auto-connects or auto-pairs, and the destructive re-pair is always user-confirmed. That pushes clusters 3 and 4 heavily toward GUIDE and user-initiated actions, with far fewer silent AUTO fixes than the net side. + +Rough distribution: ~12 AUTO, ~16 PRIV, ~12 REBOOT-TAIL, ~15 GUIDE. The buildable share is lower than net's because the adapter/firmware cluster is almost entirely PRIV/REBOOT-TAIL/GUIDE and the pairing cluster is mostly user-driven. The doctor's existing safe-auto tiers (unblock, power-on, service-restart, a2dp) all live in clusters 2 and 5 — which is exactly where the buildable share concentrates. + +** Cluster 1 — no adapter +Probe: the adapter step (=btctl.show=). New work: a dmesg firmware-load hint would name the vendor blob (ibt-*.sfi, BT_RAM_CODE_MT7961, rtl8761, QCA, BCM .hcd) instead of a generic "no adapter found" — the direct parallel to the audio cluster-1 dmesg-hint proposal. + +- btusb not auto-loaded :: PRIV — =modprobe btusb=, persist via modules-load.d. +- Intel/MediaTek/Realtek/Qualcomm/Broadcom firmware missing or mismatched :: REBOOT-TAIL — update linux-firmware or symlink/decompress the requested blob (PRIV), reboot. +- Intel "failed to send firmware data" / broken-initial-NCMD :: REBOOT-TAIL — =btusb reset=1= or a newer kernel, reboot. +- Controller lost after a kernel update (Intel AX / MT7922 / QCA) :: REBOOT-TAIL — boot linux-lts or downgrade, reboot. +- MediaTek MT7922 stuck in a residual power state :: GUIDE — full power-off, pull power ~10s, cold boot. +- Realtek/Broadcom dongle needs an out-of-tree driver or .hcd :: REBOOT-TAIL — build the DKMS driver / install the firmware, reboot. +- Fake CSR clone resets/times out :: PRIV — the kernel clone quirks or =btusb reset=1=, reload. +- MT7925 killed by USB autosuspend :: REBOOT-TAIL — =btusb.enable_autosuspend=n= on the cmdline, reboot. Also cluster 4. +- Bluetooth hard-blocked by BIOS/switch :: GUIDE — flip the switch / enable in BIOS. + +** Cluster 2 — adapter blocked, unpowered, or the daemon is down +Probe: the rfkill + service + powered steps. This is where all four of the doctor's safe auto-repairs (unblock, service-restart, power-on) live. + +- Soft-blocked adapter :: AUTO — the doctor's unblock (=rfkill unblock bluetooth=). +- Airplane / physical hard-block :: GUIDE — turn airplane mode off / flip the switch (the doctor deliberately won't touch this radio). +- AutoEnable off, powered down every boot :: PRIV — =AutoEnable=true= in main.conf. +- bluetooth.service masked / not enabled :: PRIV — =systemctl unmask / enable --now= (service-restart is the AUTO tier for a stopped-but-enabled service). +- D-Bus policy blocks org.bluez / bad unit-file after a bluez upgrade :: PRIV — reinstall bluez / reconcile the .pacnew, daemon-reload. +- WiFi rfkill toggle also killed BT (combo chip) :: AUTO — the doctor's unblock. +- TLP disables BT on startup / systemd-rfkill restores a stale block :: PRIV — remove BT from the TLP list / clear the saved rfkill state (unblock is AUTO now). +- rfkill block spins bluetoothd at 100% CPU :: AUTO — service-restart after unblock. +- Adapter powers back off after a desktop toggle :: AUTO — unblock + power-on (or re-enable from the desktop). +- BT stuck/soft-blocked on resume, needs a module reload :: PRIV — reload btusb (a sleep hook to persist). Also cluster 4. +- Wrong/missing default controller (multiple adapters) :: AUTO — =bluetoothctl select <MAC>= then power-on. + +** Cluster 3 — won't pair / bond +Probe: none in-chain — pairing is a user intent the panel offers, not something the doctor health-repairs. The doctor's one destructive tier here (re-pair = remove + pair) is always user-confirmed, never chained. New work: a "stale bond / corrupt cache" signature (a device that fails auth repeatedly with a bond present) would let the doctor *offer* the re-pair with confidence rather than guess. + +- Generic AuthenticationFailed / re-pair loop / corrupt cache survives remove :: PRIV (user-confirmed) — stop bluetooth, =rm -rf= the stale =/var/lib/bluetooth/<adapter>/<mac>=, restart, re-pair. +- Stale link key after a dual-boot re-pair :: GUIDE — remove and re-pair (or sync keys between OSes); a user decision. +- Paired but not trusted, no auto-reconnect :: AUTO — =bluetoothctl trust <mac>= (a safe, non-connecting fix). Bridges to cluster 4. +- No default pairing agent :: GUIDE — register an agent (=agent on=); mostly a headless-setup concern. +- BLE device never appears in scan / not in pairing mode / adapter not pairable :: GUIDE — enable LE transport / put the device in pairing mode / =pairable on= — user actions. +- ConnectionAttemptFailed / page-timeout after a bluez upgrade :: PRIV — restart bluetooth; downgrade bluez if it recurs. +- Headset un-pairs itself on disconnect / BLE address rotates :: GUIDE — re-pair; a device-behavior limitation. +- Legacy keyboard needs the PIN typed on-device :: GUIDE — type the passkey on the keyboard. +- Too many bonded devices, keys lost :: PRIV (user-confirmed) — remove devices below the ~14 limit, restart. +- Dual-mode device pairs on the wrong transport / seen as LE-only :: PRIV — =ControllerMode = bredr= in main.conf, restart, re-pair. Shared with cluster 5. +- Paired without a bond via blueman :: GUIDE — re-pair with =bluetoothctl pair= (which bonds). + +** Cluster 4 — paired but the link won't hold +Probe: the per-device connected-state step. A disconnected paired device is information, not a failure; reconnecting is user-initiated. New work: connection-parameter / coexistence hints (name USB3 noise, 2.4GHz coexistence, or a too-aggressive supervision timeout) so a "keeps dropping" verdict points somewhere. + +- Paired+trusted device never reconnects on boot/resume :: PRIV — install bluetooth-autoconnect, or set =ReconnectIntervals= in main.conf. +- Xbox/PS controller drops unless ERTM is off :: REBOOT-TAIL — =disable_ertm=1= modprobe option, reboot. +- USB3 port radio noise / loose antenna / short range :: GUIDE — move the device off the USB3 port, reseat the antenna, close the distance. +- WiFi/BT 2.4GHz coexistence stutter :: PRIV — move WiFi to 5GHz / tune the coexistence modprobe option (GUIDE the band move). +- btusb autosuspend input lag / adapter dead after resume :: REBOOT-TAIL — =btusb.enable_autosuspend=0= on the cmdline (PRIV sleep hook for the resume case). Shared with clusters 1/2. +- BLE supervision-timeout / connection-interval too aggressive :: PRIV — raise the timeout / set =[ConnectionParameters]= (a startup service to persist debugfs). +- RTL8761B wrong-firmware reconnect loop :: PRIV — symlink the correct blob, reload. Same cause as cluster 1's name-mismatch. +- "Software caused connection abort" on connect-after-scan :: AUTO — retry the connect without a preceding scan-stop. +- A2DP stutters with a BT mouse also connected / mouse stutters without a scan :: GUIDE — lower audio-codec airtime or move the mouse to a dongle; a contention trade-off. + +** Cluster 5 — connected but audio is wrong +Probe: the audio-profile step (=audio.pw_dump=, card/sink for the device). The doctor's a2dp repair (force the A2DP profile) is the AUTO tier here. The generic sound-server internals belong to the audio taxonomy; these are the BT-specific ones the bt panel names. + +- No A2DP sink / stale bluez state hides A2DP :: PRIV (user-confirmed) — clear the stale =/var/lib/bluetooth= record, re-pair (the a2dp repair is AUTO when the card is merely on the wrong profile). +- Stuck in HSP/HFP, autoswitch-to-headset fired :: AUTO — the doctor's a2dp repair / disable autoswitch. +- Leftover pulseaudio-bluetooth steals the endpoint / bluez SPA plugin missing :: PRIV — remove pulseaudio-bluetooth / install the SPA plugin. +- WirePlumber seat-monitoring drops the node :: PRIV — a =seat-monitoring = disabled= drop-in. +- Codec falls back to SBC / LDAC stuck low / AAC missing :: AUTO — a =bluez5.codecs= / ldac-quality drop-in (PRIV/REBOOT for the fdk-aac rebuild). +- BT sink not made default :: AUTO — =wpctl set-default= (the audio taxonomy's set-default primitive). +- Absolute-volume slider no-op :: PRIV — =enable-hw-volume = false= drop-in. +- Silent sink after resume :: AUTO — bounce the stream via set-default. +- oFono blocks native HFP / mSBC :: PRIV — remove ofono / set the native backend. +- mSBC unsupported by adapter / HFP output dead :: GUIDE — a hardware or PipeWire-version limitation. +- Device seen as LE/GATT-only, no audio class :: PRIV — =ControllerMode = bredr=, re-pair. Shared with cluster 3. +- Battery percent never reported :: PRIV — =Experimental = true= in main.conf. +- Multi-device, wrong audio default :: AUTO — =wpctl set-default= / a priority.session rule. +- HFP narrowband mic during calls :: GUIDE — an inherent A2DP/HFP trade-off. + +* Saturation test + +Deferred to a later pass, mirroring the audio taxonomy's blind-different-axis resample. When run, it must sample on a *different* axis than this by-layer first sweep — by hardware vendor, by distro, by device class (headset vs mouse vs controller vs enterprise-wifi vs VPN), weighted toward non-Arch 2024-2026 sources — and run the agents blind to these clusters, so it tests whether the eight net / five bt clusters are representative rather than re-deriving them. The audio resample forced zero new clusters from 108 fresh reports; the expectation here is the same, but it is unproven until run. + diff --git a/docs/design/maintenance-console-design-ideas.org b/docs/design/maintenance-console-design-ideas.org new file mode 100644 index 0000000..066c25d --- /dev/null +++ b/docs/design/maintenance-console-design-ideas.org @@ -0,0 +1,527 @@ +#+TITLE: Maintenance Console — Design Ideas +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-06 + +* Direction + +A single-host maintenance console (GUI, waybar-launched) that surfaces every +health metric for this Arch workstation and, where the remediation is safe, +runs it. It reshapes the earlier install-health/workstation-contract framing +(=system-monitor-design-ideas.org=) into a maintenance surface with a doctor +action. + +The console is the glanceable, single-host version of the home project's +=system-health-check.org= workflow (~1000 lines, capability-dispatched across +ratio/velox/mybitch/truenas). The console owns the routine-maintenance and +at-a-glance-status slice; the workflow stays as the escalation path for +forensic investigation and as the only sanctioned way to run updates. + +* Scope: hosts (Q2 resolved 2026-07-06) + +v1 targets *both* daily drivers — ratio and velox — as first-class hosts, not +ratio-first-velox-later. Consequences for v1: + +- *Capability dispatch is mandatory from day one*, not deferred. The console + probes the live host and runs only applicable checks (btrfs RAID1 on ratio vs + ZFS-primary on velox; AMD amd-pstate on ratio vs Intel intel_pstate on velox). + This mirrors the home workflow's Phase 0 probe. +- *velox-only metrics ship in v1*: battery health (capacity / cycles) and the + unclean-shutdown / suspend-failure rate — both currently GAPs, both driven by + velox being the machine that travels and suspends. +- The ZFS-primary remediation paths (scrub, snapshot retention, pool health) are + built and tested, not stubbed. + +* Thresholds source of truth (Q3 resolved 2026-07-06) + +One machine-readable file, =maintenance-thresholds.toml=, holds every severity +value (cache-size trigger, disk %, scrub-age warn/crit, snapshot retention +limits, temp bands, backup-staleness windows, etc.). archsetup owns and ships +it; both consumers read the *installed* path, so neither reaches into the +other's repo: + +- the maintenance console (dotfiles code) reads it at runtime, +- the system-health-check workflow reads it instead of hardcoding severity + rules in prose. + +The workflow's hard-won values migrate into the TOML as the seed content (the +snapshot MONTHLY limit that bit /home, the 10 GB cache trigger, the scrub-age +bands). Install path is an implementation sub-question — a stable location both +consumers reach (e.g. =~/.config/archsetup/maintenance-thresholds.toml=). + +* Workflow ownership — move system-health-check into archsetup (proposed 2026-07-06) + +Craig's call: the =system-health-check.org= workflow should live in archsetup, +not the home project. Rationale — the home project is scoped to finances, +health, and personal matters; system design, execution, and maintenance are +archsetup's domain. Home only owns the workflow by inherited accident, not by +fit. Moving it here also collapses the Q3 coupling: the TOML source-of-truth and +its workflow consumer end up in the same project. + +Move scope (home → archsetup), to sequence when Craig gives the go: +- =system-health-check.org= (the workflow itself) +- =homelab-inventory/*.org= (ratio/velox/mybitch/truenas capability inventories + it cross-references) +- any home-project references to the workflow (startup, project-workflows index) + +Cross-project mechanics: the archsetup side (receiving the files, wiring the +TOML) is in-scope here; the home side (removing the originals, updating home's +references) is the home project's scope — handled by a handoff note to home's +inbox or a home session, not edited blind from here. + +* Panel shape + +Three regions, driven by the Automation column of the metrics table below: + +- *Actionable* (left) — metrics with a lever. Automation = Auto or Confirm. + Two doctor actions serve this column: "Clean up" fires every Auto metric + unattended; "Review & fix" opens a preview for the Confirm metrics and acts + only on approval. +- *Diagnostic* (right) — read-only telemetry. Automation = None or Human. A + state color (green/amber/red) and the value; no button. Red here is the + signal to run the workflow. +- *Updates* (quarantined strip) — Automation = Workflow. Shown as a count with + notable packages named; the only affordance is "run the workflow." The panel + never applies updates in place. + +*Bar glyph* tracks the worst *Diagnostic* state only — not the actionable +count. A big package cache is boring; a SMART failure is a fire. Actionable +clutter must not turn the bar red or it trains you to ignore it. + +** Doctor = live output wall (Q1 resolved 2026-07-06) + +"Clean up" and "Review & fix" are not fire-and-forget buttons. Running a doctor +opens an *output wall* — one lamp per action, streaming in realtime as each +runs: + +- *amber* while the action is running, +- *green* on success (with the reclaimed amount / result inline), +- *red* on failure. + +Feedback is always shown (not just when something was reclaimed) — you watch it +happen. This is the same live-results shape that should back the other doctors +(net, bluetooth), so every doctor in the system reads the same way. See the +todo task to retrofit the net + bt doctors to realtime lamp output. + +* Automation legend + +| Value | Meaning | +|----------+-----------------------------------------------------------------------------------------------------------| +| Auto | Doctor "Clean up" button — fire unattended, reversible or harmless | +|----------+-----------------------------------------------------------------------------------------------------------| +| Confirm | Doctor "Review & fix" — preview the change, act only on one click | +|----------+-----------------------------------------------------------------------------------------------------------| +| Human | Panel nudges; a human decides and acts (no button, not the agent) | +|----------+-----------------------------------------------------------------------------------------------------------| +| Workflow | vLater — metric shown read-only; no in-panel button. Agent-workflow assistance deferred (decided | +| | 2026-07-07, see below) | +|----------+-----------------------------------------------------------------------------------------------------------| +| None | Diagnostic only — no software remediation exists (hardware, telemetry) | +|----------+-----------------------------------------------------------------------------------------------------------| + +*Workflow buttons removed (decided 2026-07-07, prototyping E5).* The panel +carries no "run workflow" affordances — not on failed units, unclean boots, +updates, or CVEs. The metrics stay on the board as read-only telemetry; AI +assistance via the system-health-check workflow is a vLater feature. v1 ships +only determinate remedies (fixed, scriptable actions with predictable +outcomes). + +*Determinate remedies adopted (decided 2026-07-07).* Every remedy below is +Confirm-tier; contextual levers appear only when the metric is off-nominal: + +- /Service restarts & enables/: failed units (systemctl restart + reset-failed), + fail2ban, cronie, chronyd (+ makestep), tailscaled, snapper timers, zram + config re-apply. DNS/NetworkManager delegates to the net panel's doctor + (deep-link, not a duplicate repair chain). +- /Deterministic maintenance ops/: btrfs balance -dusage=50 (unallocated low), + reinstall owning packages on pacman -Qkk failures, rsyncshot RUN NOW (result + streams to the output well), smartctl -t short self-test, snapshot-retention + repair (write sane TIMELINE limits + cleanup — the /home lesson as a + one-press fix). btrfs device-error counter reset stays manual: resetting + without diagnosis masks a dying drive. +- /Composite macro/: RECLAIM SPACE on the disk-usage cell — runs every reclaim + lever (cache, journal, coredumps, app logs, docker tier-1, snapper cleanup) + as one output-wall stream. +- /Disruptive but determinate/: REBOOT behind arm-to-fire, offered when + running kernel != installed. +- /Still read-only/: temps, throttling, battery, memory/OOM, taint, journal + error content, kernel/hw events, listeners, unclean-shutdown rate — physical + or investigative; vLater AI territory. + +*Updates join the Confirm layer (decided 2026-07-07).* Two levers on the +updates strip, both behind the live-update guard (mesa/hyprland/wayland +runtime in the pending set): + +- UPDATE — repo + AUR system update. +- TOPGRADE — full ecosystem run. The panel's wrapper always passes + --disable git (topgrade's git step rebase-autostashes ~/code/*/ — never + under a live session). + +*MEM·PWR: evidence + two levers + expectation tags (decided 2026-07-07).* +Mostly physics, so the category leans watch-only — with these additions: + +- /CPU mode selector/: a free segmented control (PERF · BAL · POWER) writing + the EPP hint — set the active mode to anything, not drift-repair against a + declared default. amd-pstate on ratio, intel_pstate on velox. +- /Battery charge limit/ (velox-only, capability-gated): SET 80% writes + charge_control_end_threshold — the standard longevity cap. Battery *health* + (capacity vs design, cycles) stays watch-only hardware telemetry. +- /Evidence drill-downs/ (digest idiom): top-5 RAM consumers under memory, + recent boots listed clean/unclean under the unclean-rate, and throttle/OOM + events with timestamps. Evidence makes the numbers actionable even where no + button exists. +- /KILL on top-memory items/ (revised 2026-07-07 — Craig): arm-to-fire, four + guards: the arm shows the exact victim (name + size); SIGTERM not SIGKILL, + with the outcome reported to the wall; PID + process name revalidated at + fire time so a recycled PID can't be hit; session-critical names (systemd, + the compositor, the panel itself) render a disabled key — protected. A + SIGKILL escalation for TERM-survivors is vLater. +- /Expectation-setting, panel-wide/: every leverless cell's sub-line carries an + explicit tag ("hardware — watch only", "evidence below"), and each subpanel + header shows the split — "N fixable · M watch" — so the user knows their + agency before reading a single cell. + +*Refresh cadence (decided 2026-07-07).* Four tiers, matching probe cost: + +- /Live group, panel open/: temps, memory free + top consumers, throttle + state — re-read every ~3 s while their subpanel is visible, gated on + panel-open exactly like the audio panel's meters. Stop when hidden. +- /Fast local tier, panel open/: re-probed every ~30 s while the panel is up; + additionally, any metric re-probes immediately after an action that touches + it (fire CLEAN → cache re-measured, not assumed). +- /On open/: the hydration tiers re-run (fast reads first, process probes + behind them — sub-second perceived). +- /Network tier/: checkupdates / arch-audit / AUR / firmware stay on the + hourly systemd-timer cache with age shown; refreshed on demand only. +- /Panel closed/: the waybar glyph is fed by a light background scan every + ~30 min (systemd timer writing the state file the glyph reads) — the bar + stays honest without the panel running. + +*Journal errors get a digest, not a fix (decided 2026-07-07).* No generic +remedy exists — an error-priority line is a symptom of an arbitrary subsystem — +so the panel ships four determinate assists instead: + +- /Digest/: the cell expands to errors grouped by syslog identifier — count, + message snippet, first/last seen, and the exact next command (journalctl -u + <unit> -b) when the identifier maps to a unit. Top-10 groups, read-only. +- /MARK KNOWN with a full lifecycle/: arm-to-fire showing the exact pattern + before it stores; marked groups move to a dim KNOWN section (never vanish) + with per-row UNMARK; every mark/unmark logs to the results wall; marks carry + date + example. Patterns bind to identifier + message, never a whole unit — + a muted service's *new* errors still surface. +- /Two noise layers/: shipped defaults (bluetoothd HFP, pixman, xkbcomp) in + the packaged TOML, user marks in a separate user file merged over it (a + template sync never eats curation). CLEAR MARKS (arm-to-fire) empties the + user layer and re-enables shipped defaults; unmarking a shipped default + records a disable flag in the user layer. +- /OPEN JOURNAL/: launches a terminal running journalctl -p err -b — the same + delegation pattern as NET DOCTOR. + +Ruled out: auto-restarting units that log errors (error ≠ failed) and +keyword-driven fix suggestions (vLater AI territory). + +*Full-sweep findings — all committed to v1 (decided 2026-07-07).* Every metric +was audited against the converged checklist (honest label, evidence digest +where a count hides detail, curation lifecycle where "expected" is config +knowledge, guarded per-item remedies, cross-links, watch-only tags). Adopted, +all prototyped: + +- /Storage/: disk top-consumers digest (evidence only — no file deletion + keys); per-device error rows on RAID1 when counters are nonzero, + cross-checked against SMART; SMART sub-line carries the last self-test + result. Spec note: a real scrub runs hours — the ring needs a running-% + state, not an instant reset. +- /Packages/: orphan digest (name + size) with per-package REMOVE (armed) and + KEEP — the curation lifecycle encoding "intentional, not orphaned" (the + rust lesson); batch REMOVE ALL skips kept packages. Per-file pacnew rows + tagged safe-delete (reflector-managed) vs needs-merge, MERGE delegating to + a terminal diff. CVEs named: package · CVE id · severity. AUR and firmware + names spelled out. +- /systemd/: failed units upgraded from a count-lever to a per-unit roster + (name · since · exit code · journalctl hint) with per-row RESTART + RESET; + is-system-running names its cause ("degraded — N failed units below"); + taint letters decoded. +- /Logs/: coredumps grouped by binary (count · last · coredumpctl hint), + cleared with the CLEAR action; kernel/hw events listed when not clean + (hardware — watch only). +- /Services/: docker system df breakdown (images / containers / volumes / + build cache with per-type reclaimable); stopped containers upgraded to the + full signal/expected curation lifecycle (MARK EXPECTED / UNMARK, shipped + default: winvm) with per-container START; cron expected-entries drift + roster. +- /Snapshots/: count split by type — timeline (auto-pruned) · single + (manual — escapes timeline cleanup, the pile-up risk) · pre/post — with + oldest-single named and DELETE STALE (armed, keeps newest 2) when singles + accumulate. +- /Network minors/: fail2ban shows recent-ban count; NTP shows offset. + +Rationale for prototyping everything (Craig): real estate and complexity have +bitten before — surface those limits in the disposable prototype, not after +functionality exists behind the UI. + +*Vertical compression → rotary band selector (2026-07-07).* First attempt — +the MEM·PWR three-column layout on Packages/Logs/Services — lost too much row +detail to third-width truncation (Craig's verdict after use). Replaced by a +*rotary band selector*: the amplifier input-selector idiom. A machined knob +(click to cycle) whose needle swings to engraved band labels, one per evidence +section (ORPHANS · PACNEW · ADVISORIES; SIGNAL · KNOWN NOISE · COREDUMPS · +KERNEL/HW; CONTAINERS · DOCKER DISK · CRON & BACKUPS). Each band carries its +own status lamp (section health at a glance without switching) plus a count; +the selected band gets a gold underline and the needle. One section renders at +a time at full width, restoring complete row detail. Deliberately distinct +from the category tiles, console keys, and the CPU-mode segmented control — +each selection idiom in the panel now has its own visual voice. MEM·PWR keeps +its three-column evidence strip (short rows fit fine at third-width). + +*Listeners get the same treatment (decided 2026-07-07).* The count becomes +"unexpected listeners" — evidence digest (process · port · bind address from +ss -tlnp), expected-list curation with the full MARK EXPECTED / UNMARK / +CLEAR lifecycle (shipped defaults: sshd, mpd, tailscaled; user marks in the +user layer), and guarded per-socket remedies: STOP (systemctl stop, armed) +when a unit owns the socket, KILL (SIGTERM, armed) otherwise. Severity keys +on unexpected AND public-bind (0.0.0.0/::) — a loopback listener warns, an +exposed one fails — and when ufw is down the signal header names the exposure +("ufw down — N public binds exposed"). Stopped containers likewise gained a +contextual START lever (allowlist: winvm), and the firewall its ENABLE. + +*Updates strip border is state-tiered (decided 2026-07-07).* Green when +nothing pending, amber for ordinary pending/AUR/firmware counts, red when +CVEs exist or pending exceeds the "a lot" threshold (or the update cache has +gone stale — staleness window in the TOML). The CVE badge renders only when +the count is nonzero. + +*Guard arms instead of blocking (revised 2026-07-07).* When the guard trips, +the key arms (red, "press again to run anyway — or apply from a TTY") rather +than hard-refusing. The user decides; the footgun is acknowledged and +deliberately handed over. After a system update lands, the panel offers a +reboot: a REBOOT key (arm-to-fire) appears on the updates strip and the +reboot-required metric flips. + +*No per-ecosystem update metrics.* Topgrade's step set (yay, rustup, cargo, +pipx, npm/pnpm, gem, go, flatpak, fwupd, tmux/zsh/nvim plugins, git repos) has +no cheap offline "updates available?" probe — mirroring it means a network +round-trip per registry at panel-open. Instead: one *topgrade freshness* +metric (wrapper stamps last-run time; threshold in the TOML) whose remedy is +the TOPGRADE lever, plus a *firmware updates* count in the updates strip +(fwupd refreshes metadata on its own timer; the panel reads the cache). + +Rationale for the hard lines: system updates are Workflow, never Auto — the +2026-06-07 Hyprland crash was a live -Syu swapping mesa+hyprland under the +running session, and the standing rule is never -Syu live under Hyprland when +the mesa/hyprland/wayland runtime is in the set. Hardware findings (SMART, MCE, +thermal) are None — the fix is replacing a drive or clearing a fan, not +software. + +* Metrics — Storage & filesystem integrity + +| Metric | Fix / lever | Automation | Notes | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Package cache size | paccache -r / -ruk0 | Auto | Reclaim, all re-downloadable | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| paccache -rk1 (keep 1 version) | paccache -rk1 | Confirm | Frees most; kills downgrade | +| | | | headroom | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Disk usage (df) | — | None | Fix is cache/snapshot/prune | +| | | | levers | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs unallocated space | — | None | Chunk headroom; diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs scrub age | btrfs scrub start | Confirm | GAP; ZFS has this, btrfs doesn't. | +| | | | IO-heavy, on-demand | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs device stats (per-drive | btrfs device stats --reset | None | GAP; RAID1 early-warning ahead of | +| error counters) | | | SMART; reset after review | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS pool health / errors | — | None | CRITICAL if state != ONLINE or | +| | | | errors > 0 | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS scrub age | zpool scrub | Confirm | Covered | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS capacity | — | None | Perf degrades > 80%; the ZFS | +| | | | headroom metric (no unallocated | +| | | | concept) | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS fragmentation (FRAG) | — | None | GAP; no defrag exists — the | +| | | | remedy is snapshot pruning + | +| | | | staying under 80% | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS autotrim (SSD TRIM) | zpool set autotrim=on / zpool | Confirm | GAP; velox; the fstrim.timer | +| | trim | | counterpart on ZFS | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| fstrim.timer enabled + firing | systemctl enable --now | Confirm | GAP; standard SSD hygiene | +| | fstrim.timer | | | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| pacman file integrity (Qkk) | reinstall package | Workflow | GAP; modified/missing files need | +| | | | judgment | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART health | — | None | Hardware; replace drive | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART wear / temperature | — | None | Diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| + +* Metrics — Snapshots + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapper count / retention | snapper cleanup / delete | Confirm | Manual (single) snapshots need explicit | +| | | | choice | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| ZFS snapshot count / retention | zfs destroy | Confirm | Runaway retention | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapshot auto-timer running | systemctl enable timer | Confirm | Is the auto-snapshot service firing | +|--------------------------------+--------------------------+------------+---------------------------------------------| + +* Metrics — Packages & security + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Orphaned packages | pacman -Rns (named args) | Confirm | Review first (rust looked orphaned, | +| | | | was intentional) | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Pending updates count | — | Workflow | Never auto; workflow-only | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| .pacnew files | diff + merge / delete | Confirm | Allowlist auto-deletes mirrorlist / | +| | | | locale.gen | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| arch-audit CVEs | reviewed update | Workflow | GAP; top-priority add. Fix is an | +| | | | update | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Keyring freshness | pacman -Sy archlinux-keyring | Confirm | GAP; stale keyring breaks update | +| | | | signatures | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| AUR / foreign staleness (Qm) | paru -Sua | Workflow | GAP; AUR updates reviewed, not auto | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Reboot required (kernel mismatch) | reboot | Human | GAP; uname -r vs /usr/lib/modules. | +| | | | You pick when | +|-----------------------------------+------------------------------+------------+--------------------------------------| + +* Metrics — systemd & boot + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| is-system-running (degraded?) | — | None | GAP; one-token whole-system verdict; | +| | | | candidate for the bar glyph | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Failed systemd units | restart / investigate | Workflow | Depends why it failed | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Maintenance timers enabled + | systemctl enable timer | Confirm | GAP; meta-metric — are | +| firing | | | paccache/btrfs-scrub/fstrim/reflector/snapper | +| | | | firing | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Kernel taint flag | — | None | GAP; tainted != 0 = proprietary module or | +| | | | prior crash | +|--------------------------------+------------------------+------------+-----------------------------------------------| + +* Metrics — Logs & coredumps + +| Metric | Fix / lever | Automation | Notes | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Coredump count | coredumpctl clean (keep recent) | Auto | Keep last few days for forensics | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journald disk usage | journalctl --vacuum-size/time | Auto | Bounded journal | +|------------------------------------+---------------------------------+------------+----------------------------------| +| App-log cleanup (>7d) | log-cleanup cron trigger | Auto | Already a cron; manual trigger | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journal error count (real vs | — | Workflow | Forensic; noise-filtered count | +| noise) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Kernel/hardware events | — | None | Forensic, hardware | +| (MCE/USB/thermal/GPU) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| + +* Metrics — Memory, thermal, power + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Memory free / OOM kills | — | Workflow | OOM = investigate | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Swap / zram present + healthy | — | None | Diagnostic | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| CPU / GPU temperatures | — | None | Hardware | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Thermal throttling active | — | None | Cooling issue | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Battery health (capacity / cycles) | — | None | GAP; laptop; ties to open suspend todo | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Unclean-shutdown / suspend-failure rate | — | Workflow | GAP; ratio flagged ~75% unclean 2026-06-08 | +|-----------------------------------------+-------------+------------+--------------------------------------------| + +* Metrics — Network & security posture + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+---------------------------+------------+-----------------------| +| DNS / NetworkManager reachability | restart NM | Workflow | CRITICAL if down | +|-----------------------------------+---------------------------+------------+-----------------------| +| Firewall active (ufw / nftables) | ufw enable | Confirm | GAP; security posture | +|-----------------------------------+---------------------------+------------+-----------------------| +| Unexpected listeners (ss -tlnp) | — | None | GAP; security review | +|-----------------------------------+---------------------------+------------+-----------------------| +| Tailscale peers | tailscale up | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| fail2ban running + bans | systemctl start | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| NTP sync (chrony) | systemctl restart chronyd | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| + +* Metrics — Services, backups, virt + +| Metric | Fix / lever | Automation | Notes | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| rsyncshot backup freshness | — | Workflow | CRITICAL if daily > 48h; investigate | +| | | | failure | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker/podman reclaimable | prune tier 1 / tiers 2-3 | Confirm | Tier 1 nearly Auto; 2-3 destructive | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker stopped containers | — | None | Mostly expected (WinVM on-demand) | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| libvirt VM state | — | None | Expected off | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Cron running + expected entries | systemctl enable cronie | Confirm | rsyncshot + log-cleanup entries | +|---------------------------------+--------------------------+------------+--------------------------------------------| + +* Architecture & testing (decided 2026-07-07) + +*CLI-first, GUI as a face.* The console ships as the fourth panel sibling: a +=maint= Python package in dotfiles (like =net/=, =bt/=, =audio/=) with probe +modules (read-only collectors), a remedies module, =cli.py=, and =gui.py= +driving the same code. =maint status --json= is the contract; =maint fix +<thing>= is every lever. The GUI never does anything the CLI can't. + +Safety mechanics baked into the CLI: +- global =--dry-run= prints the exact command instead of executing — free test + surface, and the GUI's arm-press can display it ("this will run: …"). +- hard read/write split: collectors never elevate; every remedy is an + allowlisted exact argv in one small auditable module. + +*Four test layers (safest → scariest):* + +1. /Unit, fake binaries — no VM, no root (~90% of surface)./ Probes are + parsers over command output: feed canned smartctl/btrfs/journalctl/pacman/ + ss/docker output via fakes on PATH (the net suite's fake-curl and audio's + fake-parec pattern). Remedies tested as command construction (assert the + argv, don't run it). The live-update guard is a pure function over a + package list. +2. /Read-only integration on the live machine./ All collectors are read-only + by design — =maint status --json= runs safely against real hosts. +3. /Remedies in a VM — archsetup's existing harness./ + =scripts/testing/run-test.sh= boots the installer VM; a maint scenario + breaks things deliberately over ssh (stop cronie, mask fstrim, orphan + packages, fill the cache), runs =maint fix …=, asserts post-state. No GUI + in the VM. Add qcow2 snapshot/restore between remedy tests so each starts + pristine and destructive remedies can't contaminate each other. Pure + pacman-level tests may use a throwaway systemd-nspawn container instead + (lighter); the VM stays for systemd/btrfs/reboot territory. +4. /GUI on the host, never in the VM./ AT-SPI smoke like the sibling panels, + driven by fixture data. The prototype's GOOD/BAD snapshots become those + fixtures — =MAINT_PANEL_FIXTURE=bad= renders the degraded board without a + degraded machine, conforming to the =maint status --json= schema. + +* Open questions + +- RESOLVED 2026-07-06 — live output wall (amber running / green done / red fail), + realtime, always shown. See "Doctor = live output wall" above. +- RESOLVED 2026-07-06 — velox is a first-class v1 target alongside ratio. See + "Scope: hosts" below. +- RESOLVED 2026-07-06 — single machine-readable thresholds file + (=maintenance-thresholds.toml=) is the source of truth, *owned by archsetup*. + Both the console and the system-health-check workflow read it, so they can + never drift. See "Thresholds source of truth" and "Workflow ownership" below. diff --git a/docs/design/system-monitor-design-ideas.org b/docs/design/system-monitor-design-ideas.org new file mode 100644 index 0000000..26619a0 --- /dev/null +++ b/docs/design/system-monitor-design-ideas.org @@ -0,0 +1,1008 @@ +#+TITLE: System Monitor Design Ideas +#+DATE: 2026-07-04 +#+TODO: TODO | DONE +#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED + +* DRAFT Status +:PROPERTIES: +:ID: system-monitor-design-ideas +:END: +- [2026-07-04 Sat] DRAFT — initial design sketch for a health monitor covering + Archangel ISO/base-install health, Archsetup workstation health, and the local + laptop as the daily canary. + +* Metadata + +| Field | Value | +|--------+--------------------------------------------| +| Status | draft | +|--------+--------------------------------------------| +| Owner | Craig Jennings | +|--------+--------------------------------------------| +| Repos | archsetup, archangel, dotfiles | +|--------+--------------------------------------------| +| Kin | net panel, bluetooth panel, audio panel | +|--------+--------------------------------------------| + +* Problem + +Archangel and Archsetup can fail in ways that are individually obvious only +after the damage is done: an ISO build goes stale against Arch or archzfs, +ZFSBootMenu or GRUB boots once but not after the first upgrade, a snapshot +hook silently disappears, a package database ages out, systemd services fail +after a reboot, or the desktop contract is technically installed but not +usable. + +The health surface should compress those risks into one operational question: +"Can I trust a fresh install, and is this current workstation drifting away +from the known-good install contract?" + +This monitor is not a generic CPU/RAM graph. It is an install-health and +workstation-contract console. CPU, memory, and temperature belong only as +secondary context unless they block install/test operations. + +* Priority Model + +Rank metrics by the cost of blindness: what happens if Craig never sees the +metric, no one mitigates it, and the next install/upgrade/reboot simply happens. + +Severity: + +- =P0= — can cause data loss, unbootable systems, or loss of rollback path. +- =P1= — can break fresh installs, upgrades, remote access, or core + workstation use. +- =P2= — causes degraded workstation behavior, security drift, or accumulating + maintenance debt. +- =P3= — useful context, not a release gate by itself. + +The panel should sort by live severity first, then by this priority. A red =P2= +row appears above a green =P0= row, but in the steady state the layout keeps the +P0/P1 rows in the first viewport. + +* Priority Ranking + +| Rank | Priority | Metric | Why this rank exists | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 1 | P0 | Storage health | Silent pool/filesystem degradation is the nearest thing to data loss. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 2 | P0 | Snapshot safety coverage | Without snapshots, upgrades lose their rollback safety net. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 3 | P0 | Bootloader and EFI redundancy | A machine that cannot boot is operationally dead. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 4 | P1 | First-upgrade bootability | Catches the classic "installed fine, broke after update" failure. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 5 | P1 | End-to-end VM install pass rate | Best release gate for the whole Archangel + Archsetup chain. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 6 | P1 | Package sync and repo freshness | Arch, archzfs, keyring, and mirror drift are leading break signals. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 7 | P1 | Archangel ISO reproducibility | If current inputs cannot build an ISO, recovery/install confidence is stale. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 8 | P1 | Post-install service health | Network, DNS, SSH, and user services decide whether the system is usable. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 9 | P1 | Archsetup state/log cleanliness | Prevents "half-installed but looks fine" machines. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 10 | P2 | Workstation contract checks | Confirms this is Craig's workstation, not just generic Arch. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 11 | P2 | Backup and rollback readiness | Catches loss of off-machine recovery and edited-file backups. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 12 | P2 | Security and hardening drift | Important, but usually less immediately destructive than boot/storage. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| + +This order intentionally puts ZFS/Btrfs health above VM install evidence. A +broken future install is expensive; silent damage to the current root or backup +chain is worse. + +* Consequence Matrix + +This is the design justification for every metric. A row earns panel space only +if blindness has a clear failure mode and Doctor has at least a useful +diagnostic or mitigation. + +| Rank | Metric | If never seen / never mitigated | Typical failure | Worst plausible failure | Doctor posture | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 1 | Storage health | Disk, pool, or metadata degradation accumulates silently. | Correctable ZFS/Btrfs errors, low EFI/root space, stale scrub. | Data loss, degraded root, failed import/mount during boot. | Diagnose + scrub/cleanup with confirmation. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 2 | Snapshot safety coverage | Upgrades proceed without a rollback point. | Missing recent snapshot or missing pre-pacman hook. | Bad upgrade cannot be rolled back cleanly; manual repair required. | Create snapshot; restore hook; never rollback. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 3 | Bootloader and EFI redundancy | Boot path rots until the next reboot or disk failure. | Missing GRUB/ZBM file on one EFI partition. | Unbootable machine after update, firmware reset, or disk loss. | Diagnose; regenerate config; advanced reinstall only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 4 | First-upgrade bootability | Installs look good until the first real system update. | Kernel/initramfs/bootloader mismatch in VM. | Fresh bare-metal install dies on first reboot after upgrade. | VM-only upgrade test; collect boot evidence. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 5 | End-to-end VM install pass rate | Unit tests give false confidence about the real workflow. | Current branch fails one filesystem path or desktop assertion. | Bare-metal install fails mid-flight after disks are wiped. | Run/schedule VM test; clean stale VM artifacts. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 6 | Package sync/repo freshness | Arch/keyring/archzfs drift surprises the next install. | 404s, stale keyring, bad mirror, stale archzfs DB. | Installer cannot pacstrap or installs mismatched ZFS/kernel bits. | Refresh DB; update keyring; reflector. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 7 | ISO build reproducibility | Recovery/install media confidence becomes historical. | AUR package fails to build; mkarchiso or DKMS breaks. | Need rescue/install media and discover no current ISO can be built. | Parse logs; clean work; explicit rebuild. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 8 | Post-install service health | System is installed but degraded in daily operation. | DNS, NetworkManager, fail2ban, tailscale, or user service down. | No remote access, no network, broken sync, or security tooling off. | Restart/re-enable classified services only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 9 | Archsetup state/log cleanliness | Half-completed provisioning masquerades as success. | Missing marker, log error, skipped step after resume. | Fresh workstation lacks critical config but looks mostly usable. | Summarize; rerun resumable archsetup. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 10 | Workstation contract checks | The box drifts from "Craig's workstation" to generic Arch. | Dotfile symlink broken, keyring wrong, missing tool/package. | Desktop/session workflow is broken during real work. | Restow, repair perms, reinstall with confirm. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 11 | Backup and rollback readiness | Recovery assumptions go stale. | Missing =.archsetup.bak=, backup timer stale, dry-run fails. | Local rollback works but important personal/system state is gone. | Dry-run, start configured job, no deletes. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 12 | Security and hardening drift | Small protective edits quietly regress. | SSH/firewall/fail2ban/sysctl/EFI mask drift. | Exposed service or weakened local boot/config protections. | Restore owned snippets; no broad rewrite. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| + +The worst cases are intentionally conservative, not dramatic. The monitor is +useful because it catches the boring early signal: stale, missing, not recently +tested, or silently degraded. + +* Product Shape + +Use the existing instrument-console panel language rather than a dashboard page: +lamps for pass/warn/fail state, engraved metric groups, dense rows, physical +console keys for explicit actions, and an output well for the last diagnosis. + +The bar module should be tiny: + +- icon: a pulse/terminal glyph or compact =SYS= label. +- state lamp: green/yellow/red/grey. +- text: one word only: =OK=, =WARN=, =FAIL=, =STALE=, =RUNNING=, =UNKNOWN=. +- click: opens the system monitor panel. +- middle click or secondary action: run a cheap refresh only, never repairs. + +The panel should be one screen with internal scroll only where needed. The +first viewport should show the decision state and the Doctor controls without +scrolling. + +* Layout + +** Faceplate + +Top row: + +- State lamp + state word: =OK= / =WARN= / =FAIL= / =STALE= / =RUNNING= / + =UNKNOWN=. +- Unit label: =SYS·01=. +- Scope segmented control: =HOST= / =INSTALL= / =BUILD=. +- Badges: =ZFS= or =BTRFS=, =VM STALE=, =DB STALE=, =SNAPSHOT=, =BACKUP=, + =ROOT= when elevated actions are available. +- Close button. + +The scope control changes the metric emphasis, not the underlying data model: + +- =HOST= is this laptop/workstation right now. +- =INSTALL= is the last Archangel+Archsetup VM install result. +- =BUILD= is ISO, AUR repo, archzfs, and test artifact health. + +** Health Stack + +Arrange metrics as four horizontal bands. Each band has a section title, a +summary lamp, two to four row lamps, and a short "age" or "count" value. + +1. =BOOT + STORAGE= + - bootloader + - EFI redundancy + - pool/filesystem + - snapshots + +2. =INSTALL PIPELINE= + - first-upgrade reboot + - ZFS VM install + - Btrfs VM install + - ISO build + +3. =PACKAGES + SERVICES= + - pacman sync freshness + - archzfs/AUR health + - failed services + - journal errors + +4. =WORKSTATION CONTRACT= + - Archsetup markers/logs + - user/dotfiles + - desktop/session + - backups/replication + +Each row is clickable. Clicking a row opens the evidence drawer in the output +well with: + +- last command run +- normalized verdict +- raw excerpt, redacted where needed +- suggested Doctor action, if any + +** Console Keys + +Use physical console-key buttons, same family as net/bt: + +| Key | Purpose | +|--------------+------------------------------------------------------------| +| REFRESH | Cheap read-only probe of host state | +|--------------+------------------------------------------------------------| +| DOCTOR | Diagnose, classify, run safe mitigations, re-check | +|--------------+------------------------------------------------------------| +| TEST VM | Run or schedule Archsetup VM validation | +|--------------+------------------------------------------------------------| +| BUILD ISO | Run or schedule Archangel ISO build | +|--------------+------------------------------------------------------------| +| CLEAN | Clean old test artifacts, package cache, stale logs | +|--------------+------------------------------------------------------------| +| SNAPSHOT | Create a manual pre-change snapshot | +|--------------+------------------------------------------------------------| + +Keys that can take a long time stream progress into the output well. Mutating +keys must use arm-first behavior: + +- first click arms for 3 seconds and explains the action. +- second click runs. +- destructive cleanup names what will be deleted before it runs. + +* Metric Details + +Details below are grouped by workflow rather than priority. The authoritative +importance order is the priority table above. + +** 1. End-to-end VM install pass rate =P1/rank 5= + +Problem overcome: unit tests can pass while the real install is broken by +mirrors, bootloader state, pacstrap, SSH, disk layout, or the actual desktop +contract. This metric fights false confidence. + +Representation: + +- Lamp row in =INSTALL PIPELINE=. +- Two child lamps: =ZFS= and =BTRFS=. +- Age chip: =last pass 2d= / =never= / =stale 14d=. +- Red if either required filesystem has no recent pass. + +Tools: + +- =scripts/testing/run-test.sh= +- =scripts/testing/create-base-vm.sh= +- =pytest= testinfra suite under =scripts/testing/tests/= +- =qemu-img=, =qemu-system-x86_64=, =sshpass= + +Doctor: + +- read-only first: summarize last =test-results/*/test-report.txt= and failing + test names. +- mitigation: offer =TEST VM= for the failed filesystem. +- cleanup: remove stale temporary VM overlays before retrying. +- no automatic retry loop if the failure is in the installer itself. + +** 2. First-upgrade bootability =P1/rank 4= + +Problem overcome: the machine can boot immediately after install but fail after +the first =pacman -Syu= because initramfs hooks, ZFS modules, GRUB, ZFSBootMenu, +or kernel packages drift. + +Representation: + +- Lamp row: =first-upgrade reboot=. +- Badge: =not run=, =passed=, =failed=. +- Evidence drawer includes boot count, kernel version, and last reachable SSH + timestamp. + +Tools: + +- VM harness +- =pacman -Syu= +- =reboot= +- =ssh= reachability checks +- =journalctl -b -1= where available + +Doctor: + +- run upgrade-in-VM only, never on host without explicit confirmation. +- if failure is ZFS, collect =zpool import=, =lsinitcpio=, =mkinitcpio.conf=, + and EFI files. +- if failure is Btrfs, collect =grub.cfg=, =crypttab=, =fstab=, and snapper + config. + +** 3. Package database freshness and sync health =P1/rank 6= + +Problem overcome: Arch rolling-release state changes faster than installer +assumptions. Stale sync databases, stale keyrings, archzfs drift, or broken +mirrors are leading indicators of a failing install. + +Representation: + +- Row lamp in =PACKAGES + SERVICES=. +- Small meter: newest sync DB age vs threshold. +- Child chips: =core=, =extra=, =multilib=, =archzfs=. +- Yellow over 48 hours, red over 7 days or failed sync. + +Tools: + +- =find /var/lib/pacman/sync= +- =pacman -Syyu --needed archlinux-keyring= +- =checkupdates= +- =reflector= +- =pacman-conf= + +Doctor: + +- safe: refresh package databases. +- safe: update =archlinux-keyring= before full upgrades. +- mitigation: run =reflector= with the configured country/age policy. +- no unattended full system upgrade from the panel unless separately approved. + +** 4. Archangel ISO build reproducibility =P1/rank 7= + +Problem overcome: an old "good" ISO can hide broken current inputs. Archiso, +archzfs, DKMS, AUR package recipes, and pacoloco cache state can all break the +next install. + +Representation: + +- Row lamp in =INSTALL PIPELINE= or =BUILD= scope. +- Shows latest ISO date, kernel version, and AUR manifest age. +- Red if latest build failed or no ISO exists. +- Yellow if latest successful ISO is older than the configured freshness + window. + +Tools: + +- =make build= in =~/code/archangel= +- =build.sh --skip-aur= for fast non-AUR iteration +- =build-aur.sh= +- =mkarchiso= +- =pacoloco= status if installed + +Doctor: + +- read-only: parse latest =out/*.log= for pacman, DKMS, archzfs, AUR, and + mkarchiso failures. +- cleanup: safe build-work cleanup only through Archangel's cleanup function + or =make clean=. +- mitigation: suggest =--skip-aur= when the failure is unrelated to baked AUR. +- build retry is explicit via =BUILD ISO=, not automatic. + +** 5. ZFS/Btrfs storage health =P0/rank 1= + +Problem overcome: the root filesystem can degrade silently before the user +notices. For ZFS this means pool errors or degraded vdevs; for Btrfs this means +device stats, scrub failures, metadata pressure, or degraded RAID. + +Representation: + +- =BOOT + STORAGE= band. +- Filesystem-specific lamp grammar: + - ZFS green: =zpool status -x= healthy. + - Btrfs green: device stats clean and recent scrub clean. +- Capacity strip for root/home/EFI. + +Tools: + +- ZFS: =zpool status -x=, =zpool list=, =zfs list=. +- Btrfs: =btrfs device stats=, =btrfs filesystem usage=, + =btrfs scrub status=. +- Common: =df -h=, =findmnt=, =lsblk=. + +Doctor: + +- safe: start a scrub only with arm-first confirmation. +- safe: clear stale Btrfs stats only after a clean scrub and explicit + confirmation. +- mitigation: warn on low EFI/root space and offer package cache cleanup. +- never destroy snapshots, pools, subvolumes, or datasets from Doctor. + +** 6. Snapshot safety coverage =P0/rank 2= + +Problem overcome: rollback safety is assumed during upgrades but can disappear +when hooks, services, or snapshot tools drift. + +Representation: + +- Row lamp: =snapshots=. +- Child chips: =genesis=, =pre-pacman=, =recent=, =pruned=. +- Yellow if no recent snapshot. +- Red if genesis or pre-transaction hook is missing. + +Tools: + +- ZFS: =zfs list -t snapshot=, =zfs-pre-snapshot=, + =/etc/pacman.d/hooks/zfs-snapshot.hook=. +- Btrfs: =snapper list=, =snap-pac=, =grub-btrfs-mkconfig=, + =/.snapshots=. +- Common: =pacman -Q= for snapshot packages. + +Doctor: + +- safe: create a manual snapshot. +- safe: reinstall or re-enable missing hook only if the expected script exists. +- cleanup: prune only snapshots matching the tool-owned policy and prefix. +- mitigation: show exact command for manual rollback; do not perform rollback + from the panel. + +** 7. Bootloader and EFI redundancy =P0/rank 3= + +Problem overcome: single-disk bootloader success can mask missing redundant EFI +installs on multi-disk systems. A system can also pass install but lose a boot +entry or generate an invalid config. + +Representation: + +- Row lamp: =bootloader=. +- Child chips: =ZBM= or =GRUB=, =EFI=, =entries=, =all disks=. +- Yellow if redundancy cannot be proven. +- Red if the expected loader/config is missing. + +Tools: + +- =bootctl status= +- =efibootmgr -v= +- =findmnt /efi /boot= +- ZFS: check =/efi/EFI/ZBM/zfsbootmenu.efi=. +- Btrfs: check =/boot/grub/grub.cfg= and grub-btrfs entries. + +Doctor: + +- read-only by default. +- mitigation: regenerate GRUB config for Btrfs with arm-first confirmation. +- mitigation: rebuild initramfs with arm-first confirmation. +- no automatic EFI reinstall without an explicit advanced flow. + +** 8. Post-install service health =P1/rank 8= + +Problem overcome: the install can complete while the real workstation is +degraded: DNS broken, NetworkManager failed, fail2ban not responding, user +services not lingering, or Docker/Tailscale/Syncthing not in their expected +state. + +Representation: + +- =PACKAGES + SERVICES= band. +- Count badge: =0 failed= or =3 failed=. +- Child lamps: =network=, =dns=, =security=, =user services=. + +Tools: + +- =systemctl --failed= +- =systemctl is-enabled/is-active= +- =resolvectl status= +- =nmcli general status= +- =fail2ban-client status= +- =loginctl show-user= + +Doctor: + +- safe: restart known flaky non-destructive services such as + =NetworkManager= only after classifying the failure. +- safe: re-enable expected services from Archsetup's contract. +- mitigation: bounce DNS resolver and re-check. +- no blanket =systemctl restart --failed=. + +** 9. Archsetup state and log cleanliness =P1/rank 9= + +Problem overcome: a resumable installer can leave a half-finished system that +looks usable until a missing marker or skipped step matters later. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Step-progress mini bar: completed markers / expected markers. +- Red if =archsetup --status= reports incomplete required steps. +- Red if latest log contains fatal errors. + +Tools: + +- =./archsetup --status= +- =/var/log/archsetup-*.log= +- marker files from the Archsetup state directory +- existing testinfra assertions in =scripts/testing/tests/test_archsetup.py= + +Doctor: + +- read-only: summarize incomplete steps and latest log errors. +- mitigation: offer to rerun =archsetup= in normal resumable mode. +- cleanup: archive old logs, keep the latest N. +- never run =--fresh= from Doctor. + +** 10. Workstation contract checks =P2/rank 10= + +Problem overcome: a fresh Arch system is not the goal. The goal is Craig's +working machine: user, shell, groups, dotfiles, Emacs, Hyprland/DWM, keyring, +VPN tools, Bluetooth tools, and local scripts. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Child lamps: =user=, =dotfiles=, =desktop=, =tools=. +- Evidence drawer mirrors the testinfra checks. + +Tools: + +- =id=, =getent passwd= +- =test -L ~/.zshrc= +- =stow= via dotfiles Makefile +- =pacman -Q=, =yay -Qi yay= +- =hyprctl=, =gdbus= portal checks when session is running + +Doctor: + +- safe: restow dotfiles with the selected profile. +- safe: repair keyring directory permissions. +- mitigation: reinstall missing official packages. +- AUR package rebuilds require confirmation and stream output. + +** 11. Security and hardening drift =P2/rank 12= + +Problem overcome: security settings are easy to regress because they are small +file edits: SSH root login, EFI mount masks, firewall, issue banner, fail2ban, +quiet printk. + +Representation: + +- Compact row under =WORKSTATION CONTRACT= or =PACKAGES + SERVICES=. +- Red only for high-risk drift, yellow for unknown/unreadable state. + +Tools: + +- =sshd -T= or config file checks +- =ufw status= +- =fail2ban-client status= +- =findmnt /efi= +- =sysctl kernel.printk= + +Doctor: + +- safe: restore known Archsetup-owned config snippets. +- safe: re-enable firewall if policy file is present. +- mitigation: write missing drop-ins only from version-controlled templates. +- no broad hardening rewrite from panel state. + +** 12. Backup and rollback readiness =P2/rank 11= + +Problem overcome: rollback only helps local state. The install also needs +backups of edited system files and confidence that personal data replication is +not silently stale. + +Representation: + +- Row lamp: =backups=. +- Chips: =system-file .bak=, =replication=, =last run=. +- Yellow if last replication exceeds policy. +- Red if expected backup files for edited system config are missing. + +Tools: + +- Archsetup backup assertions in =scripts/testing/tests/test_backups.py=. +- =zfs-replicate= if configured. +- =systemctl list-timers= for backup timers. +- =journalctl -u= relevant backup units. + +Doctor: + +- safe: create missing =.archsetup.bak= for files before editing. +- safe: run dry-run replication check. +- mitigation: start a configured backup timer/unit with confirmation. +- never delete backup targets from Doctor. + +* Doctor Model + +Doctor is a classifier with bounded mitigations, not a magic repair button. + +Flow: + +1. Probe the selected scope. +2. Normalize each metric to =ok=, =warn=, =fail=, =unknown=, or =running=. +3. Classify failures as: + - =safe-fix= — local, reversible, low risk. + - =safe-cleanup= — removes only known generated artifacts. + - =mitigation= — improves the chance of success but does not claim repair. + - =needs-confirmation= — mutating, long-running, or system-wide. + - =manual= — too dangerous or context-heavy for Doctor. +4. Run only safe actions automatically after the user presses Doctor. +5. Arm-first for anything mutating beyond safe local cleanup. +6. Re-run the affected probe. +7. Stream a verdict into the output well. + +Doctor should say exactly what it did: + +#+BEGIN_EXAMPLE +doctor: package db stale + check: core.db age 4d, archzfs.db age 4d + action: refreshed sync databases + action: updated archlinux-keyring + result: ok, newest db age 2m +#+END_EXAMPLE + +* Common Tool Drivers + +** Host probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| systemd | =systemctl --failed=, =systemctl is-active=, =journalctl= | +|------------+----------------------------------------------------------------| +| packages | =pacman=, =checkupdates=, =pacman-conf=, =yay= | +|------------+----------------------------------------------------------------| +| storage | =zpool=, =zfs=, =btrfs=, =df=, =findmnt=, =lsblk= | +|------------+----------------------------------------------------------------| +| boot | =bootctl=, =efibootmgr=, =mkinitcpio=, =grub-mkconfig= | +|------------+----------------------------------------------------------------| +| network | =nmcli=, =resolvectl=, =ping= or HTTPS probe | +|------------+----------------------------------------------------------------| +| desktop | =hyprctl=, =gdbus=, =loginctl=, dotfiles Makefile | +|------------+----------------------------------------------------------------| + +** Project probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| archangel | =make test=, =make build=, =build.sh --skip-aur= | +|------------+----------------------------------------------------------------| +| archsetup | =make test-unit=, =make test=, =scripts/testing/run-test.sh= | +|------------+----------------------------------------------------------------| +| VM | =qemu-img=, =qemu-system-x86_64=, =sshpass=, =pytest= | +|------------+----------------------------------------------------------------| +| artifacts | latest =out/*.log=, =out/*aur-manifest.tsv=, =test-results/*= | +|------------+----------------------------------------------------------------| + +* Top-family Comparison + +This monitor should borrow the mature display ideas from =top=-style tools +without becoming another CPU/process viewer. The domain objects are install +contracts, boot/storage health, package freshness, snapshots, services, and +artifacts. The interaction model is still the same: sort the thing that hurts, +filter to the thing you care about, expand one row for evidence, and act only +when the diagnosis is clear. + +** Comparison table + +| Tool | What it represents well | Sorting/filtering model | Useful pattern for system monitor | Gaps for our domain | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =htop= | Dense live table plus configurable meters; process tree; direct process actions. | Interactive sort by column, search, filter, tree toggle. | Metric table should support column sort, search, filter, and tree/group mode. | No historical artifact model; actions are process-centric. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =btop++= | Boxed dashboard: CPU, memory, disks, network, processes, battery, GPU; strong graph language; selected process detail. | Easy switching between process sort modes; filter; tree view; pause. | Use boxed bands, mini time-series, detail pane, pause/freeze, and clickable controls. | Graph-first layout can overemphasize volatile values over install risk. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =bottom/btm= | Custom widget layout, per-widget focus/expand, zoomable time windows, basic mode. | Process widget supports sort, search, tree; widgets can be filtered/configured. | Every health band should be expandable; stale/history windows should be zoomable. | Mostly resource telemetry, not remediation workflow. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =atop= | Interval deltas, critical-resource highlighting, all active processes including exited ones, long-term logs. | Resource views and interval replay; emphasizes deviations and active load. | Add history/replay for health events and show "new since last good" changes. | Lower immediate visual polish; Linux-performance scoped. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =Glances= | Broad plugin dashboard, thresholds, remote/web/API modes, export to JSON/CSV/time-series backends. | Configurable visible plugins; API/stdout selectors instead of only interactive sorting. | Use plugin architecture, threshold config, JSON output, remote/headless mode. | Too broad; can become a generic monitoring surface. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =procs= | Modern table ergonomics: custom columns, keyword search across selected fields, sort by named column, tree view. | CLI sort asc/desc by partial column name; watch mode cycles sort columns; AND/OR/NAND/NOR search. | Use named metric columns, saved views, multi-keyword filters, and value-aware coloring. | Process-only; no graphs or remediation model. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =gotop/gtop= family | Fast glanceable terminal dashboard with compact graphs and gauges. | Usually lighter than htop/btop; less important than presentation density. | Use compact sparklines/gauges for "age", "last pass", and "failure count". | Not enough evidence/action depth for this monitor. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| + +** What to pull in + +*** htop: table discipline + +Pull: + +- Column headers that are real controls: click or key-cycle to sort by + =priority=, =state=, =age=, =last_checked=, =last_pass=, =failure_count=, + =scope=, and =doctor_class=. +- Search and filter as first-class actions, not hidden debug commands. +- Tree mode for ownership: + - =system= → =boot/storage= → =bootloader= → =efi entries=. + - =archangel= → =iso build= → =aur repo= → =manifest rows=. + - =archsetup= → =state markers= → =desktop= → =dotfiles=. +- Horizontal detail access for long evidence, like htop's horizontal scrolling + for full commands. + +Equivalent-or-better requirement: + +- htop sorts processes; this monitor sorts risk. The default sort is + live-state severity, then priority rank, then age. + +*** btop++: instrument boxes and live graphs + +Pull: + +- Boxed bands with stable geometry. +- Small time-series graphs, but only where history matters: + - package DB age over time + - failed-service count + - journal error count + - snapshot count / newest snapshot age + - VM pass/fail history + - ISO build duration/result history +- Selected-row detail pane with the last command, verdict, and raw excerpt. +- Pause/freeze button so a failure does not scroll away while reading. +- Mouse-clickable controls where every visible key has the same keyboard path. + +Equivalent-or-better requirement: + +- btop's graphs answer "what is hot right now?" Our graphs answer "is the + safety margin shrinking?" Trend charts should be muted unless the threshold + is crossed. + +*** bottom: focus/expand and layout presets + +Pull: + +- Expand one band full-height: + - =BOOT + STORAGE= expands into boot files, EFI entries, pools, snapshots. + - =INSTALL PIPELINE= expands into last VM runs and build artifacts. + - =PACKAGES + SERVICES= expands into DB ages, repo status, failed units. + - =WORKSTATION CONTRACT= expands into Archsetup markers and testinfra-style + checks. +- Zoomable history windows: 24h / 7d / 30d / all artifacts. +- Layout presets: + - =compact= for bar dropdown. + - =full= for terminal/TUI. + - =host-only= for laptop health. + - =release-gate= for Archangel/Archsetup changes. + +Equivalent-or-better requirement: + +- bottom expands widgets; this monitor expands evidence and remediation state. + The expanded view must show "what changed since last good" before raw logs. + +*** atop: history and vanished failures + +Pull: + +- Permanent, compact health-event log. +- Interval deltas instead of only current values: + - new failed services since last check + - new journal errors since last check + - packages/repos newly stale + - snapshot hook present before, missing now + - bootloader file changed since last known-good +- "Show active/deviating only" mode. In normal use, hide green rows unless + their age is approaching threshold. +- Replay mode: inspect the state at the time an install/test/build failed. + +Equivalent-or-better requirement: + +- atop can report processes that already exited. This monitor should report + failures that already passed through: a transient failed unit, a VM test that + failed last night, an ISO build that failed before the current successful + build, or a package DB that was stale until Doctor fixed it. + +*** Glances: plugin/API/export model + +Pull: + +- Plugin-like probes. Each metric owns: + - =probe= + - =normalize= + - =thresholds= + - =doctor_actions= + - =redaction= + - =evidence= +- JSON output as a stable contract before GTK work. +- Optional stdout selectors: + - =system-monitor --stdout packages.state,storage.state= + - =system-monitor --json boot,snapshots= +- Remote/headless mode for VMs and bare-metal test targets. +- Threshold config in one file, not hardcoded in the UI. + +Equivalent-or-better requirement: + +- Glances is broad; this must stay opinionated. A plugin is accepted only if it + maps to install health, rollback safety, workstation contract, or recovery + readiness. + +*** procs: custom columns and query grammar + +Pull: + +- Named columns and saved views: + - =risk=: state, priority, age, doctor class. + - =install=: last pass, filesystem, artifact, branch, commit. + - =host=: state, source, last checked, command. + - =doctor=: action class, requires root, reversible, last run. +- Multi-keyword search: + - =zfs failed= + - =doctor safe-fix= + - =archangel stale= + - =service red= +- Boolean query modes: + - AND default for narrowing. + - OR for "show any boot or storage issue". + - NOT for "hide green". +- Value-aware coloring for age, severity, and units. + +Equivalent-or-better requirement: + +- procs lets the user build a process table. This monitor should let Craig + build a risk table without editing code. + +*** gotop/gtop: glance density + +Pull: + +- Small sparklines for trend, not full charts. +- Big obvious state words. +- Compact gauges for bounded values: + - EFI usage + - root/home usage + - DB age as percent of freshness window + - VM evidence age + - snapshot age +- Simple default screen that is useful without learning keys. + +Equivalent-or-better requirement: + +- The first screen should answer "am I safe to upgrade or install?" in under + two seconds. + +* Sorting and Views + +The monitor needs two sorting layers: global row ordering and per-band evidence +tables. + +** Global row ordering + +Default: + +1. =state= severity: red, yellow, unknown, running, green. +2. =priority= rank: P0 before P1 before P2. +3. =age= or =staleness=, descending. +4. =last_changed=, newest first. + +Alternate sorts: + +| Sort key | Use case | +|-----------------+------------------------------------------------------| +| =priority= | Release-gate review; keep P0/P1 at the top. | +|-----------------+------------------------------------------------------| +| =state= | Triage; show all red/yellow rows first. | +|-----------------+------------------------------------------------------| +| =age= | Find stale tests, stale package DBs, old backups. | +|-----------------+------------------------------------------------------| +| =doctor_class= | Find what Doctor can safely fix now. | +|-----------------+------------------------------------------------------| +| =scope= | Group host vs install vs build. | +|-----------------+------------------------------------------------------| +| =last_changed= | See what recently regressed. | +|-----------------+------------------------------------------------------| +| =source= | Group by archangel, archsetup, dotfiles, host. | +|-----------------+------------------------------------------------------| + +** Per-band sorts + +| Band | Sorts | +|------------------------+------------------------------------------------------------| +| =BOOT + STORAGE= | severity, mountpoint, filesystem, capacity, last scrub, newest snapshot age | +|------------------------+------------------------------------------------------------| +| =INSTALL PIPELINE= | result, filesystem, duration, artifact age, commit age, last pass | +|------------------------+------------------------------------------------------------| +| =PACKAGES + SERVICES= | severity, unit name, repo name, DB age, error count, enabled/active state | +|------------------------+------------------------------------------------------------| +| =WORKSTATION CONTRACT= | severity, check name, owner repo, last pass, doctor class | +|------------------------+------------------------------------------------------------| + +** Filters + +Quick filters should be visible as chips: + +- =red= +- =yellow= +- =doctorable= +- =needs-root= +- =stale= +- =zfs= +- =btrfs= +- =host= +- =install= +- =build= +- =changed= +- =hidden-green= + +The default view can hide healthy low-priority rows, but it must show enough +green P0/P1 summary state to prove the monitor is working. + +* Display Requirements Borrowed from Tops + +1. Every table has sortable columns and a visible sort indicator. +2. Every visible metric row has a filterable state, priority, age, and source. +3. Every row can expand to evidence without losing the list context. +4. Every graph has a threshold marker; trend without threshold is decoration. +5. Every long-running action can be paused/frozen in the display. +6. Every mutating action has an equivalent CLI command shown in the output well. +7. Every Doctor action records before/after state so fixed failures remain + visible in history. +8. Every band has a compact mode and an expanded mode. +9. Green rows are quiet; new regressions are loud. +10. The system must be useful over SSH/TUI before GTK polish. + +* Source Notes + +- =htop=: upstream README describes configurable system/process display, + interactive sorting/filtering/search, tree view, and process actions. +- =btop++=: upstream README describes resource boxes, detailed process stats, + filter, sort switching, tree view, mouse support, auto-scaling network graphs, + disk IO, battery, GPU support, and themes. +- =bottom/btm=: upstream README describes customizable widgets, process sort + and search, tree mode, expand/focus, zoomable graph intervals, filters, and + basic mode. +- =atop=: upstream README describes interval resource accounting, critical + highlighting, long-term compressed logs, exited-process visibility, cgroup + views, and active/deviation-focused output. +- =Glances=: upstream README describes plugin-style broad monitoring, web/API + modes, stdout JSON/CSV, remote monitoring, exports, and threshold-oriented + dashboard use. +- =procs=: upstream README describes configurable columns, named-column sort, + watch mode, tree view, logical keyword search, value-aware coloring, and + pager behavior. + +* Data Model + +Emit JSON from a CLI first; the panel is a client. + +#+BEGIN_SRC json +{ + "v": 1, + "scope": "host", + "state": "warn", + "ts": "2026-07-04T12:00:00-04:00", + "metrics": [ + { + "id": "packages.sync_freshness", + "label": "package databases", + "state": "warn", + "summary": "archzfs.db age 4d", + "evidence": [ + {"command": "find /var/lib/pacman/sync", "excerpt": "archzfs.db 2026-06-30"} + ], + "doctor": { + "class": "safe-fix", + "actions": ["refresh-sync-db", "update-keyring"] + } + } + ] +} +#+END_SRC + +* Implementation Notes + +- Start with a CLI: =system-monitor status --json=, =system-monitor doctor + --json=, =system-monitor refresh=. +- Keep probes read-only by default. Actions live in separate verbs. +- Cache slow probes. The bar should read a cache, not run VM tests. +- VM/build actions should create job records and stream logs; the panel follows + the job rather than blocking the UI process. +- Reuse the net/bt panel architecture if this becomes a GTK panel: GTK-free + model + fake-command unit tests + one AT-SPI smoke. +- Redact secrets from logs and JSON: WiFi PSKs, tokens, private repo URLs with + credentials, SSH material, and backup target credentials. + +* Open Decisions + +** TODO Where should the first implementation live? + +Recommendation: dotfiles owns the user-facing panel and CLI wrapper because it +is workstation UI. Archsetup owns reusable install-contract probes and testinfra +assertions. Archangel owns ISO/build probes. + +** TODO Should Doctor run elevated actions through polkit or terminal? + +Recommendation: read-only checks run unprivileged; elevated actions launch a +terminal or polkit prompt with the exact command visible. Do not hide long +privileged operations inside the panel process. + +** TODO How fresh must VM evidence be? + +Recommendation: host checks go stale after 1 hour; package DB after 48 hours; +VM install evidence after 7 days; ISO build evidence after 14 days or whenever +Archangel/Archsetup has changed since the last successful artifact. + +** TODO Which actions are allowed on bare metal? + +Recommendation: host Doctor may refresh databases, update keyring, restow +dotfiles, create snapshots, run scrub, and restart narrowly classified services. +It may not perform full upgrades, bootloader reinstalls, destructive snapshot +prune, or filesystem repair without a separate advanced flow. + +* First Build Slice + +1. CLI read-only host status: + - package DB freshness + - failed services + - ZFS/Btrfs health + - snapshot presence + - Archsetup status/log check +2. Doctor safe actions: + - refresh package DB + - update keyring + - restow dotfiles + - create manual snapshot +3. Artifact parser: + - latest Archsetup VM test result + - latest Archangel ISO build result +4. Panel prototype: + - faceplate + - four health bands + - evidence output well + - REFRESH and DOCTOR keys only |