diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/2026-06-25-testinfra-validation.org | 8 | ||||
| -rw-r--r-- | docs/design/2026-06-29-waybar-network-module-spec.org | 335 | ||||
| -rw-r--r-- | docs/design/2026-06-29-waybar-timer-module-spec.org | 2 | ||||
| -rw-r--r-- | docs/design/2026-07-02-waybar-expansion-animation-feasibility.org | 53 | ||||
| -rw-r--r-- | docs/design/maintenance-console-design-ideas.org | 527 | ||||
| -rw-r--r-- | docs/design/system-monitor-design-ideas.org | 1008 |
6 files changed, 1919 insertions, 14 deletions
diff --git a/docs/design/2026-06-25-testinfra-validation.org b/docs/design/2026-06-25-testinfra-validation.org index 5c82aa2..0f81231 100644 --- a/docs/design/2026-06-25-testinfra-validation.org +++ b/docs/design/2026-06-25-testinfra-validation.org @@ -106,16 +106,18 @@ scripts/testing/tests/ ** Example tests (parity) #+begin_src python +TEST_USER = os.environ.get("ARCHSETUP_TEST_USER", "archie") + def test_ufw_enabled(host): assert host.service("ufw").is_enabled -def test_user_cjennings_exists(host): - u = host.user("cjennings") +def test_primary_user_exists(host): + u = host.user(TEST_USER) assert u.exists assert u.shell == "/usr/bin/zsh" def test_zshrc_stowed_and_readable(host): - f = host.file("/home/cjennings/.zshrc") + f = host.file(f"/home/{TEST_USER}/.zshrc") assert f.is_symlink assert ".dotfiles/" in f.linked_to assert f.exists # not broken diff --git a/docs/design/2026-06-29-waybar-network-module-spec.org b/docs/design/2026-06-29-waybar-network-module-spec.org index b4ba1f2..3a1260c 100644 --- a/docs/design/2026-06-29-waybar-network-module-spec.org +++ b/docs/design/2026-06-29-waybar-network-module-spec.org @@ -1,5 +1,5 @@ #+TITLE: Waybar Network Module — Design Spec -#+AUTHOR: Craig Jennings & Claude +#+AUTHOR: Craig Jennings #+DATE: 2026-06-29 * Status @@ -45,9 +45,12 @@ reverses two earlier choices and widens coverage: success) and *detect + respond to every failure mode* — the full ~44-mode catalog, edge cases included, lives in the redesign task and supersedes the table below. -Phase 4 (docs / rollout) and Phase 5 (VPN) remain. The three Codex review rounds + -Craig's cj comments from the original spec are all incorporated ([31/31], no open -decisions). Phases 1-3's manual live checks are under todo.org "Manual testing and +Phase 4 (docs / rollout) and Phase 5 (VPN) remain. Review incorporated (Codex, +2026-06-30): four review rounds + Craig's cj comments are all dispositioned +([40/40], no open findings) — the fourth round reshaped the V2 panel UX (single nav +target, saved-vs-available groups, join-from-row, the auth matrix, progressive +loading, a findable diagnostics report, and the Waybar visual contract; see "V2 panel +UX"). Phases 1-3's manual live checks are under todo.org "Manual testing and validation". * Goal @@ -380,7 +383,8 @@ there's one source of truth. Per-surface rendering of the canonical string: - *Indicator* — the matching glyph + CSS class; the string is the tooltip (untruncated). -- *Notification* (=notify=) — title = the failure label, body = the string. +- *Notification* (=notify=) — title = "Networking"; body = the failure label on + its own line, then the canonical string. - *CLI* — the string on stderr; =--json= puts it in =error.message= with the evidence in =error.detail= and a stable =error.code=. - *Panel* — the string as the section banner, with the diagnostic step's evidence @@ -432,6 +436,31 @@ portal tokens never). - *DNS override cleanup unverified* — "Couldn't confirm DNS was restored after the test" → evidence: iface, attempted revert → revert DNS manually (=resolvectl revert <iface>=). +- *VPN kill-switch blocking* — "A VPN kill-switch is blocking all traffic, and the + VPN itself is down" → evidence: a block artifact present with no tunnel up → bring + the VPN back, or clear the kill-switch (the exact root command surfaced, not + auto-run). + +*VPN kill-switch detection + correction.* A kill-switch blocks all non-VPN egress when +the tunnel drops, so the link looks up (wifi, IP, gateway) but nothing reaches the +internet. This extends the =deferred-vpn= branch: when a VPN is active and the probe +fails, run a rootless cascade to tell a working tunnel from a kill-switch that's +blocking because the tunnel is down — +- =ip rule= for wg-quick's =not fwmark 0xca6c= + =suppress_prefixlength 0= (and the + PostUp =REJECT ! -o %i= rule that makes it leak-proof); +- =wg show= for an up tunnel interface; +- =nmcli connection show= for Proton's =pvpn-killswitch= / =pvpn-ipv6leak-protection= + (device =pvpnksintrf0=); +- =nft list ruleset= / =iptables -S OUTPUT= for a drop/reject table (=killswitch=, + =protonvpn=, =oifname != "wg0" ... drop=); +- =nmcli -f connection.zone= for a firewalld =drop= zone. +Classify *kill-switch-blocking* only when a block artifact exists AND no tunnel +interface is up — that's what distinguishes it from a healthy VPN. Correction is tiered +by artifact and every option needs root, so surface the exact command rather than +auto-running it: =wg-quick down <iface>=, =nmcli connection delete pvpn-killswitch +pvpn-ipv6leak-protection=, =nft delete table inet killswitch=, or =nmcli connection +modify <con> connection.zone ''=. (Sits alongside the Phase 5 VPN work; detection can +land earlier since =deferred-vpn= already exists.) Each message names whether the system was left unchanged, partially changed (with what), or fully changed, so the user knows the residue. @@ -676,6 +705,82 @@ individual repair tiers (renamed, with tooltips) plus a *Debug capture on/off* toggle (the manual side of the verbose-capture feature; a failing diagnose triggers it automatically). Speed test moves under Performance. +** V2 panel UX — the target design + +The shipped four-page stack (Connections / Diagnose / Repair / Speed test) is +*history*, not active design. V2 is the sole current target: one panel opened from +the bar, three top tabs — Connections | Diagnostics | Performance — and the page +model below is the contract for what gets built and what gets deleted, not just for +labels. + +*** Connections — saved vs available, join-from-row +Three labelled groups, never one merged list: +- *Saved* — saved NM profiles, MRU-first, rendered instantly without a scan. +- *Available now* — scan-backed in-range SSIDs with signal + security; may carry a + loading/stale hint; unsaved networks appear here. +- *Wired* — ethernet when a wired device is present. +=net list= already yields this (=connections.py= lists saved MRU-first, merges live +signal/security for in-range saved profiles, then appends unsaved in-range SSIDs with +=uuid: nil=); the panel groups and labels it. *Rescan refreshes only the +availability/signal layer* — it never gates or reloads the Saved list. + +*Progressive loading:* render the Saved group immediately on open, then overlay +availability, signal, and the unsaved Available-now rows when the scan returns. Show a +small scan-in-progress state (elapsed + last-scan age). A slow or bad radio scan must +not make the whole panel feel stuck — this is the direct answer to "why does it take +so long to see my connections?" + +*Join-from-row (no Add page):* selecting an unsaved Available-now row *is* the join +flow — SSID and security come prefilled from the scan, never retyped. Open networks +connect (confirm only if needed); WPA/WPA2/WPA3-Personal ask only for the password. +The standalone Add button + modal are deleted for visible networks. A hidden/manual +SSID join lives behind an Advanced "Join hidden network" affordance. + +*** Supported authentication classes (the join matrix) +From the scanned NM =SECURITY= value, V2 handles: +- *Inline-supported* — open, open-with-captive-portal, WPA/WPA2/WPA3-Personal + (PSK/SAE), and WPA2/WPA3 transition mode. The row shows the security label so the + user knows why a password is or isn't asked. +- *Activate-only* — 802.1X / enterprise: connect if already saved, else "edit in + nmtui/nmcli" (no add form in v1/V2, per decision 9). +- *Hidden / manual* — behind the Advanced "Join hidden network" affordance. +- *Rare / unsupported* — WEP, OWE/enhanced-open, MAC-registration, voucher, or + proxy-required: a clear in-panel explanation ("not supported here yet") plus a + non-terminal next step, never a hand-off to a terminal tool. + +*** Diagnostics owns the diagnostic story +Diagnostics holds the read-only checks, the repair stream, Get Me Online, debug +capture (Advanced), and the doctor report. A *lightweight* latency/throughput probe +runs inline as a Diagnose evidence row when internet is available (skipped offline, on +a metered/hotspot warning, or with no backend), and its result is stored in the doctor +report. The *full* speed test stays under Performance (decision 19) — which is also +the home for future throughput history, so Performance earns its tab rather than being +a lone button. + +*** Forget confirmation — future tense + verified +The destructive copy is future tense and names the scope: "This will remove the saved +NetworkManager profile and its stored password from this machine." After the op, +verify the UUID is gone, refresh the Saved list, and report "Forgot <SSID>" or "Could +not forget <SSID>; nothing changed / partial <evidence>" — the verify-every-action +decision applied to a destructive op. + +*** Findable diagnostics report +Every diagnose, repair, and speed/performance run ends with a "Copy report" / "Open +report" action in Diagnostics. The report carries the step statuses + elapsed, the +final classification, the last speed/latency result when available, scan age, +route/interface owner, the redacted event-log tail, and the bundle path when verbose +capture ran. It states explicitly whether any repair mutated state and whether +cleanup/verification passed. "Logs exist somewhere" isn't enough when the network is +already down — the report is the one artifact the user copies to hand over. + +*** Visual contract — a Waybar-attached popup +The panel reads as part of the bar, not a separate app. Match the live Waybar theme: +the dark rounded capsule (=border-radius: 1rem=), the golden border, compact monospace +text, and the =custom/net= state colors. Avoid square corners next to rounded UI, keep +cards out of cards, and use compact icon+label controls with tooltips for the advanced +repairs. Reuse any existing archsetup-owned GTK/panel conventions. (Non-blocking for +engine work; blocks final V2 UX acceptance.) + ** Panel state, cancellation, permissions State machines for: connection-list loading, rescan-in-progress, activation-in-progress, diagnose-running, repair-running, speedtest-running. Plus @@ -725,8 +830,13 @@ The concrete interaction defaults, so the GTK build isn't inventing them: - *After close*: the bar reflects the new state immediately (signal/refresh on next poll); a running speedtest/diagnose keeps running and notifies on finish (panel close doesn't cancel it). -- *Keyboard*: Esc closes; Tab moves between sections; arrows move rows; Enter - fires the section primary; the connection list is type-to-filter. +- *Keyboard*: Esc closes (wired); arrows move row focus and Enter activates a + row (GTK ListBox defaults — row-activate connects, never disconnects); Tab is + the plain GTK focus chain, widget by widget (inside a list it crawls row by + row — no section jumps); there is NO type-to-filter. Verified live via + targeted-key AT-SPI probe 2026-07-02; the earlier tab-between-sections and + type-to-filter claims were aspirational and are struck. If section-jump Tab + or filtering is ever wanted, it's a new task, not an existing behavior. * Connection management (nmcli) @@ -988,8 +1098,12 @@ a *coverage-gap pass*, not just a percentage: V2 redesign decisions (Craig, 2026-06-30): 15. *No terminals anywhere in the module* — =net-popup= is removed; every action and - result renders in the panel. Reverses the part of decision 11 that ran privileged - repairs in a terminal "so sudo/polkit can prompt". + result renders in the panel. No terminal is ever used to report information to the + user or to collect input from them: every prompt, confirmation, repair stream, and + result lives in the panel UI (Craig, cj, 2026-06-30). Reverses the part of decision + 11 that ran privileged repairs in a terminal "so sudo/polkit can prompt". (Unrelated + to the doctor's "terminal states" — that word means a final outcome, not a tty. The + one open question is the dead-GUI console-recovery path; see the VERIFY in todo.org.) 16. *Passwordless privileged path* — a root-owned helper + a narrow NOPASSWD sudoers rule scoped to it, archsetup-installed, run as =sudo <helper> <verb>=. This gates decision 15 (a worker thread can't prompt). Absorbs the earlier DoT-toggle @@ -1096,7 +1210,7 @@ back in the config and restoring their scripts. The panel is additive — not wiring its clicks leaves the bar working as before. No credential store to roll back (secrets stay in NM throughout). -* Review findings [31/31] +* Review findings [40/40] ** DONE Define the structured diagnostics contract :blocking: The spec says the engine "emits JSON" and that diagnostics "reuse =captive= @@ -1608,6 +1722,144 @@ state), row content, one primary button per section, disabled-state rules, exact confirmation wording for reset/bounce/DNS-override/remove, the live "Get me online" escalation reporting, what survives panel close, and keyboard nav. +** DONE Reconcile the panel navigation source of truth :blocking: +Disposition: accept — folded into "V2 panel UX". V2 (Connections | Diagnostics | +Performance) is the sole current target; the shipped four-page stack is marked history, +not active design. +The spec now names at least three navigation shapes: the shipped four-page stack +(Connections / Diagnose / Repair / Speed test), the V2 three-tab plan +(Connections / Diagnostics / Performance), and the redesign task's Diagnostics +sub-row (Diagnose / Get Me Online / Advanced). That leaves an implementer free +to keep extra pages and buttons even though Craig is explicitly asking for the +opposite. Make V2 the sole current target: one panel opened from the bar, top +tabs =Connections | Diagnostics | Performance=, with Diagnostics owning the +read-only checks, repair stream, debug capture, doctor report, and related +diagnostic evidence. Mark the old four-page stack as shipped history only, not +active design. This blocks the redesign because the page model determines what +code is deleted, not just labels. + +** DONE Fold speed tests into the diagnostic story :blocking: +Disposition: modify — Craig pre-decided Speed test lives under Performance (decision +19), and Performance carries future throughput history, which meets this finding's own +"keep the tab only if it carries ongoing throughput" condition. Accepted the rest: +Diagnostics runs a lightweight inline latency/throughput probe as a Diagnose evidence +row (with skip conditions for offline / metered / no-backend), and the full speed +result is stored in the doctor report. Folded into "V2 panel UX → Diagnostics owns the +diagnostic story". +Speed test is currently isolated under =Performance=, while the Goal and user +mental model treat speed, latency, and packet loss as part of "diagnostics." +That split risks another top-level button/page whose only job is a diagnostic +measurement. Keep the top-level =Performance= tab only if it carries ongoing +throughput/history later; for V2, specify that Diagnostics can run a lightweight +performance check from the same Diagnose/Get Me Online flow when internet is +available, and that the full speed test is presented as a diagnostic evidence +row or secondary action rather than a separate repair-adjacent workflow. Define +when it is skipped (offline, metered/hotspot warning, missing backend) and how +the result is stored in the doctor report. This is blocking because otherwise +the implementation preserves avoidable navigation and misses a useful failure +signal. + +** DONE Define saved-list vs available-scan semantics :blocking: +Disposition: accept — folded into "V2 panel UX → Connections". Saved / Available now / +Wired groups; Rescan refreshes only the availability/signal layer, never the Saved +list. +=net list= merges saved profiles with in-range scanned networks, while the panel +copy calls the page "Connections" and the control "Rescan." It is not clear to a +user whether they are looking at saved connections, currently available +networks, or both. The current implementation confirms the ambiguity: +=connections.py= lists saved profiles MRU-first, merges live signal/security for +saved profiles that are in range, then appends unsaved in-range SSIDs with +=uuid: nil=. Rename and specify the groups: e.g. =Saved= (instant, does not +require scan), =Available now= (scan-backed, may still be loading/stale), and +=Wired=. =Scan= should refresh only the availability/signal layer, not gate the +saved profile list. This blocks readiness because it affects loading behavior, +button enablement, and whether unsaved rows can be selected. + +** DONE Replace the Add page with join-from-row behavior :blocking: +Disposition: accept — folded into "V2 panel UX → Connections". Selecting an unsaved +Available-now row is the join flow (SSID/security prefilled); the standalone Add modal +is deleted for visible networks; hidden/manual join lives behind Advanced. +The current Add dialog asks for an SSID as free text even though a scan usually +already found the SSID and security type. That is redundant UI and a common +network-manager mistake: it turns "join this visible network" into "copy a name +from the list and type it again." V2 should remove the standalone Add button and +modal for normal visible networks. Selecting an unsaved available row should +become the join flow: the SSID/security are prefilled from the row, open +networks connect with a confirmation only if needed, WPA/WPA2/WPA3-Personal ask +only for the password, and hidden/manual SSID is tucked behind an Advanced +"Join hidden network" affordance. Keep edit/create for enterprise profiles out +of v1/V2 unless explicitly added later. This blocks the redesign because it +changes the primary connection workflow and deletes a whole page/control. + +** DONE Pin the supported authentication types in the join flow :blocking: +Disposition: accept — folded into "V2 panel UX → Supported authentication classes". +The spec says "open + WPA-PSK" and "enterprise activate-only," but cafe/hotel +networks also commonly appear as open captive portals, WPA/WPA2/WPA3-Personal +(PSK/SAE), and sometimes transition-mode networks; less commonly they use +enterprise/802.1X, WEP, OWE/enhanced-open, MAC registration, voucher portals, or +proxy-required networks. Define the V2 join matrix from the scanned NM +=SECURITY= value: supported inline (open, captive/open, WPA/WPA2/WPA3 Personal), +activate-only if already saved (802.1X/enterprise), hidden-manual behind +Advanced, and unsupported/rare types with a clear in-panel explanation plus a +non-terminal next step. If an auth type is common enough to support, support it +in the panel; if it is too rare for V2, say "not supported here yet" and keep +the user in the same UI rather than sending them to a terminal tool. Also define +what security label appears in the row so the user knows why a password is or is +not requested. This blocks because the Add/Join deletion above cannot be +implemented safely without knowing which auth classes the simplified flow covers. + +** DONE Fix destructive confirmation tense and verification +Disposition: accept — folded into "V2 panel UX → Forget confirmation". +The Forget confirmation says "The saved password is deleted" before the user has +clicked Forget. That reads as if the destructive action already happened. Change +the copy to future tense and name the scope, e.g. "This will remove the saved +NetworkManager profile and its stored password from this machine." After the +operation, verify the UUID is gone, refresh the Saved list, and report either +"Forgot <SSID>" or "Could not forget <SSID>; nothing changed / partial state +<evidence>." This is non-blocking because the existing confirm prevents an +accidental click, but the wording is misleading and the V2 "verify every action" +decision should cover it. + +** DONE Make connection loading progressive and observable :blocking: +Disposition: accept — folded into "V2 panel UX → Connections (progressive loading)". +Opening the panel currently says "Loading connections..." while =net list= +collects both saved profiles and the WiFi scan. Saved profiles do not require a +network scan, so a slow scan should not delay the saved list. Split loading into +two phases: render saved NM profiles immediately, then overlay availability, +signal, and unsaved in-range rows when the scan completes. Show a small +scan-in-progress state with elapsed time and stale-last-scan age, and make +Rescan update only the scan-backed fields. This blocks because it is the direct +answer to "why does it take so long to see the list of connections?" and keeps a +bad radio scan from making the whole panel feel broken. + +** DONE Define the visual contract with Waybar and existing Archsetup UI +Disposition: accept — folded into "V2 panel UX → Visual contract". +The panel is a layer-shell popup anchored under Waybar, but the spec does not +state the visual contract. The live Waybar theme uses a dark rounded capsule +(=border-radius: 1rem=), golden border, compact monospace text, and state colors +for =custom/net=; the GTK panel currently has a generic title, stack switcher, +default GTK controls, and square-ish/default widget corners. Add a short style +section: panel should read as a Waybar-attached popup, not a separate app; match +Waybar's palette, border/radius, spacing density, and state colors; avoid square +corners where surrounding UI is rounded; keep cards out of cards; use compact +icon+label controls with tooltips for advanced repairs. Also cite any existing +Archsetup-owned GTK/panel conventions that should be reused. This is +non-blocking for engine work but should block final V2 UX acceptance. + +** DONE Add a diagnostics report affordance that users can actually find +Disposition: accept — folded into "V2 panel UX → Findable diagnostics report". +The observability design has a JSONL event log, =diagnose --json=, automatic +verbose capture, and redacted bundles, but the panel flow does not yet define +the user affordance that turns those into an inspectable diagnosis. Add a +Diagnostics-side "Copy report" / "Open report" action after every diagnose, +repair, and speed/performance run. The report should include the current step +statuses, elapsed time, final classification, last speed/latency result when +available, scan age, route/interface owner, relevant redacted event-log tail, +and bundle path when verbose capture ran. It must explicitly say whether any +repair mutated state and whether cleanup/verification passed. This is blocking +for observability because "logs exist somewhere" is not enough when the network +is already failing. + * Review and iteration history ** 2026-06-29 Mon @ 17:00:39 -0400 — Codex — reviewer @@ -1777,3 +2029,66 @@ online" escalation reporting, what survives panel close, and keyboard nav. Failure-mode coverage, Interactions, Panel, Resolved decisions, Implementation phases. Shipped commits referenced: dotfiles =a7d7559=, =eef6b0b=, =450b7f0=. Full catalog + redesign children in todo.org "Network panel redesign". + +** 2026-06-30 Tue @ 17:29:29 -0400 — Codex — reviewer + +- *What changed or was recommended:* Rubric: =Not ready= for the V2 redesign until + the new blocking findings are dispositioned. Added a fourth review batch focused on + diagnostics/observability and the simple UX questions Craig raised: reconcile the + active panel nav, fold performance/speed testing into the diagnostics story, make + saved-vs-available list semantics explicit, replace the Add page with join-from-row, + pin supported WiFi authentication classes, fix Forget confirmation tense, make + connection loading progressive, define the Waybar/Archsetup visual contract, and add + a findable redacted diagnostics report affordance. +- *Why:* The spec has strong engine and failure-mode coverage, but the current doc and + shipped implementation still replicate common network-manager UX mistakes: ambiguous + mixed lists, scan-gated saved data, redundant SSID entry, misleading destructive + copy, diagnostics split across too many pages, and observability that exists in logs + more than in the user's workflow. +- *Artifacts:* Findings appended under =Review findings= after reading the + spec-review workflow, the target spec, todo.org's redesign task, and current + dotfiles implementation files =net/src/net/connections.py=, + =net/src/net/gui.py=, =net/src/net/panel.py=, and Waybar =style.css=. + +** 2026-06-30 Tue @ 17:36:40 -0400 — Claude Code (archsetup) — responder (4th review) + +- *What changed:* Dispositioned all nine fourth-review findings (8 accept, 1 modify, + 0 reject) and wove them into a new "V2 panel UX — the target design" section. + Accepts: V2's three-tab nav is the sole target (the four-page stack is history); + Connections splits into Saved / Available now / Wired with Rescan touching only the + scan layer; selecting an unsaved row is the join flow (the Add modal is deleted); + a join matrix pins which NM SECURITY classes are inline-supported / activate-only / + hidden / unsupported; loading renders Saved first and overlays the scan; the Forget + copy goes future-tense + verified; every run ends with a Copy/Open redacted report; + and a Waybar visual contract (rounded capsule, golden border, state colors). Modify: + the speed-test finding kept Craig's decision-19 placement (full speed test under + Performance, which carries future history) while accepting a lightweight inline + latency probe as Diagnose evidence stored in the doctor report. Cookie [40/40]. +- *Why:* Codex read the live implementation and caught the UX places where the module + still replicated common network-manager mistakes — mixed lists, scan-gated saved + data, redundant SSID entry, misleading destructive copy, diagnostics scattered + across pages, observability that lived in logs more than the workflow. +- *Artifacts:* Findings 32-40 completed in place with dispositions; the modify reason + on the speed-test finding. New "V2 panel UX" section under Panel. todo.org redesign + task updated to point the V2 build at the dispositioned design. + +** 2026-07-01 Wed @ 10:43:18 -0400 — Claude Code (archsetup) — responder (cj comments) + +- *What changed:* Folded in Craig's three cj comments from his review. (1) Notification + rendering: title = "Networking", body = the failure label on its own line then the + canonical string. (2) VPN kill-switch: added a "kill-switch blocking" failure state + plus a detection-and-correction strategy off the =deferred-vpn= branch (rootless + cascade over =ip rule= fwmark 0xca6c / =wg show= / Proton =pvpn-*= NM connections / + =nft=/=iptables= drop tables / firewalld =drop= zone; classify blocking only when a + block artifact exists AND no tunnel is up; correction surfaces the exact root command + per artifact). (3) Terminals: strengthened decision 15 to "no terminal ever reports + to or collects input from the user", disambiguated from the doctor's "terminal + states" wording. +- *Why:* Craig's review annotations. The kill-switch closes a real gap in the + VPN-routed classification; the terminal directive makes the no-terminal rule + absolute for the module UX. +- *Artifacts:* Three cj blocks removed. VPN research subagent cited wg-quick man page, + Pro Custodibus, System76/Proton killswitch docs, and local =doctor.py:42= / + =classify.py:60= / =USNY.conf:15=. One open tension filed as a VERIFY in todo.org: + the dead-GUI console-recovery path (=make online= from a TTY) vs the no-terminal + directive. diff --git a/docs/design/2026-06-29-waybar-timer-module-spec.org b/docs/design/2026-06-29-waybar-timer-module-spec.org index 4b0ed0e..e8eecbe 100644 --- a/docs/design/2026-06-29-waybar-timer-module-spec.org +++ b/docs/design/2026-06-29-waybar-timer-module-spec.org @@ -1,5 +1,5 @@ #+TITLE: Waybar Timer Module (wtimer) — Design Spec -#+AUTHOR: Craig Jennings & Claude +#+AUTHOR: Craig Jennings #+DATE: 2026-06-29 * Goal diff --git a/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org b/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org new file mode 100644 index 0000000..cb195c6 --- /dev/null +++ b/docs/design/2026-07-02-waybar-expansion-animation-feasibility.org @@ -0,0 +1,53 @@ +#+TITLE: Waybar Expansion Animation — Feasibility Assessment +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-02 + +* Question + +The todo.org task "Smooth waybar expansion animation" [#C]: the collapse/expand +jump is abrupt, and a few systray icons pop in one-by-one afterward. Can the +expansion animate smoothly? + +* How the collapse actually works + +=waybar-collapse= rewrites the module arrays in a runtime copy of the config +(=$XDG_RUNTIME_DIR/waybar/config=) and sends waybar SIGUSR2. Waybar reloads: +it tears down every module widget and rebuilds the bar from the new config. + +* Findings + +1. *No widget survives the reload, so nothing can transition.* GTK3 CSS + transitions animate property changes on live widgets. The collapse + mechanism replaces the whole widget tree; there is no widget on both sides + of the change to interpolate. The jump is structural, not stylistic. +2. *GTK3 doesn't animate widget add/remove anyway.* Smooth insert/remove needs + =GtkRevealer= wrapping, which waybar does not use for modules. Making it do + so is an upstream waybar patch, not a config or CSS matter. +3. *The layer surface resize isn't animatable either.* Hyprland layerrules can + animate map/unmap (slide/fade), but the bar stays mapped through a collapse + — the same surface changes width. No compositor-side hook exists for that. +4. *A CSS-only fake covers custom modules at best.* Custom modules could emit + a "collapsed" class and transition font-size/padding toward zero (GTK3 CSS + can animate those). But the collapsed set includes built-ins — tray, + pulseaudio, workspaces — which take no script-driven classes. The result + would be half the modules gliding and half popping: worse than the clean + jump. +5. *Tray icons popping in one-by-one is separate and unfixable here.* That's + asynchronous StatusNotifier re-registration after the reload; each app + answers on its own schedule. Only keeping the tray alive across the change + (i.e. not reloading) avoids it. + +* Conclusion + +Not feasible with the current collapse mechanism, and no acceptable partial +measure exists. A real animation requires waybar itself to support dynamic +module sets with Revealer-style transitions (an upstream feature), or +replacing the collapse-by-reload design entirely. + +* Recommendation + +Close the task as infeasible-for-now (or park at [#D] with a pointer here). +Revisit only if waybar upstream gains dynamic module visibility (worth a +check at major waybar releases) or if the bar ever migrates to a custom +GTK4 shell — the Blueprint pipeline from the net panel would make Revealer +transitions natural there. diff --git a/docs/design/maintenance-console-design-ideas.org b/docs/design/maintenance-console-design-ideas.org new file mode 100644 index 0000000..066c25d --- /dev/null +++ b/docs/design/maintenance-console-design-ideas.org @@ -0,0 +1,527 @@ +#+TITLE: Maintenance Console — Design Ideas +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-06 + +* Direction + +A single-host maintenance console (GUI, waybar-launched) that surfaces every +health metric for this Arch workstation and, where the remediation is safe, +runs it. It reshapes the earlier install-health/workstation-contract framing +(=system-monitor-design-ideas.org=) into a maintenance surface with a doctor +action. + +The console is the glanceable, single-host version of the home project's +=system-health-check.org= workflow (~1000 lines, capability-dispatched across +ratio/velox/mybitch/truenas). The console owns the routine-maintenance and +at-a-glance-status slice; the workflow stays as the escalation path for +forensic investigation and as the only sanctioned way to run updates. + +* Scope: hosts (Q2 resolved 2026-07-06) + +v1 targets *both* daily drivers — ratio and velox — as first-class hosts, not +ratio-first-velox-later. Consequences for v1: + +- *Capability dispatch is mandatory from day one*, not deferred. The console + probes the live host and runs only applicable checks (btrfs RAID1 on ratio vs + ZFS-primary on velox; AMD amd-pstate on ratio vs Intel intel_pstate on velox). + This mirrors the home workflow's Phase 0 probe. +- *velox-only metrics ship in v1*: battery health (capacity / cycles) and the + unclean-shutdown / suspend-failure rate — both currently GAPs, both driven by + velox being the machine that travels and suspends. +- The ZFS-primary remediation paths (scrub, snapshot retention, pool health) are + built and tested, not stubbed. + +* Thresholds source of truth (Q3 resolved 2026-07-06) + +One machine-readable file, =maintenance-thresholds.toml=, holds every severity +value (cache-size trigger, disk %, scrub-age warn/crit, snapshot retention +limits, temp bands, backup-staleness windows, etc.). archsetup owns and ships +it; both consumers read the *installed* path, so neither reaches into the +other's repo: + +- the maintenance console (dotfiles code) reads it at runtime, +- the system-health-check workflow reads it instead of hardcoding severity + rules in prose. + +The workflow's hard-won values migrate into the TOML as the seed content (the +snapshot MONTHLY limit that bit /home, the 10 GB cache trigger, the scrub-age +bands). Install path is an implementation sub-question — a stable location both +consumers reach (e.g. =~/.config/archsetup/maintenance-thresholds.toml=). + +* Workflow ownership — move system-health-check into archsetup (proposed 2026-07-06) + +Craig's call: the =system-health-check.org= workflow should live in archsetup, +not the home project. Rationale — the home project is scoped to finances, +health, and personal matters; system design, execution, and maintenance are +archsetup's domain. Home only owns the workflow by inherited accident, not by +fit. Moving it here also collapses the Q3 coupling: the TOML source-of-truth and +its workflow consumer end up in the same project. + +Move scope (home → archsetup), to sequence when Craig gives the go: +- =system-health-check.org= (the workflow itself) +- =homelab-inventory/*.org= (ratio/velox/mybitch/truenas capability inventories + it cross-references) +- any home-project references to the workflow (startup, project-workflows index) + +Cross-project mechanics: the archsetup side (receiving the files, wiring the +TOML) is in-scope here; the home side (removing the originals, updating home's +references) is the home project's scope — handled by a handoff note to home's +inbox or a home session, not edited blind from here. + +* Panel shape + +Three regions, driven by the Automation column of the metrics table below: + +- *Actionable* (left) — metrics with a lever. Automation = Auto or Confirm. + Two doctor actions serve this column: "Clean up" fires every Auto metric + unattended; "Review & fix" opens a preview for the Confirm metrics and acts + only on approval. +- *Diagnostic* (right) — read-only telemetry. Automation = None or Human. A + state color (green/amber/red) and the value; no button. Red here is the + signal to run the workflow. +- *Updates* (quarantined strip) — Automation = Workflow. Shown as a count with + notable packages named; the only affordance is "run the workflow." The panel + never applies updates in place. + +*Bar glyph* tracks the worst *Diagnostic* state only — not the actionable +count. A big package cache is boring; a SMART failure is a fire. Actionable +clutter must not turn the bar red or it trains you to ignore it. + +** Doctor = live output wall (Q1 resolved 2026-07-06) + +"Clean up" and "Review & fix" are not fire-and-forget buttons. Running a doctor +opens an *output wall* — one lamp per action, streaming in realtime as each +runs: + +- *amber* while the action is running, +- *green* on success (with the reclaimed amount / result inline), +- *red* on failure. + +Feedback is always shown (not just when something was reclaimed) — you watch it +happen. This is the same live-results shape that should back the other doctors +(net, bluetooth), so every doctor in the system reads the same way. See the +todo task to retrofit the net + bt doctors to realtime lamp output. + +* Automation legend + +| Value | Meaning | +|----------+-----------------------------------------------------------------------------------------------------------| +| Auto | Doctor "Clean up" button — fire unattended, reversible or harmless | +|----------+-----------------------------------------------------------------------------------------------------------| +| Confirm | Doctor "Review & fix" — preview the change, act only on one click | +|----------+-----------------------------------------------------------------------------------------------------------| +| Human | Panel nudges; a human decides and acts (no button, not the agent) | +|----------+-----------------------------------------------------------------------------------------------------------| +| Workflow | vLater — metric shown read-only; no in-panel button. Agent-workflow assistance deferred (decided | +| | 2026-07-07, see below) | +|----------+-----------------------------------------------------------------------------------------------------------| +| None | Diagnostic only — no software remediation exists (hardware, telemetry) | +|----------+-----------------------------------------------------------------------------------------------------------| + +*Workflow buttons removed (decided 2026-07-07, prototyping E5).* The panel +carries no "run workflow" affordances — not on failed units, unclean boots, +updates, or CVEs. The metrics stay on the board as read-only telemetry; AI +assistance via the system-health-check workflow is a vLater feature. v1 ships +only determinate remedies (fixed, scriptable actions with predictable +outcomes). + +*Determinate remedies adopted (decided 2026-07-07).* Every remedy below is +Confirm-tier; contextual levers appear only when the metric is off-nominal: + +- /Service restarts & enables/: failed units (systemctl restart + reset-failed), + fail2ban, cronie, chronyd (+ makestep), tailscaled, snapper timers, zram + config re-apply. DNS/NetworkManager delegates to the net panel's doctor + (deep-link, not a duplicate repair chain). +- /Deterministic maintenance ops/: btrfs balance -dusage=50 (unallocated low), + reinstall owning packages on pacman -Qkk failures, rsyncshot RUN NOW (result + streams to the output well), smartctl -t short self-test, snapshot-retention + repair (write sane TIMELINE limits + cleanup — the /home lesson as a + one-press fix). btrfs device-error counter reset stays manual: resetting + without diagnosis masks a dying drive. +- /Composite macro/: RECLAIM SPACE on the disk-usage cell — runs every reclaim + lever (cache, journal, coredumps, app logs, docker tier-1, snapper cleanup) + as one output-wall stream. +- /Disruptive but determinate/: REBOOT behind arm-to-fire, offered when + running kernel != installed. +- /Still read-only/: temps, throttling, battery, memory/OOM, taint, journal + error content, kernel/hw events, listeners, unclean-shutdown rate — physical + or investigative; vLater AI territory. + +*Updates join the Confirm layer (decided 2026-07-07).* Two levers on the +updates strip, both behind the live-update guard (mesa/hyprland/wayland +runtime in the pending set): + +- UPDATE — repo + AUR system update. +- TOPGRADE — full ecosystem run. The panel's wrapper always passes + --disable git (topgrade's git step rebase-autostashes ~/code/*/ — never + under a live session). + +*MEM·PWR: evidence + two levers + expectation tags (decided 2026-07-07).* +Mostly physics, so the category leans watch-only — with these additions: + +- /CPU mode selector/: a free segmented control (PERF · BAL · POWER) writing + the EPP hint — set the active mode to anything, not drift-repair against a + declared default. amd-pstate on ratio, intel_pstate on velox. +- /Battery charge limit/ (velox-only, capability-gated): SET 80% writes + charge_control_end_threshold — the standard longevity cap. Battery *health* + (capacity vs design, cycles) stays watch-only hardware telemetry. +- /Evidence drill-downs/ (digest idiom): top-5 RAM consumers under memory, + recent boots listed clean/unclean under the unclean-rate, and throttle/OOM + events with timestamps. Evidence makes the numbers actionable even where no + button exists. +- /KILL on top-memory items/ (revised 2026-07-07 — Craig): arm-to-fire, four + guards: the arm shows the exact victim (name + size); SIGTERM not SIGKILL, + with the outcome reported to the wall; PID + process name revalidated at + fire time so a recycled PID can't be hit; session-critical names (systemd, + the compositor, the panel itself) render a disabled key — protected. A + SIGKILL escalation for TERM-survivors is vLater. +- /Expectation-setting, panel-wide/: every leverless cell's sub-line carries an + explicit tag ("hardware — watch only", "evidence below"), and each subpanel + header shows the split — "N fixable · M watch" — so the user knows their + agency before reading a single cell. + +*Refresh cadence (decided 2026-07-07).* Four tiers, matching probe cost: + +- /Live group, panel open/: temps, memory free + top consumers, throttle + state — re-read every ~3 s while their subpanel is visible, gated on + panel-open exactly like the audio panel's meters. Stop when hidden. +- /Fast local tier, panel open/: re-probed every ~30 s while the panel is up; + additionally, any metric re-probes immediately after an action that touches + it (fire CLEAN → cache re-measured, not assumed). +- /On open/: the hydration tiers re-run (fast reads first, process probes + behind them — sub-second perceived). +- /Network tier/: checkupdates / arch-audit / AUR / firmware stay on the + hourly systemd-timer cache with age shown; refreshed on demand only. +- /Panel closed/: the waybar glyph is fed by a light background scan every + ~30 min (systemd timer writing the state file the glyph reads) — the bar + stays honest without the panel running. + +*Journal errors get a digest, not a fix (decided 2026-07-07).* No generic +remedy exists — an error-priority line is a symptom of an arbitrary subsystem — +so the panel ships four determinate assists instead: + +- /Digest/: the cell expands to errors grouped by syslog identifier — count, + message snippet, first/last seen, and the exact next command (journalctl -u + <unit> -b) when the identifier maps to a unit. Top-10 groups, read-only. +- /MARK KNOWN with a full lifecycle/: arm-to-fire showing the exact pattern + before it stores; marked groups move to a dim KNOWN section (never vanish) + with per-row UNMARK; every mark/unmark logs to the results wall; marks carry + date + example. Patterns bind to identifier + message, never a whole unit — + a muted service's *new* errors still surface. +- /Two noise layers/: shipped defaults (bluetoothd HFP, pixman, xkbcomp) in + the packaged TOML, user marks in a separate user file merged over it (a + template sync never eats curation). CLEAR MARKS (arm-to-fire) empties the + user layer and re-enables shipped defaults; unmarking a shipped default + records a disable flag in the user layer. +- /OPEN JOURNAL/: launches a terminal running journalctl -p err -b — the same + delegation pattern as NET DOCTOR. + +Ruled out: auto-restarting units that log errors (error ≠ failed) and +keyword-driven fix suggestions (vLater AI territory). + +*Full-sweep findings — all committed to v1 (decided 2026-07-07).* Every metric +was audited against the converged checklist (honest label, evidence digest +where a count hides detail, curation lifecycle where "expected" is config +knowledge, guarded per-item remedies, cross-links, watch-only tags). Adopted, +all prototyped: + +- /Storage/: disk top-consumers digest (evidence only — no file deletion + keys); per-device error rows on RAID1 when counters are nonzero, + cross-checked against SMART; SMART sub-line carries the last self-test + result. Spec note: a real scrub runs hours — the ring needs a running-% + state, not an instant reset. +- /Packages/: orphan digest (name + size) with per-package REMOVE (armed) and + KEEP — the curation lifecycle encoding "intentional, not orphaned" (the + rust lesson); batch REMOVE ALL skips kept packages. Per-file pacnew rows + tagged safe-delete (reflector-managed) vs needs-merge, MERGE delegating to + a terminal diff. CVEs named: package · CVE id · severity. AUR and firmware + names spelled out. +- /systemd/: failed units upgraded from a count-lever to a per-unit roster + (name · since · exit code · journalctl hint) with per-row RESTART + RESET; + is-system-running names its cause ("degraded — N failed units below"); + taint letters decoded. +- /Logs/: coredumps grouped by binary (count · last · coredumpctl hint), + cleared with the CLEAR action; kernel/hw events listed when not clean + (hardware — watch only). +- /Services/: docker system df breakdown (images / containers / volumes / + build cache with per-type reclaimable); stopped containers upgraded to the + full signal/expected curation lifecycle (MARK EXPECTED / UNMARK, shipped + default: winvm) with per-container START; cron expected-entries drift + roster. +- /Snapshots/: count split by type — timeline (auto-pruned) · single + (manual — escapes timeline cleanup, the pile-up risk) · pre/post — with + oldest-single named and DELETE STALE (armed, keeps newest 2) when singles + accumulate. +- /Network minors/: fail2ban shows recent-ban count; NTP shows offset. + +Rationale for prototyping everything (Craig): real estate and complexity have +bitten before — surface those limits in the disposable prototype, not after +functionality exists behind the UI. + +*Vertical compression → rotary band selector (2026-07-07).* First attempt — +the MEM·PWR three-column layout on Packages/Logs/Services — lost too much row +detail to third-width truncation (Craig's verdict after use). Replaced by a +*rotary band selector*: the amplifier input-selector idiom. A machined knob +(click to cycle) whose needle swings to engraved band labels, one per evidence +section (ORPHANS · PACNEW · ADVISORIES; SIGNAL · KNOWN NOISE · COREDUMPS · +KERNEL/HW; CONTAINERS · DOCKER DISK · CRON & BACKUPS). Each band carries its +own status lamp (section health at a glance without switching) plus a count; +the selected band gets a gold underline and the needle. One section renders at +a time at full width, restoring complete row detail. Deliberately distinct +from the category tiles, console keys, and the CPU-mode segmented control — +each selection idiom in the panel now has its own visual voice. MEM·PWR keeps +its three-column evidence strip (short rows fit fine at third-width). + +*Listeners get the same treatment (decided 2026-07-07).* The count becomes +"unexpected listeners" — evidence digest (process · port · bind address from +ss -tlnp), expected-list curation with the full MARK EXPECTED / UNMARK / +CLEAR lifecycle (shipped defaults: sshd, mpd, tailscaled; user marks in the +user layer), and guarded per-socket remedies: STOP (systemctl stop, armed) +when a unit owns the socket, KILL (SIGTERM, armed) otherwise. Severity keys +on unexpected AND public-bind (0.0.0.0/::) — a loopback listener warns, an +exposed one fails — and when ufw is down the signal header names the exposure +("ufw down — N public binds exposed"). Stopped containers likewise gained a +contextual START lever (allowlist: winvm), and the firewall its ENABLE. + +*Updates strip border is state-tiered (decided 2026-07-07).* Green when +nothing pending, amber for ordinary pending/AUR/firmware counts, red when +CVEs exist or pending exceeds the "a lot" threshold (or the update cache has +gone stale — staleness window in the TOML). The CVE badge renders only when +the count is nonzero. + +*Guard arms instead of blocking (revised 2026-07-07).* When the guard trips, +the key arms (red, "press again to run anyway — or apply from a TTY") rather +than hard-refusing. The user decides; the footgun is acknowledged and +deliberately handed over. After a system update lands, the panel offers a +reboot: a REBOOT key (arm-to-fire) appears on the updates strip and the +reboot-required metric flips. + +*No per-ecosystem update metrics.* Topgrade's step set (yay, rustup, cargo, +pipx, npm/pnpm, gem, go, flatpak, fwupd, tmux/zsh/nvim plugins, git repos) has +no cheap offline "updates available?" probe — mirroring it means a network +round-trip per registry at panel-open. Instead: one *topgrade freshness* +metric (wrapper stamps last-run time; threshold in the TOML) whose remedy is +the TOPGRADE lever, plus a *firmware updates* count in the updates strip +(fwupd refreshes metadata on its own timer; the panel reads the cache). + +Rationale for the hard lines: system updates are Workflow, never Auto — the +2026-06-07 Hyprland crash was a live -Syu swapping mesa+hyprland under the +running session, and the standing rule is never -Syu live under Hyprland when +the mesa/hyprland/wayland runtime is in the set. Hardware findings (SMART, MCE, +thermal) are None — the fix is replacing a drive or clearing a fan, not +software. + +* Metrics — Storage & filesystem integrity + +| Metric | Fix / lever | Automation | Notes | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Package cache size | paccache -r / -ruk0 | Auto | Reclaim, all re-downloadable | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| paccache -rk1 (keep 1 version) | paccache -rk1 | Confirm | Frees most; kills downgrade | +| | | | headroom | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Disk usage (df) | — | None | Fix is cache/snapshot/prune | +| | | | levers | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs unallocated space | — | None | Chunk headroom; diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs scrub age | btrfs scrub start | Confirm | GAP; ZFS has this, btrfs doesn't. | +| | | | IO-heavy, on-demand | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs device stats (per-drive | btrfs device stats --reset | None | GAP; RAID1 early-warning ahead of | +| error counters) | | | SMART; reset after review | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS pool health / errors | — | None | CRITICAL if state != ONLINE or | +| | | | errors > 0 | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS scrub age | zpool scrub | Confirm | Covered | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS capacity | — | None | Perf degrades > 80%; the ZFS | +| | | | headroom metric (no unallocated | +| | | | concept) | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS fragmentation (FRAG) | — | None | GAP; no defrag exists — the | +| | | | remedy is snapshot pruning + | +| | | | staying under 80% | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS autotrim (SSD TRIM) | zpool set autotrim=on / zpool | Confirm | GAP; velox; the fstrim.timer | +| | trim | | counterpart on ZFS | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| fstrim.timer enabled + firing | systemctl enable --now | Confirm | GAP; standard SSD hygiene | +| | fstrim.timer | | | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| pacman file integrity (Qkk) | reinstall package | Workflow | GAP; modified/missing files need | +| | | | judgment | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART health | — | None | Hardware; replace drive | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART wear / temperature | — | None | Diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| + +* Metrics — Snapshots + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapper count / retention | snapper cleanup / delete | Confirm | Manual (single) snapshots need explicit | +| | | | choice | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| ZFS snapshot count / retention | zfs destroy | Confirm | Runaway retention | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapshot auto-timer running | systemctl enable timer | Confirm | Is the auto-snapshot service firing | +|--------------------------------+--------------------------+------------+---------------------------------------------| + +* Metrics — Packages & security + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Orphaned packages | pacman -Rns (named args) | Confirm | Review first (rust looked orphaned, | +| | | | was intentional) | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Pending updates count | — | Workflow | Never auto; workflow-only | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| .pacnew files | diff + merge / delete | Confirm | Allowlist auto-deletes mirrorlist / | +| | | | locale.gen | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| arch-audit CVEs | reviewed update | Workflow | GAP; top-priority add. Fix is an | +| | | | update | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Keyring freshness | pacman -Sy archlinux-keyring | Confirm | GAP; stale keyring breaks update | +| | | | signatures | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| AUR / foreign staleness (Qm) | paru -Sua | Workflow | GAP; AUR updates reviewed, not auto | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Reboot required (kernel mismatch) | reboot | Human | GAP; uname -r vs /usr/lib/modules. | +| | | | You pick when | +|-----------------------------------+------------------------------+------------+--------------------------------------| + +* Metrics — systemd & boot + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| is-system-running (degraded?) | — | None | GAP; one-token whole-system verdict; | +| | | | candidate for the bar glyph | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Failed systemd units | restart / investigate | Workflow | Depends why it failed | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Maintenance timers enabled + | systemctl enable timer | Confirm | GAP; meta-metric — are | +| firing | | | paccache/btrfs-scrub/fstrim/reflector/snapper | +| | | | firing | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Kernel taint flag | — | None | GAP; tainted != 0 = proprietary module or | +| | | | prior crash | +|--------------------------------+------------------------+------------+-----------------------------------------------| + +* Metrics — Logs & coredumps + +| Metric | Fix / lever | Automation | Notes | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Coredump count | coredumpctl clean (keep recent) | Auto | Keep last few days for forensics | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journald disk usage | journalctl --vacuum-size/time | Auto | Bounded journal | +|------------------------------------+---------------------------------+------------+----------------------------------| +| App-log cleanup (>7d) | log-cleanup cron trigger | Auto | Already a cron; manual trigger | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journal error count (real vs | — | Workflow | Forensic; noise-filtered count | +| noise) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Kernel/hardware events | — | None | Forensic, hardware | +| (MCE/USB/thermal/GPU) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| + +* Metrics — Memory, thermal, power + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Memory free / OOM kills | — | Workflow | OOM = investigate | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Swap / zram present + healthy | — | None | Diagnostic | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| CPU / GPU temperatures | — | None | Hardware | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Thermal throttling active | — | None | Cooling issue | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Battery health (capacity / cycles) | — | None | GAP; laptop; ties to open suspend todo | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Unclean-shutdown / suspend-failure rate | — | Workflow | GAP; ratio flagged ~75% unclean 2026-06-08 | +|-----------------------------------------+-------------+------------+--------------------------------------------| + +* Metrics — Network & security posture + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+---------------------------+------------+-----------------------| +| DNS / NetworkManager reachability | restart NM | Workflow | CRITICAL if down | +|-----------------------------------+---------------------------+------------+-----------------------| +| Firewall active (ufw / nftables) | ufw enable | Confirm | GAP; security posture | +|-----------------------------------+---------------------------+------------+-----------------------| +| Unexpected listeners (ss -tlnp) | — | None | GAP; security review | +|-----------------------------------+---------------------------+------------+-----------------------| +| Tailscale peers | tailscale up | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| fail2ban running + bans | systemctl start | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| NTP sync (chrony) | systemctl restart chronyd | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| + +* Metrics — Services, backups, virt + +| Metric | Fix / lever | Automation | Notes | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| rsyncshot backup freshness | — | Workflow | CRITICAL if daily > 48h; investigate | +| | | | failure | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker/podman reclaimable | prune tier 1 / tiers 2-3 | Confirm | Tier 1 nearly Auto; 2-3 destructive | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker stopped containers | — | None | Mostly expected (WinVM on-demand) | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| libvirt VM state | — | None | Expected off | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Cron running + expected entries | systemctl enable cronie | Confirm | rsyncshot + log-cleanup entries | +|---------------------------------+--------------------------+------------+--------------------------------------------| + +* Architecture & testing (decided 2026-07-07) + +*CLI-first, GUI as a face.* The console ships as the fourth panel sibling: a +=maint= Python package in dotfiles (like =net/=, =bt/=, =audio/=) with probe +modules (read-only collectors), a remedies module, =cli.py=, and =gui.py= +driving the same code. =maint status --json= is the contract; =maint fix +<thing>= is every lever. The GUI never does anything the CLI can't. + +Safety mechanics baked into the CLI: +- global =--dry-run= prints the exact command instead of executing — free test + surface, and the GUI's arm-press can display it ("this will run: …"). +- hard read/write split: collectors never elevate; every remedy is an + allowlisted exact argv in one small auditable module. + +*Four test layers (safest → scariest):* + +1. /Unit, fake binaries — no VM, no root (~90% of surface)./ Probes are + parsers over command output: feed canned smartctl/btrfs/journalctl/pacman/ + ss/docker output via fakes on PATH (the net suite's fake-curl and audio's + fake-parec pattern). Remedies tested as command construction (assert the + argv, don't run it). The live-update guard is a pure function over a + package list. +2. /Read-only integration on the live machine./ All collectors are read-only + by design — =maint status --json= runs safely against real hosts. +3. /Remedies in a VM — archsetup's existing harness./ + =scripts/testing/run-test.sh= boots the installer VM; a maint scenario + breaks things deliberately over ssh (stop cronie, mask fstrim, orphan + packages, fill the cache), runs =maint fix …=, asserts post-state. No GUI + in the VM. Add qcow2 snapshot/restore between remedy tests so each starts + pristine and destructive remedies can't contaminate each other. Pure + pacman-level tests may use a throwaway systemd-nspawn container instead + (lighter); the VM stays for systemd/btrfs/reboot territory. +4. /GUI on the host, never in the VM./ AT-SPI smoke like the sibling panels, + driven by fixture data. The prototype's GOOD/BAD snapshots become those + fixtures — =MAINT_PANEL_FIXTURE=bad= renders the degraded board without a + degraded machine, conforming to the =maint status --json= schema. + +* Open questions + +- RESOLVED 2026-07-06 — live output wall (amber running / green done / red fail), + realtime, always shown. See "Doctor = live output wall" above. +- RESOLVED 2026-07-06 — velox is a first-class v1 target alongside ratio. See + "Scope: hosts" below. +- RESOLVED 2026-07-06 — single machine-readable thresholds file + (=maintenance-thresholds.toml=) is the source of truth, *owned by archsetup*. + Both the console and the system-health-check workflow read it, so they can + never drift. See "Thresholds source of truth" and "Workflow ownership" below. diff --git a/docs/design/system-monitor-design-ideas.org b/docs/design/system-monitor-design-ideas.org new file mode 100644 index 0000000..26619a0 --- /dev/null +++ b/docs/design/system-monitor-design-ideas.org @@ -0,0 +1,1008 @@ +#+TITLE: System Monitor Design Ideas +#+DATE: 2026-07-04 +#+TODO: TODO | DONE +#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED + +* DRAFT Status +:PROPERTIES: +:ID: system-monitor-design-ideas +:END: +- [2026-07-04 Sat] DRAFT — initial design sketch for a health monitor covering + Archangel ISO/base-install health, Archsetup workstation health, and the local + laptop as the daily canary. + +* Metadata + +| Field | Value | +|--------+--------------------------------------------| +| Status | draft | +|--------+--------------------------------------------| +| Owner | Craig Jennings | +|--------+--------------------------------------------| +| Repos | archsetup, archangel, dotfiles | +|--------+--------------------------------------------| +| Kin | net panel, bluetooth panel, audio panel | +|--------+--------------------------------------------| + +* Problem + +Archangel and Archsetup can fail in ways that are individually obvious only +after the damage is done: an ISO build goes stale against Arch or archzfs, +ZFSBootMenu or GRUB boots once but not after the first upgrade, a snapshot +hook silently disappears, a package database ages out, systemd services fail +after a reboot, or the desktop contract is technically installed but not +usable. + +The health surface should compress those risks into one operational question: +"Can I trust a fresh install, and is this current workstation drifting away +from the known-good install contract?" + +This monitor is not a generic CPU/RAM graph. It is an install-health and +workstation-contract console. CPU, memory, and temperature belong only as +secondary context unless they block install/test operations. + +* Priority Model + +Rank metrics by the cost of blindness: what happens if Craig never sees the +metric, no one mitigates it, and the next install/upgrade/reboot simply happens. + +Severity: + +- =P0= — can cause data loss, unbootable systems, or loss of rollback path. +- =P1= — can break fresh installs, upgrades, remote access, or core + workstation use. +- =P2= — causes degraded workstation behavior, security drift, or accumulating + maintenance debt. +- =P3= — useful context, not a release gate by itself. + +The panel should sort by live severity first, then by this priority. A red =P2= +row appears above a green =P0= row, but in the steady state the layout keeps the +P0/P1 rows in the first viewport. + +* Priority Ranking + +| Rank | Priority | Metric | Why this rank exists | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 1 | P0 | Storage health | Silent pool/filesystem degradation is the nearest thing to data loss. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 2 | P0 | Snapshot safety coverage | Without snapshots, upgrades lose their rollback safety net. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 3 | P0 | Bootloader and EFI redundancy | A machine that cannot boot is operationally dead. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 4 | P1 | First-upgrade bootability | Catches the classic "installed fine, broke after update" failure. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 5 | P1 | End-to-end VM install pass rate | Best release gate for the whole Archangel + Archsetup chain. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 6 | P1 | Package sync and repo freshness | Arch, archzfs, keyring, and mirror drift are leading break signals. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 7 | P1 | Archangel ISO reproducibility | If current inputs cannot build an ISO, recovery/install confidence is stale. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 8 | P1 | Post-install service health | Network, DNS, SSH, and user services decide whether the system is usable. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 9 | P1 | Archsetup state/log cleanliness | Prevents "half-installed but looks fine" machines. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 10 | P2 | Workstation contract checks | Confirms this is Craig's workstation, not just generic Arch. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 11 | P2 | Backup and rollback readiness | Catches loss of off-machine recovery and edited-file backups. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 12 | P2 | Security and hardening drift | Important, but usually less immediately destructive than boot/storage. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| + +This order intentionally puts ZFS/Btrfs health above VM install evidence. A +broken future install is expensive; silent damage to the current root or backup +chain is worse. + +* Consequence Matrix + +This is the design justification for every metric. A row earns panel space only +if blindness has a clear failure mode and Doctor has at least a useful +diagnostic or mitigation. + +| Rank | Metric | If never seen / never mitigated | Typical failure | Worst plausible failure | Doctor posture | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 1 | Storage health | Disk, pool, or metadata degradation accumulates silently. | Correctable ZFS/Btrfs errors, low EFI/root space, stale scrub. | Data loss, degraded root, failed import/mount during boot. | Diagnose + scrub/cleanup with confirmation. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 2 | Snapshot safety coverage | Upgrades proceed without a rollback point. | Missing recent snapshot or missing pre-pacman hook. | Bad upgrade cannot be rolled back cleanly; manual repair required. | Create snapshot; restore hook; never rollback. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 3 | Bootloader and EFI redundancy | Boot path rots until the next reboot or disk failure. | Missing GRUB/ZBM file on one EFI partition. | Unbootable machine after update, firmware reset, or disk loss. | Diagnose; regenerate config; advanced reinstall only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 4 | First-upgrade bootability | Installs look good until the first real system update. | Kernel/initramfs/bootloader mismatch in VM. | Fresh bare-metal install dies on first reboot after upgrade. | VM-only upgrade test; collect boot evidence. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 5 | End-to-end VM install pass rate | Unit tests give false confidence about the real workflow. | Current branch fails one filesystem path or desktop assertion. | Bare-metal install fails mid-flight after disks are wiped. | Run/schedule VM test; clean stale VM artifacts. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 6 | Package sync/repo freshness | Arch/keyring/archzfs drift surprises the next install. | 404s, stale keyring, bad mirror, stale archzfs DB. | Installer cannot pacstrap or installs mismatched ZFS/kernel bits. | Refresh DB; update keyring; reflector. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 7 | ISO build reproducibility | Recovery/install media confidence becomes historical. | AUR package fails to build; mkarchiso or DKMS breaks. | Need rescue/install media and discover no current ISO can be built. | Parse logs; clean work; explicit rebuild. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 8 | Post-install service health | System is installed but degraded in daily operation. | DNS, NetworkManager, fail2ban, tailscale, or user service down. | No remote access, no network, broken sync, or security tooling off. | Restart/re-enable classified services only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 9 | Archsetup state/log cleanliness | Half-completed provisioning masquerades as success. | Missing marker, log error, skipped step after resume. | Fresh workstation lacks critical config but looks mostly usable. | Summarize; rerun resumable archsetup. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 10 | Workstation contract checks | The box drifts from "Craig's workstation" to generic Arch. | Dotfile symlink broken, keyring wrong, missing tool/package. | Desktop/session workflow is broken during real work. | Restow, repair perms, reinstall with confirm. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 11 | Backup and rollback readiness | Recovery assumptions go stale. | Missing =.archsetup.bak=, backup timer stale, dry-run fails. | Local rollback works but important personal/system state is gone. | Dry-run, start configured job, no deletes. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 12 | Security and hardening drift | Small protective edits quietly regress. | SSH/firewall/fail2ban/sysctl/EFI mask drift. | Exposed service or weakened local boot/config protections. | Restore owned snippets; no broad rewrite. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| + +The worst cases are intentionally conservative, not dramatic. The monitor is +useful because it catches the boring early signal: stale, missing, not recently +tested, or silently degraded. + +* Product Shape + +Use the existing instrument-console panel language rather than a dashboard page: +lamps for pass/warn/fail state, engraved metric groups, dense rows, physical +console keys for explicit actions, and an output well for the last diagnosis. + +The bar module should be tiny: + +- icon: a pulse/terminal glyph or compact =SYS= label. +- state lamp: green/yellow/red/grey. +- text: one word only: =OK=, =WARN=, =FAIL=, =STALE=, =RUNNING=, =UNKNOWN=. +- click: opens the system monitor panel. +- middle click or secondary action: run a cheap refresh only, never repairs. + +The panel should be one screen with internal scroll only where needed. The +first viewport should show the decision state and the Doctor controls without +scrolling. + +* Layout + +** Faceplate + +Top row: + +- State lamp + state word: =OK= / =WARN= / =FAIL= / =STALE= / =RUNNING= / + =UNKNOWN=. +- Unit label: =SYS·01=. +- Scope segmented control: =HOST= / =INSTALL= / =BUILD=. +- Badges: =ZFS= or =BTRFS=, =VM STALE=, =DB STALE=, =SNAPSHOT=, =BACKUP=, + =ROOT= when elevated actions are available. +- Close button. + +The scope control changes the metric emphasis, not the underlying data model: + +- =HOST= is this laptop/workstation right now. +- =INSTALL= is the last Archangel+Archsetup VM install result. +- =BUILD= is ISO, AUR repo, archzfs, and test artifact health. + +** Health Stack + +Arrange metrics as four horizontal bands. Each band has a section title, a +summary lamp, two to four row lamps, and a short "age" or "count" value. + +1. =BOOT + STORAGE= + - bootloader + - EFI redundancy + - pool/filesystem + - snapshots + +2. =INSTALL PIPELINE= + - first-upgrade reboot + - ZFS VM install + - Btrfs VM install + - ISO build + +3. =PACKAGES + SERVICES= + - pacman sync freshness + - archzfs/AUR health + - failed services + - journal errors + +4. =WORKSTATION CONTRACT= + - Archsetup markers/logs + - user/dotfiles + - desktop/session + - backups/replication + +Each row is clickable. Clicking a row opens the evidence drawer in the output +well with: + +- last command run +- normalized verdict +- raw excerpt, redacted where needed +- suggested Doctor action, if any + +** Console Keys + +Use physical console-key buttons, same family as net/bt: + +| Key | Purpose | +|--------------+------------------------------------------------------------| +| REFRESH | Cheap read-only probe of host state | +|--------------+------------------------------------------------------------| +| DOCTOR | Diagnose, classify, run safe mitigations, re-check | +|--------------+------------------------------------------------------------| +| TEST VM | Run or schedule Archsetup VM validation | +|--------------+------------------------------------------------------------| +| BUILD ISO | Run or schedule Archangel ISO build | +|--------------+------------------------------------------------------------| +| CLEAN | Clean old test artifacts, package cache, stale logs | +|--------------+------------------------------------------------------------| +| SNAPSHOT | Create a manual pre-change snapshot | +|--------------+------------------------------------------------------------| + +Keys that can take a long time stream progress into the output well. Mutating +keys must use arm-first behavior: + +- first click arms for 3 seconds and explains the action. +- second click runs. +- destructive cleanup names what will be deleted before it runs. + +* Metric Details + +Details below are grouped by workflow rather than priority. The authoritative +importance order is the priority table above. + +** 1. End-to-end VM install pass rate =P1/rank 5= + +Problem overcome: unit tests can pass while the real install is broken by +mirrors, bootloader state, pacstrap, SSH, disk layout, or the actual desktop +contract. This metric fights false confidence. + +Representation: + +- Lamp row in =INSTALL PIPELINE=. +- Two child lamps: =ZFS= and =BTRFS=. +- Age chip: =last pass 2d= / =never= / =stale 14d=. +- Red if either required filesystem has no recent pass. + +Tools: + +- =scripts/testing/run-test.sh= +- =scripts/testing/create-base-vm.sh= +- =pytest= testinfra suite under =scripts/testing/tests/= +- =qemu-img=, =qemu-system-x86_64=, =sshpass= + +Doctor: + +- read-only first: summarize last =test-results/*/test-report.txt= and failing + test names. +- mitigation: offer =TEST VM= for the failed filesystem. +- cleanup: remove stale temporary VM overlays before retrying. +- no automatic retry loop if the failure is in the installer itself. + +** 2. First-upgrade bootability =P1/rank 4= + +Problem overcome: the machine can boot immediately after install but fail after +the first =pacman -Syu= because initramfs hooks, ZFS modules, GRUB, ZFSBootMenu, +or kernel packages drift. + +Representation: + +- Lamp row: =first-upgrade reboot=. +- Badge: =not run=, =passed=, =failed=. +- Evidence drawer includes boot count, kernel version, and last reachable SSH + timestamp. + +Tools: + +- VM harness +- =pacman -Syu= +- =reboot= +- =ssh= reachability checks +- =journalctl -b -1= where available + +Doctor: + +- run upgrade-in-VM only, never on host without explicit confirmation. +- if failure is ZFS, collect =zpool import=, =lsinitcpio=, =mkinitcpio.conf=, + and EFI files. +- if failure is Btrfs, collect =grub.cfg=, =crypttab=, =fstab=, and snapper + config. + +** 3. Package database freshness and sync health =P1/rank 6= + +Problem overcome: Arch rolling-release state changes faster than installer +assumptions. Stale sync databases, stale keyrings, archzfs drift, or broken +mirrors are leading indicators of a failing install. + +Representation: + +- Row lamp in =PACKAGES + SERVICES=. +- Small meter: newest sync DB age vs threshold. +- Child chips: =core=, =extra=, =multilib=, =archzfs=. +- Yellow over 48 hours, red over 7 days or failed sync. + +Tools: + +- =find /var/lib/pacman/sync= +- =pacman -Syyu --needed archlinux-keyring= +- =checkupdates= +- =reflector= +- =pacman-conf= + +Doctor: + +- safe: refresh package databases. +- safe: update =archlinux-keyring= before full upgrades. +- mitigation: run =reflector= with the configured country/age policy. +- no unattended full system upgrade from the panel unless separately approved. + +** 4. Archangel ISO build reproducibility =P1/rank 7= + +Problem overcome: an old "good" ISO can hide broken current inputs. Archiso, +archzfs, DKMS, AUR package recipes, and pacoloco cache state can all break the +next install. + +Representation: + +- Row lamp in =INSTALL PIPELINE= or =BUILD= scope. +- Shows latest ISO date, kernel version, and AUR manifest age. +- Red if latest build failed or no ISO exists. +- Yellow if latest successful ISO is older than the configured freshness + window. + +Tools: + +- =make build= in =~/code/archangel= +- =build.sh --skip-aur= for fast non-AUR iteration +- =build-aur.sh= +- =mkarchiso= +- =pacoloco= status if installed + +Doctor: + +- read-only: parse latest =out/*.log= for pacman, DKMS, archzfs, AUR, and + mkarchiso failures. +- cleanup: safe build-work cleanup only through Archangel's cleanup function + or =make clean=. +- mitigation: suggest =--skip-aur= when the failure is unrelated to baked AUR. +- build retry is explicit via =BUILD ISO=, not automatic. + +** 5. ZFS/Btrfs storage health =P0/rank 1= + +Problem overcome: the root filesystem can degrade silently before the user +notices. For ZFS this means pool errors or degraded vdevs; for Btrfs this means +device stats, scrub failures, metadata pressure, or degraded RAID. + +Representation: + +- =BOOT + STORAGE= band. +- Filesystem-specific lamp grammar: + - ZFS green: =zpool status -x= healthy. + - Btrfs green: device stats clean and recent scrub clean. +- Capacity strip for root/home/EFI. + +Tools: + +- ZFS: =zpool status -x=, =zpool list=, =zfs list=. +- Btrfs: =btrfs device stats=, =btrfs filesystem usage=, + =btrfs scrub status=. +- Common: =df -h=, =findmnt=, =lsblk=. + +Doctor: + +- safe: start a scrub only with arm-first confirmation. +- safe: clear stale Btrfs stats only after a clean scrub and explicit + confirmation. +- mitigation: warn on low EFI/root space and offer package cache cleanup. +- never destroy snapshots, pools, subvolumes, or datasets from Doctor. + +** 6. Snapshot safety coverage =P0/rank 2= + +Problem overcome: rollback safety is assumed during upgrades but can disappear +when hooks, services, or snapshot tools drift. + +Representation: + +- Row lamp: =snapshots=. +- Child chips: =genesis=, =pre-pacman=, =recent=, =pruned=. +- Yellow if no recent snapshot. +- Red if genesis or pre-transaction hook is missing. + +Tools: + +- ZFS: =zfs list -t snapshot=, =zfs-pre-snapshot=, + =/etc/pacman.d/hooks/zfs-snapshot.hook=. +- Btrfs: =snapper list=, =snap-pac=, =grub-btrfs-mkconfig=, + =/.snapshots=. +- Common: =pacman -Q= for snapshot packages. + +Doctor: + +- safe: create a manual snapshot. +- safe: reinstall or re-enable missing hook only if the expected script exists. +- cleanup: prune only snapshots matching the tool-owned policy and prefix. +- mitigation: show exact command for manual rollback; do not perform rollback + from the panel. + +** 7. Bootloader and EFI redundancy =P0/rank 3= + +Problem overcome: single-disk bootloader success can mask missing redundant EFI +installs on multi-disk systems. A system can also pass install but lose a boot +entry or generate an invalid config. + +Representation: + +- Row lamp: =bootloader=. +- Child chips: =ZBM= or =GRUB=, =EFI=, =entries=, =all disks=. +- Yellow if redundancy cannot be proven. +- Red if the expected loader/config is missing. + +Tools: + +- =bootctl status= +- =efibootmgr -v= +- =findmnt /efi /boot= +- ZFS: check =/efi/EFI/ZBM/zfsbootmenu.efi=. +- Btrfs: check =/boot/grub/grub.cfg= and grub-btrfs entries. + +Doctor: + +- read-only by default. +- mitigation: regenerate GRUB config for Btrfs with arm-first confirmation. +- mitigation: rebuild initramfs with arm-first confirmation. +- no automatic EFI reinstall without an explicit advanced flow. + +** 8. Post-install service health =P1/rank 8= + +Problem overcome: the install can complete while the real workstation is +degraded: DNS broken, NetworkManager failed, fail2ban not responding, user +services not lingering, or Docker/Tailscale/Syncthing not in their expected +state. + +Representation: + +- =PACKAGES + SERVICES= band. +- Count badge: =0 failed= or =3 failed=. +- Child lamps: =network=, =dns=, =security=, =user services=. + +Tools: + +- =systemctl --failed= +- =systemctl is-enabled/is-active= +- =resolvectl status= +- =nmcli general status= +- =fail2ban-client status= +- =loginctl show-user= + +Doctor: + +- safe: restart known flaky non-destructive services such as + =NetworkManager= only after classifying the failure. +- safe: re-enable expected services from Archsetup's contract. +- mitigation: bounce DNS resolver and re-check. +- no blanket =systemctl restart --failed=. + +** 9. Archsetup state and log cleanliness =P1/rank 9= + +Problem overcome: a resumable installer can leave a half-finished system that +looks usable until a missing marker or skipped step matters later. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Step-progress mini bar: completed markers / expected markers. +- Red if =archsetup --status= reports incomplete required steps. +- Red if latest log contains fatal errors. + +Tools: + +- =./archsetup --status= +- =/var/log/archsetup-*.log= +- marker files from the Archsetup state directory +- existing testinfra assertions in =scripts/testing/tests/test_archsetup.py= + +Doctor: + +- read-only: summarize incomplete steps and latest log errors. +- mitigation: offer to rerun =archsetup= in normal resumable mode. +- cleanup: archive old logs, keep the latest N. +- never run =--fresh= from Doctor. + +** 10. Workstation contract checks =P2/rank 10= + +Problem overcome: a fresh Arch system is not the goal. The goal is Craig's +working machine: user, shell, groups, dotfiles, Emacs, Hyprland/DWM, keyring, +VPN tools, Bluetooth tools, and local scripts. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Child lamps: =user=, =dotfiles=, =desktop=, =tools=. +- Evidence drawer mirrors the testinfra checks. + +Tools: + +- =id=, =getent passwd= +- =test -L ~/.zshrc= +- =stow= via dotfiles Makefile +- =pacman -Q=, =yay -Qi yay= +- =hyprctl=, =gdbus= portal checks when session is running + +Doctor: + +- safe: restow dotfiles with the selected profile. +- safe: repair keyring directory permissions. +- mitigation: reinstall missing official packages. +- AUR package rebuilds require confirmation and stream output. + +** 11. Security and hardening drift =P2/rank 12= + +Problem overcome: security settings are easy to regress because they are small +file edits: SSH root login, EFI mount masks, firewall, issue banner, fail2ban, +quiet printk. + +Representation: + +- Compact row under =WORKSTATION CONTRACT= or =PACKAGES + SERVICES=. +- Red only for high-risk drift, yellow for unknown/unreadable state. + +Tools: + +- =sshd -T= or config file checks +- =ufw status= +- =fail2ban-client status= +- =findmnt /efi= +- =sysctl kernel.printk= + +Doctor: + +- safe: restore known Archsetup-owned config snippets. +- safe: re-enable firewall if policy file is present. +- mitigation: write missing drop-ins only from version-controlled templates. +- no broad hardening rewrite from panel state. + +** 12. Backup and rollback readiness =P2/rank 11= + +Problem overcome: rollback only helps local state. The install also needs +backups of edited system files and confidence that personal data replication is +not silently stale. + +Representation: + +- Row lamp: =backups=. +- Chips: =system-file .bak=, =replication=, =last run=. +- Yellow if last replication exceeds policy. +- Red if expected backup files for edited system config are missing. + +Tools: + +- Archsetup backup assertions in =scripts/testing/tests/test_backups.py=. +- =zfs-replicate= if configured. +- =systemctl list-timers= for backup timers. +- =journalctl -u= relevant backup units. + +Doctor: + +- safe: create missing =.archsetup.bak= for files before editing. +- safe: run dry-run replication check. +- mitigation: start a configured backup timer/unit with confirmation. +- never delete backup targets from Doctor. + +* Doctor Model + +Doctor is a classifier with bounded mitigations, not a magic repair button. + +Flow: + +1. Probe the selected scope. +2. Normalize each metric to =ok=, =warn=, =fail=, =unknown=, or =running=. +3. Classify failures as: + - =safe-fix= — local, reversible, low risk. + - =safe-cleanup= — removes only known generated artifacts. + - =mitigation= — improves the chance of success but does not claim repair. + - =needs-confirmation= — mutating, long-running, or system-wide. + - =manual= — too dangerous or context-heavy for Doctor. +4. Run only safe actions automatically after the user presses Doctor. +5. Arm-first for anything mutating beyond safe local cleanup. +6. Re-run the affected probe. +7. Stream a verdict into the output well. + +Doctor should say exactly what it did: + +#+BEGIN_EXAMPLE +doctor: package db stale + check: core.db age 4d, archzfs.db age 4d + action: refreshed sync databases + action: updated archlinux-keyring + result: ok, newest db age 2m +#+END_EXAMPLE + +* Common Tool Drivers + +** Host probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| systemd | =systemctl --failed=, =systemctl is-active=, =journalctl= | +|------------+----------------------------------------------------------------| +| packages | =pacman=, =checkupdates=, =pacman-conf=, =yay= | +|------------+----------------------------------------------------------------| +| storage | =zpool=, =zfs=, =btrfs=, =df=, =findmnt=, =lsblk= | +|------------+----------------------------------------------------------------| +| boot | =bootctl=, =efibootmgr=, =mkinitcpio=, =grub-mkconfig= | +|------------+----------------------------------------------------------------| +| network | =nmcli=, =resolvectl=, =ping= or HTTPS probe | +|------------+----------------------------------------------------------------| +| desktop | =hyprctl=, =gdbus=, =loginctl=, dotfiles Makefile | +|------------+----------------------------------------------------------------| + +** Project probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| archangel | =make test=, =make build=, =build.sh --skip-aur= | +|------------+----------------------------------------------------------------| +| archsetup | =make test-unit=, =make test=, =scripts/testing/run-test.sh= | +|------------+----------------------------------------------------------------| +| VM | =qemu-img=, =qemu-system-x86_64=, =sshpass=, =pytest= | +|------------+----------------------------------------------------------------| +| artifacts | latest =out/*.log=, =out/*aur-manifest.tsv=, =test-results/*= | +|------------+----------------------------------------------------------------| + +* Top-family Comparison + +This monitor should borrow the mature display ideas from =top=-style tools +without becoming another CPU/process viewer. The domain objects are install +contracts, boot/storage health, package freshness, snapshots, services, and +artifacts. The interaction model is still the same: sort the thing that hurts, +filter to the thing you care about, expand one row for evidence, and act only +when the diagnosis is clear. + +** Comparison table + +| Tool | What it represents well | Sorting/filtering model | Useful pattern for system monitor | Gaps for our domain | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =htop= | Dense live table plus configurable meters; process tree; direct process actions. | Interactive sort by column, search, filter, tree toggle. | Metric table should support column sort, search, filter, and tree/group mode. | No historical artifact model; actions are process-centric. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =btop++= | Boxed dashboard: CPU, memory, disks, network, processes, battery, GPU; strong graph language; selected process detail. | Easy switching between process sort modes; filter; tree view; pause. | Use boxed bands, mini time-series, detail pane, pause/freeze, and clickable controls. | Graph-first layout can overemphasize volatile values over install risk. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =bottom/btm= | Custom widget layout, per-widget focus/expand, zoomable time windows, basic mode. | Process widget supports sort, search, tree; widgets can be filtered/configured. | Every health band should be expandable; stale/history windows should be zoomable. | Mostly resource telemetry, not remediation workflow. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =atop= | Interval deltas, critical-resource highlighting, all active processes including exited ones, long-term logs. | Resource views and interval replay; emphasizes deviations and active load. | Add history/replay for health events and show "new since last good" changes. | Lower immediate visual polish; Linux-performance scoped. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =Glances= | Broad plugin dashboard, thresholds, remote/web/API modes, export to JSON/CSV/time-series backends. | Configurable visible plugins; API/stdout selectors instead of only interactive sorting. | Use plugin architecture, threshold config, JSON output, remote/headless mode. | Too broad; can become a generic monitoring surface. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =procs= | Modern table ergonomics: custom columns, keyword search across selected fields, sort by named column, tree view. | CLI sort asc/desc by partial column name; watch mode cycles sort columns; AND/OR/NAND/NOR search. | Use named metric columns, saved views, multi-keyword filters, and value-aware coloring. | Process-only; no graphs or remediation model. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =gotop/gtop= family | Fast glanceable terminal dashboard with compact graphs and gauges. | Usually lighter than htop/btop; less important than presentation density. | Use compact sparklines/gauges for "age", "last pass", and "failure count". | Not enough evidence/action depth for this monitor. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| + +** What to pull in + +*** htop: table discipline + +Pull: + +- Column headers that are real controls: click or key-cycle to sort by + =priority=, =state=, =age=, =last_checked=, =last_pass=, =failure_count=, + =scope=, and =doctor_class=. +- Search and filter as first-class actions, not hidden debug commands. +- Tree mode for ownership: + - =system= → =boot/storage= → =bootloader= → =efi entries=. + - =archangel= → =iso build= → =aur repo= → =manifest rows=. + - =archsetup= → =state markers= → =desktop= → =dotfiles=. +- Horizontal detail access for long evidence, like htop's horizontal scrolling + for full commands. + +Equivalent-or-better requirement: + +- htop sorts processes; this monitor sorts risk. The default sort is + live-state severity, then priority rank, then age. + +*** btop++: instrument boxes and live graphs + +Pull: + +- Boxed bands with stable geometry. +- Small time-series graphs, but only where history matters: + - package DB age over time + - failed-service count + - journal error count + - snapshot count / newest snapshot age + - VM pass/fail history + - ISO build duration/result history +- Selected-row detail pane with the last command, verdict, and raw excerpt. +- Pause/freeze button so a failure does not scroll away while reading. +- Mouse-clickable controls where every visible key has the same keyboard path. + +Equivalent-or-better requirement: + +- btop's graphs answer "what is hot right now?" Our graphs answer "is the + safety margin shrinking?" Trend charts should be muted unless the threshold + is crossed. + +*** bottom: focus/expand and layout presets + +Pull: + +- Expand one band full-height: + - =BOOT + STORAGE= expands into boot files, EFI entries, pools, snapshots. + - =INSTALL PIPELINE= expands into last VM runs and build artifacts. + - =PACKAGES + SERVICES= expands into DB ages, repo status, failed units. + - =WORKSTATION CONTRACT= expands into Archsetup markers and testinfra-style + checks. +- Zoomable history windows: 24h / 7d / 30d / all artifacts. +- Layout presets: + - =compact= for bar dropdown. + - =full= for terminal/TUI. + - =host-only= for laptop health. + - =release-gate= for Archangel/Archsetup changes. + +Equivalent-or-better requirement: + +- bottom expands widgets; this monitor expands evidence and remediation state. + The expanded view must show "what changed since last good" before raw logs. + +*** atop: history and vanished failures + +Pull: + +- Permanent, compact health-event log. +- Interval deltas instead of only current values: + - new failed services since last check + - new journal errors since last check + - packages/repos newly stale + - snapshot hook present before, missing now + - bootloader file changed since last known-good +- "Show active/deviating only" mode. In normal use, hide green rows unless + their age is approaching threshold. +- Replay mode: inspect the state at the time an install/test/build failed. + +Equivalent-or-better requirement: + +- atop can report processes that already exited. This monitor should report + failures that already passed through: a transient failed unit, a VM test that + failed last night, an ISO build that failed before the current successful + build, or a package DB that was stale until Doctor fixed it. + +*** Glances: plugin/API/export model + +Pull: + +- Plugin-like probes. Each metric owns: + - =probe= + - =normalize= + - =thresholds= + - =doctor_actions= + - =redaction= + - =evidence= +- JSON output as a stable contract before GTK work. +- Optional stdout selectors: + - =system-monitor --stdout packages.state,storage.state= + - =system-monitor --json boot,snapshots= +- Remote/headless mode for VMs and bare-metal test targets. +- Threshold config in one file, not hardcoded in the UI. + +Equivalent-or-better requirement: + +- Glances is broad; this must stay opinionated. A plugin is accepted only if it + maps to install health, rollback safety, workstation contract, or recovery + readiness. + +*** procs: custom columns and query grammar + +Pull: + +- Named columns and saved views: + - =risk=: state, priority, age, doctor class. + - =install=: last pass, filesystem, artifact, branch, commit. + - =host=: state, source, last checked, command. + - =doctor=: action class, requires root, reversible, last run. +- Multi-keyword search: + - =zfs failed= + - =doctor safe-fix= + - =archangel stale= + - =service red= +- Boolean query modes: + - AND default for narrowing. + - OR for "show any boot or storage issue". + - NOT for "hide green". +- Value-aware coloring for age, severity, and units. + +Equivalent-or-better requirement: + +- procs lets the user build a process table. This monitor should let Craig + build a risk table without editing code. + +*** gotop/gtop: glance density + +Pull: + +- Small sparklines for trend, not full charts. +- Big obvious state words. +- Compact gauges for bounded values: + - EFI usage + - root/home usage + - DB age as percent of freshness window + - VM evidence age + - snapshot age +- Simple default screen that is useful without learning keys. + +Equivalent-or-better requirement: + +- The first screen should answer "am I safe to upgrade or install?" in under + two seconds. + +* Sorting and Views + +The monitor needs two sorting layers: global row ordering and per-band evidence +tables. + +** Global row ordering + +Default: + +1. =state= severity: red, yellow, unknown, running, green. +2. =priority= rank: P0 before P1 before P2. +3. =age= or =staleness=, descending. +4. =last_changed=, newest first. + +Alternate sorts: + +| Sort key | Use case | +|-----------------+------------------------------------------------------| +| =priority= | Release-gate review; keep P0/P1 at the top. | +|-----------------+------------------------------------------------------| +| =state= | Triage; show all red/yellow rows first. | +|-----------------+------------------------------------------------------| +| =age= | Find stale tests, stale package DBs, old backups. | +|-----------------+------------------------------------------------------| +| =doctor_class= | Find what Doctor can safely fix now. | +|-----------------+------------------------------------------------------| +| =scope= | Group host vs install vs build. | +|-----------------+------------------------------------------------------| +| =last_changed= | See what recently regressed. | +|-----------------+------------------------------------------------------| +| =source= | Group by archangel, archsetup, dotfiles, host. | +|-----------------+------------------------------------------------------| + +** Per-band sorts + +| Band | Sorts | +|------------------------+------------------------------------------------------------| +| =BOOT + STORAGE= | severity, mountpoint, filesystem, capacity, last scrub, newest snapshot age | +|------------------------+------------------------------------------------------------| +| =INSTALL PIPELINE= | result, filesystem, duration, artifact age, commit age, last pass | +|------------------------+------------------------------------------------------------| +| =PACKAGES + SERVICES= | severity, unit name, repo name, DB age, error count, enabled/active state | +|------------------------+------------------------------------------------------------| +| =WORKSTATION CONTRACT= | severity, check name, owner repo, last pass, doctor class | +|------------------------+------------------------------------------------------------| + +** Filters + +Quick filters should be visible as chips: + +- =red= +- =yellow= +- =doctorable= +- =needs-root= +- =stale= +- =zfs= +- =btrfs= +- =host= +- =install= +- =build= +- =changed= +- =hidden-green= + +The default view can hide healthy low-priority rows, but it must show enough +green P0/P1 summary state to prove the monitor is working. + +* Display Requirements Borrowed from Tops + +1. Every table has sortable columns and a visible sort indicator. +2. Every visible metric row has a filterable state, priority, age, and source. +3. Every row can expand to evidence without losing the list context. +4. Every graph has a threshold marker; trend without threshold is decoration. +5. Every long-running action can be paused/frozen in the display. +6. Every mutating action has an equivalent CLI command shown in the output well. +7. Every Doctor action records before/after state so fixed failures remain + visible in history. +8. Every band has a compact mode and an expanded mode. +9. Green rows are quiet; new regressions are loud. +10. The system must be useful over SSH/TUI before GTK polish. + +* Source Notes + +- =htop=: upstream README describes configurable system/process display, + interactive sorting/filtering/search, tree view, and process actions. +- =btop++=: upstream README describes resource boxes, detailed process stats, + filter, sort switching, tree view, mouse support, auto-scaling network graphs, + disk IO, battery, GPU support, and themes. +- =bottom/btm=: upstream README describes customizable widgets, process sort + and search, tree mode, expand/focus, zoomable graph intervals, filters, and + basic mode. +- =atop=: upstream README describes interval resource accounting, critical + highlighting, long-term compressed logs, exited-process visibility, cgroup + views, and active/deviation-focused output. +- =Glances=: upstream README describes plugin-style broad monitoring, web/API + modes, stdout JSON/CSV, remote monitoring, exports, and threshold-oriented + dashboard use. +- =procs=: upstream README describes configurable columns, named-column sort, + watch mode, tree view, logical keyword search, value-aware coloring, and + pager behavior. + +* Data Model + +Emit JSON from a CLI first; the panel is a client. + +#+BEGIN_SRC json +{ + "v": 1, + "scope": "host", + "state": "warn", + "ts": "2026-07-04T12:00:00-04:00", + "metrics": [ + { + "id": "packages.sync_freshness", + "label": "package databases", + "state": "warn", + "summary": "archzfs.db age 4d", + "evidence": [ + {"command": "find /var/lib/pacman/sync", "excerpt": "archzfs.db 2026-06-30"} + ], + "doctor": { + "class": "safe-fix", + "actions": ["refresh-sync-db", "update-keyring"] + } + } + ] +} +#+END_SRC + +* Implementation Notes + +- Start with a CLI: =system-monitor status --json=, =system-monitor doctor + --json=, =system-monitor refresh=. +- Keep probes read-only by default. Actions live in separate verbs. +- Cache slow probes. The bar should read a cache, not run VM tests. +- VM/build actions should create job records and stream logs; the panel follows + the job rather than blocking the UI process. +- Reuse the net/bt panel architecture if this becomes a GTK panel: GTK-free + model + fake-command unit tests + one AT-SPI smoke. +- Redact secrets from logs and JSON: WiFi PSKs, tokens, private repo URLs with + credentials, SSH material, and backup target credentials. + +* Open Decisions + +** TODO Where should the first implementation live? + +Recommendation: dotfiles owns the user-facing panel and CLI wrapper because it +is workstation UI. Archsetup owns reusable install-contract probes and testinfra +assertions. Archangel owns ISO/build probes. + +** TODO Should Doctor run elevated actions through polkit or terminal? + +Recommendation: read-only checks run unprivileged; elevated actions launch a +terminal or polkit prompt with the exact command visible. Do not hide long +privileged operations inside the panel process. + +** TODO How fresh must VM evidence be? + +Recommendation: host checks go stale after 1 hour; package DB after 48 hours; +VM install evidence after 7 days; ISO build evidence after 14 days or whenever +Archangel/Archsetup has changed since the last successful artifact. + +** TODO Which actions are allowed on bare metal? + +Recommendation: host Doctor may refresh databases, update keyring, restow +dotfiles, create snapshots, run scrub, and restart narrowly classified services. +It may not perform full upgrades, bootloader reinstalls, destructive snapshot +prune, or filesystem repair without a separate advanced flow. + +* First Build Slice + +1. CLI read-only host status: + - package DB freshness + - failed services + - ZFS/Btrfs health + - snapshot presence + - Archsetup status/log check +2. Doctor safe actions: + - refresh package DB + - update keyring + - restow dotfiles + - create manual snapshot +3. Artifact parser: + - latest Archsetup VM test result + - latest Archangel ISO build result +4. Panel prototype: + - faceplate + - four health bands + - evidence output well + - REFRESH and DOCTOR keys only |