diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/maintenance-console-design-ideas.org | 527 | ||||
| -rw-r--r-- | docs/design/system-monitor-design-ideas.org | 1008 |
2 files changed, 1535 insertions, 0 deletions
diff --git a/docs/design/maintenance-console-design-ideas.org b/docs/design/maintenance-console-design-ideas.org new file mode 100644 index 0000000..066c25d --- /dev/null +++ b/docs/design/maintenance-console-design-ideas.org @@ -0,0 +1,527 @@ +#+TITLE: Maintenance Console — Design Ideas +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-06 + +* Direction + +A single-host maintenance console (GUI, waybar-launched) that surfaces every +health metric for this Arch workstation and, where the remediation is safe, +runs it. It reshapes the earlier install-health/workstation-contract framing +(=system-monitor-design-ideas.org=) into a maintenance surface with a doctor +action. + +The console is the glanceable, single-host version of the home project's +=system-health-check.org= workflow (~1000 lines, capability-dispatched across +ratio/velox/mybitch/truenas). The console owns the routine-maintenance and +at-a-glance-status slice; the workflow stays as the escalation path for +forensic investigation and as the only sanctioned way to run updates. + +* Scope: hosts (Q2 resolved 2026-07-06) + +v1 targets *both* daily drivers — ratio and velox — as first-class hosts, not +ratio-first-velox-later. Consequences for v1: + +- *Capability dispatch is mandatory from day one*, not deferred. The console + probes the live host and runs only applicable checks (btrfs RAID1 on ratio vs + ZFS-primary on velox; AMD amd-pstate on ratio vs Intel intel_pstate on velox). + This mirrors the home workflow's Phase 0 probe. +- *velox-only metrics ship in v1*: battery health (capacity / cycles) and the + unclean-shutdown / suspend-failure rate — both currently GAPs, both driven by + velox being the machine that travels and suspends. +- The ZFS-primary remediation paths (scrub, snapshot retention, pool health) are + built and tested, not stubbed. + +* Thresholds source of truth (Q3 resolved 2026-07-06) + +One machine-readable file, =maintenance-thresholds.toml=, holds every severity +value (cache-size trigger, disk %, scrub-age warn/crit, snapshot retention +limits, temp bands, backup-staleness windows, etc.). archsetup owns and ships +it; both consumers read the *installed* path, so neither reaches into the +other's repo: + +- the maintenance console (dotfiles code) reads it at runtime, +- the system-health-check workflow reads it instead of hardcoding severity + rules in prose. + +The workflow's hard-won values migrate into the TOML as the seed content (the +snapshot MONTHLY limit that bit /home, the 10 GB cache trigger, the scrub-age +bands). Install path is an implementation sub-question — a stable location both +consumers reach (e.g. =~/.config/archsetup/maintenance-thresholds.toml=). + +* Workflow ownership — move system-health-check into archsetup (proposed 2026-07-06) + +Craig's call: the =system-health-check.org= workflow should live in archsetup, +not the home project. Rationale — the home project is scoped to finances, +health, and personal matters; system design, execution, and maintenance are +archsetup's domain. Home only owns the workflow by inherited accident, not by +fit. Moving it here also collapses the Q3 coupling: the TOML source-of-truth and +its workflow consumer end up in the same project. + +Move scope (home → archsetup), to sequence when Craig gives the go: +- =system-health-check.org= (the workflow itself) +- =homelab-inventory/*.org= (ratio/velox/mybitch/truenas capability inventories + it cross-references) +- any home-project references to the workflow (startup, project-workflows index) + +Cross-project mechanics: the archsetup side (receiving the files, wiring the +TOML) is in-scope here; the home side (removing the originals, updating home's +references) is the home project's scope — handled by a handoff note to home's +inbox or a home session, not edited blind from here. + +* Panel shape + +Three regions, driven by the Automation column of the metrics table below: + +- *Actionable* (left) — metrics with a lever. Automation = Auto or Confirm. + Two doctor actions serve this column: "Clean up" fires every Auto metric + unattended; "Review & fix" opens a preview for the Confirm metrics and acts + only on approval. +- *Diagnostic* (right) — read-only telemetry. Automation = None or Human. A + state color (green/amber/red) and the value; no button. Red here is the + signal to run the workflow. +- *Updates* (quarantined strip) — Automation = Workflow. Shown as a count with + notable packages named; the only affordance is "run the workflow." The panel + never applies updates in place. + +*Bar glyph* tracks the worst *Diagnostic* state only — not the actionable +count. A big package cache is boring; a SMART failure is a fire. Actionable +clutter must not turn the bar red or it trains you to ignore it. + +** Doctor = live output wall (Q1 resolved 2026-07-06) + +"Clean up" and "Review & fix" are not fire-and-forget buttons. Running a doctor +opens an *output wall* — one lamp per action, streaming in realtime as each +runs: + +- *amber* while the action is running, +- *green* on success (with the reclaimed amount / result inline), +- *red* on failure. + +Feedback is always shown (not just when something was reclaimed) — you watch it +happen. This is the same live-results shape that should back the other doctors +(net, bluetooth), so every doctor in the system reads the same way. See the +todo task to retrofit the net + bt doctors to realtime lamp output. + +* Automation legend + +| Value | Meaning | +|----------+-----------------------------------------------------------------------------------------------------------| +| Auto | Doctor "Clean up" button — fire unattended, reversible or harmless | +|----------+-----------------------------------------------------------------------------------------------------------| +| Confirm | Doctor "Review & fix" — preview the change, act only on one click | +|----------+-----------------------------------------------------------------------------------------------------------| +| Human | Panel nudges; a human decides and acts (no button, not the agent) | +|----------+-----------------------------------------------------------------------------------------------------------| +| Workflow | vLater — metric shown read-only; no in-panel button. Agent-workflow assistance deferred (decided | +| | 2026-07-07, see below) | +|----------+-----------------------------------------------------------------------------------------------------------| +| None | Diagnostic only — no software remediation exists (hardware, telemetry) | +|----------+-----------------------------------------------------------------------------------------------------------| + +*Workflow buttons removed (decided 2026-07-07, prototyping E5).* The panel +carries no "run workflow" affordances — not on failed units, unclean boots, +updates, or CVEs. The metrics stay on the board as read-only telemetry; AI +assistance via the system-health-check workflow is a vLater feature. v1 ships +only determinate remedies (fixed, scriptable actions with predictable +outcomes). + +*Determinate remedies adopted (decided 2026-07-07).* Every remedy below is +Confirm-tier; contextual levers appear only when the metric is off-nominal: + +- /Service restarts & enables/: failed units (systemctl restart + reset-failed), + fail2ban, cronie, chronyd (+ makestep), tailscaled, snapper timers, zram + config re-apply. DNS/NetworkManager delegates to the net panel's doctor + (deep-link, not a duplicate repair chain). +- /Deterministic maintenance ops/: btrfs balance -dusage=50 (unallocated low), + reinstall owning packages on pacman -Qkk failures, rsyncshot RUN NOW (result + streams to the output well), smartctl -t short self-test, snapshot-retention + repair (write sane TIMELINE limits + cleanup — the /home lesson as a + one-press fix). btrfs device-error counter reset stays manual: resetting + without diagnosis masks a dying drive. +- /Composite macro/: RECLAIM SPACE on the disk-usage cell — runs every reclaim + lever (cache, journal, coredumps, app logs, docker tier-1, snapper cleanup) + as one output-wall stream. +- /Disruptive but determinate/: REBOOT behind arm-to-fire, offered when + running kernel != installed. +- /Still read-only/: temps, throttling, battery, memory/OOM, taint, journal + error content, kernel/hw events, listeners, unclean-shutdown rate — physical + or investigative; vLater AI territory. + +*Updates join the Confirm layer (decided 2026-07-07).* Two levers on the +updates strip, both behind the live-update guard (mesa/hyprland/wayland +runtime in the pending set): + +- UPDATE — repo + AUR system update. +- TOPGRADE — full ecosystem run. The panel's wrapper always passes + --disable git (topgrade's git step rebase-autostashes ~/code/*/ — never + under a live session). + +*MEM·PWR: evidence + two levers + expectation tags (decided 2026-07-07).* +Mostly physics, so the category leans watch-only — with these additions: + +- /CPU mode selector/: a free segmented control (PERF · BAL · POWER) writing + the EPP hint — set the active mode to anything, not drift-repair against a + declared default. amd-pstate on ratio, intel_pstate on velox. +- /Battery charge limit/ (velox-only, capability-gated): SET 80% writes + charge_control_end_threshold — the standard longevity cap. Battery *health* + (capacity vs design, cycles) stays watch-only hardware telemetry. +- /Evidence drill-downs/ (digest idiom): top-5 RAM consumers under memory, + recent boots listed clean/unclean under the unclean-rate, and throttle/OOM + events with timestamps. Evidence makes the numbers actionable even where no + button exists. +- /KILL on top-memory items/ (revised 2026-07-07 — Craig): arm-to-fire, four + guards: the arm shows the exact victim (name + size); SIGTERM not SIGKILL, + with the outcome reported to the wall; PID + process name revalidated at + fire time so a recycled PID can't be hit; session-critical names (systemd, + the compositor, the panel itself) render a disabled key — protected. A + SIGKILL escalation for TERM-survivors is vLater. +- /Expectation-setting, panel-wide/: every leverless cell's sub-line carries an + explicit tag ("hardware — watch only", "evidence below"), and each subpanel + header shows the split — "N fixable · M watch" — so the user knows their + agency before reading a single cell. + +*Refresh cadence (decided 2026-07-07).* Four tiers, matching probe cost: + +- /Live group, panel open/: temps, memory free + top consumers, throttle + state — re-read every ~3 s while their subpanel is visible, gated on + panel-open exactly like the audio panel's meters. Stop when hidden. +- /Fast local tier, panel open/: re-probed every ~30 s while the panel is up; + additionally, any metric re-probes immediately after an action that touches + it (fire CLEAN → cache re-measured, not assumed). +- /On open/: the hydration tiers re-run (fast reads first, process probes + behind them — sub-second perceived). +- /Network tier/: checkupdates / arch-audit / AUR / firmware stay on the + hourly systemd-timer cache with age shown; refreshed on demand only. +- /Panel closed/: the waybar glyph is fed by a light background scan every + ~30 min (systemd timer writing the state file the glyph reads) — the bar + stays honest without the panel running. + +*Journal errors get a digest, not a fix (decided 2026-07-07).* No generic +remedy exists — an error-priority line is a symptom of an arbitrary subsystem — +so the panel ships four determinate assists instead: + +- /Digest/: the cell expands to errors grouped by syslog identifier — count, + message snippet, first/last seen, and the exact next command (journalctl -u + <unit> -b) when the identifier maps to a unit. Top-10 groups, read-only. +- /MARK KNOWN with a full lifecycle/: arm-to-fire showing the exact pattern + before it stores; marked groups move to a dim KNOWN section (never vanish) + with per-row UNMARK; every mark/unmark logs to the results wall; marks carry + date + example. Patterns bind to identifier + message, never a whole unit — + a muted service's *new* errors still surface. +- /Two noise layers/: shipped defaults (bluetoothd HFP, pixman, xkbcomp) in + the packaged TOML, user marks in a separate user file merged over it (a + template sync never eats curation). CLEAR MARKS (arm-to-fire) empties the + user layer and re-enables shipped defaults; unmarking a shipped default + records a disable flag in the user layer. +- /OPEN JOURNAL/: launches a terminal running journalctl -p err -b — the same + delegation pattern as NET DOCTOR. + +Ruled out: auto-restarting units that log errors (error ≠ failed) and +keyword-driven fix suggestions (vLater AI territory). + +*Full-sweep findings — all committed to v1 (decided 2026-07-07).* Every metric +was audited against the converged checklist (honest label, evidence digest +where a count hides detail, curation lifecycle where "expected" is config +knowledge, guarded per-item remedies, cross-links, watch-only tags). Adopted, +all prototyped: + +- /Storage/: disk top-consumers digest (evidence only — no file deletion + keys); per-device error rows on RAID1 when counters are nonzero, + cross-checked against SMART; SMART sub-line carries the last self-test + result. Spec note: a real scrub runs hours — the ring needs a running-% + state, not an instant reset. +- /Packages/: orphan digest (name + size) with per-package REMOVE (armed) and + KEEP — the curation lifecycle encoding "intentional, not orphaned" (the + rust lesson); batch REMOVE ALL skips kept packages. Per-file pacnew rows + tagged safe-delete (reflector-managed) vs needs-merge, MERGE delegating to + a terminal diff. CVEs named: package · CVE id · severity. AUR and firmware + names spelled out. +- /systemd/: failed units upgraded from a count-lever to a per-unit roster + (name · since · exit code · journalctl hint) with per-row RESTART + RESET; + is-system-running names its cause ("degraded — N failed units below"); + taint letters decoded. +- /Logs/: coredumps grouped by binary (count · last · coredumpctl hint), + cleared with the CLEAR action; kernel/hw events listed when not clean + (hardware — watch only). +- /Services/: docker system df breakdown (images / containers / volumes / + build cache with per-type reclaimable); stopped containers upgraded to the + full signal/expected curation lifecycle (MARK EXPECTED / UNMARK, shipped + default: winvm) with per-container START; cron expected-entries drift + roster. +- /Snapshots/: count split by type — timeline (auto-pruned) · single + (manual — escapes timeline cleanup, the pile-up risk) · pre/post — with + oldest-single named and DELETE STALE (armed, keeps newest 2) when singles + accumulate. +- /Network minors/: fail2ban shows recent-ban count; NTP shows offset. + +Rationale for prototyping everything (Craig): real estate and complexity have +bitten before — surface those limits in the disposable prototype, not after +functionality exists behind the UI. + +*Vertical compression → rotary band selector (2026-07-07).* First attempt — +the MEM·PWR three-column layout on Packages/Logs/Services — lost too much row +detail to third-width truncation (Craig's verdict after use). Replaced by a +*rotary band selector*: the amplifier input-selector idiom. A machined knob +(click to cycle) whose needle swings to engraved band labels, one per evidence +section (ORPHANS · PACNEW · ADVISORIES; SIGNAL · KNOWN NOISE · COREDUMPS · +KERNEL/HW; CONTAINERS · DOCKER DISK · CRON & BACKUPS). Each band carries its +own status lamp (section health at a glance without switching) plus a count; +the selected band gets a gold underline and the needle. One section renders at +a time at full width, restoring complete row detail. Deliberately distinct +from the category tiles, console keys, and the CPU-mode segmented control — +each selection idiom in the panel now has its own visual voice. MEM·PWR keeps +its three-column evidence strip (short rows fit fine at third-width). + +*Listeners get the same treatment (decided 2026-07-07).* The count becomes +"unexpected listeners" — evidence digest (process · port · bind address from +ss -tlnp), expected-list curation with the full MARK EXPECTED / UNMARK / +CLEAR lifecycle (shipped defaults: sshd, mpd, tailscaled; user marks in the +user layer), and guarded per-socket remedies: STOP (systemctl stop, armed) +when a unit owns the socket, KILL (SIGTERM, armed) otherwise. Severity keys +on unexpected AND public-bind (0.0.0.0/::) — a loopback listener warns, an +exposed one fails — and when ufw is down the signal header names the exposure +("ufw down — N public binds exposed"). Stopped containers likewise gained a +contextual START lever (allowlist: winvm), and the firewall its ENABLE. + +*Updates strip border is state-tiered (decided 2026-07-07).* Green when +nothing pending, amber for ordinary pending/AUR/firmware counts, red when +CVEs exist or pending exceeds the "a lot" threshold (or the update cache has +gone stale — staleness window in the TOML). The CVE badge renders only when +the count is nonzero. + +*Guard arms instead of blocking (revised 2026-07-07).* When the guard trips, +the key arms (red, "press again to run anyway — or apply from a TTY") rather +than hard-refusing. The user decides; the footgun is acknowledged and +deliberately handed over. After a system update lands, the panel offers a +reboot: a REBOOT key (arm-to-fire) appears on the updates strip and the +reboot-required metric flips. + +*No per-ecosystem update metrics.* Topgrade's step set (yay, rustup, cargo, +pipx, npm/pnpm, gem, go, flatpak, fwupd, tmux/zsh/nvim plugins, git repos) has +no cheap offline "updates available?" probe — mirroring it means a network +round-trip per registry at panel-open. Instead: one *topgrade freshness* +metric (wrapper stamps last-run time; threshold in the TOML) whose remedy is +the TOPGRADE lever, plus a *firmware updates* count in the updates strip +(fwupd refreshes metadata on its own timer; the panel reads the cache). + +Rationale for the hard lines: system updates are Workflow, never Auto — the +2026-06-07 Hyprland crash was a live -Syu swapping mesa+hyprland under the +running session, and the standing rule is never -Syu live under Hyprland when +the mesa/hyprland/wayland runtime is in the set. Hardware findings (SMART, MCE, +thermal) are None — the fix is replacing a drive or clearing a fan, not +software. + +* Metrics — Storage & filesystem integrity + +| Metric | Fix / lever | Automation | Notes | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Package cache size | paccache -r / -ruk0 | Auto | Reclaim, all re-downloadable | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| paccache -rk1 (keep 1 version) | paccache -rk1 | Confirm | Frees most; kills downgrade | +| | | | headroom | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Disk usage (df) | — | None | Fix is cache/snapshot/prune | +| | | | levers | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs unallocated space | — | None | Chunk headroom; diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs scrub age | btrfs scrub start | Confirm | GAP; ZFS has this, btrfs doesn't. | +| | | | IO-heavy, on-demand | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| Btrfs device stats (per-drive | btrfs device stats --reset | None | GAP; RAID1 early-warning ahead of | +| error counters) | | | SMART; reset after review | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS pool health / errors | — | None | CRITICAL if state != ONLINE or | +| | | | errors > 0 | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS scrub age | zpool scrub | Confirm | Covered | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS capacity | — | None | Perf degrades > 80%; the ZFS | +| | | | headroom metric (no unallocated | +| | | | concept) | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS fragmentation (FRAG) | — | None | GAP; no defrag exists — the | +| | | | remedy is snapshot pruning + | +| | | | staying under 80% | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| ZFS autotrim (SSD TRIM) | zpool set autotrim=on / zpool | Confirm | GAP; velox; the fstrim.timer | +| | trim | | counterpart on ZFS | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| fstrim.timer enabled + firing | systemctl enable --now | Confirm | GAP; standard SSD hygiene | +| | fstrim.timer | | | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| pacman file integrity (Qkk) | reinstall package | Workflow | GAP; modified/missing files need | +| | | | judgment | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART health | — | None | Hardware; replace drive | +|----------------------------------+----------------------------------+------------+-----------------------------------| +| SMART wear / temperature | — | None | Diagnostic | +|----------------------------------+----------------------------------+------------+-----------------------------------| + +* Metrics — Snapshots + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapper count / retention | snapper cleanup / delete | Confirm | Manual (single) snapshots need explicit | +| | | | choice | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| ZFS snapshot count / retention | zfs destroy | Confirm | Runaway retention | +|--------------------------------+--------------------------+------------+---------------------------------------------| +| Snapshot auto-timer running | systemctl enable timer | Confirm | Is the auto-snapshot service firing | +|--------------------------------+--------------------------+------------+---------------------------------------------| + +* Metrics — Packages & security + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Orphaned packages | pacman -Rns (named args) | Confirm | Review first (rust looked orphaned, | +| | | | was intentional) | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Pending updates count | — | Workflow | Never auto; workflow-only | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| .pacnew files | diff + merge / delete | Confirm | Allowlist auto-deletes mirrorlist / | +| | | | locale.gen | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| arch-audit CVEs | reviewed update | Workflow | GAP; top-priority add. Fix is an | +| | | | update | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Keyring freshness | pacman -Sy archlinux-keyring | Confirm | GAP; stale keyring breaks update | +| | | | signatures | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| AUR / foreign staleness (Qm) | paru -Sua | Workflow | GAP; AUR updates reviewed, not auto | +|-----------------------------------+------------------------------+------------+--------------------------------------| +| Reboot required (kernel mismatch) | reboot | Human | GAP; uname -r vs /usr/lib/modules. | +| | | | You pick when | +|-----------------------------------+------------------------------+------------+--------------------------------------| + +* Metrics — systemd & boot + +| Metric | Fix / lever | Automation | Notes | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| is-system-running (degraded?) | — | None | GAP; one-token whole-system verdict; | +| | | | candidate for the bar glyph | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Failed systemd units | restart / investigate | Workflow | Depends why it failed | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Maintenance timers enabled + | systemctl enable timer | Confirm | GAP; meta-metric — are | +| firing | | | paccache/btrfs-scrub/fstrim/reflector/snapper | +| | | | firing | +|--------------------------------+------------------------+------------+-----------------------------------------------| +| Kernel taint flag | — | None | GAP; tainted != 0 = proprietary module or | +| | | | prior crash | +|--------------------------------+------------------------+------------+-----------------------------------------------| + +* Metrics — Logs & coredumps + +| Metric | Fix / lever | Automation | Notes | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Coredump count | coredumpctl clean (keep recent) | Auto | Keep last few days for forensics | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journald disk usage | journalctl --vacuum-size/time | Auto | Bounded journal | +|------------------------------------+---------------------------------+------------+----------------------------------| +| App-log cleanup (>7d) | log-cleanup cron trigger | Auto | Already a cron; manual trigger | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Journal error count (real vs | — | Workflow | Forensic; noise-filtered count | +| noise) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| +| Kernel/hardware events | — | None | Forensic, hardware | +| (MCE/USB/thermal/GPU) | | | | +|------------------------------------+---------------------------------+------------+----------------------------------| + +* Metrics — Memory, thermal, power + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Memory free / OOM kills | — | Workflow | OOM = investigate | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Swap / zram present + healthy | — | None | Diagnostic | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| CPU / GPU temperatures | — | None | Hardware | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Thermal throttling active | — | None | Cooling issue | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Battery health (capacity / cycles) | — | None | GAP; laptop; ties to open suspend todo | +|-----------------------------------------+-------------+------------+--------------------------------------------| +| Unclean-shutdown / suspend-failure rate | — | Workflow | GAP; ratio flagged ~75% unclean 2026-06-08 | +|-----------------------------------------+-------------+------------+--------------------------------------------| + +* Metrics — Network & security posture + +| Metric | Fix / lever | Automation | Notes | +|-----------------------------------+---------------------------+------------+-----------------------| +| DNS / NetworkManager reachability | restart NM | Workflow | CRITICAL if down | +|-----------------------------------+---------------------------+------------+-----------------------| +| Firewall active (ufw / nftables) | ufw enable | Confirm | GAP; security posture | +|-----------------------------------+---------------------------+------------+-----------------------| +| Unexpected listeners (ss -tlnp) | — | None | GAP; security review | +|-----------------------------------+---------------------------+------------+-----------------------| +| Tailscale peers | tailscale up | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| fail2ban running + bans | systemctl start | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| +| NTP sync (chrony) | systemctl restart chronyd | Confirm | Covered | +|-----------------------------------+---------------------------+------------+-----------------------| + +* Metrics — Services, backups, virt + +| Metric | Fix / lever | Automation | Notes | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| rsyncshot backup freshness | — | Workflow | CRITICAL if daily > 48h; investigate | +| | | | failure | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker/podman reclaimable | prune tier 1 / tiers 2-3 | Confirm | Tier 1 nearly Auto; 2-3 destructive | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Docker stopped containers | — | None | Mostly expected (WinVM on-demand) | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| libvirt VM state | — | None | Expected off | +|---------------------------------+--------------------------+------------+--------------------------------------------| +| Cron running + expected entries | systemctl enable cronie | Confirm | rsyncshot + log-cleanup entries | +|---------------------------------+--------------------------+------------+--------------------------------------------| + +* Architecture & testing (decided 2026-07-07) + +*CLI-first, GUI as a face.* The console ships as the fourth panel sibling: a +=maint= Python package in dotfiles (like =net/=, =bt/=, =audio/=) with probe +modules (read-only collectors), a remedies module, =cli.py=, and =gui.py= +driving the same code. =maint status --json= is the contract; =maint fix +<thing>= is every lever. The GUI never does anything the CLI can't. + +Safety mechanics baked into the CLI: +- global =--dry-run= prints the exact command instead of executing — free test + surface, and the GUI's arm-press can display it ("this will run: …"). +- hard read/write split: collectors never elevate; every remedy is an + allowlisted exact argv in one small auditable module. + +*Four test layers (safest → scariest):* + +1. /Unit, fake binaries — no VM, no root (~90% of surface)./ Probes are + parsers over command output: feed canned smartctl/btrfs/journalctl/pacman/ + ss/docker output via fakes on PATH (the net suite's fake-curl and audio's + fake-parec pattern). Remedies tested as command construction (assert the + argv, don't run it). The live-update guard is a pure function over a + package list. +2. /Read-only integration on the live machine./ All collectors are read-only + by design — =maint status --json= runs safely against real hosts. +3. /Remedies in a VM — archsetup's existing harness./ + =scripts/testing/run-test.sh= boots the installer VM; a maint scenario + breaks things deliberately over ssh (stop cronie, mask fstrim, orphan + packages, fill the cache), runs =maint fix …=, asserts post-state. No GUI + in the VM. Add qcow2 snapshot/restore between remedy tests so each starts + pristine and destructive remedies can't contaminate each other. Pure + pacman-level tests may use a throwaway systemd-nspawn container instead + (lighter); the VM stays for systemd/btrfs/reboot territory. +4. /GUI on the host, never in the VM./ AT-SPI smoke like the sibling panels, + driven by fixture data. The prototype's GOOD/BAD snapshots become those + fixtures — =MAINT_PANEL_FIXTURE=bad= renders the degraded board without a + degraded machine, conforming to the =maint status --json= schema. + +* Open questions + +- RESOLVED 2026-07-06 — live output wall (amber running / green done / red fail), + realtime, always shown. See "Doctor = live output wall" above. +- RESOLVED 2026-07-06 — velox is a first-class v1 target alongside ratio. See + "Scope: hosts" below. +- RESOLVED 2026-07-06 — single machine-readable thresholds file + (=maintenance-thresholds.toml=) is the source of truth, *owned by archsetup*. + Both the console and the system-health-check workflow read it, so they can + never drift. See "Thresholds source of truth" and "Workflow ownership" below. diff --git a/docs/design/system-monitor-design-ideas.org b/docs/design/system-monitor-design-ideas.org new file mode 100644 index 0000000..26619a0 --- /dev/null +++ b/docs/design/system-monitor-design-ideas.org @@ -0,0 +1,1008 @@ +#+TITLE: System Monitor Design Ideas +#+DATE: 2026-07-04 +#+TODO: TODO | DONE +#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED + +* DRAFT Status +:PROPERTIES: +:ID: system-monitor-design-ideas +:END: +- [2026-07-04 Sat] DRAFT — initial design sketch for a health monitor covering + Archangel ISO/base-install health, Archsetup workstation health, and the local + laptop as the daily canary. + +* Metadata + +| Field | Value | +|--------+--------------------------------------------| +| Status | draft | +|--------+--------------------------------------------| +| Owner | Craig Jennings | +|--------+--------------------------------------------| +| Repos | archsetup, archangel, dotfiles | +|--------+--------------------------------------------| +| Kin | net panel, bluetooth panel, audio panel | +|--------+--------------------------------------------| + +* Problem + +Archangel and Archsetup can fail in ways that are individually obvious only +after the damage is done: an ISO build goes stale against Arch or archzfs, +ZFSBootMenu or GRUB boots once but not after the first upgrade, a snapshot +hook silently disappears, a package database ages out, systemd services fail +after a reboot, or the desktop contract is technically installed but not +usable. + +The health surface should compress those risks into one operational question: +"Can I trust a fresh install, and is this current workstation drifting away +from the known-good install contract?" + +This monitor is not a generic CPU/RAM graph. It is an install-health and +workstation-contract console. CPU, memory, and temperature belong only as +secondary context unless they block install/test operations. + +* Priority Model + +Rank metrics by the cost of blindness: what happens if Craig never sees the +metric, no one mitigates it, and the next install/upgrade/reboot simply happens. + +Severity: + +- =P0= — can cause data loss, unbootable systems, or loss of rollback path. +- =P1= — can break fresh installs, upgrades, remote access, or core + workstation use. +- =P2= — causes degraded workstation behavior, security drift, or accumulating + maintenance debt. +- =P3= — useful context, not a release gate by itself. + +The panel should sort by live severity first, then by this priority. A red =P2= +row appears above a green =P0= row, but in the steady state the layout keeps the +P0/P1 rows in the first viewport. + +* Priority Ranking + +| Rank | Priority | Metric | Why this rank exists | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 1 | P0 | Storage health | Silent pool/filesystem degradation is the nearest thing to data loss. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 2 | P0 | Snapshot safety coverage | Without snapshots, upgrades lose their rollback safety net. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 3 | P0 | Bootloader and EFI redundancy | A machine that cannot boot is operationally dead. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 4 | P1 | First-upgrade bootability | Catches the classic "installed fine, broke after update" failure. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 5 | P1 | End-to-end VM install pass rate | Best release gate for the whole Archangel + Archsetup chain. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 6 | P1 | Package sync and repo freshness | Arch, archzfs, keyring, and mirror drift are leading break signals. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 7 | P1 | Archangel ISO reproducibility | If current inputs cannot build an ISO, recovery/install confidence is stale. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 8 | P1 | Post-install service health | Network, DNS, SSH, and user services decide whether the system is usable. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 9 | P1 | Archsetup state/log cleanliness | Prevents "half-installed but looks fine" machines. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 10 | P2 | Workstation contract checks | Confirms this is Craig's workstation, not just generic Arch. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 11 | P2 | Backup and rollback readiness | Catches loss of off-machine recovery and edited-file backups. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 12 | P2 | Security and hardening drift | Important, but usually less immediately destructive than boot/storage. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| + +This order intentionally puts ZFS/Btrfs health above VM install evidence. A +broken future install is expensive; silent damage to the current root or backup +chain is worse. + +* Consequence Matrix + +This is the design justification for every metric. A row earns panel space only +if blindness has a clear failure mode and Doctor has at least a useful +diagnostic or mitigation. + +| Rank | Metric | If never seen / never mitigated | Typical failure | Worst plausible failure | Doctor posture | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 1 | Storage health | Disk, pool, or metadata degradation accumulates silently. | Correctable ZFS/Btrfs errors, low EFI/root space, stale scrub. | Data loss, degraded root, failed import/mount during boot. | Diagnose + scrub/cleanup with confirmation. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 2 | Snapshot safety coverage | Upgrades proceed without a rollback point. | Missing recent snapshot or missing pre-pacman hook. | Bad upgrade cannot be rolled back cleanly; manual repair required. | Create snapshot; restore hook; never rollback. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 3 | Bootloader and EFI redundancy | Boot path rots until the next reboot or disk failure. | Missing GRUB/ZBM file on one EFI partition. | Unbootable machine after update, firmware reset, or disk loss. | Diagnose; regenerate config; advanced reinstall only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 4 | First-upgrade bootability | Installs look good until the first real system update. | Kernel/initramfs/bootloader mismatch in VM. | Fresh bare-metal install dies on first reboot after upgrade. | VM-only upgrade test; collect boot evidence. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 5 | End-to-end VM install pass rate | Unit tests give false confidence about the real workflow. | Current branch fails one filesystem path or desktop assertion. | Bare-metal install fails mid-flight after disks are wiped. | Run/schedule VM test; clean stale VM artifacts. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 6 | Package sync/repo freshness | Arch/keyring/archzfs drift surprises the next install. | 404s, stale keyring, bad mirror, stale archzfs DB. | Installer cannot pacstrap or installs mismatched ZFS/kernel bits. | Refresh DB; update keyring; reflector. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 7 | ISO build reproducibility | Recovery/install media confidence becomes historical. | AUR package fails to build; mkarchiso or DKMS breaks. | Need rescue/install media and discover no current ISO can be built. | Parse logs; clean work; explicit rebuild. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 8 | Post-install service health | System is installed but degraded in daily operation. | DNS, NetworkManager, fail2ban, tailscale, or user service down. | No remote access, no network, broken sync, or security tooling off. | Restart/re-enable classified services only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 9 | Archsetup state/log cleanliness | Half-completed provisioning masquerades as success. | Missing marker, log error, skipped step after resume. | Fresh workstation lacks critical config but looks mostly usable. | Summarize; rerun resumable archsetup. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 10 | Workstation contract checks | The box drifts from "Craig's workstation" to generic Arch. | Dotfile symlink broken, keyring wrong, missing tool/package. | Desktop/session workflow is broken during real work. | Restow, repair perms, reinstall with confirm. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 11 | Backup and rollback readiness | Recovery assumptions go stale. | Missing =.archsetup.bak=, backup timer stale, dry-run fails. | Local rollback works but important personal/system state is gone. | Dry-run, start configured job, no deletes. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 12 | Security and hardening drift | Small protective edits quietly regress. | SSH/firewall/fail2ban/sysctl/EFI mask drift. | Exposed service or weakened local boot/config protections. | Restore owned snippets; no broad rewrite. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| + +The worst cases are intentionally conservative, not dramatic. The monitor is +useful because it catches the boring early signal: stale, missing, not recently +tested, or silently degraded. + +* Product Shape + +Use the existing instrument-console panel language rather than a dashboard page: +lamps for pass/warn/fail state, engraved metric groups, dense rows, physical +console keys for explicit actions, and an output well for the last diagnosis. + +The bar module should be tiny: + +- icon: a pulse/terminal glyph or compact =SYS= label. +- state lamp: green/yellow/red/grey. +- text: one word only: =OK=, =WARN=, =FAIL=, =STALE=, =RUNNING=, =UNKNOWN=. +- click: opens the system monitor panel. +- middle click or secondary action: run a cheap refresh only, never repairs. + +The panel should be one screen with internal scroll only where needed. The +first viewport should show the decision state and the Doctor controls without +scrolling. + +* Layout + +** Faceplate + +Top row: + +- State lamp + state word: =OK= / =WARN= / =FAIL= / =STALE= / =RUNNING= / + =UNKNOWN=. +- Unit label: =SYS·01=. +- Scope segmented control: =HOST= / =INSTALL= / =BUILD=. +- Badges: =ZFS= or =BTRFS=, =VM STALE=, =DB STALE=, =SNAPSHOT=, =BACKUP=, + =ROOT= when elevated actions are available. +- Close button. + +The scope control changes the metric emphasis, not the underlying data model: + +- =HOST= is this laptop/workstation right now. +- =INSTALL= is the last Archangel+Archsetup VM install result. +- =BUILD= is ISO, AUR repo, archzfs, and test artifact health. + +** Health Stack + +Arrange metrics as four horizontal bands. Each band has a section title, a +summary lamp, two to four row lamps, and a short "age" or "count" value. + +1. =BOOT + STORAGE= + - bootloader + - EFI redundancy + - pool/filesystem + - snapshots + +2. =INSTALL PIPELINE= + - first-upgrade reboot + - ZFS VM install + - Btrfs VM install + - ISO build + +3. =PACKAGES + SERVICES= + - pacman sync freshness + - archzfs/AUR health + - failed services + - journal errors + +4. =WORKSTATION CONTRACT= + - Archsetup markers/logs + - user/dotfiles + - desktop/session + - backups/replication + +Each row is clickable. Clicking a row opens the evidence drawer in the output +well with: + +- last command run +- normalized verdict +- raw excerpt, redacted where needed +- suggested Doctor action, if any + +** Console Keys + +Use physical console-key buttons, same family as net/bt: + +| Key | Purpose | +|--------------+------------------------------------------------------------| +| REFRESH | Cheap read-only probe of host state | +|--------------+------------------------------------------------------------| +| DOCTOR | Diagnose, classify, run safe mitigations, re-check | +|--------------+------------------------------------------------------------| +| TEST VM | Run or schedule Archsetup VM validation | +|--------------+------------------------------------------------------------| +| BUILD ISO | Run or schedule Archangel ISO build | +|--------------+------------------------------------------------------------| +| CLEAN | Clean old test artifacts, package cache, stale logs | +|--------------+------------------------------------------------------------| +| SNAPSHOT | Create a manual pre-change snapshot | +|--------------+------------------------------------------------------------| + +Keys that can take a long time stream progress into the output well. Mutating +keys must use arm-first behavior: + +- first click arms for 3 seconds and explains the action. +- second click runs. +- destructive cleanup names what will be deleted before it runs. + +* Metric Details + +Details below are grouped by workflow rather than priority. The authoritative +importance order is the priority table above. + +** 1. End-to-end VM install pass rate =P1/rank 5= + +Problem overcome: unit tests can pass while the real install is broken by +mirrors, bootloader state, pacstrap, SSH, disk layout, or the actual desktop +contract. This metric fights false confidence. + +Representation: + +- Lamp row in =INSTALL PIPELINE=. +- Two child lamps: =ZFS= and =BTRFS=. +- Age chip: =last pass 2d= / =never= / =stale 14d=. +- Red if either required filesystem has no recent pass. + +Tools: + +- =scripts/testing/run-test.sh= +- =scripts/testing/create-base-vm.sh= +- =pytest= testinfra suite under =scripts/testing/tests/= +- =qemu-img=, =qemu-system-x86_64=, =sshpass= + +Doctor: + +- read-only first: summarize last =test-results/*/test-report.txt= and failing + test names. +- mitigation: offer =TEST VM= for the failed filesystem. +- cleanup: remove stale temporary VM overlays before retrying. +- no automatic retry loop if the failure is in the installer itself. + +** 2. First-upgrade bootability =P1/rank 4= + +Problem overcome: the machine can boot immediately after install but fail after +the first =pacman -Syu= because initramfs hooks, ZFS modules, GRUB, ZFSBootMenu, +or kernel packages drift. + +Representation: + +- Lamp row: =first-upgrade reboot=. +- Badge: =not run=, =passed=, =failed=. +- Evidence drawer includes boot count, kernel version, and last reachable SSH + timestamp. + +Tools: + +- VM harness +- =pacman -Syu= +- =reboot= +- =ssh= reachability checks +- =journalctl -b -1= where available + +Doctor: + +- run upgrade-in-VM only, never on host without explicit confirmation. +- if failure is ZFS, collect =zpool import=, =lsinitcpio=, =mkinitcpio.conf=, + and EFI files. +- if failure is Btrfs, collect =grub.cfg=, =crypttab=, =fstab=, and snapper + config. + +** 3. Package database freshness and sync health =P1/rank 6= + +Problem overcome: Arch rolling-release state changes faster than installer +assumptions. Stale sync databases, stale keyrings, archzfs drift, or broken +mirrors are leading indicators of a failing install. + +Representation: + +- Row lamp in =PACKAGES + SERVICES=. +- Small meter: newest sync DB age vs threshold. +- Child chips: =core=, =extra=, =multilib=, =archzfs=. +- Yellow over 48 hours, red over 7 days or failed sync. + +Tools: + +- =find /var/lib/pacman/sync= +- =pacman -Syyu --needed archlinux-keyring= +- =checkupdates= +- =reflector= +- =pacman-conf= + +Doctor: + +- safe: refresh package databases. +- safe: update =archlinux-keyring= before full upgrades. +- mitigation: run =reflector= with the configured country/age policy. +- no unattended full system upgrade from the panel unless separately approved. + +** 4. Archangel ISO build reproducibility =P1/rank 7= + +Problem overcome: an old "good" ISO can hide broken current inputs. Archiso, +archzfs, DKMS, AUR package recipes, and pacoloco cache state can all break the +next install. + +Representation: + +- Row lamp in =INSTALL PIPELINE= or =BUILD= scope. +- Shows latest ISO date, kernel version, and AUR manifest age. +- Red if latest build failed or no ISO exists. +- Yellow if latest successful ISO is older than the configured freshness + window. + +Tools: + +- =make build= in =~/code/archangel= +- =build.sh --skip-aur= for fast non-AUR iteration +- =build-aur.sh= +- =mkarchiso= +- =pacoloco= status if installed + +Doctor: + +- read-only: parse latest =out/*.log= for pacman, DKMS, archzfs, AUR, and + mkarchiso failures. +- cleanup: safe build-work cleanup only through Archangel's cleanup function + or =make clean=. +- mitigation: suggest =--skip-aur= when the failure is unrelated to baked AUR. +- build retry is explicit via =BUILD ISO=, not automatic. + +** 5. ZFS/Btrfs storage health =P0/rank 1= + +Problem overcome: the root filesystem can degrade silently before the user +notices. For ZFS this means pool errors or degraded vdevs; for Btrfs this means +device stats, scrub failures, metadata pressure, or degraded RAID. + +Representation: + +- =BOOT + STORAGE= band. +- Filesystem-specific lamp grammar: + - ZFS green: =zpool status -x= healthy. + - Btrfs green: device stats clean and recent scrub clean. +- Capacity strip for root/home/EFI. + +Tools: + +- ZFS: =zpool status -x=, =zpool list=, =zfs list=. +- Btrfs: =btrfs device stats=, =btrfs filesystem usage=, + =btrfs scrub status=. +- Common: =df -h=, =findmnt=, =lsblk=. + +Doctor: + +- safe: start a scrub only with arm-first confirmation. +- safe: clear stale Btrfs stats only after a clean scrub and explicit + confirmation. +- mitigation: warn on low EFI/root space and offer package cache cleanup. +- never destroy snapshots, pools, subvolumes, or datasets from Doctor. + +** 6. Snapshot safety coverage =P0/rank 2= + +Problem overcome: rollback safety is assumed during upgrades but can disappear +when hooks, services, or snapshot tools drift. + +Representation: + +- Row lamp: =snapshots=. +- Child chips: =genesis=, =pre-pacman=, =recent=, =pruned=. +- Yellow if no recent snapshot. +- Red if genesis or pre-transaction hook is missing. + +Tools: + +- ZFS: =zfs list -t snapshot=, =zfs-pre-snapshot=, + =/etc/pacman.d/hooks/zfs-snapshot.hook=. +- Btrfs: =snapper list=, =snap-pac=, =grub-btrfs-mkconfig=, + =/.snapshots=. +- Common: =pacman -Q= for snapshot packages. + +Doctor: + +- safe: create a manual snapshot. +- safe: reinstall or re-enable missing hook only if the expected script exists. +- cleanup: prune only snapshots matching the tool-owned policy and prefix. +- mitigation: show exact command for manual rollback; do not perform rollback + from the panel. + +** 7. Bootloader and EFI redundancy =P0/rank 3= + +Problem overcome: single-disk bootloader success can mask missing redundant EFI +installs on multi-disk systems. A system can also pass install but lose a boot +entry or generate an invalid config. + +Representation: + +- Row lamp: =bootloader=. +- Child chips: =ZBM= or =GRUB=, =EFI=, =entries=, =all disks=. +- Yellow if redundancy cannot be proven. +- Red if the expected loader/config is missing. + +Tools: + +- =bootctl status= +- =efibootmgr -v= +- =findmnt /efi /boot= +- ZFS: check =/efi/EFI/ZBM/zfsbootmenu.efi=. +- Btrfs: check =/boot/grub/grub.cfg= and grub-btrfs entries. + +Doctor: + +- read-only by default. +- mitigation: regenerate GRUB config for Btrfs with arm-first confirmation. +- mitigation: rebuild initramfs with arm-first confirmation. +- no automatic EFI reinstall without an explicit advanced flow. + +** 8. Post-install service health =P1/rank 8= + +Problem overcome: the install can complete while the real workstation is +degraded: DNS broken, NetworkManager failed, fail2ban not responding, user +services not lingering, or Docker/Tailscale/Syncthing not in their expected +state. + +Representation: + +- =PACKAGES + SERVICES= band. +- Count badge: =0 failed= or =3 failed=. +- Child lamps: =network=, =dns=, =security=, =user services=. + +Tools: + +- =systemctl --failed= +- =systemctl is-enabled/is-active= +- =resolvectl status= +- =nmcli general status= +- =fail2ban-client status= +- =loginctl show-user= + +Doctor: + +- safe: restart known flaky non-destructive services such as + =NetworkManager= only after classifying the failure. +- safe: re-enable expected services from Archsetup's contract. +- mitigation: bounce DNS resolver and re-check. +- no blanket =systemctl restart --failed=. + +** 9. Archsetup state and log cleanliness =P1/rank 9= + +Problem overcome: a resumable installer can leave a half-finished system that +looks usable until a missing marker or skipped step matters later. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Step-progress mini bar: completed markers / expected markers. +- Red if =archsetup --status= reports incomplete required steps. +- Red if latest log contains fatal errors. + +Tools: + +- =./archsetup --status= +- =/var/log/archsetup-*.log= +- marker files from the Archsetup state directory +- existing testinfra assertions in =scripts/testing/tests/test_archsetup.py= + +Doctor: + +- read-only: summarize incomplete steps and latest log errors. +- mitigation: offer to rerun =archsetup= in normal resumable mode. +- cleanup: archive old logs, keep the latest N. +- never run =--fresh= from Doctor. + +** 10. Workstation contract checks =P2/rank 10= + +Problem overcome: a fresh Arch system is not the goal. The goal is Craig's +working machine: user, shell, groups, dotfiles, Emacs, Hyprland/DWM, keyring, +VPN tools, Bluetooth tools, and local scripts. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Child lamps: =user=, =dotfiles=, =desktop=, =tools=. +- Evidence drawer mirrors the testinfra checks. + +Tools: + +- =id=, =getent passwd= +- =test -L ~/.zshrc= +- =stow= via dotfiles Makefile +- =pacman -Q=, =yay -Qi yay= +- =hyprctl=, =gdbus= portal checks when session is running + +Doctor: + +- safe: restow dotfiles with the selected profile. +- safe: repair keyring directory permissions. +- mitigation: reinstall missing official packages. +- AUR package rebuilds require confirmation and stream output. + +** 11. Security and hardening drift =P2/rank 12= + +Problem overcome: security settings are easy to regress because they are small +file edits: SSH root login, EFI mount masks, firewall, issue banner, fail2ban, +quiet printk. + +Representation: + +- Compact row under =WORKSTATION CONTRACT= or =PACKAGES + SERVICES=. +- Red only for high-risk drift, yellow for unknown/unreadable state. + +Tools: + +- =sshd -T= or config file checks +- =ufw status= +- =fail2ban-client status= +- =findmnt /efi= +- =sysctl kernel.printk= + +Doctor: + +- safe: restore known Archsetup-owned config snippets. +- safe: re-enable firewall if policy file is present. +- mitigation: write missing drop-ins only from version-controlled templates. +- no broad hardening rewrite from panel state. + +** 12. Backup and rollback readiness =P2/rank 11= + +Problem overcome: rollback only helps local state. The install also needs +backups of edited system files and confidence that personal data replication is +not silently stale. + +Representation: + +- Row lamp: =backups=. +- Chips: =system-file .bak=, =replication=, =last run=. +- Yellow if last replication exceeds policy. +- Red if expected backup files for edited system config are missing. + +Tools: + +- Archsetup backup assertions in =scripts/testing/tests/test_backups.py=. +- =zfs-replicate= if configured. +- =systemctl list-timers= for backup timers. +- =journalctl -u= relevant backup units. + +Doctor: + +- safe: create missing =.archsetup.bak= for files before editing. +- safe: run dry-run replication check. +- mitigation: start a configured backup timer/unit with confirmation. +- never delete backup targets from Doctor. + +* Doctor Model + +Doctor is a classifier with bounded mitigations, not a magic repair button. + +Flow: + +1. Probe the selected scope. +2. Normalize each metric to =ok=, =warn=, =fail=, =unknown=, or =running=. +3. Classify failures as: + - =safe-fix= — local, reversible, low risk. + - =safe-cleanup= — removes only known generated artifacts. + - =mitigation= — improves the chance of success but does not claim repair. + - =needs-confirmation= — mutating, long-running, or system-wide. + - =manual= — too dangerous or context-heavy for Doctor. +4. Run only safe actions automatically after the user presses Doctor. +5. Arm-first for anything mutating beyond safe local cleanup. +6. Re-run the affected probe. +7. Stream a verdict into the output well. + +Doctor should say exactly what it did: + +#+BEGIN_EXAMPLE +doctor: package db stale + check: core.db age 4d, archzfs.db age 4d + action: refreshed sync databases + action: updated archlinux-keyring + result: ok, newest db age 2m +#+END_EXAMPLE + +* Common Tool Drivers + +** Host probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| systemd | =systemctl --failed=, =systemctl is-active=, =journalctl= | +|------------+----------------------------------------------------------------| +| packages | =pacman=, =checkupdates=, =pacman-conf=, =yay= | +|------------+----------------------------------------------------------------| +| storage | =zpool=, =zfs=, =btrfs=, =df=, =findmnt=, =lsblk= | +|------------+----------------------------------------------------------------| +| boot | =bootctl=, =efibootmgr=, =mkinitcpio=, =grub-mkconfig= | +|------------+----------------------------------------------------------------| +| network | =nmcli=, =resolvectl=, =ping= or HTTPS probe | +|------------+----------------------------------------------------------------| +| desktop | =hyprctl=, =gdbus=, =loginctl=, dotfiles Makefile | +|------------+----------------------------------------------------------------| + +** Project probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| archangel | =make test=, =make build=, =build.sh --skip-aur= | +|------------+----------------------------------------------------------------| +| archsetup | =make test-unit=, =make test=, =scripts/testing/run-test.sh= | +|------------+----------------------------------------------------------------| +| VM | =qemu-img=, =qemu-system-x86_64=, =sshpass=, =pytest= | +|------------+----------------------------------------------------------------| +| artifacts | latest =out/*.log=, =out/*aur-manifest.tsv=, =test-results/*= | +|------------+----------------------------------------------------------------| + +* Top-family Comparison + +This monitor should borrow the mature display ideas from =top=-style tools +without becoming another CPU/process viewer. The domain objects are install +contracts, boot/storage health, package freshness, snapshots, services, and +artifacts. The interaction model is still the same: sort the thing that hurts, +filter to the thing you care about, expand one row for evidence, and act only +when the diagnosis is clear. + +** Comparison table + +| Tool | What it represents well | Sorting/filtering model | Useful pattern for system monitor | Gaps for our domain | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =htop= | Dense live table plus configurable meters; process tree; direct process actions. | Interactive sort by column, search, filter, tree toggle. | Metric table should support column sort, search, filter, and tree/group mode. | No historical artifact model; actions are process-centric. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =btop++= | Boxed dashboard: CPU, memory, disks, network, processes, battery, GPU; strong graph language; selected process detail. | Easy switching between process sort modes; filter; tree view; pause. | Use boxed bands, mini time-series, detail pane, pause/freeze, and clickable controls. | Graph-first layout can overemphasize volatile values over install risk. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =bottom/btm= | Custom widget layout, per-widget focus/expand, zoomable time windows, basic mode. | Process widget supports sort, search, tree; widgets can be filtered/configured. | Every health band should be expandable; stale/history windows should be zoomable. | Mostly resource telemetry, not remediation workflow. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =atop= | Interval deltas, critical-resource highlighting, all active processes including exited ones, long-term logs. | Resource views and interval replay; emphasizes deviations and active load. | Add history/replay for health events and show "new since last good" changes. | Lower immediate visual polish; Linux-performance scoped. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =Glances= | Broad plugin dashboard, thresholds, remote/web/API modes, export to JSON/CSV/time-series backends. | Configurable visible plugins; API/stdout selectors instead of only interactive sorting. | Use plugin architecture, threshold config, JSON output, remote/headless mode. | Too broad; can become a generic monitoring surface. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =procs= | Modern table ergonomics: custom columns, keyword search across selected fields, sort by named column, tree view. | CLI sort asc/desc by partial column name; watch mode cycles sort columns; AND/OR/NAND/NOR search. | Use named metric columns, saved views, multi-keyword filters, and value-aware coloring. | Process-only; no graphs or remediation model. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =gotop/gtop= family | Fast glanceable terminal dashboard with compact graphs and gauges. | Usually lighter than htop/btop; less important than presentation density. | Use compact sparklines/gauges for "age", "last pass", and "failure count". | Not enough evidence/action depth for this monitor. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| + +** What to pull in + +*** htop: table discipline + +Pull: + +- Column headers that are real controls: click or key-cycle to sort by + =priority=, =state=, =age=, =last_checked=, =last_pass=, =failure_count=, + =scope=, and =doctor_class=. +- Search and filter as first-class actions, not hidden debug commands. +- Tree mode for ownership: + - =system= → =boot/storage= → =bootloader= → =efi entries=. + - =archangel= → =iso build= → =aur repo= → =manifest rows=. + - =archsetup= → =state markers= → =desktop= → =dotfiles=. +- Horizontal detail access for long evidence, like htop's horizontal scrolling + for full commands. + +Equivalent-or-better requirement: + +- htop sorts processes; this monitor sorts risk. The default sort is + live-state severity, then priority rank, then age. + +*** btop++: instrument boxes and live graphs + +Pull: + +- Boxed bands with stable geometry. +- Small time-series graphs, but only where history matters: + - package DB age over time + - failed-service count + - journal error count + - snapshot count / newest snapshot age + - VM pass/fail history + - ISO build duration/result history +- Selected-row detail pane with the last command, verdict, and raw excerpt. +- Pause/freeze button so a failure does not scroll away while reading. +- Mouse-clickable controls where every visible key has the same keyboard path. + +Equivalent-or-better requirement: + +- btop's graphs answer "what is hot right now?" Our graphs answer "is the + safety margin shrinking?" Trend charts should be muted unless the threshold + is crossed. + +*** bottom: focus/expand and layout presets + +Pull: + +- Expand one band full-height: + - =BOOT + STORAGE= expands into boot files, EFI entries, pools, snapshots. + - =INSTALL PIPELINE= expands into last VM runs and build artifacts. + - =PACKAGES + SERVICES= expands into DB ages, repo status, failed units. + - =WORKSTATION CONTRACT= expands into Archsetup markers and testinfra-style + checks. +- Zoomable history windows: 24h / 7d / 30d / all artifacts. +- Layout presets: + - =compact= for bar dropdown. + - =full= for terminal/TUI. + - =host-only= for laptop health. + - =release-gate= for Archangel/Archsetup changes. + +Equivalent-or-better requirement: + +- bottom expands widgets; this monitor expands evidence and remediation state. + The expanded view must show "what changed since last good" before raw logs. + +*** atop: history and vanished failures + +Pull: + +- Permanent, compact health-event log. +- Interval deltas instead of only current values: + - new failed services since last check + - new journal errors since last check + - packages/repos newly stale + - snapshot hook present before, missing now + - bootloader file changed since last known-good +- "Show active/deviating only" mode. In normal use, hide green rows unless + their age is approaching threshold. +- Replay mode: inspect the state at the time an install/test/build failed. + +Equivalent-or-better requirement: + +- atop can report processes that already exited. This monitor should report + failures that already passed through: a transient failed unit, a VM test that + failed last night, an ISO build that failed before the current successful + build, or a package DB that was stale until Doctor fixed it. + +*** Glances: plugin/API/export model + +Pull: + +- Plugin-like probes. Each metric owns: + - =probe= + - =normalize= + - =thresholds= + - =doctor_actions= + - =redaction= + - =evidence= +- JSON output as a stable contract before GTK work. +- Optional stdout selectors: + - =system-monitor --stdout packages.state,storage.state= + - =system-monitor --json boot,snapshots= +- Remote/headless mode for VMs and bare-metal test targets. +- Threshold config in one file, not hardcoded in the UI. + +Equivalent-or-better requirement: + +- Glances is broad; this must stay opinionated. A plugin is accepted only if it + maps to install health, rollback safety, workstation contract, or recovery + readiness. + +*** procs: custom columns and query grammar + +Pull: + +- Named columns and saved views: + - =risk=: state, priority, age, doctor class. + - =install=: last pass, filesystem, artifact, branch, commit. + - =host=: state, source, last checked, command. + - =doctor=: action class, requires root, reversible, last run. +- Multi-keyword search: + - =zfs failed= + - =doctor safe-fix= + - =archangel stale= + - =service red= +- Boolean query modes: + - AND default for narrowing. + - OR for "show any boot or storage issue". + - NOT for "hide green". +- Value-aware coloring for age, severity, and units. + +Equivalent-or-better requirement: + +- procs lets the user build a process table. This monitor should let Craig + build a risk table without editing code. + +*** gotop/gtop: glance density + +Pull: + +- Small sparklines for trend, not full charts. +- Big obvious state words. +- Compact gauges for bounded values: + - EFI usage + - root/home usage + - DB age as percent of freshness window + - VM evidence age + - snapshot age +- Simple default screen that is useful without learning keys. + +Equivalent-or-better requirement: + +- The first screen should answer "am I safe to upgrade or install?" in under + two seconds. + +* Sorting and Views + +The monitor needs two sorting layers: global row ordering and per-band evidence +tables. + +** Global row ordering + +Default: + +1. =state= severity: red, yellow, unknown, running, green. +2. =priority= rank: P0 before P1 before P2. +3. =age= or =staleness=, descending. +4. =last_changed=, newest first. + +Alternate sorts: + +| Sort key | Use case | +|-----------------+------------------------------------------------------| +| =priority= | Release-gate review; keep P0/P1 at the top. | +|-----------------+------------------------------------------------------| +| =state= | Triage; show all red/yellow rows first. | +|-----------------+------------------------------------------------------| +| =age= | Find stale tests, stale package DBs, old backups. | +|-----------------+------------------------------------------------------| +| =doctor_class= | Find what Doctor can safely fix now. | +|-----------------+------------------------------------------------------| +| =scope= | Group host vs install vs build. | +|-----------------+------------------------------------------------------| +| =last_changed= | See what recently regressed. | +|-----------------+------------------------------------------------------| +| =source= | Group by archangel, archsetup, dotfiles, host. | +|-----------------+------------------------------------------------------| + +** Per-band sorts + +| Band | Sorts | +|------------------------+------------------------------------------------------------| +| =BOOT + STORAGE= | severity, mountpoint, filesystem, capacity, last scrub, newest snapshot age | +|------------------------+------------------------------------------------------------| +| =INSTALL PIPELINE= | result, filesystem, duration, artifact age, commit age, last pass | +|------------------------+------------------------------------------------------------| +| =PACKAGES + SERVICES= | severity, unit name, repo name, DB age, error count, enabled/active state | +|------------------------+------------------------------------------------------------| +| =WORKSTATION CONTRACT= | severity, check name, owner repo, last pass, doctor class | +|------------------------+------------------------------------------------------------| + +** Filters + +Quick filters should be visible as chips: + +- =red= +- =yellow= +- =doctorable= +- =needs-root= +- =stale= +- =zfs= +- =btrfs= +- =host= +- =install= +- =build= +- =changed= +- =hidden-green= + +The default view can hide healthy low-priority rows, but it must show enough +green P0/P1 summary state to prove the monitor is working. + +* Display Requirements Borrowed from Tops + +1. Every table has sortable columns and a visible sort indicator. +2. Every visible metric row has a filterable state, priority, age, and source. +3. Every row can expand to evidence without losing the list context. +4. Every graph has a threshold marker; trend without threshold is decoration. +5. Every long-running action can be paused/frozen in the display. +6. Every mutating action has an equivalent CLI command shown in the output well. +7. Every Doctor action records before/after state so fixed failures remain + visible in history. +8. Every band has a compact mode and an expanded mode. +9. Green rows are quiet; new regressions are loud. +10. The system must be useful over SSH/TUI before GTK polish. + +* Source Notes + +- =htop=: upstream README describes configurable system/process display, + interactive sorting/filtering/search, tree view, and process actions. +- =btop++=: upstream README describes resource boxes, detailed process stats, + filter, sort switching, tree view, mouse support, auto-scaling network graphs, + disk IO, battery, GPU support, and themes. +- =bottom/btm=: upstream README describes customizable widgets, process sort + and search, tree mode, expand/focus, zoomable graph intervals, filters, and + basic mode. +- =atop=: upstream README describes interval resource accounting, critical + highlighting, long-term compressed logs, exited-process visibility, cgroup + views, and active/deviation-focused output. +- =Glances=: upstream README describes plugin-style broad monitoring, web/API + modes, stdout JSON/CSV, remote monitoring, exports, and threshold-oriented + dashboard use. +- =procs=: upstream README describes configurable columns, named-column sort, + watch mode, tree view, logical keyword search, value-aware coloring, and + pager behavior. + +* Data Model + +Emit JSON from a CLI first; the panel is a client. + +#+BEGIN_SRC json +{ + "v": 1, + "scope": "host", + "state": "warn", + "ts": "2026-07-04T12:00:00-04:00", + "metrics": [ + { + "id": "packages.sync_freshness", + "label": "package databases", + "state": "warn", + "summary": "archzfs.db age 4d", + "evidence": [ + {"command": "find /var/lib/pacman/sync", "excerpt": "archzfs.db 2026-06-30"} + ], + "doctor": { + "class": "safe-fix", + "actions": ["refresh-sync-db", "update-keyring"] + } + } + ] +} +#+END_SRC + +* Implementation Notes + +- Start with a CLI: =system-monitor status --json=, =system-monitor doctor + --json=, =system-monitor refresh=. +- Keep probes read-only by default. Actions live in separate verbs. +- Cache slow probes. The bar should read a cache, not run VM tests. +- VM/build actions should create job records and stream logs; the panel follows + the job rather than blocking the UI process. +- Reuse the net/bt panel architecture if this becomes a GTK panel: GTK-free + model + fake-command unit tests + one AT-SPI smoke. +- Redact secrets from logs and JSON: WiFi PSKs, tokens, private repo URLs with + credentials, SSH material, and backup target credentials. + +* Open Decisions + +** TODO Where should the first implementation live? + +Recommendation: dotfiles owns the user-facing panel and CLI wrapper because it +is workstation UI. Archsetup owns reusable install-contract probes and testinfra +assertions. Archangel owns ISO/build probes. + +** TODO Should Doctor run elevated actions through polkit or terminal? + +Recommendation: read-only checks run unprivileged; elevated actions launch a +terminal or polkit prompt with the exact command visible. Do not hide long +privileged operations inside the panel process. + +** TODO How fresh must VM evidence be? + +Recommendation: host checks go stale after 1 hour; package DB after 48 hours; +VM install evidence after 7 days; ISO build evidence after 14 days or whenever +Archangel/Archsetup has changed since the last successful artifact. + +** TODO Which actions are allowed on bare metal? + +Recommendation: host Doctor may refresh databases, update keyring, restow +dotfiles, create snapshots, run scrub, and restart narrowly classified services. +It may not perform full upgrades, bootloader reinstalls, destructive snapshot +prune, or filesystem repair without a separate advanced flow. + +* First Build Slice + +1. CLI read-only host status: + - package DB freshness + - failed services + - ZFS/Btrfs health + - snapshot presence + - Archsetup status/log check +2. Doctor safe actions: + - refresh package DB + - update keyring + - restow dotfiles + - create manual snapshot +3. Artifact parser: + - latest Archsetup VM test result + - latest Archangel ISO build result +4. Panel prototype: + - faceplate + - four health bands + - evidence output well + - REFRESH and DOCTOR keys only |