path: root/docs/design/maintenance-console-design-ideas.org

#+TITLE: Maintenance Console — Design Ideas
#+AUTHOR: Craig Jennings
#+DATE: 2026-07-06

* Direction

A single-host maintenance console (GUI, waybar-launched) that surfaces every
health metric for this Arch workstation and, where the remediation is safe,
runs it. It reshapes the earlier install-health/workstation-contract framing
(=system-monitor-design-ideas.org=) into a maintenance surface with a doctor
action.

The console is the glanceable, single-host version of the home project's
=system-health-check.org= workflow (~1000 lines, capability-dispatched across
ratio/velox/mybitch/truenas). The console owns the routine-maintenance and
at-a-glance-status slice; the workflow stays as the escalation path for
forensic investigation and as the only sanctioned way to run updates.

* Scope: hosts (Q2 resolved 2026-07-06)

v1 targets *both* daily drivers — ratio and velox — as first-class hosts, not
ratio-first-velox-later. Consequences for v1:

- *Capability dispatch is mandatory from day one*, not deferred. The console
  probes the live host and runs only applicable checks (btrfs RAID1 on ratio vs
  ZFS-primary on velox; AMD amd-pstate on ratio vs Intel intel_pstate on velox).
  This mirrors the home workflow's Phase 0 probe.
- *velox-only metrics ship in v1*: battery health (capacity / cycles) and the
  unclean-shutdown / suspend-failure rate — both currently GAPs, both driven by
  velox being the machine that travels and suspends.
- The ZFS-primary remediation paths (scrub, snapshot retention, pool health) are
  built and tested, not stubbed.

* Thresholds source of truth (Q3 resolved 2026-07-06)

One machine-readable file, =maintenance-thresholds.toml=, holds every severity
value (cache-size trigger, disk %, scrub-age warn/crit, snapshot retention
limits, temp bands, backup-staleness windows, etc.). archsetup owns and ships
it; both consumers read the *installed* path, so neither reaches into the
other's repo:

- the maintenance console (dotfiles code) reads it at runtime,
- the system-health-check workflow reads it instead of hardcoding severity
  rules in prose.

The workflow's hard-won values migrate into the TOML as the seed content (the
snapshot MONTHLY limit that bit /home, the 10 GB cache trigger, the scrub-age
bands). Install path is an implementation sub-question — a stable location both
consumers reach (e.g. =~/.config/archsetup/maintenance-thresholds.toml=).

* Workflow ownership — move system-health-check into archsetup (proposed 2026-07-06)

Craig's call: the =system-health-check.org= workflow should live in archsetup,
not the home project. Rationale — the home project is scoped to finances,
health, and personal matters; system design, execution, and maintenance are
archsetup's domain. Home only owns the workflow by inherited accident, not by
fit. Moving it here also collapses the Q3 coupling: the TOML source-of-truth and
its workflow consumer end up in the same project.

Move scope (home → archsetup), to sequence when Craig gives the go:
- =system-health-check.org= (the workflow itself)
- =homelab-inventory/*.org= (ratio/velox/mybitch/truenas capability inventories
  it cross-references)
- any home-project references to the workflow (startup, project-workflows index)

Cross-project mechanics: the archsetup side (receiving the files, wiring the
TOML) is in-scope here; the home side (removing the originals, updating home's
references) is the home project's scope — handled by a handoff note to home's
inbox or a home session, not edited blind from here.

* Panel shape

Three regions, driven by the Automation column of the metrics table below:

- *Actionable* (left) — metrics with a lever. Automation = Auto or Confirm.
  Two doctor actions serve this column: "Clean up" fires every Auto metric
  unattended; "Review & fix" opens a preview for the Confirm metrics and acts
  only on approval.
- *Diagnostic* (right) — read-only telemetry. Automation = None or Human. A
  state color (green/amber/red) and the value; no button. Red here is the
  signal to run the workflow.
- *Updates* (quarantined strip) — Automation = Workflow. Shown as a count with
  notable packages named; the only affordance is "run the workflow." The panel
  never applies updates in place.

*Bar glyph* tracks the worst *Diagnostic* state only — not the actionable
count. A big package cache is boring; a SMART failure is a fire. Actionable
clutter must not turn the bar red or it trains you to ignore it.

** Doctor = live output wall (Q1 resolved 2026-07-06)

"Clean up" and "Review & fix" are not fire-and-forget buttons. Running a doctor
opens an *output wall* — one lamp per action, streaming in realtime as each
runs:

- *amber* while the action is running,
- *green* on success (with the reclaimed amount / result inline),
- *red* on failure.

Feedback is always shown (not just when something was reclaimed) — you watch it
happen. This is the same live-results shape that should back the other doctors
(net, bluetooth), so every doctor in the system reads the same way. See the
todo task to retrofit the net + bt doctors to realtime lamp output.

* Automation legend

| Value    | Meaning                                                                                                   |
|----------+-----------------------------------------------------------------------------------------------------------|
| Auto     | Doctor "Clean up" button — fire unattended, reversible or harmless                                        |
|----------+-----------------------------------------------------------------------------------------------------------|
| Confirm  | Doctor "Review & fix" — preview the change, act only on one click                                         |
|----------+-----------------------------------------------------------------------------------------------------------|
| Human    | Panel nudges; a human decides and acts (no button, not the agent)                                         |
|----------+-----------------------------------------------------------------------------------------------------------|
| Workflow | vLater — metric shown read-only; no in-panel button. Agent-workflow assistance deferred (decided          |
|          | 2026-07-07, see below)                                                                                    |
|----------+-----------------------------------------------------------------------------------------------------------|
| None     | Diagnostic only — no software remediation exists (hardware, telemetry)                                    |
|----------+-----------------------------------------------------------------------------------------------------------|

*Workflow buttons removed (decided 2026-07-07, prototyping E5).* The panel
carries no "run workflow" affordances — not on failed units, unclean boots,
updates, or CVEs. The metrics stay on the board as read-only telemetry; AI
assistance via the system-health-check workflow is a vLater feature. v1 ships
only determinate remedies (fixed, scriptable actions with predictable
outcomes).

*Determinate remedies adopted (decided 2026-07-07).* Every remedy below is
Confirm-tier; contextual levers appear only when the metric is off-nominal:

- /Service restarts & enables/: failed units (systemctl restart + reset-failed),
  fail2ban, cronie, chronyd (+ makestep), tailscaled, snapper timers, zram
  config re-apply. DNS/NetworkManager delegates to the net panel's doctor
  (deep-link, not a duplicate repair chain).
- /Deterministic maintenance ops/: btrfs balance -dusage=50 (unallocated low),
  reinstall owning packages on pacman -Qkk failures, rsyncshot RUN NOW (result
  streams to the output well), smartctl -t short self-test, snapshot-retention
  repair (write sane TIMELINE limits + cleanup — the /home lesson as a
  one-press fix). btrfs device-error counter reset stays manual: resetting
  without diagnosis masks a dying drive.
- /Composite macro/: RECLAIM SPACE on the disk-usage cell — runs every reclaim
  lever (cache, journal, coredumps, app logs, docker tier-1, snapper cleanup)
  as one output-wall stream.
- /Disruptive but determinate/: REBOOT behind arm-to-fire, offered when
  running kernel != installed.
- /Still read-only/: temps, throttling, battery, memory/OOM, taint, journal
  error content, kernel/hw events, listeners, unclean-shutdown rate — physical
  or investigative; vLater AI territory.

*Updates join the Confirm layer (decided 2026-07-07).* Two levers on the
updates strip, both behind the live-update guard (mesa/hyprland/wayland
runtime in the pending set):

- UPDATE — repo + AUR system update.
- TOPGRADE — full ecosystem run. The panel's wrapper always passes
  --disable git (topgrade's git step rebase-autostashes ~/code/*/ — never
  under a live session).

*MEM·PWR: evidence + two levers + expectation tags (decided 2026-07-07).*
Mostly physics, so the category leans watch-only — with these additions:

- /CPU mode selector/: a free segmented control (PERF · BAL · POWER) writing
  the EPP hint — set the active mode to anything, not drift-repair against a
  declared default. amd-pstate on ratio, intel_pstate on velox.
- /Battery charge limit/ (velox-only, capability-gated): SET 80% writes
  charge_control_end_threshold — the standard longevity cap. Battery *health*
  (capacity vs design, cycles) stays watch-only hardware telemetry.
- /Evidence drill-downs/ (digest idiom): top-5 RAM consumers under memory,
  recent boots listed clean/unclean under the unclean-rate, and throttle/OOM
  events with timestamps. Evidence makes the numbers actionable even where no
  button exists.
- /KILL on top-memory items/ (revised 2026-07-07 — Craig): arm-to-fire, four
  guards: the arm shows the exact victim (name + size); SIGTERM not SIGKILL,
  with the outcome reported to the wall; PID + process name revalidated at
  fire time so a recycled PID can't be hit; session-critical names (systemd,
  the compositor, the panel itself) render a disabled key — protected. A
  SIGKILL escalation for TERM-survivors is vLater.
- /Expectation-setting, panel-wide/: every leverless cell's sub-line carries an
  explicit tag ("hardware — watch only", "evidence below"), and each subpanel
  header shows the split — "N fixable · M watch" — so the user knows their
  agency before reading a single cell.

*Refresh cadence (decided 2026-07-07).* Four tiers, matching probe cost:

- /Live group, panel open/: temps, memory free + top consumers, throttle
  state — re-read every ~3 s while their subpanel is visible, gated on
  panel-open exactly like the audio panel's meters. Stop when hidden.
- /Fast local tier, panel open/: re-probed every ~30 s while the panel is up;
  additionally, any metric re-probes immediately after an action that touches
  it (fire CLEAN → cache re-measured, not assumed).
- /On open/: the hydration tiers re-run (fast reads first, process probes
  behind them — sub-second perceived).
- /Network tier/: checkupdates / arch-audit / AUR / firmware stay on the
  hourly systemd-timer cache with age shown; refreshed on demand only.
- /Panel closed/: the waybar glyph is fed by a light background scan every
  ~30 min (systemd timer writing the state file the glyph reads) — the bar
  stays honest without the panel running.

*Journal errors get a digest, not a fix (decided 2026-07-07).* No generic
remedy exists — an error-priority line is a symptom of an arbitrary subsystem —
so the panel ships four determinate assists instead:

- /Digest/: the cell expands to errors grouped by syslog identifier — count,
  message snippet, first/last seen, and the exact next command (journalctl -u
  <unit> -b) when the identifier maps to a unit. Top-10 groups, read-only.
- /MARK KNOWN with a full lifecycle/: arm-to-fire showing the exact pattern
  before it stores; marked groups move to a dim KNOWN section (never vanish)
  with per-row UNMARK; every mark/unmark logs to the results wall; marks carry
  date + example. Patterns bind to identifier + message, never a whole unit —
  a muted service's *new* errors still surface.
- /Two noise layers/: shipped defaults (bluetoothd HFP, pixman, xkbcomp) in
  the packaged TOML, user marks in a separate user file merged over it (a
  template sync never eats curation). CLEAR MARKS (arm-to-fire) empties the
  user layer and re-enables shipped defaults; unmarking a shipped default
  records a disable flag in the user layer.
- /OPEN JOURNAL/: launches a terminal running journalctl -p err -b — the same
  delegation pattern as NET DOCTOR.

Ruled out: auto-restarting units that log errors (error ≠ failed) and
keyword-driven fix suggestions (vLater AI territory).

*Full-sweep findings — all committed to v1 (decided 2026-07-07).* Every metric
was audited against the converged checklist (honest label, evidence digest
where a count hides detail, curation lifecycle where "expected" is config
knowledge, guarded per-item remedies, cross-links, watch-only tags). Adopted,
all prototyped:

- /Storage/: disk top-consumers digest (evidence only — no file deletion
  keys); per-device error rows on RAID1 when counters are nonzero,
  cross-checked against SMART; SMART sub-line carries the last self-test
  result. Spec note: a real scrub runs hours — the ring needs a running-%
  state, not an instant reset.
- /Packages/: orphan digest (name + size) with per-package REMOVE (armed) and
  KEEP — the curation lifecycle encoding "intentional, not orphaned" (the
  rust lesson); batch REMOVE ALL skips kept packages. Per-file pacnew rows
  tagged safe-delete (reflector-managed) vs needs-merge, MERGE delegating to
  a terminal diff. CVEs named: package · CVE id · severity. AUR and firmware
  names spelled out.
- /systemd/: failed units upgraded from a count-lever to a per-unit roster
  (name · since · exit code · journalctl hint) with per-row RESTART + RESET;
  is-system-running names its cause ("degraded — N failed units below");
  taint letters decoded.
- /Logs/: coredumps grouped by binary (count · last · coredumpctl hint),
  cleared with the CLEAR action; kernel/hw events listed when not clean
  (hardware — watch only).
- /Services/: docker system df breakdown (images / containers / volumes /
  build cache with per-type reclaimable); stopped containers upgraded to the
  full signal/expected curation lifecycle (MARK EXPECTED / UNMARK, shipped
  default: winvm) with per-container START; cron expected-entries drift
  roster.
- /Snapshots/: count split by type — timeline (auto-pruned) · single
  (manual — escapes timeline cleanup, the pile-up risk) · pre/post — with
  oldest-single named and DELETE STALE (armed, keeps newest 2) when singles
  accumulate.
- /Network minors/: fail2ban shows recent-ban count; NTP shows offset.

Rationale for prototyping everything (Craig): real estate and complexity have
bitten before — surface those limits in the disposable prototype, not after
functionality exists behind the UI.

*Vertical compression → rotary band selector (2026-07-07).* First attempt —
the MEM·PWR three-column layout on Packages/Logs/Services — lost too much row
detail to third-width truncation (Craig's verdict after use). Replaced by a
*rotary band selector*: the amplifier input-selector idiom. A machined knob
(click to cycle) whose needle swings to engraved band labels, one per evidence
section (ORPHANS · PACNEW · ADVISORIES; SIGNAL · KNOWN NOISE · COREDUMPS ·
KERNEL/HW; CONTAINERS · DOCKER DISK · CRON & BACKUPS). Each band carries its
own status lamp (section health at a glance without switching) plus a count;
the selected band gets a gold underline and the needle. One section renders at
a time at full width, restoring complete row detail. Deliberately distinct
from the category tiles, console keys, and the CPU-mode segmented control —
each selection idiom in the panel now has its own visual voice. MEM·PWR keeps
its three-column evidence strip (short rows fit fine at third-width).

*Listeners get the same treatment (decided 2026-07-07).* The count becomes
"unexpected listeners" — evidence digest (process · port · bind address from
ss -tlnp), expected-list curation with the full MARK EXPECTED / UNMARK /
CLEAR lifecycle (shipped defaults: sshd, mpd, tailscaled; user marks in the
user layer), and guarded per-socket remedies: STOP (systemctl stop, armed)
when a unit owns the socket, KILL (SIGTERM, armed) otherwise. Severity keys
on unexpected AND public-bind (0.0.0.0/::) — a loopback listener warns, an
exposed one fails — and when ufw is down the signal header names the exposure
("ufw down — N public binds exposed"). Stopped containers likewise gained a
contextual START lever (allowlist: winvm), and the firewall its ENABLE.

*Updates strip border is state-tiered (decided 2026-07-07).* Green when
nothing pending, amber for ordinary pending/AUR/firmware counts, red when
CVEs exist or pending exceeds the "a lot" threshold (or the update cache has
gone stale — staleness window in the TOML). The CVE badge renders only when
the count is nonzero.

*Guard arms instead of blocking (revised 2026-07-07).* When the guard trips,
the key arms (red, "press again to run anyway — or apply from a TTY") rather
than hard-refusing. The user decides; the footgun is acknowledged and
deliberately handed over. After a system update lands, the panel offers a
reboot: a REBOOT key (arm-to-fire) appears on the updates strip and the
reboot-required metric flips.

*No per-ecosystem update metrics.* Topgrade's step set (yay, rustup, cargo,
pipx, npm/pnpm, gem, go, flatpak, fwupd, tmux/zsh/nvim plugins, git repos) has
no cheap offline "updates available?" probe — mirroring it means a network
round-trip per registry at panel-open. Instead: one *topgrade freshness*
metric (wrapper stamps last-run time; threshold in the TOML) whose remedy is
the TOPGRADE lever, plus a *firmware updates* count in the updates strip
(fwupd refreshes metadata on its own timer; the panel reads the cache).

Rationale for the hard lines: system updates are Workflow, never Auto — the
2026-06-07 Hyprland crash was a live -Syu swapping mesa+hyprland under the
running session, and the standing rule is never -Syu live under Hyprland when
the mesa/hyprland/wayland runtime is in the set. Hardware findings (SMART, MCE,
thermal) are None — the fix is replacing a drive or clearing a fan, not
software.

* Metrics — Storage & filesystem integrity

| Metric                           | Fix / lever                      | Automation | Notes                             |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| Package cache size               | paccache -r / -ruk0              | Auto       | Reclaim, all re-downloadable      |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| paccache -rk1 (keep 1 version)   | paccache -rk1                    | Confirm    | Frees most; kills downgrade       |
|                                  |                                  |            | headroom                          |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| Disk usage (df)                  | —                                | None       | Fix is cache/snapshot/prune       |
|                                  |                                  |            | levers                            |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| Btrfs unallocated space          | —                                | None       | Chunk headroom; diagnostic        |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| Btrfs scrub age                  | btrfs scrub start                | Confirm    | GAP; ZFS has this, btrfs doesn't. |
|                                  |                                  |            | IO-heavy, on-demand               |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| Btrfs device stats (per-drive    | btrfs device stats --reset       | None       | GAP; RAID1 early-warning ahead of |
| error counters)                  |                                  |            | SMART; reset after review         |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| ZFS pool health / errors         | —                                | None       | CRITICAL if state != ONLINE or    |
|                                  |                                  |            | errors > 0                        |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| ZFS scrub age                    | zpool scrub                      | Confirm    | Covered                           |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| ZFS capacity                     | —                                | None       | Perf degrades > 80%; the ZFS      |
|                                  |                                  |            | headroom metric (no unallocated   |
|                                  |                                  |            | concept)                          |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| ZFS fragmentation (FRAG)         | —                                | None       | GAP; no defrag exists — the       |
|                                  |                                  |            | remedy is snapshot pruning +      |
|                                  |                                  |            | staying under 80%                 |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| ZFS autotrim (SSD TRIM)          | zpool set autotrim=on / zpool    | Confirm    | GAP; velox; the fstrim.timer      |
|                                  | trim                             |            | counterpart on ZFS                |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| fstrim.timer enabled + firing    | systemctl enable --now           | Confirm    | GAP; standard SSD hygiene         |
|                                  | fstrim.timer                     |            |                                   |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| pacman file integrity (Qkk)      | reinstall package                | Workflow   | GAP; modified/missing files need  |
|                                  |                                  |            | judgment                          |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| SMART health                     | —                                | None       | Hardware; replace drive           |
|----------------------------------+----------------------------------+------------+-----------------------------------|
| SMART wear / temperature         | —                                | None       | Diagnostic                        |
|----------------------------------+----------------------------------+------------+-----------------------------------|

* Metrics — Snapshots

| Metric                         | Fix / lever              | Automation | Notes                                       |
|--------------------------------+--------------------------+------------+---------------------------------------------|
| Snapper count / retention      | snapper cleanup / delete | Confirm    | Manual (single) snapshots need explicit     |
|                                |                          |            | choice                                      |
|--------------------------------+--------------------------+------------+---------------------------------------------|
| ZFS snapshot count / retention | zfs destroy              | Confirm    | Runaway retention                           |
|--------------------------------+--------------------------+------------+---------------------------------------------|
| Snapshot auto-timer running    | systemctl enable timer   | Confirm    | Is the auto-snapshot service firing         |
|--------------------------------+--------------------------+------------+---------------------------------------------|

* Metrics — Packages & security

| Metric                            | Fix / lever                  | Automation | Notes                                |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| Orphaned packages                 | pacman -Rns (named args)     | Confirm    | Review first (rust looked orphaned,  |
|                                   |                              |            | was intentional)                     |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| Pending updates count             | —                            | Workflow   | Never auto; workflow-only            |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| .pacnew files                     | diff + merge / delete        | Confirm    | Allowlist auto-deletes mirrorlist /  |
|                                   |                              |            | locale.gen                           |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| arch-audit CVEs                   | reviewed update              | Workflow   | GAP; top-priority add. Fix is an     |
|                                   |                              |            | update                               |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| Keyring freshness                 | pacman -Sy archlinux-keyring | Confirm    | GAP; stale keyring breaks update     |
|                                   |                              |            | signatures                           |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| AUR / foreign staleness (Qm)      | paru -Sua                    | Workflow   | GAP; AUR updates reviewed, not auto  |
|-----------------------------------+------------------------------+------------+--------------------------------------|
| Reboot required (kernel mismatch) | reboot                       | Human      | GAP; uname -r vs /usr/lib/modules.   |
|                                   |                              |            | You pick when                        |
|-----------------------------------+------------------------------+------------+--------------------------------------|

* Metrics — systemd & boot

| Metric                         | Fix / lever            | Automation | Notes                                         |
|--------------------------------+------------------------+------------+-----------------------------------------------|
| is-system-running (degraded?)  | —                      | None       | GAP; one-token whole-system verdict;          |
|                                |                        |            | candidate for the bar glyph                   |
|--------------------------------+------------------------+------------+-----------------------------------------------|
| Failed systemd units           | restart / investigate  | Workflow   | Depends why it failed                         |
|--------------------------------+------------------------+------------+-----------------------------------------------|
| Maintenance timers enabled +   | systemctl enable timer | Confirm    | GAP; meta-metric — are                        |
| firing                         |                        |            | paccache/btrfs-scrub/fstrim/reflector/snapper |
|                                |                        |            | firing                                        |
|--------------------------------+------------------------+------------+-----------------------------------------------|
| Kernel taint flag              | —                      | None       | GAP; tainted != 0 = proprietary module or     |
|                                |                        |            | prior crash                                   |
|--------------------------------+------------------------+------------+-----------------------------------------------|

* Metrics — Logs & coredumps

| Metric                             | Fix / lever                     | Automation | Notes                            |
|------------------------------------+---------------------------------+------------+----------------------------------|
| Coredump count                     | coredumpctl clean (keep recent) | Auto       | Keep last few days for forensics |
|------------------------------------+---------------------------------+------------+----------------------------------|
| Journald disk usage                | journalctl --vacuum-size/time   | Auto       | Bounded journal                  |
|------------------------------------+---------------------------------+------------+----------------------------------|
| App-log cleanup (>7d)              | log-cleanup cron trigger        | Auto       | Already a cron; manual trigger   |
|------------------------------------+---------------------------------+------------+----------------------------------|
| Journal error count (real vs       | —                               | Workflow   | Forensic; noise-filtered count   |
| noise)                             |                                 |            |                                  |
|------------------------------------+---------------------------------+------------+----------------------------------|
| Kernel/hardware events             | —                               | None       | Forensic, hardware               |
| (MCE/USB/thermal/GPU)              |                                 |            |                                  |
|------------------------------------+---------------------------------+------------+----------------------------------|

* Metrics — Memory, thermal, power

| Metric                                  | Fix / lever | Automation | Notes                                      |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| Memory free / OOM kills                 | —           | Workflow   | OOM = investigate                          |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| Swap / zram present + healthy           | —           | None       | Diagnostic                                 |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| CPU / GPU temperatures                  | —           | None       | Hardware                                   |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| Thermal throttling active               | —           | None       | Cooling issue                              |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| Battery health (capacity / cycles)      | —           | None       | GAP; laptop; ties to open suspend todo     |
|-----------------------------------------+-------------+------------+--------------------------------------------|
| Unclean-shutdown / suspend-failure rate | —           | Workflow   | GAP; ratio flagged ~75% unclean 2026-06-08 |
|-----------------------------------------+-------------+------------+--------------------------------------------|

* Metrics — Network & security posture

| Metric                            | Fix / lever               | Automation | Notes                 |
|-----------------------------------+---------------------------+------------+-----------------------|
| DNS / NetworkManager reachability | restart NM                | Workflow   | CRITICAL if down      |
|-----------------------------------+---------------------------+------------+-----------------------|
| Firewall active (ufw / nftables)  | ufw enable                | Confirm    | GAP; security posture |
|-----------------------------------+---------------------------+------------+-----------------------|
| Unexpected listeners (ss -tlnp)   | —                         | None       | GAP; security review  |
|-----------------------------------+---------------------------+------------+-----------------------|
| Tailscale peers                   | tailscale up              | Confirm    | Covered               |
|-----------------------------------+---------------------------+------------+-----------------------|
| fail2ban running + bans           | systemctl start           | Confirm    | Covered               |
|-----------------------------------+---------------------------+------------+-----------------------|
| NTP sync (chrony)                 | systemctl restart chronyd | Confirm    | Covered               |
|-----------------------------------+---------------------------+------------+-----------------------|

* Metrics — Services, backups, virt

| Metric                          | Fix / lever              | Automation | Notes                                      |
|---------------------------------+--------------------------+------------+--------------------------------------------|
| rsyncshot backup freshness      | —                        | Workflow   | CRITICAL if daily > 48h; investigate       |
|                                 |                          |            | failure                                    |
|---------------------------------+--------------------------+------------+--------------------------------------------|
| Docker/podman reclaimable       | prune tier 1 / tiers 2-3 | Confirm    | Tier 1 nearly Auto; 2-3 destructive        |
|---------------------------------+--------------------------+------------+--------------------------------------------|
| Docker stopped containers       | —                        | None       | Mostly expected (WinVM on-demand)          |
|---------------------------------+--------------------------+------------+--------------------------------------------|
| libvirt VM state                | —                        | None       | Expected off                               |
|---------------------------------+--------------------------+------------+--------------------------------------------|
| Cron running + expected entries | systemctl enable cronie  | Confirm    | rsyncshot + log-cleanup entries            |
|---------------------------------+--------------------------+------------+--------------------------------------------|

* Architecture & testing (decided 2026-07-07)

*CLI-first, GUI as a face.* The console ships as the fourth panel sibling: a
=maint= Python package in dotfiles (like =net/=, =bt/=, =audio/=) with probe
modules (read-only collectors), a remedies module, =cli.py=, and =gui.py=
driving the same code. =maint status --json= is the contract; =maint fix
<thing>= is every lever. The GUI never does anything the CLI can't.

Safety mechanics baked into the CLI:
- global =--dry-run= prints the exact command instead of executing — free test
  surface, and the GUI's arm-press can display it ("this will run: …").
- hard read/write split: collectors never elevate; every remedy is an
  allowlisted exact argv in one small auditable module.

*Four test layers (safest → scariest):*

1. /Unit, fake binaries — no VM, no root (~90% of surface)./ Probes are
   parsers over command output: feed canned smartctl/btrfs/journalctl/pacman/
   ss/docker output via fakes on PATH (the net suite's fake-curl and audio's
   fake-parec pattern). Remedies tested as command construction (assert the
   argv, don't run it). The live-update guard is a pure function over a
   package list.
2. /Read-only integration on the live machine./ All collectors are read-only
   by design — =maint status --json= runs safely against real hosts.
3. /Remedies in a VM — archsetup's existing harness./
   =scripts/testing/run-test.sh= boots the installer VM; a maint scenario
   breaks things deliberately over ssh (stop cronie, mask fstrim, orphan
   packages, fill the cache), runs =maint fix …=, asserts post-state. No GUI
   in the VM. Add qcow2 snapshot/restore between remedy tests so each starts
   pristine and destructive remedies can't contaminate each other. Pure
   pacman-level tests may use a throwaway systemd-nspawn container instead
   (lighter); the VM stays for systemd/btrfs/reboot territory.
4. /GUI on the host, never in the VM./ AT-SPI smoke like the sibling panels,
   driven by fixture data. The prototype's GOOD/BAD snapshots become those
   fixtures — =MAINT_PANEL_FIXTURE=bad= renders the degraded board without a
   degraded machine, conforming to the =maint status --json= schema.

* Open questions

- RESOLVED 2026-07-06 — live output wall (amber running / green done / red fail),
  realtime, always shown. See "Doctor = live output wall" above.
- RESOLVED 2026-07-06 — velox is a first-class v1 target alongside ratio. See
  "Scope: hosts" below.
- RESOLVED 2026-07-06 — single machine-readable thresholds file
  (=maintenance-thresholds.toml=) is the source of truth, *owned by archsetup*.
  Both the console and the system-health-check workflow read it, so they can
  never drift. See "Thresholds source of truth" and "Workflow ownership" below.