#+TITLE: Maintenance Console — Design Ideas #+AUTHOR: Craig Jennings #+DATE: 2026-07-06 * Direction A single-host maintenance console (GUI, waybar-launched) that surfaces every health metric for this Arch workstation and, where the remediation is safe, runs it. It reshapes the earlier install-health/workstation-contract framing (=system-monitor-design-ideas.org=) into a maintenance surface with a doctor action. The console is the glanceable, single-host version of the home project's =system-health-check.org= workflow (~1000 lines, capability-dispatched across ratio/velox/mybitch/truenas). The console owns the routine-maintenance and at-a-glance-status slice; the workflow stays as the escalation path for forensic investigation and as the only sanctioned way to run updates. * Scope: hosts (Q2 resolved 2026-07-06) v1 targets *both* daily drivers — ratio and velox — as first-class hosts, not ratio-first-velox-later. Consequences for v1: - *Capability dispatch is mandatory from day one*, not deferred. The console probes the live host and runs only applicable checks (btrfs RAID1 on ratio vs ZFS-primary on velox; AMD amd-pstate on ratio vs Intel intel_pstate on velox). This mirrors the home workflow's Phase 0 probe. - *velox-only metrics ship in v1*: battery health (capacity / cycles) and the unclean-shutdown / suspend-failure rate — both currently GAPs, both driven by velox being the machine that travels and suspends. - The ZFS-primary remediation paths (scrub, snapshot retention, pool health) are built and tested, not stubbed. * Thresholds source of truth (Q3 resolved 2026-07-06) One machine-readable file, =maintenance-thresholds.toml=, holds every severity value (cache-size trigger, disk %, scrub-age warn/crit, snapshot retention limits, temp bands, backup-staleness windows, etc.). archsetup owns and ships it; both consumers read the *installed* path, so neither reaches into the other's repo: - the maintenance console (dotfiles code) reads it at runtime, - the system-health-check workflow reads it instead of hardcoding severity rules in prose. The workflow's hard-won values migrate into the TOML as the seed content (the snapshot MONTHLY limit that bit /home, the 10 GB cache trigger, the scrub-age bands). Install path is an implementation sub-question — a stable location both consumers reach (e.g. =~/.config/archsetup/maintenance-thresholds.toml=). * Workflow ownership — move system-health-check into archsetup (proposed 2026-07-06) Craig's call: the =system-health-check.org= workflow should live in archsetup, not the home project. Rationale — the home project is scoped to finances, health, and personal matters; system design, execution, and maintenance are archsetup's domain. Home only owns the workflow by inherited accident, not by fit. Moving it here also collapses the Q3 coupling: the TOML source-of-truth and its workflow consumer end up in the same project. Move scope (home → archsetup), to sequence when Craig gives the go: - =system-health-check.org= (the workflow itself) - =homelab-inventory/*.org= (ratio/velox/mybitch/truenas capability inventories it cross-references) - any home-project references to the workflow (startup, project-workflows index) Cross-project mechanics: the archsetup side (receiving the files, wiring the TOML) is in-scope here; the home side (removing the originals, updating home's references) is the home project's scope — handled by a handoff note to home's inbox or a home session, not edited blind from here. * Panel shape Three regions, driven by the Automation column of the metrics table below: - *Actionable* (left) — metrics with a lever. Automation = Auto or Confirm. Two doctor actions serve this column: "Clean up" fires every Auto metric unattended; "Review & fix" opens a preview for the Confirm metrics and acts only on approval. - *Diagnostic* (right) — read-only telemetry. Automation = None or Human. A state color (green/amber/red) and the value; no button. Red here is the signal to run the workflow. - *Updates* (quarantined strip) — Automation = Workflow. Shown as a count with notable packages named; the only affordance is "run the workflow." The panel never applies updates in place. *Bar glyph* tracks the worst *Diagnostic* state only — not the actionable count. A big package cache is boring; a SMART failure is a fire. Actionable clutter must not turn the bar red or it trains you to ignore it. ** Doctor = live output wall (Q1 resolved 2026-07-06) "Clean up" and "Review & fix" are not fire-and-forget buttons. Running a doctor opens an *output wall* — one lamp per action, streaming in realtime as each runs: - *amber* while the action is running, - *green* on success (with the reclaimed amount / result inline), - *red* on failure. Feedback is always shown (not just when something was reclaimed) — you watch it happen. This is the same live-results shape that should back the other doctors (net, bluetooth), so every doctor in the system reads the same way. See the todo task to retrofit the net + bt doctors to realtime lamp output. * Automation legend | Value | Meaning | |----------+-----------------------------------------------------------------------------------------------------------| | Auto | Doctor "Clean up" button — fire unattended, reversible or harmless | |----------+-----------------------------------------------------------------------------------------------------------| | Confirm | Doctor "Review & fix" — preview the change, act only on one click | |----------+-----------------------------------------------------------------------------------------------------------| | Human | Panel nudges; a human decides and acts (no button, not the agent) | |----------+-----------------------------------------------------------------------------------------------------------| | Workflow | vLater — metric shown read-only; no in-panel button. Agent-workflow assistance deferred (decided | | | 2026-07-07, see below) | |----------+-----------------------------------------------------------------------------------------------------------| | None | Diagnostic only — no software remediation exists (hardware, telemetry) | |----------+-----------------------------------------------------------------------------------------------------------| *Workflow buttons removed (decided 2026-07-07, prototyping E5).* The panel carries no "run workflow" affordances — not on failed units, unclean boots, updates, or CVEs. The metrics stay on the board as read-only telemetry; AI assistance via the system-health-check workflow is a vLater feature. v1 ships only determinate remedies (fixed, scriptable actions with predictable outcomes). *Determinate remedies adopted (decided 2026-07-07).* Every remedy below is Confirm-tier; contextual levers appear only when the metric is off-nominal: - /Service restarts & enables/: failed units (systemctl restart + reset-failed), fail2ban, cronie, chronyd (+ makestep), tailscaled, snapper timers, zram config re-apply. DNS/NetworkManager delegates to the net panel's doctor (deep-link, not a duplicate repair chain). - /Deterministic maintenance ops/: btrfs balance -dusage=50 (unallocated low), reinstall owning packages on pacman -Qkk failures, rsyncshot RUN NOW (result streams to the output well), smartctl -t short self-test, snapshot-retention repair (write sane TIMELINE limits + cleanup — the /home lesson as a one-press fix). btrfs device-error counter reset stays manual: resetting without diagnosis masks a dying drive. - /Composite macro/: RECLAIM SPACE on the disk-usage cell — runs every reclaim lever (cache, journal, coredumps, app logs, docker tier-1, snapper cleanup) as one output-wall stream. - /Disruptive but determinate/: REBOOT behind arm-to-fire, offered when running kernel != installed. - /Still read-only/: temps, throttling, battery, memory/OOM, taint, journal error content, kernel/hw events, listeners, unclean-shutdown rate — physical or investigative; vLater AI territory. *Updates join the Confirm layer (decided 2026-07-07).* Two levers on the updates strip, both behind the live-update guard (mesa/hyprland/wayland runtime in the pending set): - UPDATE — repo + AUR system update. - TOPGRADE — full ecosystem run. The panel's wrapper always passes --disable git (topgrade's git step rebase-autostashes ~/code/*/ — never under a live session). *MEM·PWR: evidence + two levers + expectation tags (decided 2026-07-07).* Mostly physics, so the category leans watch-only — with these additions: - /CPU mode selector/: a free segmented control (PERF · BAL · POWER) writing the EPP hint — set the active mode to anything, not drift-repair against a declared default. amd-pstate on ratio, intel_pstate on velox. - /Battery charge limit/ (velox-only, capability-gated): SET 80% writes charge_control_end_threshold — the standard longevity cap. Battery *health* (capacity vs design, cycles) stays watch-only hardware telemetry. - /Evidence drill-downs/ (digest idiom): top-5 RAM consumers under memory, recent boots listed clean/unclean under the unclean-rate, and throttle/OOM events with timestamps. Evidence makes the numbers actionable even where no button exists. - /KILL on top-memory items/ (revised 2026-07-07 — Craig): arm-to-fire, four guards: the arm shows the exact victim (name + size); SIGTERM not SIGKILL, with the outcome reported to the wall; PID + process name revalidated at fire time so a recycled PID can't be hit; session-critical names (systemd, the compositor, the panel itself) render a disabled key — protected. A SIGKILL escalation for TERM-survivors is vLater. - /Expectation-setting, panel-wide/: every leverless cell's sub-line carries an explicit tag ("hardware — watch only", "evidence below"), and each subpanel header shows the split — "N fixable · M watch" — so the user knows their agency before reading a single cell. *Refresh cadence (decided 2026-07-07).* Four tiers, matching probe cost: - /Live group, panel open/: temps, memory free + top consumers, throttle state — re-read every ~3 s while their subpanel is visible, gated on panel-open exactly like the audio panel's meters. Stop when hidden. - /Fast local tier, panel open/: re-probed every ~30 s while the panel is up; additionally, any metric re-probes immediately after an action that touches it (fire CLEAN → cache re-measured, not assumed). - /On open/: the hydration tiers re-run (fast reads first, process probes behind them — sub-second perceived). - /Network tier/: checkupdates / arch-audit / AUR / firmware stay on the hourly systemd-timer cache with age shown; refreshed on demand only. - /Panel closed/: the waybar glyph is fed by a light background scan every ~30 min (systemd timer writing the state file the glyph reads) — the bar stays honest without the panel running. *Journal errors get a digest, not a fix (decided 2026-07-07).* No generic remedy exists — an error-priority line is a symptom of an arbitrary subsystem — so the panel ships four determinate assists instead: - /Digest/: the cell expands to errors grouped by syslog identifier — count, message snippet, first/last seen, and the exact next command (journalctl -u -b) when the identifier maps to a unit. Top-10 groups, read-only. - /MARK KNOWN with a full lifecycle/: arm-to-fire showing the exact pattern before it stores; marked groups move to a dim KNOWN section (never vanish) with per-row UNMARK; every mark/unmark logs to the results wall; marks carry date + example. Patterns bind to identifier + message, never a whole unit — a muted service's *new* errors still surface. - /Two noise layers/: shipped defaults (bluetoothd HFP, pixman, xkbcomp) in the packaged TOML, user marks in a separate user file merged over it (a template sync never eats curation). CLEAR MARKS (arm-to-fire) empties the user layer and re-enables shipped defaults; unmarking a shipped default records a disable flag in the user layer. - /OPEN JOURNAL/: launches a terminal running journalctl -p err -b — the same delegation pattern as NET DOCTOR. Ruled out: auto-restarting units that log errors (error ≠ failed) and keyword-driven fix suggestions (vLater AI territory). *Full-sweep findings — all committed to v1 (decided 2026-07-07).* Every metric was audited against the converged checklist (honest label, evidence digest where a count hides detail, curation lifecycle where "expected" is config knowledge, guarded per-item remedies, cross-links, watch-only tags). Adopted, all prototyped: - /Storage/: disk top-consumers digest (evidence only — no file deletion keys); per-device error rows on RAID1 when counters are nonzero, cross-checked against SMART; SMART sub-line carries the last self-test result. Spec note: a real scrub runs hours — the ring needs a running-% state, not an instant reset. - /Packages/: orphan digest (name + size) with per-package REMOVE (armed) and KEEP — the curation lifecycle encoding "intentional, not orphaned" (the rust lesson); batch REMOVE ALL skips kept packages. Per-file pacnew rows tagged safe-delete (reflector-managed) vs needs-merge, MERGE delegating to a terminal diff. CVEs named: package · CVE id · severity. AUR and firmware names spelled out. - /systemd/: failed units upgraded from a count-lever to a per-unit roster (name · since · exit code · journalctl hint) with per-row RESTART + RESET; is-system-running names its cause ("degraded — N failed units below"); taint letters decoded. - /Logs/: coredumps grouped by binary (count · last · coredumpctl hint), cleared with the CLEAR action; kernel/hw events listed when not clean (hardware — watch only). - /Services/: docker system df breakdown (images / containers / volumes / build cache with per-type reclaimable); stopped containers upgraded to the full signal/expected curation lifecycle (MARK EXPECTED / UNMARK, shipped default: winvm) with per-container START; cron expected-entries drift roster. - /Snapshots/: count split by type — timeline (auto-pruned) · single (manual — escapes timeline cleanup, the pile-up risk) · pre/post — with oldest-single named and DELETE STALE (armed, keeps newest 2) when singles accumulate. - /Network minors/: fail2ban shows recent-ban count; NTP shows offset. Rationale for prototyping everything (Craig): real estate and complexity have bitten before — surface those limits in the disposable prototype, not after functionality exists behind the UI. *Vertical compression → rotary band selector (2026-07-07).* First attempt — the MEM·PWR three-column layout on Packages/Logs/Services — lost too much row detail to third-width truncation (Craig's verdict after use). Replaced by a *rotary band selector*: the amplifier input-selector idiom. A machined knob (click to cycle) whose needle swings to engraved band labels, one per evidence section (ORPHANS · PACNEW · ADVISORIES; SIGNAL · KNOWN NOISE · COREDUMPS · KERNEL/HW; CONTAINERS · DOCKER DISK · CRON & BACKUPS). Each band carries its own status lamp (section health at a glance without switching) plus a count; the selected band gets a gold underline and the needle. One section renders at a time at full width, restoring complete row detail. Deliberately distinct from the category tiles, console keys, and the CPU-mode segmented control — each selection idiom in the panel now has its own visual voice. MEM·PWR keeps its three-column evidence strip (short rows fit fine at third-width). *Listeners get the same treatment (decided 2026-07-07).* The count becomes "unexpected listeners" — evidence digest (process · port · bind address from ss -tlnp), expected-list curation with the full MARK EXPECTED / UNMARK / CLEAR lifecycle (shipped defaults: sshd, mpd, tailscaled; user marks in the user layer), and guarded per-socket remedies: STOP (systemctl stop, armed) when a unit owns the socket, KILL (SIGTERM, armed) otherwise. Severity keys on unexpected AND public-bind (0.0.0.0/::) — a loopback listener warns, an exposed one fails — and when ufw is down the signal header names the exposure ("ufw down — N public binds exposed"). Stopped containers likewise gained a contextual START lever (allowlist: winvm), and the firewall its ENABLE. *Updates strip border is state-tiered (decided 2026-07-07).* Green when nothing pending, amber for ordinary pending/AUR/firmware counts, red when CVEs exist or pending exceeds the "a lot" threshold (or the update cache has gone stale — staleness window in the TOML). The CVE badge renders only when the count is nonzero. *Guard arms instead of blocking (revised 2026-07-07).* When the guard trips, the key arms (red, "press again to run anyway — or apply from a TTY") rather than hard-refusing. The user decides; the footgun is acknowledged and deliberately handed over. After a system update lands, the panel offers a reboot: a REBOOT key (arm-to-fire) appears on the updates strip and the reboot-required metric flips. *No per-ecosystem update metrics.* Topgrade's step set (yay, rustup, cargo, pipx, npm/pnpm, gem, go, flatpak, fwupd, tmux/zsh/nvim plugins, git repos) has no cheap offline "updates available?" probe — mirroring it means a network round-trip per registry at panel-open. Instead: one *topgrade freshness* metric (wrapper stamps last-run time; threshold in the TOML) whose remedy is the TOPGRADE lever, plus a *firmware updates* count in the updates strip (fwupd refreshes metadata on its own timer; the panel reads the cache). Rationale for the hard lines: system updates are Workflow, never Auto — the 2026-06-07 Hyprland crash was a live -Syu swapping mesa+hyprland under the running session, and the standing rule is never -Syu live under Hyprland when the mesa/hyprland/wayland runtime is in the set. Hardware findings (SMART, MCE, thermal) are None — the fix is replacing a drive or clearing a fan, not software. * Metrics — Storage & filesystem integrity | Metric | Fix / lever | Automation | Notes | |----------------------------------+----------------------------------+------------+-----------------------------------| | Package cache size | paccache -r / -ruk0 | Auto | Reclaim, all re-downloadable | |----------------------------------+----------------------------------+------------+-----------------------------------| | paccache -rk1 (keep 1 version) | paccache -rk1 | Confirm | Frees most; kills downgrade | | | | | headroom | |----------------------------------+----------------------------------+------------+-----------------------------------| | Disk usage (df) | — | None | Fix is cache/snapshot/prune | | | | | levers | |----------------------------------+----------------------------------+------------+-----------------------------------| | Btrfs unallocated space | — | None | Chunk headroom; diagnostic | |----------------------------------+----------------------------------+------------+-----------------------------------| | Btrfs scrub age | btrfs scrub start | Confirm | GAP; ZFS has this, btrfs doesn't. | | | | | IO-heavy, on-demand | |----------------------------------+----------------------------------+------------+-----------------------------------| | Btrfs device stats (per-drive | btrfs device stats --reset | None | GAP; RAID1 early-warning ahead of | | error counters) | | | SMART; reset after review | |----------------------------------+----------------------------------+------------+-----------------------------------| | ZFS pool health / errors | — | None | CRITICAL if state != ONLINE or | | | | | errors > 0 | |----------------------------------+----------------------------------+------------+-----------------------------------| | ZFS scrub age | zpool scrub | Confirm | Covered | |----------------------------------+----------------------------------+------------+-----------------------------------| | ZFS capacity | — | None | Perf degrades > 80%; the ZFS | | | | | headroom metric (no unallocated | | | | | concept) | |----------------------------------+----------------------------------+------------+-----------------------------------| | ZFS fragmentation (FRAG) | — | None | GAP; no defrag exists — the | | | | | remedy is snapshot pruning + | | | | | staying under 80% | |----------------------------------+----------------------------------+------------+-----------------------------------| | ZFS autotrim (SSD TRIM) | zpool set autotrim=on / zpool | Confirm | GAP; velox; the fstrim.timer | | | trim | | counterpart on ZFS | |----------------------------------+----------------------------------+------------+-----------------------------------| | fstrim.timer enabled + firing | systemctl enable --now | Confirm | GAP; standard SSD hygiene | | | fstrim.timer | | | |----------------------------------+----------------------------------+------------+-----------------------------------| | pacman file integrity (Qkk) | reinstall package | Workflow | GAP; modified/missing files need | | | | | judgment | |----------------------------------+----------------------------------+------------+-----------------------------------| | SMART health | — | None | Hardware; replace drive | |----------------------------------+----------------------------------+------------+-----------------------------------| | SMART wear / temperature | — | None | Diagnostic | |----------------------------------+----------------------------------+------------+-----------------------------------| * Metrics — Snapshots | Metric | Fix / lever | Automation | Notes | |--------------------------------+--------------------------+------------+---------------------------------------------| | Snapper count / retention | snapper cleanup / delete | Confirm | Manual (single) snapshots need explicit | | | | | choice | |--------------------------------+--------------------------+------------+---------------------------------------------| | ZFS snapshot count / retention | zfs destroy | Confirm | Runaway retention | |--------------------------------+--------------------------+------------+---------------------------------------------| | Snapshot auto-timer running | systemctl enable timer | Confirm | Is the auto-snapshot service firing | |--------------------------------+--------------------------+------------+---------------------------------------------| * Metrics — Packages & security | Metric | Fix / lever | Automation | Notes | |-----------------------------------+------------------------------+------------+--------------------------------------| | Orphaned packages | pacman -Rns (named args) | Confirm | Review first (rust looked orphaned, | | | | | was intentional) | |-----------------------------------+------------------------------+------------+--------------------------------------| | Pending updates count | — | Workflow | Never auto; workflow-only | |-----------------------------------+------------------------------+------------+--------------------------------------| | .pacnew files | diff + merge / delete | Confirm | Allowlist auto-deletes mirrorlist / | | | | | locale.gen | |-----------------------------------+------------------------------+------------+--------------------------------------| | arch-audit CVEs | reviewed update | Workflow | GAP; top-priority add. Fix is an | | | | | update | |-----------------------------------+------------------------------+------------+--------------------------------------| | Keyring freshness | pacman -Sy archlinux-keyring | Confirm | GAP; stale keyring breaks update | | | | | signatures | |-----------------------------------+------------------------------+------------+--------------------------------------| | AUR / foreign staleness (Qm) | paru -Sua | Workflow | GAP; AUR updates reviewed, not auto | |-----------------------------------+------------------------------+------------+--------------------------------------| | Reboot required (kernel mismatch) | reboot | Human | GAP; uname -r vs /usr/lib/modules. | | | | | You pick when | |-----------------------------------+------------------------------+------------+--------------------------------------| * Metrics — systemd & boot | Metric | Fix / lever | Automation | Notes | |--------------------------------+------------------------+------------+-----------------------------------------------| | is-system-running (degraded?) | — | None | GAP; one-token whole-system verdict; | | | | | candidate for the bar glyph | |--------------------------------+------------------------+------------+-----------------------------------------------| | Failed systemd units | restart / investigate | Workflow | Depends why it failed | |--------------------------------+------------------------+------------+-----------------------------------------------| | Maintenance timers enabled + | systemctl enable timer | Confirm | GAP; meta-metric — are | | firing | | | paccache/btrfs-scrub/fstrim/reflector/snapper | | | | | firing | |--------------------------------+------------------------+------------+-----------------------------------------------| | Kernel taint flag | — | None | GAP; tainted != 0 = proprietary module or | | | | | prior crash | |--------------------------------+------------------------+------------+-----------------------------------------------| * Metrics — Logs & coredumps | Metric | Fix / lever | Automation | Notes | |------------------------------------+---------------------------------+------------+----------------------------------| | Coredump count | coredumpctl clean (keep recent) | Auto | Keep last few days for forensics | |------------------------------------+---------------------------------+------------+----------------------------------| | Journald disk usage | journalctl --vacuum-size/time | Auto | Bounded journal | |------------------------------------+---------------------------------+------------+----------------------------------| | App-log cleanup (>7d) | log-cleanup cron trigger | Auto | Already a cron; manual trigger | |------------------------------------+---------------------------------+------------+----------------------------------| | Journal error count (real vs | — | Workflow | Forensic; noise-filtered count | | noise) | | | | |------------------------------------+---------------------------------+------------+----------------------------------| | Kernel/hardware events | — | None | Forensic, hardware | | (MCE/USB/thermal/GPU) | | | | |------------------------------------+---------------------------------+------------+----------------------------------| * Metrics — Memory, thermal, power | Metric | Fix / lever | Automation | Notes | |-----------------------------------------+-------------+------------+--------------------------------------------| | Memory free / OOM kills | — | Workflow | OOM = investigate | |-----------------------------------------+-------------+------------+--------------------------------------------| | Swap / zram present + healthy | — | None | Diagnostic | |-----------------------------------------+-------------+------------+--------------------------------------------| | CPU / GPU temperatures | — | None | Hardware | |-----------------------------------------+-------------+------------+--------------------------------------------| | Thermal throttling active | — | None | Cooling issue | |-----------------------------------------+-------------+------------+--------------------------------------------| | Battery health (capacity / cycles) | — | None | GAP; laptop; ties to open suspend todo | |-----------------------------------------+-------------+------------+--------------------------------------------| | Unclean-shutdown / suspend-failure rate | — | Workflow | GAP; ratio flagged ~75% unclean 2026-06-08 | |-----------------------------------------+-------------+------------+--------------------------------------------| * Metrics — Network & security posture | Metric | Fix / lever | Automation | Notes | |-----------------------------------+---------------------------+------------+-----------------------| | DNS / NetworkManager reachability | restart NM | Workflow | CRITICAL if down | |-----------------------------------+---------------------------+------------+-----------------------| | Firewall active (ufw / nftables) | ufw enable | Confirm | GAP; security posture | |-----------------------------------+---------------------------+------------+-----------------------| | Unexpected listeners (ss -tlnp) | — | None | GAP; security review | |-----------------------------------+---------------------------+------------+-----------------------| | Tailscale peers | tailscale up | Confirm | Covered | |-----------------------------------+---------------------------+------------+-----------------------| | fail2ban running + bans | systemctl start | Confirm | Covered | |-----------------------------------+---------------------------+------------+-----------------------| | NTP sync (chrony) | systemctl restart chronyd | Confirm | Covered | |-----------------------------------+---------------------------+------------+-----------------------| * Metrics — Services, backups, virt | Metric | Fix / lever | Automation | Notes | |---------------------------------+--------------------------+------------+--------------------------------------------| | rsyncshot backup freshness | — | Workflow | CRITICAL if daily > 48h; investigate | | | | | failure | |---------------------------------+--------------------------+------------+--------------------------------------------| | Docker/podman reclaimable | prune tier 1 / tiers 2-3 | Confirm | Tier 1 nearly Auto; 2-3 destructive | |---------------------------------+--------------------------+------------+--------------------------------------------| | Docker stopped containers | — | None | Mostly expected (WinVM on-demand) | |---------------------------------+--------------------------+------------+--------------------------------------------| | libvirt VM state | — | None | Expected off | |---------------------------------+--------------------------+------------+--------------------------------------------| | Cron running + expected entries | systemctl enable cronie | Confirm | rsyncshot + log-cleanup entries | |---------------------------------+--------------------------+------------+--------------------------------------------| * Architecture & testing (decided 2026-07-07) *CLI-first, GUI as a face.* The console ships as the fourth panel sibling: a =maint= Python package in dotfiles (like =net/=, =bt/=, =audio/=) with probe modules (read-only collectors), a remedies module, =cli.py=, and =gui.py= driving the same code. =maint status --json= is the contract; =maint fix = is every lever. The GUI never does anything the CLI can't. Safety mechanics baked into the CLI: - global =--dry-run= prints the exact command instead of executing — free test surface, and the GUI's arm-press can display it ("this will run: …"). - hard read/write split: collectors never elevate; every remedy is an allowlisted exact argv in one small auditable module. *Four test layers (safest → scariest):* 1. /Unit, fake binaries — no VM, no root (~90% of surface)./ Probes are parsers over command output: feed canned smartctl/btrfs/journalctl/pacman/ ss/docker output via fakes on PATH (the net suite's fake-curl and audio's fake-parec pattern). Remedies tested as command construction (assert the argv, don't run it). The live-update guard is a pure function over a package list. 2. /Read-only integration on the live machine./ All collectors are read-only by design — =maint status --json= runs safely against real hosts. 3. /Remedies in a VM — archsetup's existing harness./ =scripts/testing/run-test.sh= boots the installer VM; a maint scenario breaks things deliberately over ssh (stop cronie, mask fstrim, orphan packages, fill the cache), runs =maint fix …=, asserts post-state. No GUI in the VM. Add qcow2 snapshot/restore between remedy tests so each starts pristine and destructive remedies can't contaminate each other. Pure pacman-level tests may use a throwaway systemd-nspawn container instead (lighter); the VM stays for systemd/btrfs/reboot territory. 4. /GUI on the host, never in the VM./ AT-SPI smoke like the sibling panels, driven by fixture data. The prototype's GOOD/BAD snapshots become those fixtures — =MAINT_PANEL_FIXTURE=bad= renders the degraded board without a degraded machine, conforming to the =maint status --json= schema. * Open questions - RESOLVED 2026-07-06 — live output wall (amber running / green done / red fail), realtime, always shown. See "Doctor = live output wall" above. - RESOLVED 2026-07-06 — velox is a first-class v1 target alongside ratio. See "Scope: hosts" below. - RESOLVED 2026-07-06 — single machine-readable thresholds file (=maintenance-thresholds.toml=) is the source of truth, *owned by archsetup*. Both the console and the system-health-check workflow read it, so they can never drift. See "Thresholds source of truth" and "Workflow ownership" below.