path: root/docs/design/system-monitor-design-ideas.org

#+TITLE: System Monitor Design Ideas
#+DATE: 2026-07-04
#+TODO: TODO | DONE
#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED

* DRAFT Status
:PROPERTIES:
:ID:       system-monitor-design-ideas
:END:
- [2026-07-04 Sat] DRAFT — initial design sketch for a health monitor covering
  Archangel ISO/base-install health, Archsetup workstation health, and the local
  laptop as the daily canary.

* Metadata

| Field  | Value                                      |
|--------+--------------------------------------------|
| Status | draft                                      |
|--------+--------------------------------------------|
| Owner  | Craig Jennings                             |
|--------+--------------------------------------------|
| Repos  | archsetup, archangel, dotfiles             |
|--------+--------------------------------------------|
| Kin    | net panel, bluetooth panel, audio panel    |
|--------+--------------------------------------------|

* Problem

Archangel and Archsetup can fail in ways that are individually obvious only
after the damage is done: an ISO build goes stale against Arch or archzfs,
ZFSBootMenu or GRUB boots once but not after the first upgrade, a snapshot
hook silently disappears, a package database ages out, systemd services fail
after a reboot, or the desktop contract is technically installed but not
usable.

The health surface should compress those risks into one operational question:
"Can I trust a fresh install, and is this current workstation drifting away
from the known-good install contract?"

This monitor is not a generic CPU/RAM graph. It is an install-health and
workstation-contract console. CPU, memory, and temperature belong only as
secondary context unless they block install/test operations.

* Priority Model

Rank metrics by the cost of blindness: what happens if Craig never sees the
metric, no one mitigates it, and the next install/upgrade/reboot simply happens.

Severity:

- =P0= — can cause data loss, unbootable systems, or loss of rollback path.
- =P1= — can break fresh installs, upgrades, remote access, or core
  workstation use.
- =P2= — causes degraded workstation behavior, security drift, or accumulating
  maintenance debt.
- =P3= — useful context, not a release gate by itself.

The panel should sort by live severity first, then by this priority. A red =P2=
row appears above a green =P0= row, but in the steady state the layout keeps the
P0/P1 rows in the first viewport.

* Priority Ranking

| Rank | Priority | Metric                          | Why this rank exists                                                         |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    1 | P0       | Storage health                  | Silent pool/filesystem degradation is the nearest thing to data loss.        |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    2 | P0       | Snapshot safety coverage        | Without snapshots, upgrades lose their rollback safety net.                  |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    3 | P0       | Bootloader and EFI redundancy   | A machine that cannot boot is operationally dead.                            |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    4 | P1       | First-upgrade bootability       | Catches the classic "installed fine, broke after update" failure.            |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    5 | P1       | End-to-end VM install pass rate | Best release gate for the whole Archangel + Archsetup chain.                 |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    6 | P1       | Package sync and repo freshness | Arch, archzfs, keyring, and mirror drift are leading break signals.          |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    7 | P1       | Archangel ISO reproducibility   | If current inputs cannot build an ISO, recovery/install confidence is stale. |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    8 | P1       | Post-install service health     | Network, DNS, SSH, and user services decide whether the system is usable.    |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|    9 | P1       | Archsetup state/log cleanliness | Prevents "half-installed but looks fine" machines.                           |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|   10 | P2       | Workstation contract checks     | Confirms this is Craig's workstation, not just generic Arch.                 |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|   11 | P2       | Backup and rollback readiness   | Catches loss of off-machine recovery and edited-file backups.                |
|------+----------+---------------------------------+------------------------------------------------------------------------------|
|   12 | P2       | Security and hardening drift    | Important, but usually less immediately destructive than boot/storage.       |
|------+----------+---------------------------------+------------------------------------------------------------------------------|

This order intentionally puts ZFS/Btrfs health above VM install evidence. A
broken future install is expensive; silent damage to the current root or backup
chain is worse.

* Consequence Matrix

This is the design justification for every metric. A row earns panel space only
if blindness has a clear failure mode and Doctor has at least a useful
diagnostic or mitigation.

| Rank | Metric                          | If never seen / never mitigated                          | Typical failure                                                | Worst plausible failure                                           | Doctor posture                                  |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    1 | Storage health                  | Disk, pool, or metadata degradation accumulates silently. | Correctable ZFS/Btrfs errors, low EFI/root space, stale scrub. | Data loss, degraded root, failed import/mount during boot.         | Diagnose + scrub/cleanup with confirmation.    |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    2 | Snapshot safety coverage        | Upgrades proceed without a rollback point.                | Missing recent snapshot or missing pre-pacman hook.            | Bad upgrade cannot be rolled back cleanly; manual repair required. | Create snapshot; restore hook; never rollback. |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    3 | Bootloader and EFI redundancy   | Boot path rots until the next reboot or disk failure.     | Missing GRUB/ZBM file on one EFI partition.                    | Unbootable machine after update, firmware reset, or disk loss.     | Diagnose; regenerate config; advanced reinstall only. |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    4 | First-upgrade bootability       | Installs look good until the first real system update.    | Kernel/initramfs/bootloader mismatch in VM.                    | Fresh bare-metal install dies on first reboot after upgrade.       | VM-only upgrade test; collect boot evidence.   |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    5 | End-to-end VM install pass rate | Unit tests give false confidence about the real workflow. | Current branch fails one filesystem path or desktop assertion. | Bare-metal install fails mid-flight after disks are wiped.         | Run/schedule VM test; clean stale VM artifacts. |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    6 | Package sync/repo freshness     | Arch/keyring/archzfs drift surprises the next install.    | 404s, stale keyring, bad mirror, stale archzfs DB.             | Installer cannot pacstrap or installs mismatched ZFS/kernel bits.  | Refresh DB; update keyring; reflector.         |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    7 | ISO build reproducibility       | Recovery/install media confidence becomes historical.     | AUR package fails to build; mkarchiso or DKMS breaks.          | Need rescue/install media and discover no current ISO can be built. | Parse logs; clean work; explicit rebuild.      |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    8 | Post-install service health     | System is installed but degraded in daily operation.      | DNS, NetworkManager, fail2ban, tailscale, or user service down. | No remote access, no network, broken sync, or security tooling off. | Restart/re-enable classified services only.    |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|    9 | Archsetup state/log cleanliness | Half-completed provisioning masquerades as success.       | Missing marker, log error, skipped step after resume.          | Fresh workstation lacks critical config but looks mostly usable.   | Summarize; rerun resumable archsetup.          |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|   10 | Workstation contract checks     | The box drifts from "Craig's workstation" to generic Arch. | Dotfile symlink broken, keyring wrong, missing tool/package.   | Desktop/session workflow is broken during real work.              | Restow, repair perms, reinstall with confirm.  |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|   11 | Backup and rollback readiness   | Recovery assumptions go stale.                            | Missing =.archsetup.bak=, backup timer stale, dry-run fails.   | Local rollback works but important personal/system state is gone.  | Dry-run, start configured job, no deletes.     |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|
|   12 | Security and hardening drift    | Small protective edits quietly regress.                   | SSH/firewall/fail2ban/sysctl/EFI mask drift.                   | Exposed service or weakened local boot/config protections.         | Restore owned snippets; no broad rewrite.      |
|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------|

The worst cases are intentionally conservative, not dramatic. The monitor is
useful because it catches the boring early signal: stale, missing, not recently
tested, or silently degraded.

* Product Shape

Use the existing instrument-console panel language rather than a dashboard page:
lamps for pass/warn/fail state, engraved metric groups, dense rows, physical
console keys for explicit actions, and an output well for the last diagnosis.

The bar module should be tiny:

- icon: a pulse/terminal glyph or compact =SYS= label.
- state lamp: green/yellow/red/grey.
- text: one word only: =OK=, =WARN=, =FAIL=, =STALE=, =RUNNING=, =UNKNOWN=.
- click: opens the system monitor panel.
- middle click or secondary action: run a cheap refresh only, never repairs.

The panel should be one screen with internal scroll only where needed. The
first viewport should show the decision state and the Doctor controls without
scrolling.

* Layout

** Faceplate

Top row:

- State lamp + state word: =OK= / =WARN= / =FAIL= / =STALE= / =RUNNING= /
  =UNKNOWN=.
- Unit label: =SYS·01=.
- Scope segmented control: =HOST= / =INSTALL= / =BUILD=.
- Badges: =ZFS= or =BTRFS=, =VM STALE=, =DB STALE=, =SNAPSHOT=, =BACKUP=,
  =ROOT= when elevated actions are available.
- Close button.

The scope control changes the metric emphasis, not the underlying data model:

- =HOST= is this laptop/workstation right now.
- =INSTALL= is the last Archangel+Archsetup VM install result.
- =BUILD= is ISO, AUR repo, archzfs, and test artifact health.

** Health Stack

Arrange metrics as four horizontal bands. Each band has a section title, a
summary lamp, two to four row lamps, and a short "age" or "count" value.

1. =BOOT + STORAGE=
   - bootloader
   - EFI redundancy
   - pool/filesystem
   - snapshots

2. =INSTALL PIPELINE=
   - first-upgrade reboot
   - ZFS VM install
   - Btrfs VM install
   - ISO build

3. =PACKAGES + SERVICES=
   - pacman sync freshness
   - archzfs/AUR health
   - failed services
   - journal errors

4. =WORKSTATION CONTRACT=
   - Archsetup markers/logs
   - user/dotfiles
   - desktop/session
   - backups/replication

Each row is clickable. Clicking a row opens the evidence drawer in the output
well with:

- last command run
- normalized verdict
- raw excerpt, redacted where needed
- suggested Doctor action, if any

** Console Keys

Use physical console-key buttons, same family as net/bt:

| Key          | Purpose                                                    |
|--------------+------------------------------------------------------------|
| REFRESH      | Cheap read-only probe of host state                        |
|--------------+------------------------------------------------------------|
| DOCTOR       | Diagnose, classify, run safe mitigations, re-check         |
|--------------+------------------------------------------------------------|
| TEST VM      | Run or schedule Archsetup VM validation                    |
|--------------+------------------------------------------------------------|
| BUILD ISO    | Run or schedule Archangel ISO build                        |
|--------------+------------------------------------------------------------|
| CLEAN        | Clean old test artifacts, package cache, stale logs        |
|--------------+------------------------------------------------------------|
| SNAPSHOT     | Create a manual pre-change snapshot                        |
|--------------+------------------------------------------------------------|

Keys that can take a long time stream progress into the output well. Mutating
keys must use arm-first behavior:

- first click arms for 3 seconds and explains the action.
- second click runs.
- destructive cleanup names what will be deleted before it runs.

* Metric Details

Details below are grouped by workflow rather than priority. The authoritative
importance order is the priority table above.

** 1. End-to-end VM install pass rate =P1/rank 5=

Problem overcome: unit tests can pass while the real install is broken by
mirrors, bootloader state, pacstrap, SSH, disk layout, or the actual desktop
contract. This metric fights false confidence.

Representation:

- Lamp row in =INSTALL PIPELINE=.
- Two child lamps: =ZFS= and =BTRFS=.
- Age chip: =last pass 2d= / =never= / =stale 14d=.
- Red if either required filesystem has no recent pass.

Tools:

- =scripts/testing/run-test.sh=
- =scripts/testing/create-base-vm.sh=
- =pytest= testinfra suite under =scripts/testing/tests/=
- =qemu-img=, =qemu-system-x86_64=, =sshpass=

Doctor:

- read-only first: summarize last =test-results/*/test-report.txt= and failing
  test names.
- mitigation: offer =TEST VM= for the failed filesystem.
- cleanup: remove stale temporary VM overlays before retrying.
- no automatic retry loop if the failure is in the installer itself.

** 2. First-upgrade bootability =P1/rank 4=

Problem overcome: the machine can boot immediately after install but fail after
the first =pacman -Syu= because initramfs hooks, ZFS modules, GRUB, ZFSBootMenu,
or kernel packages drift.

Representation:

- Lamp row: =first-upgrade reboot=.
- Badge: =not run=, =passed=, =failed=.
- Evidence drawer includes boot count, kernel version, and last reachable SSH
  timestamp.

Tools:

- VM harness
- =pacman -Syu=
- =reboot=
- =ssh= reachability checks
- =journalctl -b -1= where available

Doctor:

- run upgrade-in-VM only, never on host without explicit confirmation.
- if failure is ZFS, collect =zpool import=, =lsinitcpio=, =mkinitcpio.conf=,
  and EFI files.
- if failure is Btrfs, collect =grub.cfg=, =crypttab=, =fstab=, and snapper
  config.

** 3. Package database freshness and sync health =P1/rank 6=

Problem overcome: Arch rolling-release state changes faster than installer
assumptions. Stale sync databases, stale keyrings, archzfs drift, or broken
mirrors are leading indicators of a failing install.

Representation:

- Row lamp in =PACKAGES + SERVICES=.
- Small meter: newest sync DB age vs threshold.
- Child chips: =core=, =extra=, =multilib=, =archzfs=.
- Yellow over 48 hours, red over 7 days or failed sync.

Tools:

- =find /var/lib/pacman/sync=
- =pacman -Syyu --needed archlinux-keyring=
- =checkupdates=
- =reflector=
- =pacman-conf=

Doctor:

- safe: refresh package databases.
- safe: update =archlinux-keyring= before full upgrades.
- mitigation: run =reflector= with the configured country/age policy.
- no unattended full system upgrade from the panel unless separately approved.

** 4. Archangel ISO build reproducibility =P1/rank 7=

Problem overcome: an old "good" ISO can hide broken current inputs. Archiso,
archzfs, DKMS, AUR package recipes, and pacoloco cache state can all break the
next install.

Representation:

- Row lamp in =INSTALL PIPELINE= or =BUILD= scope.
- Shows latest ISO date, kernel version, and AUR manifest age.
- Red if latest build failed or no ISO exists.
- Yellow if latest successful ISO is older than the configured freshness
  window.

Tools:

- =make build= in =~/code/archangel=
- =build.sh --skip-aur= for fast non-AUR iteration
- =build-aur.sh=
- =mkarchiso=
- =pacoloco= status if installed

Doctor:

- read-only: parse latest =out/*.log= for pacman, DKMS, archzfs, AUR, and
  mkarchiso failures.
- cleanup: safe build-work cleanup only through Archangel's cleanup function
  or =make clean=.
- mitigation: suggest =--skip-aur= when the failure is unrelated to baked AUR.
- build retry is explicit via =BUILD ISO=, not automatic.

** 5. ZFS/Btrfs storage health =P0/rank 1=

Problem overcome: the root filesystem can degrade silently before the user
notices. For ZFS this means pool errors or degraded vdevs; for Btrfs this means
device stats, scrub failures, metadata pressure, or degraded RAID.

Representation:

- =BOOT + STORAGE= band.
- Filesystem-specific lamp grammar:
  - ZFS green: =zpool status -x= healthy.
  - Btrfs green: device stats clean and recent scrub clean.
- Capacity strip for root/home/EFI.

Tools:

- ZFS: =zpool status -x=, =zpool list=, =zfs list=.
- Btrfs: =btrfs device stats=, =btrfs filesystem usage=,
  =btrfs scrub status=.
- Common: =df -h=, =findmnt=, =lsblk=.

Doctor:

- safe: start a scrub only with arm-first confirmation.
- safe: clear stale Btrfs stats only after a clean scrub and explicit
  confirmation.
- mitigation: warn on low EFI/root space and offer package cache cleanup.
- never destroy snapshots, pools, subvolumes, or datasets from Doctor.

** 6. Snapshot safety coverage =P0/rank 2=

Problem overcome: rollback safety is assumed during upgrades but can disappear
when hooks, services, or snapshot tools drift.

Representation:

- Row lamp: =snapshots=.
- Child chips: =genesis=, =pre-pacman=, =recent=, =pruned=.
- Yellow if no recent snapshot.
- Red if genesis or pre-transaction hook is missing.

Tools:

- ZFS: =zfs list -t snapshot=, =zfs-pre-snapshot=,
  =/etc/pacman.d/hooks/zfs-snapshot.hook=.
- Btrfs: =snapper list=, =snap-pac=, =grub-btrfs-mkconfig=,
  =/.snapshots=.
- Common: =pacman -Q= for snapshot packages.

Doctor:

- safe: create a manual snapshot.
- safe: reinstall or re-enable missing hook only if the expected script exists.
- cleanup: prune only snapshots matching the tool-owned policy and prefix.
- mitigation: show exact command for manual rollback; do not perform rollback
  from the panel.

** 7. Bootloader and EFI redundancy =P0/rank 3=

Problem overcome: single-disk bootloader success can mask missing redundant EFI
installs on multi-disk systems. A system can also pass install but lose a boot
entry or generate an invalid config.

Representation:

- Row lamp: =bootloader=.
- Child chips: =ZBM= or =GRUB=, =EFI=, =entries=, =all disks=.
- Yellow if redundancy cannot be proven.
- Red if the expected loader/config is missing.

Tools:

- =bootctl status=
- =efibootmgr -v=
- =findmnt /efi /boot=
- ZFS: check =/efi/EFI/ZBM/zfsbootmenu.efi=.
- Btrfs: check =/boot/grub/grub.cfg= and grub-btrfs entries.

Doctor:

- read-only by default.
- mitigation: regenerate GRUB config for Btrfs with arm-first confirmation.
- mitigation: rebuild initramfs with arm-first confirmation.
- no automatic EFI reinstall without an explicit advanced flow.

** 8. Post-install service health =P1/rank 8=

Problem overcome: the install can complete while the real workstation is
degraded: DNS broken, NetworkManager failed, fail2ban not responding, user
services not lingering, or Docker/Tailscale/Syncthing not in their expected
state.

Representation:

- =PACKAGES + SERVICES= band.
- Count badge: =0 failed= or =3 failed=.
- Child lamps: =network=, =dns=, =security=, =user services=.

Tools:

- =systemctl --failed=
- =systemctl is-enabled/is-active=
- =resolvectl status=
- =nmcli general status=
- =fail2ban-client status=
- =loginctl show-user=

Doctor:

- safe: restart known flaky non-destructive services such as
  =NetworkManager= only after classifying the failure.
- safe: re-enable expected services from Archsetup's contract.
- mitigation: bounce DNS resolver and re-check.
- no blanket =systemctl restart --failed=.

** 9. Archsetup state and log cleanliness =P1/rank 9=

Problem overcome: a resumable installer can leave a half-finished system that
looks usable until a missing marker or skipped step matters later.

Representation:

- =WORKSTATION CONTRACT= band.
- Step-progress mini bar: completed markers / expected markers.
- Red if =archsetup --status= reports incomplete required steps.
- Red if latest log contains fatal errors.

Tools:

- =./archsetup --status=
- =/var/log/archsetup-*.log=
- marker files from the Archsetup state directory
- existing testinfra assertions in =scripts/testing/tests/test_archsetup.py=

Doctor:

- read-only: summarize incomplete steps and latest log errors.
- mitigation: offer to rerun =archsetup= in normal resumable mode.
- cleanup: archive old logs, keep the latest N.
- never run =--fresh= from Doctor.

** 10. Workstation contract checks =P2/rank 10=

Problem overcome: a fresh Arch system is not the goal. The goal is Craig's
working machine: user, shell, groups, dotfiles, Emacs, Hyprland/DWM, keyring,
VPN tools, Bluetooth tools, and local scripts.

Representation:

- =WORKSTATION CONTRACT= band.
- Child lamps: =user=, =dotfiles=, =desktop=, =tools=.
- Evidence drawer mirrors the testinfra checks.

Tools:

- =id=, =getent passwd=
- =test -L ~/.zshrc=
- =stow= via dotfiles Makefile
- =pacman -Q=, =yay -Qi yay=
- =hyprctl=, =gdbus= portal checks when session is running

Doctor:

- safe: restow dotfiles with the selected profile.
- safe: repair keyring directory permissions.
- mitigation: reinstall missing official packages.
- AUR package rebuilds require confirmation and stream output.

** 11. Security and hardening drift =P2/rank 12=

Problem overcome: security settings are easy to regress because they are small
file edits: SSH root login, EFI mount masks, firewall, issue banner, fail2ban,
quiet printk.

Representation:

- Compact row under =WORKSTATION CONTRACT= or =PACKAGES + SERVICES=.
- Red only for high-risk drift, yellow for unknown/unreadable state.

Tools:

- =sshd -T= or config file checks
- =ufw status=
- =fail2ban-client status=
- =findmnt /efi=
- =sysctl kernel.printk=

Doctor:

- safe: restore known Archsetup-owned config snippets.
- safe: re-enable firewall if policy file is present.
- mitigation: write missing drop-ins only from version-controlled templates.
- no broad hardening rewrite from panel state.

** 12. Backup and rollback readiness =P2/rank 11=

Problem overcome: rollback only helps local state. The install also needs
backups of edited system files and confidence that personal data replication is
not silently stale.

Representation:

- Row lamp: =backups=.
- Chips: =system-file .bak=, =replication=, =last run=.
- Yellow if last replication exceeds policy.
- Red if expected backup files for edited system config are missing.

Tools:

- Archsetup backup assertions in =scripts/testing/tests/test_backups.py=.
- =zfs-replicate= if configured.
- =systemctl list-timers= for backup timers.
- =journalctl -u= relevant backup units.

Doctor:

- safe: create missing =.archsetup.bak= for files before editing.
- safe: run dry-run replication check.
- mitigation: start a configured backup timer/unit with confirmation.
- never delete backup targets from Doctor.

* Doctor Model

Doctor is a classifier with bounded mitigations, not a magic repair button.

Flow:

1. Probe the selected scope.
2. Normalize each metric to =ok=, =warn=, =fail=, =unknown=, or =running=.
3. Classify failures as:
   - =safe-fix= — local, reversible, low risk.
   - =safe-cleanup= — removes only known generated artifacts.
   - =mitigation= — improves the chance of success but does not claim repair.
   - =needs-confirmation= — mutating, long-running, or system-wide.
   - =manual= — too dangerous or context-heavy for Doctor.
4. Run only safe actions automatically after the user presses Doctor.
5. Arm-first for anything mutating beyond safe local cleanup.
6. Re-run the affected probe.
7. Stream a verdict into the output well.

Doctor should say exactly what it did:

#+BEGIN_EXAMPLE
doctor: package db stale
  check: core.db age 4d, archzfs.db age 4d
  action: refreshed sync databases
  action: updated archlinux-keyring
  result: ok, newest db age 2m
#+END_EXAMPLE

* Common Tool Drivers

** Host probes

| Area       | Commands                                                       |
|------------+----------------------------------------------------------------|
| systemd    | =systemctl --failed=, =systemctl is-active=, =journalctl=       |
|------------+----------------------------------------------------------------|
| packages   | =pacman=, =checkupdates=, =pacman-conf=, =yay=                 |
|------------+----------------------------------------------------------------|
| storage    | =zpool=, =zfs=, =btrfs=, =df=, =findmnt=, =lsblk=              |
|------------+----------------------------------------------------------------|
| boot       | =bootctl=, =efibootmgr=, =mkinitcpio=, =grub-mkconfig=         |
|------------+----------------------------------------------------------------|
| network    | =nmcli=, =resolvectl=, =ping= or HTTPS probe                   |
|------------+----------------------------------------------------------------|
| desktop    | =hyprctl=, =gdbus=, =loginctl=, dotfiles Makefile              |
|------------+----------------------------------------------------------------|

** Project probes

| Area       | Commands                                                       |
|------------+----------------------------------------------------------------|
| archangel  | =make test=, =make build=, =build.sh --skip-aur=               |
|------------+----------------------------------------------------------------|
| archsetup  | =make test-unit=, =make test=, =scripts/testing/run-test.sh=   |
|------------+----------------------------------------------------------------|
| VM         | =qemu-img=, =qemu-system-x86_64=, =sshpass=, =pytest=          |
|------------+----------------------------------------------------------------|
| artifacts  | latest =out/*.log=, =out/*aur-manifest.tsv=, =test-results/*= |
|------------+----------------------------------------------------------------|

* Top-family Comparison

This monitor should borrow the mature display ideas from =top=-style tools
without becoming another CPU/process viewer. The domain objects are install
contracts, boot/storage health, package freshness, snapshots, services, and
artifacts. The interaction model is still the same: sort the thing that hurts,
filter to the thing you care about, expand one row for evidence, and act only
when the diagnosis is clear.

** Comparison table

| Tool              | What it represents well                                      | Sorting/filtering model                                      | Useful pattern for system monitor                              | Gaps for our domain                                      |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =htop=            | Dense live table plus configurable meters; process tree; direct process actions. | Interactive sort by column, search, filter, tree toggle.      | Metric table should support column sort, search, filter, and tree/group mode. | No historical artifact model; actions are process-centric. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =btop++=          | Boxed dashboard: CPU, memory, disks, network, processes, battery, GPU; strong graph language; selected process detail. | Easy switching between process sort modes; filter; tree view; pause. | Use boxed bands, mini time-series, detail pane, pause/freeze, and clickable controls. | Graph-first layout can overemphasize volatile values over install risk. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =bottom/btm=      | Custom widget layout, per-widget focus/expand, zoomable time windows, basic mode. | Process widget supports sort, search, tree; widgets can be filtered/configured. | Every health band should be expandable; stale/history windows should be zoomable. | Mostly resource telemetry, not remediation workflow. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =atop=            | Interval deltas, critical-resource highlighting, all active processes including exited ones, long-term logs. | Resource views and interval replay; emphasizes deviations and active load. | Add history/replay for health events and show "new since last good" changes. | Lower immediate visual polish; Linux-performance scoped. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =Glances=         | Broad plugin dashboard, thresholds, remote/web/API modes, export to JSON/CSV/time-series backends. | Configurable visible plugins; API/stdout selectors instead of only interactive sorting. | Use plugin architecture, threshold config, JSON output, remote/headless mode. | Too broad; can become a generic monitoring surface. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =procs=           | Modern table ergonomics: custom columns, keyword search across selected fields, sort by named column, tree view. | CLI sort asc/desc by partial column name; watch mode cycles sort columns; AND/OR/NAND/NOR search. | Use named metric columns, saved views, multi-keyword filters, and value-aware coloring. | Process-only; no graphs or remediation model. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|
| =gotop/gtop= family | Fast glanceable terminal dashboard with compact graphs and gauges. | Usually lighter than htop/btop; less important than presentation density. | Use compact sparklines/gauges for "age", "last pass", and "failure count". | Not enough evidence/action depth for this monitor. |
|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------|

** What to pull in

*** htop: table discipline

Pull:

- Column headers that are real controls: click or key-cycle to sort by
  =priority=, =state=, =age=, =last_checked=, =last_pass=, =failure_count=,
  =scope=, and =doctor_class=.
- Search and filter as first-class actions, not hidden debug commands.
- Tree mode for ownership:
  - =system= → =boot/storage= → =bootloader= → =efi entries=.
  - =archangel= → =iso build= → =aur repo= → =manifest rows=.
  - =archsetup= → =state markers= → =desktop= → =dotfiles=.
- Horizontal detail access for long evidence, like htop's horizontal scrolling
  for full commands.

Equivalent-or-better requirement:

- htop sorts processes; this monitor sorts risk. The default sort is
  live-state severity, then priority rank, then age.

*** btop++: instrument boxes and live graphs

Pull:

- Boxed bands with stable geometry.
- Small time-series graphs, but only where history matters:
  - package DB age over time
  - failed-service count
  - journal error count
  - snapshot count / newest snapshot age
  - VM pass/fail history
  - ISO build duration/result history
- Selected-row detail pane with the last command, verdict, and raw excerpt.
- Pause/freeze button so a failure does not scroll away while reading.
- Mouse-clickable controls where every visible key has the same keyboard path.

Equivalent-or-better requirement:

- btop's graphs answer "what is hot right now?" Our graphs answer "is the
  safety margin shrinking?" Trend charts should be muted unless the threshold
  is crossed.

*** bottom: focus/expand and layout presets

Pull:

- Expand one band full-height:
  - =BOOT + STORAGE= expands into boot files, EFI entries, pools, snapshots.
  - =INSTALL PIPELINE= expands into last VM runs and build artifacts.
  - =PACKAGES + SERVICES= expands into DB ages, repo status, failed units.
  - =WORKSTATION CONTRACT= expands into Archsetup markers and testinfra-style
    checks.
- Zoomable history windows: 24h / 7d / 30d / all artifacts.
- Layout presets:
  - =compact= for bar dropdown.
  - =full= for terminal/TUI.
  - =host-only= for laptop health.
  - =release-gate= for Archangel/Archsetup changes.

Equivalent-or-better requirement:

- bottom expands widgets; this monitor expands evidence and remediation state.
  The expanded view must show "what changed since last good" before raw logs.

*** atop: history and vanished failures

Pull:

- Permanent, compact health-event log.
- Interval deltas instead of only current values:
  - new failed services since last check
  - new journal errors since last check
  - packages/repos newly stale
  - snapshot hook present before, missing now
  - bootloader file changed since last known-good
- "Show active/deviating only" mode. In normal use, hide green rows unless
  their age is approaching threshold.
- Replay mode: inspect the state at the time an install/test/build failed.

Equivalent-or-better requirement:

- atop can report processes that already exited. This monitor should report
  failures that already passed through: a transient failed unit, a VM test that
  failed last night, an ISO build that failed before the current successful
  build, or a package DB that was stale until Doctor fixed it.

*** Glances: plugin/API/export model

Pull:

- Plugin-like probes. Each metric owns:
  - =probe=
  - =normalize=
  - =thresholds=
  - =doctor_actions=
  - =redaction=
  - =evidence=
- JSON output as a stable contract before GTK work.
- Optional stdout selectors:
  - =system-monitor --stdout packages.state,storage.state=
  - =system-monitor --json boot,snapshots=
- Remote/headless mode for VMs and bare-metal test targets.
- Threshold config in one file, not hardcoded in the UI.

Equivalent-or-better requirement:

- Glances is broad; this must stay opinionated. A plugin is accepted only if it
  maps to install health, rollback safety, workstation contract, or recovery
  readiness.

*** procs: custom columns and query grammar

Pull:

- Named columns and saved views:
  - =risk=: state, priority, age, doctor class.
  - =install=: last pass, filesystem, artifact, branch, commit.
  - =host=: state, source, last checked, command.
  - =doctor=: action class, requires root, reversible, last run.
- Multi-keyword search:
  - =zfs failed=
  - =doctor safe-fix=
  - =archangel stale=
  - =service red=
- Boolean query modes:
  - AND default for narrowing.
  - OR for "show any boot or storage issue".
  - NOT for "hide green".
- Value-aware coloring for age, severity, and units.

Equivalent-or-better requirement:

- procs lets the user build a process table. This monitor should let Craig
  build a risk table without editing code.

*** gotop/gtop: glance density

Pull:

- Small sparklines for trend, not full charts.
- Big obvious state words.
- Compact gauges for bounded values:
  - EFI usage
  - root/home usage
  - DB age as percent of freshness window
  - VM evidence age
  - snapshot age
- Simple default screen that is useful without learning keys.

Equivalent-or-better requirement:

- The first screen should answer "am I safe to upgrade or install?" in under
  two seconds.

* Sorting and Views

The monitor needs two sorting layers: global row ordering and per-band evidence
tables.

** Global row ordering

Default:

1. =state= severity: red, yellow, unknown, running, green.
2. =priority= rank: P0 before P1 before P2.
3. =age= or =staleness=, descending.
4. =last_changed=, newest first.

Alternate sorts:

| Sort key        | Use case                                             |
|-----------------+------------------------------------------------------|
| =priority=      | Release-gate review; keep P0/P1 at the top.          |
|-----------------+------------------------------------------------------|
| =state=         | Triage; show all red/yellow rows first.              |
|-----------------+------------------------------------------------------|
| =age=           | Find stale tests, stale package DBs, old backups.    |
|-----------------+------------------------------------------------------|
| =doctor_class=  | Find what Doctor can safely fix now.                 |
|-----------------+------------------------------------------------------|
| =scope=         | Group host vs install vs build.                      |
|-----------------+------------------------------------------------------|
| =last_changed=  | See what recently regressed.                         |
|-----------------+------------------------------------------------------|
| =source=        | Group by archangel, archsetup, dotfiles, host.       |
|-----------------+------------------------------------------------------|

** Per-band sorts

| Band                   | Sorts                                                      |
|------------------------+------------------------------------------------------------|
| =BOOT + STORAGE=       | severity, mountpoint, filesystem, capacity, last scrub, newest snapshot age |
|------------------------+------------------------------------------------------------|
| =INSTALL PIPELINE=     | result, filesystem, duration, artifact age, commit age, last pass |
|------------------------+------------------------------------------------------------|
| =PACKAGES + SERVICES=  | severity, unit name, repo name, DB age, error count, enabled/active state |
|------------------------+------------------------------------------------------------|
| =WORKSTATION CONTRACT= | severity, check name, owner repo, last pass, doctor class  |
|------------------------+------------------------------------------------------------|

** Filters

Quick filters should be visible as chips:

- =red=
- =yellow=
- =doctorable=
- =needs-root=
- =stale=
- =zfs=
- =btrfs=
- =host=
- =install=
- =build=
- =changed=
- =hidden-green=

The default view can hide healthy low-priority rows, but it must show enough
green P0/P1 summary state to prove the monitor is working.

* Display Requirements Borrowed from Tops

1. Every table has sortable columns and a visible sort indicator.
2. Every visible metric row has a filterable state, priority, age, and source.
3. Every row can expand to evidence without losing the list context.
4. Every graph has a threshold marker; trend without threshold is decoration.
5. Every long-running action can be paused/frozen in the display.
6. Every mutating action has an equivalent CLI command shown in the output well.
7. Every Doctor action records before/after state so fixed failures remain
   visible in history.
8. Every band has a compact mode and an expanded mode.
9. Green rows are quiet; new regressions are loud.
10. The system must be useful over SSH/TUI before GTK polish.

* Source Notes

- =htop=: upstream README describes configurable system/process display,
  interactive sorting/filtering/search, tree view, and process actions.
- =btop++=: upstream README describes resource boxes, detailed process stats,
  filter, sort switching, tree view, mouse support, auto-scaling network graphs,
  disk IO, battery, GPU support, and themes.
- =bottom/btm=: upstream README describes customizable widgets, process sort
  and search, tree mode, expand/focus, zoomable graph intervals, filters, and
  basic mode.
- =atop=: upstream README describes interval resource accounting, critical
  highlighting, long-term compressed logs, exited-process visibility, cgroup
  views, and active/deviation-focused output.
- =Glances=: upstream README describes plugin-style broad monitoring, web/API
  modes, stdout JSON/CSV, remote monitoring, exports, and threshold-oriented
  dashboard use.
- =procs=: upstream README describes configurable columns, named-column sort,
  watch mode, tree view, logical keyword search, value-aware coloring, and
  pager behavior.

* Data Model

Emit JSON from a CLI first; the panel is a client.

#+BEGIN_SRC json
{
  "v": 1,
  "scope": "host",
  "state": "warn",
  "ts": "2026-07-04T12:00:00-04:00",
  "metrics": [
    {
      "id": "packages.sync_freshness",
      "label": "package databases",
      "state": "warn",
      "summary": "archzfs.db age 4d",
      "evidence": [
        {"command": "find /var/lib/pacman/sync", "excerpt": "archzfs.db 2026-06-30"}
      ],
      "doctor": {
        "class": "safe-fix",
        "actions": ["refresh-sync-db", "update-keyring"]
      }
    }
  ]
}
#+END_SRC

* Implementation Notes

- Start with a CLI: =system-monitor status --json=, =system-monitor doctor
  --json=, =system-monitor refresh=.
- Keep probes read-only by default. Actions live in separate verbs.
- Cache slow probes. The bar should read a cache, not run VM tests.
- VM/build actions should create job records and stream logs; the panel follows
  the job rather than blocking the UI process.
- Reuse the net/bt panel architecture if this becomes a GTK panel: GTK-free
  model + fake-command unit tests + one AT-SPI smoke.
- Redact secrets from logs and JSON: WiFi PSKs, tokens, private repo URLs with
  credentials, SSH material, and backup target credentials.

* Open Decisions

** TODO Where should the first implementation live?

Recommendation: dotfiles owns the user-facing panel and CLI wrapper because it
is workstation UI. Archsetup owns reusable install-contract probes and testinfra
assertions. Archangel owns ISO/build probes.

** TODO Should Doctor run elevated actions through polkit or terminal?

Recommendation: read-only checks run unprivileged; elevated actions launch a
terminal or polkit prompt with the exact command visible. Do not hide long
privileged operations inside the panel process.

** TODO How fresh must VM evidence be?

Recommendation: host checks go stale after 1 hour; package DB after 48 hours;
VM install evidence after 7 days; ISO build evidence after 14 days or whenever
Archangel/Archsetup has changed since the last successful artifact.

** TODO Which actions are allowed on bare metal?

Recommendation: host Doctor may refresh databases, update keyring, restow
dotfiles, create snapshots, run scrub, and restart narrowly classified services.
It may not perform full upgrades, bootloader reinstalls, destructive snapshot
prune, or filesystem repair without a separate advanced flow.

* First Build Slice

1. CLI read-only host status:
   - package DB freshness
   - failed services
   - ZFS/Btrfs health
   - snapshot presence
   - Archsetup status/log check
2. Doctor safe actions:
   - refresh package DB
   - update keyring
   - restow dotfiles
   - create manual snapshot
3. Artifact parser:
   - latest Archsetup VM test result
   - latest Archangel ISO build result
4. Panel prototype:
   - faceplate
   - four health bands
   - evidence output well
   - REFRESH and DOCTOR keys only