diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/system-monitor-design-ideas.org | 1008 |
1 files changed, 1008 insertions, 0 deletions
diff --git a/docs/design/system-monitor-design-ideas.org b/docs/design/system-monitor-design-ideas.org new file mode 100644 index 0000000..26619a0 --- /dev/null +++ b/docs/design/system-monitor-design-ideas.org @@ -0,0 +1,1008 @@ +#+TITLE: System Monitor Design Ideas +#+DATE: 2026-07-04 +#+TODO: TODO | DONE +#+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED + +* DRAFT Status +:PROPERTIES: +:ID: system-monitor-design-ideas +:END: +- [2026-07-04 Sat] DRAFT — initial design sketch for a health monitor covering + Archangel ISO/base-install health, Archsetup workstation health, and the local + laptop as the daily canary. + +* Metadata + +| Field | Value | +|--------+--------------------------------------------| +| Status | draft | +|--------+--------------------------------------------| +| Owner | Craig Jennings | +|--------+--------------------------------------------| +| Repos | archsetup, archangel, dotfiles | +|--------+--------------------------------------------| +| Kin | net panel, bluetooth panel, audio panel | +|--------+--------------------------------------------| + +* Problem + +Archangel and Archsetup can fail in ways that are individually obvious only +after the damage is done: an ISO build goes stale against Arch or archzfs, +ZFSBootMenu or GRUB boots once but not after the first upgrade, a snapshot +hook silently disappears, a package database ages out, systemd services fail +after a reboot, or the desktop contract is technically installed but not +usable. + +The health surface should compress those risks into one operational question: +"Can I trust a fresh install, and is this current workstation drifting away +from the known-good install contract?" + +This monitor is not a generic CPU/RAM graph. It is an install-health and +workstation-contract console. CPU, memory, and temperature belong only as +secondary context unless they block install/test operations. + +* Priority Model + +Rank metrics by the cost of blindness: what happens if Craig never sees the +metric, no one mitigates it, and the next install/upgrade/reboot simply happens. + +Severity: + +- =P0= — can cause data loss, unbootable systems, or loss of rollback path. +- =P1= — can break fresh installs, upgrades, remote access, or core + workstation use. +- =P2= — causes degraded workstation behavior, security drift, or accumulating + maintenance debt. +- =P3= — useful context, not a release gate by itself. + +The panel should sort by live severity first, then by this priority. A red =P2= +row appears above a green =P0= row, but in the steady state the layout keeps the +P0/P1 rows in the first viewport. + +* Priority Ranking + +| Rank | Priority | Metric | Why this rank exists | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 1 | P0 | Storage health | Silent pool/filesystem degradation is the nearest thing to data loss. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 2 | P0 | Snapshot safety coverage | Without snapshots, upgrades lose their rollback safety net. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 3 | P0 | Bootloader and EFI redundancy | A machine that cannot boot is operationally dead. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 4 | P1 | First-upgrade bootability | Catches the classic "installed fine, broke after update" failure. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 5 | P1 | End-to-end VM install pass rate | Best release gate for the whole Archangel + Archsetup chain. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 6 | P1 | Package sync and repo freshness | Arch, archzfs, keyring, and mirror drift are leading break signals. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 7 | P1 | Archangel ISO reproducibility | If current inputs cannot build an ISO, recovery/install confidence is stale. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 8 | P1 | Post-install service health | Network, DNS, SSH, and user services decide whether the system is usable. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 9 | P1 | Archsetup state/log cleanliness | Prevents "half-installed but looks fine" machines. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 10 | P2 | Workstation contract checks | Confirms this is Craig's workstation, not just generic Arch. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 11 | P2 | Backup and rollback readiness | Catches loss of off-machine recovery and edited-file backups. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| +| 12 | P2 | Security and hardening drift | Important, but usually less immediately destructive than boot/storage. | +|------+----------+---------------------------------+------------------------------------------------------------------------------| + +This order intentionally puts ZFS/Btrfs health above VM install evidence. A +broken future install is expensive; silent damage to the current root or backup +chain is worse. + +* Consequence Matrix + +This is the design justification for every metric. A row earns panel space only +if blindness has a clear failure mode and Doctor has at least a useful +diagnostic or mitigation. + +| Rank | Metric | If never seen / never mitigated | Typical failure | Worst plausible failure | Doctor posture | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 1 | Storage health | Disk, pool, or metadata degradation accumulates silently. | Correctable ZFS/Btrfs errors, low EFI/root space, stale scrub. | Data loss, degraded root, failed import/mount during boot. | Diagnose + scrub/cleanup with confirmation. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 2 | Snapshot safety coverage | Upgrades proceed without a rollback point. | Missing recent snapshot or missing pre-pacman hook. | Bad upgrade cannot be rolled back cleanly; manual repair required. | Create snapshot; restore hook; never rollback. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 3 | Bootloader and EFI redundancy | Boot path rots until the next reboot or disk failure. | Missing GRUB/ZBM file on one EFI partition. | Unbootable machine after update, firmware reset, or disk loss. | Diagnose; regenerate config; advanced reinstall only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 4 | First-upgrade bootability | Installs look good until the first real system update. | Kernel/initramfs/bootloader mismatch in VM. | Fresh bare-metal install dies on first reboot after upgrade. | VM-only upgrade test; collect boot evidence. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 5 | End-to-end VM install pass rate | Unit tests give false confidence about the real workflow. | Current branch fails one filesystem path or desktop assertion. | Bare-metal install fails mid-flight after disks are wiped. | Run/schedule VM test; clean stale VM artifacts. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 6 | Package sync/repo freshness | Arch/keyring/archzfs drift surprises the next install. | 404s, stale keyring, bad mirror, stale archzfs DB. | Installer cannot pacstrap or installs mismatched ZFS/kernel bits. | Refresh DB; update keyring; reflector. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 7 | ISO build reproducibility | Recovery/install media confidence becomes historical. | AUR package fails to build; mkarchiso or DKMS breaks. | Need rescue/install media and discover no current ISO can be built. | Parse logs; clean work; explicit rebuild. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 8 | Post-install service health | System is installed but degraded in daily operation. | DNS, NetworkManager, fail2ban, tailscale, or user service down. | No remote access, no network, broken sync, or security tooling off. | Restart/re-enable classified services only. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 9 | Archsetup state/log cleanliness | Half-completed provisioning masquerades as success. | Missing marker, log error, skipped step after resume. | Fresh workstation lacks critical config but looks mostly usable. | Summarize; rerun resumable archsetup. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 10 | Workstation contract checks | The box drifts from "Craig's workstation" to generic Arch. | Dotfile symlink broken, keyring wrong, missing tool/package. | Desktop/session workflow is broken during real work. | Restow, repair perms, reinstall with confirm. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 11 | Backup and rollback readiness | Recovery assumptions go stale. | Missing =.archsetup.bak=, backup timer stale, dry-run fails. | Local rollback works but important personal/system state is gone. | Dry-run, start configured job, no deletes. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| +| 12 | Security and hardening drift | Small protective edits quietly regress. | SSH/firewall/fail2ban/sysctl/EFI mask drift. | Exposed service or weakened local boot/config protections. | Restore owned snippets; no broad rewrite. | +|------+---------------------------------+-----------------------------------------------------------+----------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------| + +The worst cases are intentionally conservative, not dramatic. The monitor is +useful because it catches the boring early signal: stale, missing, not recently +tested, or silently degraded. + +* Product Shape + +Use the existing instrument-console panel language rather than a dashboard page: +lamps for pass/warn/fail state, engraved metric groups, dense rows, physical +console keys for explicit actions, and an output well for the last diagnosis. + +The bar module should be tiny: + +- icon: a pulse/terminal glyph or compact =SYS= label. +- state lamp: green/yellow/red/grey. +- text: one word only: =OK=, =WARN=, =FAIL=, =STALE=, =RUNNING=, =UNKNOWN=. +- click: opens the system monitor panel. +- middle click or secondary action: run a cheap refresh only, never repairs. + +The panel should be one screen with internal scroll only where needed. The +first viewport should show the decision state and the Doctor controls without +scrolling. + +* Layout + +** Faceplate + +Top row: + +- State lamp + state word: =OK= / =WARN= / =FAIL= / =STALE= / =RUNNING= / + =UNKNOWN=. +- Unit label: =SYS·01=. +- Scope segmented control: =HOST= / =INSTALL= / =BUILD=. +- Badges: =ZFS= or =BTRFS=, =VM STALE=, =DB STALE=, =SNAPSHOT=, =BACKUP=, + =ROOT= when elevated actions are available. +- Close button. + +The scope control changes the metric emphasis, not the underlying data model: + +- =HOST= is this laptop/workstation right now. +- =INSTALL= is the last Archangel+Archsetup VM install result. +- =BUILD= is ISO, AUR repo, archzfs, and test artifact health. + +** Health Stack + +Arrange metrics as four horizontal bands. Each band has a section title, a +summary lamp, two to four row lamps, and a short "age" or "count" value. + +1. =BOOT + STORAGE= + - bootloader + - EFI redundancy + - pool/filesystem + - snapshots + +2. =INSTALL PIPELINE= + - first-upgrade reboot + - ZFS VM install + - Btrfs VM install + - ISO build + +3. =PACKAGES + SERVICES= + - pacman sync freshness + - archzfs/AUR health + - failed services + - journal errors + +4. =WORKSTATION CONTRACT= + - Archsetup markers/logs + - user/dotfiles + - desktop/session + - backups/replication + +Each row is clickable. Clicking a row opens the evidence drawer in the output +well with: + +- last command run +- normalized verdict +- raw excerpt, redacted where needed +- suggested Doctor action, if any + +** Console Keys + +Use physical console-key buttons, same family as net/bt: + +| Key | Purpose | +|--------------+------------------------------------------------------------| +| REFRESH | Cheap read-only probe of host state | +|--------------+------------------------------------------------------------| +| DOCTOR | Diagnose, classify, run safe mitigations, re-check | +|--------------+------------------------------------------------------------| +| TEST VM | Run or schedule Archsetup VM validation | +|--------------+------------------------------------------------------------| +| BUILD ISO | Run or schedule Archangel ISO build | +|--------------+------------------------------------------------------------| +| CLEAN | Clean old test artifacts, package cache, stale logs | +|--------------+------------------------------------------------------------| +| SNAPSHOT | Create a manual pre-change snapshot | +|--------------+------------------------------------------------------------| + +Keys that can take a long time stream progress into the output well. Mutating +keys must use arm-first behavior: + +- first click arms for 3 seconds and explains the action. +- second click runs. +- destructive cleanup names what will be deleted before it runs. + +* Metric Details + +Details below are grouped by workflow rather than priority. The authoritative +importance order is the priority table above. + +** 1. End-to-end VM install pass rate =P1/rank 5= + +Problem overcome: unit tests can pass while the real install is broken by +mirrors, bootloader state, pacstrap, SSH, disk layout, or the actual desktop +contract. This metric fights false confidence. + +Representation: + +- Lamp row in =INSTALL PIPELINE=. +- Two child lamps: =ZFS= and =BTRFS=. +- Age chip: =last pass 2d= / =never= / =stale 14d=. +- Red if either required filesystem has no recent pass. + +Tools: + +- =scripts/testing/run-test.sh= +- =scripts/testing/create-base-vm.sh= +- =pytest= testinfra suite under =scripts/testing/tests/= +- =qemu-img=, =qemu-system-x86_64=, =sshpass= + +Doctor: + +- read-only first: summarize last =test-results/*/test-report.txt= and failing + test names. +- mitigation: offer =TEST VM= for the failed filesystem. +- cleanup: remove stale temporary VM overlays before retrying. +- no automatic retry loop if the failure is in the installer itself. + +** 2. First-upgrade bootability =P1/rank 4= + +Problem overcome: the machine can boot immediately after install but fail after +the first =pacman -Syu= because initramfs hooks, ZFS modules, GRUB, ZFSBootMenu, +or kernel packages drift. + +Representation: + +- Lamp row: =first-upgrade reboot=. +- Badge: =not run=, =passed=, =failed=. +- Evidence drawer includes boot count, kernel version, and last reachable SSH + timestamp. + +Tools: + +- VM harness +- =pacman -Syu= +- =reboot= +- =ssh= reachability checks +- =journalctl -b -1= where available + +Doctor: + +- run upgrade-in-VM only, never on host without explicit confirmation. +- if failure is ZFS, collect =zpool import=, =lsinitcpio=, =mkinitcpio.conf=, + and EFI files. +- if failure is Btrfs, collect =grub.cfg=, =crypttab=, =fstab=, and snapper + config. + +** 3. Package database freshness and sync health =P1/rank 6= + +Problem overcome: Arch rolling-release state changes faster than installer +assumptions. Stale sync databases, stale keyrings, archzfs drift, or broken +mirrors are leading indicators of a failing install. + +Representation: + +- Row lamp in =PACKAGES + SERVICES=. +- Small meter: newest sync DB age vs threshold. +- Child chips: =core=, =extra=, =multilib=, =archzfs=. +- Yellow over 48 hours, red over 7 days or failed sync. + +Tools: + +- =find /var/lib/pacman/sync= +- =pacman -Syyu --needed archlinux-keyring= +- =checkupdates= +- =reflector= +- =pacman-conf= + +Doctor: + +- safe: refresh package databases. +- safe: update =archlinux-keyring= before full upgrades. +- mitigation: run =reflector= with the configured country/age policy. +- no unattended full system upgrade from the panel unless separately approved. + +** 4. Archangel ISO build reproducibility =P1/rank 7= + +Problem overcome: an old "good" ISO can hide broken current inputs. Archiso, +archzfs, DKMS, AUR package recipes, and pacoloco cache state can all break the +next install. + +Representation: + +- Row lamp in =INSTALL PIPELINE= or =BUILD= scope. +- Shows latest ISO date, kernel version, and AUR manifest age. +- Red if latest build failed or no ISO exists. +- Yellow if latest successful ISO is older than the configured freshness + window. + +Tools: + +- =make build= in =~/code/archangel= +- =build.sh --skip-aur= for fast non-AUR iteration +- =build-aur.sh= +- =mkarchiso= +- =pacoloco= status if installed + +Doctor: + +- read-only: parse latest =out/*.log= for pacman, DKMS, archzfs, AUR, and + mkarchiso failures. +- cleanup: safe build-work cleanup only through Archangel's cleanup function + or =make clean=. +- mitigation: suggest =--skip-aur= when the failure is unrelated to baked AUR. +- build retry is explicit via =BUILD ISO=, not automatic. + +** 5. ZFS/Btrfs storage health =P0/rank 1= + +Problem overcome: the root filesystem can degrade silently before the user +notices. For ZFS this means pool errors or degraded vdevs; for Btrfs this means +device stats, scrub failures, metadata pressure, or degraded RAID. + +Representation: + +- =BOOT + STORAGE= band. +- Filesystem-specific lamp grammar: + - ZFS green: =zpool status -x= healthy. + - Btrfs green: device stats clean and recent scrub clean. +- Capacity strip for root/home/EFI. + +Tools: + +- ZFS: =zpool status -x=, =zpool list=, =zfs list=. +- Btrfs: =btrfs device stats=, =btrfs filesystem usage=, + =btrfs scrub status=. +- Common: =df -h=, =findmnt=, =lsblk=. + +Doctor: + +- safe: start a scrub only with arm-first confirmation. +- safe: clear stale Btrfs stats only after a clean scrub and explicit + confirmation. +- mitigation: warn on low EFI/root space and offer package cache cleanup. +- never destroy snapshots, pools, subvolumes, or datasets from Doctor. + +** 6. Snapshot safety coverage =P0/rank 2= + +Problem overcome: rollback safety is assumed during upgrades but can disappear +when hooks, services, or snapshot tools drift. + +Representation: + +- Row lamp: =snapshots=. +- Child chips: =genesis=, =pre-pacman=, =recent=, =pruned=. +- Yellow if no recent snapshot. +- Red if genesis or pre-transaction hook is missing. + +Tools: + +- ZFS: =zfs list -t snapshot=, =zfs-pre-snapshot=, + =/etc/pacman.d/hooks/zfs-snapshot.hook=. +- Btrfs: =snapper list=, =snap-pac=, =grub-btrfs-mkconfig=, + =/.snapshots=. +- Common: =pacman -Q= for snapshot packages. + +Doctor: + +- safe: create a manual snapshot. +- safe: reinstall or re-enable missing hook only if the expected script exists. +- cleanup: prune only snapshots matching the tool-owned policy and prefix. +- mitigation: show exact command for manual rollback; do not perform rollback + from the panel. + +** 7. Bootloader and EFI redundancy =P0/rank 3= + +Problem overcome: single-disk bootloader success can mask missing redundant EFI +installs on multi-disk systems. A system can also pass install but lose a boot +entry or generate an invalid config. + +Representation: + +- Row lamp: =bootloader=. +- Child chips: =ZBM= or =GRUB=, =EFI=, =entries=, =all disks=. +- Yellow if redundancy cannot be proven. +- Red if the expected loader/config is missing. + +Tools: + +- =bootctl status= +- =efibootmgr -v= +- =findmnt /efi /boot= +- ZFS: check =/efi/EFI/ZBM/zfsbootmenu.efi=. +- Btrfs: check =/boot/grub/grub.cfg= and grub-btrfs entries. + +Doctor: + +- read-only by default. +- mitigation: regenerate GRUB config for Btrfs with arm-first confirmation. +- mitigation: rebuild initramfs with arm-first confirmation. +- no automatic EFI reinstall without an explicit advanced flow. + +** 8. Post-install service health =P1/rank 8= + +Problem overcome: the install can complete while the real workstation is +degraded: DNS broken, NetworkManager failed, fail2ban not responding, user +services not lingering, or Docker/Tailscale/Syncthing not in their expected +state. + +Representation: + +- =PACKAGES + SERVICES= band. +- Count badge: =0 failed= or =3 failed=. +- Child lamps: =network=, =dns=, =security=, =user services=. + +Tools: + +- =systemctl --failed= +- =systemctl is-enabled/is-active= +- =resolvectl status= +- =nmcli general status= +- =fail2ban-client status= +- =loginctl show-user= + +Doctor: + +- safe: restart known flaky non-destructive services such as + =NetworkManager= only after classifying the failure. +- safe: re-enable expected services from Archsetup's contract. +- mitigation: bounce DNS resolver and re-check. +- no blanket =systemctl restart --failed=. + +** 9. Archsetup state and log cleanliness =P1/rank 9= + +Problem overcome: a resumable installer can leave a half-finished system that +looks usable until a missing marker or skipped step matters later. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Step-progress mini bar: completed markers / expected markers. +- Red if =archsetup --status= reports incomplete required steps. +- Red if latest log contains fatal errors. + +Tools: + +- =./archsetup --status= +- =/var/log/archsetup-*.log= +- marker files from the Archsetup state directory +- existing testinfra assertions in =scripts/testing/tests/test_archsetup.py= + +Doctor: + +- read-only: summarize incomplete steps and latest log errors. +- mitigation: offer to rerun =archsetup= in normal resumable mode. +- cleanup: archive old logs, keep the latest N. +- never run =--fresh= from Doctor. + +** 10. Workstation contract checks =P2/rank 10= + +Problem overcome: a fresh Arch system is not the goal. The goal is Craig's +working machine: user, shell, groups, dotfiles, Emacs, Hyprland/DWM, keyring, +VPN tools, Bluetooth tools, and local scripts. + +Representation: + +- =WORKSTATION CONTRACT= band. +- Child lamps: =user=, =dotfiles=, =desktop=, =tools=. +- Evidence drawer mirrors the testinfra checks. + +Tools: + +- =id=, =getent passwd= +- =test -L ~/.zshrc= +- =stow= via dotfiles Makefile +- =pacman -Q=, =yay -Qi yay= +- =hyprctl=, =gdbus= portal checks when session is running + +Doctor: + +- safe: restow dotfiles with the selected profile. +- safe: repair keyring directory permissions. +- mitigation: reinstall missing official packages. +- AUR package rebuilds require confirmation and stream output. + +** 11. Security and hardening drift =P2/rank 12= + +Problem overcome: security settings are easy to regress because they are small +file edits: SSH root login, EFI mount masks, firewall, issue banner, fail2ban, +quiet printk. + +Representation: + +- Compact row under =WORKSTATION CONTRACT= or =PACKAGES + SERVICES=. +- Red only for high-risk drift, yellow for unknown/unreadable state. + +Tools: + +- =sshd -T= or config file checks +- =ufw status= +- =fail2ban-client status= +- =findmnt /efi= +- =sysctl kernel.printk= + +Doctor: + +- safe: restore known Archsetup-owned config snippets. +- safe: re-enable firewall if policy file is present. +- mitigation: write missing drop-ins only from version-controlled templates. +- no broad hardening rewrite from panel state. + +** 12. Backup and rollback readiness =P2/rank 11= + +Problem overcome: rollback only helps local state. The install also needs +backups of edited system files and confidence that personal data replication is +not silently stale. + +Representation: + +- Row lamp: =backups=. +- Chips: =system-file .bak=, =replication=, =last run=. +- Yellow if last replication exceeds policy. +- Red if expected backup files for edited system config are missing. + +Tools: + +- Archsetup backup assertions in =scripts/testing/tests/test_backups.py=. +- =zfs-replicate= if configured. +- =systemctl list-timers= for backup timers. +- =journalctl -u= relevant backup units. + +Doctor: + +- safe: create missing =.archsetup.bak= for files before editing. +- safe: run dry-run replication check. +- mitigation: start a configured backup timer/unit with confirmation. +- never delete backup targets from Doctor. + +* Doctor Model + +Doctor is a classifier with bounded mitigations, not a magic repair button. + +Flow: + +1. Probe the selected scope. +2. Normalize each metric to =ok=, =warn=, =fail=, =unknown=, or =running=. +3. Classify failures as: + - =safe-fix= — local, reversible, low risk. + - =safe-cleanup= — removes only known generated artifacts. + - =mitigation= — improves the chance of success but does not claim repair. + - =needs-confirmation= — mutating, long-running, or system-wide. + - =manual= — too dangerous or context-heavy for Doctor. +4. Run only safe actions automatically after the user presses Doctor. +5. Arm-first for anything mutating beyond safe local cleanup. +6. Re-run the affected probe. +7. Stream a verdict into the output well. + +Doctor should say exactly what it did: + +#+BEGIN_EXAMPLE +doctor: package db stale + check: core.db age 4d, archzfs.db age 4d + action: refreshed sync databases + action: updated archlinux-keyring + result: ok, newest db age 2m +#+END_EXAMPLE + +* Common Tool Drivers + +** Host probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| systemd | =systemctl --failed=, =systemctl is-active=, =journalctl= | +|------------+----------------------------------------------------------------| +| packages | =pacman=, =checkupdates=, =pacman-conf=, =yay= | +|------------+----------------------------------------------------------------| +| storage | =zpool=, =zfs=, =btrfs=, =df=, =findmnt=, =lsblk= | +|------------+----------------------------------------------------------------| +| boot | =bootctl=, =efibootmgr=, =mkinitcpio=, =grub-mkconfig= | +|------------+----------------------------------------------------------------| +| network | =nmcli=, =resolvectl=, =ping= or HTTPS probe | +|------------+----------------------------------------------------------------| +| desktop | =hyprctl=, =gdbus=, =loginctl=, dotfiles Makefile | +|------------+----------------------------------------------------------------| + +** Project probes + +| Area | Commands | +|------------+----------------------------------------------------------------| +| archangel | =make test=, =make build=, =build.sh --skip-aur= | +|------------+----------------------------------------------------------------| +| archsetup | =make test-unit=, =make test=, =scripts/testing/run-test.sh= | +|------------+----------------------------------------------------------------| +| VM | =qemu-img=, =qemu-system-x86_64=, =sshpass=, =pytest= | +|------------+----------------------------------------------------------------| +| artifacts | latest =out/*.log=, =out/*aur-manifest.tsv=, =test-results/*= | +|------------+----------------------------------------------------------------| + +* Top-family Comparison + +This monitor should borrow the mature display ideas from =top=-style tools +without becoming another CPU/process viewer. The domain objects are install +contracts, boot/storage health, package freshness, snapshots, services, and +artifacts. The interaction model is still the same: sort the thing that hurts, +filter to the thing you care about, expand one row for evidence, and act only +when the diagnosis is clear. + +** Comparison table + +| Tool | What it represents well | Sorting/filtering model | Useful pattern for system monitor | Gaps for our domain | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =htop= | Dense live table plus configurable meters; process tree; direct process actions. | Interactive sort by column, search, filter, tree toggle. | Metric table should support column sort, search, filter, and tree/group mode. | No historical artifact model; actions are process-centric. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =btop++= | Boxed dashboard: CPU, memory, disks, network, processes, battery, GPU; strong graph language; selected process detail. | Easy switching between process sort modes; filter; tree view; pause. | Use boxed bands, mini time-series, detail pane, pause/freeze, and clickable controls. | Graph-first layout can overemphasize volatile values over install risk. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =bottom/btm= | Custom widget layout, per-widget focus/expand, zoomable time windows, basic mode. | Process widget supports sort, search, tree; widgets can be filtered/configured. | Every health band should be expandable; stale/history windows should be zoomable. | Mostly resource telemetry, not remediation workflow. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =atop= | Interval deltas, critical-resource highlighting, all active processes including exited ones, long-term logs. | Resource views and interval replay; emphasizes deviations and active load. | Add history/replay for health events and show "new since last good" changes. | Lower immediate visual polish; Linux-performance scoped. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =Glances= | Broad plugin dashboard, thresholds, remote/web/API modes, export to JSON/CSV/time-series backends. | Configurable visible plugins; API/stdout selectors instead of only interactive sorting. | Use plugin architecture, threshold config, JSON output, remote/headless mode. | Too broad; can become a generic monitoring surface. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =procs= | Modern table ergonomics: custom columns, keyword search across selected fields, sort by named column, tree view. | CLI sort asc/desc by partial column name; watch mode cycles sort columns; AND/OR/NAND/NOR search. | Use named metric columns, saved views, multi-keyword filters, and value-aware coloring. | Process-only; no graphs or remediation model. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| +| =gotop/gtop= family | Fast glanceable terminal dashboard with compact graphs and gauges. | Usually lighter than htop/btop; less important than presentation density. | Use compact sparklines/gauges for "age", "last pass", and "failure count". | Not enough evidence/action depth for this monitor. | +|-------------------+---------------------------------------------------------------+---------------------------------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------| + +** What to pull in + +*** htop: table discipline + +Pull: + +- Column headers that are real controls: click or key-cycle to sort by + =priority=, =state=, =age=, =last_checked=, =last_pass=, =failure_count=, + =scope=, and =doctor_class=. +- Search and filter as first-class actions, not hidden debug commands. +- Tree mode for ownership: + - =system= → =boot/storage= → =bootloader= → =efi entries=. + - =archangel= → =iso build= → =aur repo= → =manifest rows=. + - =archsetup= → =state markers= → =desktop= → =dotfiles=. +- Horizontal detail access for long evidence, like htop's horizontal scrolling + for full commands. + +Equivalent-or-better requirement: + +- htop sorts processes; this monitor sorts risk. The default sort is + live-state severity, then priority rank, then age. + +*** btop++: instrument boxes and live graphs + +Pull: + +- Boxed bands with stable geometry. +- Small time-series graphs, but only where history matters: + - package DB age over time + - failed-service count + - journal error count + - snapshot count / newest snapshot age + - VM pass/fail history + - ISO build duration/result history +- Selected-row detail pane with the last command, verdict, and raw excerpt. +- Pause/freeze button so a failure does not scroll away while reading. +- Mouse-clickable controls where every visible key has the same keyboard path. + +Equivalent-or-better requirement: + +- btop's graphs answer "what is hot right now?" Our graphs answer "is the + safety margin shrinking?" Trend charts should be muted unless the threshold + is crossed. + +*** bottom: focus/expand and layout presets + +Pull: + +- Expand one band full-height: + - =BOOT + STORAGE= expands into boot files, EFI entries, pools, snapshots. + - =INSTALL PIPELINE= expands into last VM runs and build artifacts. + - =PACKAGES + SERVICES= expands into DB ages, repo status, failed units. + - =WORKSTATION CONTRACT= expands into Archsetup markers and testinfra-style + checks. +- Zoomable history windows: 24h / 7d / 30d / all artifacts. +- Layout presets: + - =compact= for bar dropdown. + - =full= for terminal/TUI. + - =host-only= for laptop health. + - =release-gate= for Archangel/Archsetup changes. + +Equivalent-or-better requirement: + +- bottom expands widgets; this monitor expands evidence and remediation state. + The expanded view must show "what changed since last good" before raw logs. + +*** atop: history and vanished failures + +Pull: + +- Permanent, compact health-event log. +- Interval deltas instead of only current values: + - new failed services since last check + - new journal errors since last check + - packages/repos newly stale + - snapshot hook present before, missing now + - bootloader file changed since last known-good +- "Show active/deviating only" mode. In normal use, hide green rows unless + their age is approaching threshold. +- Replay mode: inspect the state at the time an install/test/build failed. + +Equivalent-or-better requirement: + +- atop can report processes that already exited. This monitor should report + failures that already passed through: a transient failed unit, a VM test that + failed last night, an ISO build that failed before the current successful + build, or a package DB that was stale until Doctor fixed it. + +*** Glances: plugin/API/export model + +Pull: + +- Plugin-like probes. Each metric owns: + - =probe= + - =normalize= + - =thresholds= + - =doctor_actions= + - =redaction= + - =evidence= +- JSON output as a stable contract before GTK work. +- Optional stdout selectors: + - =system-monitor --stdout packages.state,storage.state= + - =system-monitor --json boot,snapshots= +- Remote/headless mode for VMs and bare-metal test targets. +- Threshold config in one file, not hardcoded in the UI. + +Equivalent-or-better requirement: + +- Glances is broad; this must stay opinionated. A plugin is accepted only if it + maps to install health, rollback safety, workstation contract, or recovery + readiness. + +*** procs: custom columns and query grammar + +Pull: + +- Named columns and saved views: + - =risk=: state, priority, age, doctor class. + - =install=: last pass, filesystem, artifact, branch, commit. + - =host=: state, source, last checked, command. + - =doctor=: action class, requires root, reversible, last run. +- Multi-keyword search: + - =zfs failed= + - =doctor safe-fix= + - =archangel stale= + - =service red= +- Boolean query modes: + - AND default for narrowing. + - OR for "show any boot or storage issue". + - NOT for "hide green". +- Value-aware coloring for age, severity, and units. + +Equivalent-or-better requirement: + +- procs lets the user build a process table. This monitor should let Craig + build a risk table without editing code. + +*** gotop/gtop: glance density + +Pull: + +- Small sparklines for trend, not full charts. +- Big obvious state words. +- Compact gauges for bounded values: + - EFI usage + - root/home usage + - DB age as percent of freshness window + - VM evidence age + - snapshot age +- Simple default screen that is useful without learning keys. + +Equivalent-or-better requirement: + +- The first screen should answer "am I safe to upgrade or install?" in under + two seconds. + +* Sorting and Views + +The monitor needs two sorting layers: global row ordering and per-band evidence +tables. + +** Global row ordering + +Default: + +1. =state= severity: red, yellow, unknown, running, green. +2. =priority= rank: P0 before P1 before P2. +3. =age= or =staleness=, descending. +4. =last_changed=, newest first. + +Alternate sorts: + +| Sort key | Use case | +|-----------------+------------------------------------------------------| +| =priority= | Release-gate review; keep P0/P1 at the top. | +|-----------------+------------------------------------------------------| +| =state= | Triage; show all red/yellow rows first. | +|-----------------+------------------------------------------------------| +| =age= | Find stale tests, stale package DBs, old backups. | +|-----------------+------------------------------------------------------| +| =doctor_class= | Find what Doctor can safely fix now. | +|-----------------+------------------------------------------------------| +| =scope= | Group host vs install vs build. | +|-----------------+------------------------------------------------------| +| =last_changed= | See what recently regressed. | +|-----------------+------------------------------------------------------| +| =source= | Group by archangel, archsetup, dotfiles, host. | +|-----------------+------------------------------------------------------| + +** Per-band sorts + +| Band | Sorts | +|------------------------+------------------------------------------------------------| +| =BOOT + STORAGE= | severity, mountpoint, filesystem, capacity, last scrub, newest snapshot age | +|------------------------+------------------------------------------------------------| +| =INSTALL PIPELINE= | result, filesystem, duration, artifact age, commit age, last pass | +|------------------------+------------------------------------------------------------| +| =PACKAGES + SERVICES= | severity, unit name, repo name, DB age, error count, enabled/active state | +|------------------------+------------------------------------------------------------| +| =WORKSTATION CONTRACT= | severity, check name, owner repo, last pass, doctor class | +|------------------------+------------------------------------------------------------| + +** Filters + +Quick filters should be visible as chips: + +- =red= +- =yellow= +- =doctorable= +- =needs-root= +- =stale= +- =zfs= +- =btrfs= +- =host= +- =install= +- =build= +- =changed= +- =hidden-green= + +The default view can hide healthy low-priority rows, but it must show enough +green P0/P1 summary state to prove the monitor is working. + +* Display Requirements Borrowed from Tops + +1. Every table has sortable columns and a visible sort indicator. +2. Every visible metric row has a filterable state, priority, age, and source. +3. Every row can expand to evidence without losing the list context. +4. Every graph has a threshold marker; trend without threshold is decoration. +5. Every long-running action can be paused/frozen in the display. +6. Every mutating action has an equivalent CLI command shown in the output well. +7. Every Doctor action records before/after state so fixed failures remain + visible in history. +8. Every band has a compact mode and an expanded mode. +9. Green rows are quiet; new regressions are loud. +10. The system must be useful over SSH/TUI before GTK polish. + +* Source Notes + +- =htop=: upstream README describes configurable system/process display, + interactive sorting/filtering/search, tree view, and process actions. +- =btop++=: upstream README describes resource boxes, detailed process stats, + filter, sort switching, tree view, mouse support, auto-scaling network graphs, + disk IO, battery, GPU support, and themes. +- =bottom/btm=: upstream README describes customizable widgets, process sort + and search, tree mode, expand/focus, zoomable graph intervals, filters, and + basic mode. +- =atop=: upstream README describes interval resource accounting, critical + highlighting, long-term compressed logs, exited-process visibility, cgroup + views, and active/deviation-focused output. +- =Glances=: upstream README describes plugin-style broad monitoring, web/API + modes, stdout JSON/CSV, remote monitoring, exports, and threshold-oriented + dashboard use. +- =procs=: upstream README describes configurable columns, named-column sort, + watch mode, tree view, logical keyword search, value-aware coloring, and + pager behavior. + +* Data Model + +Emit JSON from a CLI first; the panel is a client. + +#+BEGIN_SRC json +{ + "v": 1, + "scope": "host", + "state": "warn", + "ts": "2026-07-04T12:00:00-04:00", + "metrics": [ + { + "id": "packages.sync_freshness", + "label": "package databases", + "state": "warn", + "summary": "archzfs.db age 4d", + "evidence": [ + {"command": "find /var/lib/pacman/sync", "excerpt": "archzfs.db 2026-06-30"} + ], + "doctor": { + "class": "safe-fix", + "actions": ["refresh-sync-db", "update-keyring"] + } + } + ] +} +#+END_SRC + +* Implementation Notes + +- Start with a CLI: =system-monitor status --json=, =system-monitor doctor + --json=, =system-monitor refresh=. +- Keep probes read-only by default. Actions live in separate verbs. +- Cache slow probes. The bar should read a cache, not run VM tests. +- VM/build actions should create job records and stream logs; the panel follows + the job rather than blocking the UI process. +- Reuse the net/bt panel architecture if this becomes a GTK panel: GTK-free + model + fake-command unit tests + one AT-SPI smoke. +- Redact secrets from logs and JSON: WiFi PSKs, tokens, private repo URLs with + credentials, SSH material, and backup target credentials. + +* Open Decisions + +** TODO Where should the first implementation live? + +Recommendation: dotfiles owns the user-facing panel and CLI wrapper because it +is workstation UI. Archsetup owns reusable install-contract probes and testinfra +assertions. Archangel owns ISO/build probes. + +** TODO Should Doctor run elevated actions through polkit or terminal? + +Recommendation: read-only checks run unprivileged; elevated actions launch a +terminal or polkit prompt with the exact command visible. Do not hide long +privileged operations inside the panel process. + +** TODO How fresh must VM evidence be? + +Recommendation: host checks go stale after 1 hour; package DB after 48 hours; +VM install evidence after 7 days; ISO build evidence after 14 days or whenever +Archangel/Archsetup has changed since the last successful artifact. + +** TODO Which actions are allowed on bare metal? + +Recommendation: host Doctor may refresh databases, update keyring, restow +dotfiles, create snapshots, run scrub, and restart narrowly classified services. +It may not perform full upgrades, bootloader reinstalls, destructive snapshot +prune, or filesystem repair without a separate advanced flow. + +* First Build Slice + +1. CLI read-only host status: + - package DB freshness + - failed services + - ZFS/Btrfs health + - snapshot presence + - Archsetup status/log check +2. Doctor safe actions: + - refresh package DB + - update keyring + - restow dotfiles + - create manual snapshot +3. Artifact parser: + - latest Archsetup VM test result + - latest Archangel ISO build result +4. Panel prototype: + - faceplate + - four health bands + - evidence output well + - REFRESH and DOCTOR keys only |