docs: log overnight code audit and file triaged installer bugs

Overnight sentry ran its hygiene passes plus a code-inspection pass over the installer and the scripts/ directory. It archived 6 completed tasks to Resolved and logged 19 bugs and 12 refactors in docs/design/2026-07-19-sentry-code-findings.org. I filed the bugs as tasks graded by severity times frequency. Nothing in the codebase changed, this was detection only.

Two things stand out. The installer does no partitioning (it's a post-install config layer), so the real data-loss surface is boot config, and it overwrites GRUB_CMDLINE wholesale, which can drop a cryptdevice or resume param and leave a machine unbootable. Three WireGuard configs with live private keys are tracked in git, flagged for a rotate-or-keep decision.

1 files changed, 174 insertions, 0 deletions

diff --git a/docs/design/2026-07-19-sentry-code-findings.org b/docs/design/2026-07-19-sentry-code-findings.org
new file mode 100644
index 0000000..3816ebc
--- /dev/null
+++ b/docs/design/2026-07-19-sentry-code-findings.org
@@ -0,0 +1,174 @@
+#+TITLE: Sentry Code Findings — 2026-07-19 overnight run
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-07-19
+
+* About
+
+Bug and refactoring findings from the overnight sentry code-inspection pass
+(Craig's request, 2026-07-19). Detection only — nothing in the codebase was
+changed. Each finding is a candidate to triage into todo.org (bug priority via
+the severity x frequency matrix; refactors as :refactor: tasks) or discard, at
+the morning sentry teardown.
+
+Scope: the archsetup repo's own code (installer, scripts/, VM test framework,
+Python, elisp). Dotfiles-side findings, if any, are tagged [dotfiles] since
+that is a separate repo.
+
+Severity legend for bugs: Critical (data loss / unbootable / security),
+Major (feature broken), Minor (edge case / wrong under rare input),
+Cosmetic. Refactors: High / Medium / Low value.
+
+* Coverage tracker
+
+Each fire takes the next uncovered slice. Mark done as fires complete.
+
+- [X] S1 — archsetup installer, lines ~1-1050 (pre-flight, config, install primitives, GPU detect) — NB partitioning/filesystem code is past line 1050, so S2/S3 cover it
+- [X] S2 — archsetup installer, lines ~1050-2100 (post-chroot system config: locale, mirrors, user, dotfiles/stow, AUR, services, snapshot tooling, WM packages). NB still NO partitioning/mkfs/mount in this range either — the disk path is in S3 or delegated to a separate tool.
+- [X] S3 — archsetup installer, lines ~2100-3124 (desktop/WM, gaming, boot_ux: mkinitcpio/GRUB, finalize). RESOLVED: archsetup does NO partitioning/mkfs/mount anywhere in the file — it is a post-install config layer; disk setup is upstream (archinstall/separate tool).
+- [X] S4 — scripts/*.sh production scripts (audit-packages, cmail-setup-finish, post-install, import-wireguard, normalize-notify-sounds, generate-palette, easyeffects, games, setup-chess)
+- [ ] S5 — scripts/testing/ framework + lib (run-test, vm-utils, validation, logging, network-diagnostics, testinfra, run-maint*, create-base-vm, cleanup, debug, setup-testing-env)
+- [ ] S6 — Python: gen_tokens + test suites (refactor lens; test-quality per testing.md)
+- [ ] S7 — elisp (gallery-widget.el)
+
+* Findings
+
+(appended per fire, newest slice last. Findings are unverified detection candidates for the morning triage — severity is the scanner's estimate, not a confirmed grade.)
+
+** S1 — archsetup installer lines 1-1050 (fire 1, 2026-07-20 00:05)
+
+*** [BUG Major] archsetup:487 — df available-space parse breaks on wrapped output
+=available_kb=$(df / | awk 'NR==2 {print $4}')= parses df's second line. df wraps to two lines when the source device name is long (common on a live ISO / device-mapper root). When wrapped, NR==2 is the device-name line and $4 is empty, so available_kb="" -> available_gb=0 -> the script wrongly aborts with "Insufficient disk space" on a disk with plenty. Fails safe (blocks) but blocks a valid install.
+Fix: =df -P / | awk 'NR==2 {print $4}'= (POSIX -P forces single-line) or =df --output=avail / | tail -1=.
+
+*** [BUG Major] archsetup:298-305, 690-695 — run_step drops the state marker on a non-fatal step failure
+run_step marks a step complete only when $step_func returns 0, but error_warn (the "non-fatal, continue" path) and run_task return 1 on failure. If a step function's last statement is a run_task/error_warn that fails, the function returns 1, so run_step prints FAILED and never writes the state marker even though the error was designated non-fatal. On resume the step re-runs every time, and =if $step_func; then= in main can abort on a survivable warning. (Full confirmation needs step bodies past line 1050.)
+Fix: step functions =return 0= explicitly after their work, or have run_step gate on a per-step error flag rather than the function's last-command status.
+
+*** [BUG Minor] archsetup:1034 — crash message reports the wrong exit code
+=$refresh_ok || error_fatal "$action" "$?" ...= — $? is the status of evaluating $refresh_ok (the command =false=), not pacman's. The crash always reports "error: 1" regardless of why =pacman -Syu= failed, discarding the real code.
+Fix: capture pacman's code inside the retry loop (last_rc=$?) and pass that.
+
+*** [BUG Minor] archsetup:290-291, 318-320 — run_step locals leak to global scope
+run_step sets step_name/step_func (and timestamp/step) without =local=. They leak globally; step_name persists across calls, a latent aliasing hazard if a step function reads a same-named global. No current failure, but fragile.
+Fix: =local step_name step_func= (and local timestamp/step).
+
+*** [BUG Minor] archsetup:488 — integer truncation biases the disk-space gate against the user
+=available_gb=$((available_kb / 1024 / 1024))= truncates. Real df "available" runs slightly under nominal, so a nominally-20GB-free disk truncates to 19 and is rejected against min_disk_space_gb=20.
+Fix: compare in KB directly (min_kb=$((min_disk_space_gb*1024*1024)); test available_kb -ge min_kb).
+
+*** [REFACTOR High] archsetup:435-455, 908-934 — GPU modalias scan duplicated
+The DRM-glob-then-PCI-fallback modalias scan is implemented twice — nvidia_preflight_report (NVIDIA-only) and install_gpu_drivers (Intel/AMD/NVIDIA) — with duplicated bc03 display-class filtering and the same =*v000010DE*|*v000010de*= casing. Two copies drift independently.
+Fix: extract one scan_gpu_modalias helper (walk DRM then PCI, echo the detected vendor set); both callers consume it.
+
+*** [REFACTOR Medium] archsetup:715-729, 762-786, 1026-1033 — bounded retry loop hand-rolled three times
+retry_install, git_install, and the pacman -Syu refresh each re-derive attempt, the <= MAX_INSTALL_RETRIES guard, and the "retrying (attempt N/M)" message. The $?-capture subtlety must be re-remembered at each site (and the refresh loop at 1034 already got it wrong — see the Minor bug).
+Fix: a =retry_cmd <desc> <max> -- cmd...= primitive that runs the loop, captures the real exit code, and emits the message.
+
+*** [REFACTOR Low] archsetup:183-198 — four near-identical yes/no validation blocks
+AUTOLOGIN, NO_GPU_DRIVERS, INSTALL_CLAUDE_CODE, INSTALL_DEVICE_UDEV_RULES each get a copy-pasted five-line yes/no check. A =validate_yesno <VARNAME> <value>= helper looped over the key list removes the duplication.
+
+*** [REFACTOR Low] archsetup:459,471,476 — NVIDIA driver floor 535 is a magic literal in three spots
+Repeated in the guidance text, the numeric comparison, and the error message. A NVIDIA_MIN_DRIVER=535 constant referenced in all three prevents drift.
+
+S1 verdict: 5 bugs (2 Major, 3 Minor), 4 refactors (1 High, 1 Medium, 2 Low).
+
+** S2 — archsetup installer lines 1050-2100 (fire 2, 2026-07-20 00:55)
+
+Global note: =set -e= is commented out at archsetup:21, so any command without an explicit =|| error_*= guard fails silently and continues. Resume granularity is the top-level STEPS array (line 262), so a re-run repeats every command inside a failed step. Both facts amplify the bugs below.
+
+*** [BUG Major] archsetup:1168 — chpasswd failure silently leaves an unloggable primary user
+=echo "$username:$password" | chpasswd= has no guard, then =unset password= runs next line. With set -e off, a chpasswd failure is swallowed; useradd made the account with no password (locked), so the installed system has an unloggable primary user and no log diagnostic. Every other privileged step guards with error_fatal/error_warn; this one doesn't.
+Fix: =... | chpasswd 2>> "$logfile" || error_fatal "setting password for $username" "$?" "set it by hand: passwd $username"= before unsetting.
+
+*** [BUG Minor] archsetup:1820-1821 — unguarded cp of zfs-replicate leaves a service pointing at a missing binary
+=cp "$user_archsetup_dir/scripts/zfs-replicate" /usr/local/bin/zfs-replicate= + chmod, no guard/redirect. The dotfiles/archsetup clone at 1231 is non-fatal; if it left no tree, the cp prints to console and continues, chmod fails, and the zfs-replicate.service (1824) has an ExecStart to a binary never installed.
+Fix: =(cp ... && chmod +x ...) >> "$logfile" 2>&1 || error_warn "installing zfs-replicate script" "$?"=.
+
+*** [BUG Minor] archsetup:1943-1944 — same unguarded cp for zfs-pre-snapshot -> broken pacman hook
+Same pattern as 1820. A missing source silently yields a pacman PreTransaction hook (written at 1953) whose Exec target doesn't exist, so every future pacman transaction logs a hook failure with no record of why.
+Fix: wrap in =(...) >> "$logfile" 2>&1 || error_warn ...=.
+
+*** [BUG Minor] archsetup:1713 — crontab log-cleanup line duplicates on resume
+=(crontab -l 2>/dev/null; echo "0 12 * * * ...log-cleanup") | crontab -= appends unconditionally. essential_services is one resume STEP; a failure after this point plus a re-run appends a second identical cron line each time.
+Fix: guard on absence — =crontab -l 2>/dev/null | grep -qF 'cron/log-cleanup' || { ...; }=.
+
+*** [BUG Minor] archsetup:1146 — blind copy of sudoers.pacnew risks a hard lockout
+=[ -f /etc/sudoers.pacnew ] && cp /etc/sudoers.pacnew /etc/sudoers= with no =visudo -c= validation. A malformed pacnew replaces /etc/sudoers with a file sudo refuses to parse, locking out privilege escalation right before the NOPASSWD rule at 1183. Low probability, hard consequence.
+Fix: =visudo -cf /etc/sudoers.pacnew && cp ... || error_warn ...=.
+
+*** [BUG Minor] archsetup:1857 — zfs scrub timer uses an arbitrary pool via head -1
+=zfs_pool=$(zpool list -H -o name | head -1)= then enables =zfs-scrub-weekly@${zfs_pool}.timer=. head -1 picks a wrong pool when several exist; an empty result yields the malformed unit =zfs-scrub-weekly@.timer=.
+Fix: derive the pool hosting the root dataset explicitly; skip enabling when empty.
+
+*** [REFACTOR Medium] archsetup:1263-1283 — stow invocation copy-pasted four times
+The stow block (=(cd "$dotfiles_dir" && stow --target=... --no-folding --adopt <pkg> >> log) || error_warn=) repeats for common/desktop_env/host_tier/minimal, differing only in package name. A flag change means four edits.
+Fix: extract =stow_pkg() { ... }= and call per package.
+
+*** [REFACTOR Low] archsetup:1901-1907 — seven snapper set-config calls, mixed guarding
+Seven consecutive =snapper -c root set-config "TIMELINE_...=N"= calls, most unguarded, mixing guarded/unguarded forms. Loop over a KEY=VALUE array with one uniform guard.
+
+*** [REFACTOR Low] archsetup:1290-1300 — waybar battery pruned with position-dependent sed on JSON
+=prune_waybar_battery= edits JSON with three positional =sed -i= including a trailing-comma fixup that assumes the battery entry's position. A reordering of the modules array silently produces invalid JSON. A jq-based edit (or a battery-less config variant in the hyprland tier) is robust.
+
+S2 verdict: 6 bugs (1 Major, 5 Minor), 3 refactors (1 Medium, 2 Low). Range is post-chroot config, not partitioning.
+
+** S3 — archsetup installer lines 2100-3124 (fire 3, 2026-07-20 02:00)
+
+RESOLVED architectural fact: archsetup performs NO disk partitioning, formatting, or mounting. Grep of all 3124 lines finds no wipefs/sgdisk/parted/mkfs/mkswap/genfstab/cryptsetup/zpool-create/zfs-create. The only mounts are a tmpfs AUR-build RAM disk (1188), a btrfs /.snapshots remount (1878-1896), and an /efi fstab permission tweak (2900-2904). The installer assumes an already-partitioned, formatted, mounted base system with a bootloader present; disk layout and encryption are done upstream (archinstall or a separate tool). archsetup is a post-install package-and-config layer that only edits the existing bootloader/initramfs. This reframes the whole risk model: the data-loss surface is boot config, not partitioning.
+
+*** [BUG Major] archsetup:3054 — GRUB_CMDLINE overwrite drops boot-critical params -> unbootable
+=sed -i "s/.*GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT=\"rw loglevel=2 ... quiet splash\"/g"= replaces the whole cmdline with a fixed string. The file sets no cryptdevice/root=ZFS/resume= anywhere, so any boot-critical param the base install placed there (encrypted-root cryptdevice=, hibernation resume=, a ZFS hint, a hardware quirk) is silently dropped, and the grub-mkconfig at 3059 then bakes an unbootable config. Highest boot-risk line in the range.
+Fix: read the existing value and append only the missing tokens; assert the result still contains any pre-existing cryptdevice/root/resume before grub-mkconfig.
+
+*** [BUG Major] archsetup:2910-2921, 2943-2948 — nvme early module never built into initramfs on ZFS-root non-Framework
+add_nvme_early_module writes =MODULES=(nvme)= into mkinitcpio.conf, but the only rebuild in boot_ux (=mkinitcpio -P= in configure_initramfs_hook) runs =if ! is_zfs_root=, and trim_firmware's rebuild is Framework-only. So on a ZFS-root NVMe machine that is not a Framework 13 — exactly the target — the MODULES edit is never compiled in. Not fatal (autoload still boots), but the intended early-load hardening silently does nothing on the systems it was written for.
+Fix: run =mkinitcpio -P= after the MODULES edit regardless of ZFS (or an unconditional rebuild at the end of boot_ux), guarded with error_warn.
+
+*** [BUG Minor] archsetup:2918 — nvme presence check scans the whole file, not the MODULES line
+=... && ! grep -q "nvme" /etc/mkinitcpio.conf= concludes nvme is present if "nvme" appears anywhere (a comment, a HOOKS entry), so the module is never inserted into MODULES.
+Fix: scope it: =! grep -qE '^MODULES=\([^)]*\bnvme\b' /etc/mkinitcpio.conf=.
+
+*** [BUG Minor] archsetup:2419 — gamemode enabled via systemctl --user, which the script knows fails at install time
+=run_task ... sudo -u "$username" systemctl --user enable gamemoded.service=. The script documents at 2333-2338 that =systemctl --user= fails during install (no user session bus) and hand-creates the symlink there; gaming() ignores that lesson, so gamemoded is likely left un-enabled (silent warn, set -e off).
+Fix: enable it the way syncthing is — manual wants-symlink under /home/$username/.config/systemd/user/, chown to the user.
+
+*** [BUG Minor] archsetup:2108, 2144 — unguarded chmod after a non-fatal cp/heredoc
+=chmod 755 /usr/local/bin/hypr-live-update-guard= and =chmod 644 .../10-hypr-live-update-guard.hook= run unguarded right after a run_task/cat that could have failed; with set -e off, the chmod hits a missing/partial file and fails silently.
+Fix: fold the chmod into the guarded step or add =|| error_warn=.
+
+*** [REFACTOR Medium] archsetup:2148-2183 — display_server() and window_manager() duplicate the same case dispatch
+Near-identical case statements over $desktop_env, including a byte-for-byte duplicated =error_fatal "Unknown DESKTOP_ENV..."= default arm. Extract one dispatch helper, or validate $desktop_env once up front.
+
+*** [REFACTOR Low] archsetup:2952-2963 — identical fsck drop-in heredoc written twice
+The same =[Service] StandardOutput=null StandardError=journal+console= drop-in goes to systemd-fsck-root.service.d and systemd-fsck@.service.d. Write once to a variable, loop over the two dirs.
+
+*** [REFACTOR Low] archsetup:3049-3054 — five unguarded GRUB sed lines
+Five =sed -i "s/.*GRUB_.*=.*/.../g"= lines, all unguarded, each silently no-ops or mis-edits. Drive from a key->value table in a loop, add error_warn.
+
+S3 verdict: 5 bugs (2 Major, 3 Minor), 3 refactors (1 Medium, 2 Low). Installer fully covered (S1-S3): 16 bugs, 10 refactors total.
+
+** S4 — scripts/*.sh production scripts (fire 4, 2026-07-20 03:05)
+
+Most scripts are solid (SHA-pinned downloads, atomic mv, trap cleanup, guarded rm). Findings:
+
+*** [BUG Major] scripts/cmail-setup-finish.sh:52-54 — mail password decrypted world-readable before chmod
+=gpg --decrypt --output "$cmailpass_plain" "$cmailpass_enc"= then =chmod 600 "$cmailpass_plain"= on the next line. gpg creates the file at the process umask (typically 0644), so the plaintext mail password is world-readable in the window between decrypt and chmod. import-wireguard-configs.sh uses the safe pattern (mktemp -d 0700 first) this should mirror.
+Fix: =(umask 077; gpg ... --output "$cmailpass_plain" ...)= or decrypt into a mktemp 0600 file and mv into place.
+
+*** [BUG Minor] scripts/import-wireguard-configs.sh:51-62 — a failed modify leaves the full-tunnel VPN live
+=nmcli connection import= brings the 0.0.0.0/0 tunnel up immediately; the script renames then =down=s it. Under set -e, if the modify (58) fails, the script aborts before the down (62), leaving a live full-tunnel VPN routing all traffic through Proton — the exact "unasked-for VPN" the header says it prevents. The next-run guard catches the leftover profile but not the still-active tunnel.
+Fix: bring the connection down right after parsing the UUID, before the rename.
+
+*** [BUG Minor] scripts/normalize-notify-sounds.sh:39-46 — no temp trap + non-atomic write can corrupt a tracked repo file
+=tmp=$(mktemp ...); ffmpeg ... "$tmp"; cat "$tmp" > "$f"; rm -f "$tmp"=. No EXIT trap (a mid-loop ffmpeg failure orphans the temp), and =cat "$tmp" > "$f"= truncates first, so a zero-byte/failed encode writes a corrupt/empty file. Since $f is a stow symlink into the repo, that corruption lands in the tracked file with no rollback.
+Fix: EXIT trap to remove temp; =[ -s "$tmp" ]= before the write-through; write to $f.tmp and overwrite on success.
+
+*** [REFACTOR Low] scripts/setup-chess.sh:354,377,489 — Maia ELO range spelled out three times
+=for elo in 1100 1200 ... 1900= at 354 and 377, =range(1100,2000,100)= at 489, plus the MAIA_SHA256 map. Adding/dropping a level means four edits. Define one MAIA_ELOS array.
+
+*** [REFACTOR Low] multiple — per-script log-helper boilerplate re-defined
+info/ok/skip/warn/err printf helpers re-defined in setup-chess.sh:61-65, cmail-setup-finish.sh:29-31, and ad hoc elsewhere. A sourced lib/log.sh would dedupe, but these are standalone installers so it is defensible. Low priority.
+
+Per-file: audit-packages clean; cmail 1 (Major); post-install clean (deliberate best-effort); import-wireguard 1 (Minor, key handling itself correct); normalize-notify 1 (Minor); games clean; setup-chess 2 refactors; generate-palette clean; easyeffects clean.
+
+S4 verdict: 3 bugs (1 Major, 2 Minor), 2 refactors (2 Low). Plus a SECURITY item escalated to the approval queue (tracked WireGuard private keys).