From 2e78f55638ee80da6ce4998d37996bea73076efa Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Thu, 13 Aug 2026 22:35:41 -0500 Subject: chore(todo): velox reinstall runbook and the tasks it surfaced The runbook drove today's rebuild: preflight, a required salvage pass, the install, and post-install verification. It stays as the working record until velox is settled. Filed from the rebuild: reseating the input-cover ribbon, which is [#A] for tomorrow because phantom power-button presses have killed the machine four times and the last one cut power below systemd, where no setting can catch it. The rest are hibernate in the settings dial, a truenas VM for long-running sessions, the keyboard-backlight binding, and the screen-lock suite that fails 21 of 23 on ratio. --- todo.org | 121 +++++++++++++++- .../velox-reinstall/velox-reinstall-runbook.org | 154 +++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 working/velox-reinstall/velox-reinstall-runbook.org diff --git a/todo.org b/todo.org index d9629ba..1757a38 100644 --- a/todo.org +++ b/todo.org @@ -45,7 +45,126 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work -** TODO [#A] Move secrets out of public dotfiles → private repo + combined personal ISO :feature:security:dotfiles: +** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: +DEADLINE: <2026-08-14 Fri> +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +Machine off, lift the input cover (Framework QR-guided procedure, 5 +fasteners), reseat its ribbon connector to the mainboard — disturbed in the +2026-08-13 board swap. Root cause of every "mystery reboot" that day: +chassis flex (flash-drive touch, ethernet bump, lid partially lowered) +fired phantom power-button presses — journalctl -b -1 showed "Power key +pressed short." → orderly logind poweroff, then the glitching button +powered it back on. While in there, reseat the USB expansion cards too — +the flaky slot (two hard resets, one no-enumeration) is likely the same +flex problem. +THIRD SYMPTOM (2026-08-13 evening): touchpad delivers ZERO input events — +15s synchronized libinput debug-events capture while swiping caught +nothing, though i2c enumeration and a driver rebind handshake are clean. +Signature of a dead interrupt line on the same ribbon. Keyboard + power +LED lines work; BT mouse is the interim pointer. +ESCALATED 2026-08-13 21:00: a fourth event killed the machine THROUGH the +shield. Previous boot's journal ends mid-line (tailscaled chatter) with no +shutdown sequence at all — a hard power cut, not logind acting. So the +glitch now reaches the EC/hardware power path, which no software setting +can intercept. The reseat is the only fix, and this is a +lose-work-without-warning failure mode, not an inconvenience. +Interim shield (already live): /etc/systemd/logind.conf.d/powerkey.conf +sets HandlePowerKey=ignore — phantom presses log but do nothing; EC-level +10s hold still force-cuts. Consider keeping it even after the repair. +Verify after reseat: flex the chassis edges + partially lower the lid, then +grep the journal for new "Power key pressed" lines — zero means fixed. +Must be done before the Sunday flight — a phantom press mid-travel with the +shield on is survivable, but the connector should not be trusted at 30,000 +feet on the loose setting. +** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: +DEADLINE: <2026-08-15 Sat> +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +Mainboard swapped Intel→AMD (Ryzen AI 9 HX 370); new NVRAM has no boot entry. +Decision: full reinstall via archangel+archsetup, run deliberately as a +disaster-recovery drill before the Sunday flight. Runbook (live checklist): +[[file:working/velox-reinstall/velox-reinstall-runbook.org][working/velox-reinstall/velox-reinstall-runbook.org]] +Done 2026-08-13: ISO rebuilt (archangel-2026-08-13, archsetup baked with AMD +microcode detection, velox profiles at /root/, .ai/inbox excluded — build.sh +edits pending commit in archangel), contents verified, dotfiles swept clean of +Intel assumptions. +Finding folded in: velox's truenas backups silently stopped ~Jul 6 (newest is +DAILY.0 Jul 6; wolf.conf.gpg from Jul 29 is in NO backup). Salvage pass in the +runbook is therefore REQUIRED before partitioning, and the fresh install must +fix + verify the backup timer (runbook Phase 5). +** TODO [#B] Truenas session-host VM for long-running agent sessions :feature:tooling: +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +A small VM on truenas (TrueNAS SCALE KVM) as the home for long-running / +away-mode agent sessions. The case, per Craig 2026-08-13: truenas is the +only machine on ethernet, so network recovery after an outage is automatic +(wifi hosts may never reassociate unattended); it's UPS-backed through +blip-to-hours outages; it has the Comet KVM for out-of-band recovery; and +appliance uptime discipline means it doesn't reboot for workstation +reasons. Tonight's live demonstration of anchor-staleness risk (39 min +unlogged during a bare-metal recovery) is the motivating incident — the +session's durability equals the anchor's lag at interruption, so a host +that doesn't get interrupted is worth real money. +Costs to engineer around: a third environment to keep synced (repos, +rulesets, tailnet identity); credential provisioning — GATED on the +secrets-repo work (the [#A] secrets task above): the VM should be the +secrets bundle's second consumer after the personal ISO, not another +hand-copied key sprawl; a firm RAM carve-out so builds don't fight the +ZFS ARC; headless only — desktop-coupled sessions stay on ratio/velox. +Build deliberately AFTER the vacation, not before Sunday. +Companion idea (cheaper, complementary): put ratio on the UPS. +** TODO [#C] screen-lock test suite red on ratio :bug:test:dotfiles: +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +tests.screen-lock.test_screen_lock fails 21 of 23 (+1 error) on ratio, +verified pre-existing with unrelated changes stashed (2026-08-13). Breaks +the make test green bar for every dotfiles commit until triaged. Suspect +environmental (the suite exercises hyprlock/relaunch behavior that may +need session state this shell lacks) or a regression from a recent +screen-lock commit — diagnose, then fix or mark/skip with a reason. +Grading: Major severity (blinds the pre-commit gate for the whole repo) × +every-commit frequency on this machine, but test-infra only, no user +impact = [#C] judgment call rather than matrix-dictated. +** TODO [#C] Keyboard backlight binding + boot default :feature:dotfiles:velox: +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +Velox's kbd backlight (chromeos::kbd_backlight since the AMD board) boots +at 0 and the dotfiles carry no keyboard-brightness keybinding at all (swept +2026-08-13 — never existed). Add: Hyprland binds (XF86KbdBrightness* or a +chord) driving brightnessctl -d "chromeos::kbd_backlight", a sane boot +default, and a udev rule granting the video/input group write access so it +works without sudo (a bare ssh session got EPERM). Check whether Fn+Space +(EC-handled on Frameworks) already cycles it — if so the bind is a +complement, not the only path. Ratio: n/a (desktop). +** TODO [#B] Hibernate in the settings dial power actions :feature:dotfiles: +:PROPERTIES: +:CREATED: [2026-08-13 Thu] +:LAST_REVIEWED: 2026-08-13 +:END: +Add hibernate alongside suspend/lock in the settings module's dial power +actions. Sequencing (Craig confirmed the dial placement 2026-08-13): +1. Prove hibernate on velox first — systemctl hibernate through a real + resume; the chain (LUKS swap p3, keyfile-in-initramfs, encrypt+resume + hooks, resume= on the ZBM cmdline) went live with the 2026-08-13 + reinstall but is untested on this AMD board. +2. Then consider suspend-then-hibernate as the default lid behavior + (systemd sleep.conf HibernateDelaySec) — hibernate's savings with no + button at all; possibly a "deep sleep" toggle in the module. +3. Then the dial action itself. +Note: ratio has no swap partition, so hibernate stays velox-only until +ratio gets one; the dial entry should degrade gracefully where there's no +resume target. :PROPERTIES: :CREATED: [2026-08-11 Tue] :LAST_REVIEWED: 2026-08-11 diff --git a/working/velox-reinstall/velox-reinstall-runbook.org b/working/velox-reinstall/velox-reinstall-runbook.org new file mode 100644 index 0000000..02671e2 --- /dev/null +++ b/working/velox-reinstall/velox-reinstall-runbook.org @@ -0,0 +1,154 @@ +#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup +#+AUTHOR: Craig Jennings +#+DATE: 2026-08-13 + +Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon +890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry, +and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the +pool. Decision: full reinstall via archangel + archsetup, run deliberately as +a disaster-recovery test of the ISO and scripts before the Sunday flight. +Recent backup in hand; ratio available as the working machine. + +Fallback ordering if the test finds a real gap: +- Before partitioning starts: the old system is intact — the ZBM repair + route (efibootmgr entry pointing at the ZBM loader on the ESP, then + amd-ucode swap in a chroot) is still available. +- After partitioning: the floor is a manual Arch install; the backup makes + that survivable. + +* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk) + +- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates + the microcode vendor-detection fix (archsetup, 2026-08-08) and was built + without ARCHSETUP_DIR at all. + #+begin_src sh + cd ~/code/archangel && sudo ARCHSETUP_DIR=~/code/archsetup ./build.sh + #+end_src +- [ ] build.sh fixes before the final rebuild (archangel repo): + - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the + credential audit — off the portable USB stick). + - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile + is on the ISO at =/root/velox-zfs.conf=. +- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=), + =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs + spot-check. +- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy + drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16 + archangel ISOs removed. Boot the stick and pick + =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu. + +* Phase 1 — UEFI setup on velox (BIOS screen, before any boot) + +- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned; + the new board ships with it enforced by factory default. +- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01; + a wrong clock breaks TLS and pacman signature checks in the live env. + Rough accuracy is fine — NTP tightens it once networked. +- [ ] While you're in setup: check boot-order UI shows the USB. + +* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED + +NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is +DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke +around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio +and mybitch backed up today). Everything since Jul 6 exists only on the old +SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO +backup at all. This pass also keeps the repair fallback alive until +partitioning starts. + +- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=. +- [ ] Import the old pool read-only and unlock: + #+begin_src sh + zpool import -N -o readonly=on -R /mnt zroot + zfs load-key zroot # passphrase prompt + zfs mount zroot/ROOT/default + zfs mount -a 2>/dev/null # home datasets etc.; ignore failures + #+end_src +- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout + the backup job uses (etc + home), into a clearly-named one-off dir: + #+begin_src sh + rsync -aHAX --info=progress2 /mnt/etc /mnt/home \ + truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/ + #+end_src + (=/usr= is in the regular backups but is all reinstallable — skip unless + paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.) +- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest + files in =~/documents= and =~/downloads=. +- [ ] Check for uncommitted repo work and either push or note it: + =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge), + =~/.dotfiles=, anything under =~/code=. +- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=. + +* Phase 3 — Install (the actual DR test) + +- [ ] Review the profile, then run the installer: + #+begin_src sh + less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme + archangel --config-file /root/velox-zfs.conf + #+end_src + Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome= + placeholders — fine for install; both change post-install (=zfs change-key + zroot= for the pool, =passwd= for root). +- [ ] Record every rough edge as a DR-test finding — that's the point of + running it this way. Anything that needs a manual nudge gets a todo entry + in archangel or archsetup afterward. +- [ ] Reboot into ZBM → boot the new environment. + +* Phase 4 — archsetup (first boot of the installed system) + +- [ ] Log in as root, network up, then verify the clock synced. +- [ ] Get archsetup — two paths, test the offline one since this is a DR + drill (the online curl path is the everyday alternative): + #+begin_src sh + # offline: mount the install USB and copy the baked tree + mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt + cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup + ./archsetup + #+end_src +- [ ] Expected on the new board: =install_cpu_microcode= detects + AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests). + Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all + in the installer now — none need manual application afterward. +- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel + assumptions (swept 2026-08-13); maint's capability probe runtime-detects + amd-pstate. + +* Phase 5 — Post-install restore + verification + +- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The + secrets-bundle-in-ISO design is not built yet — manual restore is the + known gap, not a test failure. +- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=; + re-import the NM profile (autoconnect off, as before). +- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=. +- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known + idle instability (Panel Self Refresh hangs/reboots the machine; hit during + the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the + installed system's kernel command line — velox boots via ZBM, so set it on + the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default= + (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a + kernel that fixes PSR on Strix Point — track via the Framework issue + tracker (SoftwareFirmwareIssueTracker #110). +- [ ] New-hardware spot-checks: + - =journalctl -k | grep -i microcode= — amd-ucode applied. + - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect + amd-pstate(-epp). + - wifi + bluetooth up (new board radios), touchpad behavior, camera. + - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV. +- [ ] Fresh clones automatically carry the post-purge rewritten git history — + closes the clone-reconcile rider from 2026-08-11 without action. +- [ ] Fix and verify the backup timer on the fresh install — it was silently + broken since ~Jul 6. After the first manual run succeeds, confirm a new + DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it + broke (timer unit dead? mount failure? credential?) if the old journal + survives in the salvage copy. +- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate), + both daily drivers AMD. Fix the stale =intel_pstate= comment in + =airplane-mode= line 6 while at it (cosmetic). +- [ ] File every DR-test finding in the owning project's todo. + +* Timing + +Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or +Friday, leaving Saturday as pure buffer. If the install stalls past Friday +evening, cut losses to the manual-install floor. -- cgit v1.2.3