diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/2026-08-13-velox-reinstall-runbook.org | 184 | ||||
| -rw-r--r-- | docs/2026-08-15-velox-uefi-boot-entry-reference.org | 108 | ||||
| -rw-r--r-- | docs/design/2026-07-15-velox-boot-failure-handoff.org | 61 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-1.org | 137 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-2.org | 63 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-3.org | 98 | ||||
| -rw-r--r-- | docs/workflows/system-health-check.org | 4 |
7 files changed, 653 insertions, 2 deletions
diff --git a/docs/2026-08-13-velox-reinstall-runbook.org b/docs/2026-08-13-velox-reinstall-runbook.org new file mode 100644 index 0000000..2d99fc9 --- /dev/null +++ b/docs/2026-08-13-velox-reinstall-runbook.org @@ -0,0 +1,184 @@ +#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup +#+AUTHOR: Craig Jennings +#+DATE: 2026-08-13 + +Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon +890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry, +and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the +pool. Decision: full reinstall via archangel + archsetup, run deliberately as +a disaster-recovery test of the ISO and scripts before the Sunday flight. +Recent backup in hand; ratio available as the working machine. + +* Outcome (recorded 2026-09-13 Sun) + +The drill ran on 2026-08-13 and 14 and velox came back as a working daily +driver: fresh install from the archangel ISO, keys and data restored from +the salvage backup, 23 repos re-cloned, rsyncshot reinstalled, hibernate +proven end to end. The checklist below was the live plan; it was not ticked +as the phases ran, so read it as the plan, not a log of each step. + +What the drill found, each filed as its own task rather than fixed in place: +velox's truenas backups had silently stopped on 2026-07-06 (found on 08-13 +before partitioning, which is what made the salvage pass required); four phantom +reboots were a ribbon disturbed by the board swap; a fresh install never +clones rulesets, never links the .emacs.d systemd user units, ships no +brightness udev rule, and loses gcalcli and the signal-cli registration. +Those live in archsetup's todo as the post-rebuild verification pass and its +siblings. The one commit that existed only on the old disk (emacs-wttrin +bf0457f) was rescued as a bundle and has its own task. + +Companion documents: the UEFI boot-entry recovery reference +([[file:2026-08-15-velox-uefi-boot-entry-reference.org][2026-08-15-velox-uefi-boot-entry-reference.org]]) +and the three gap reports under docs/design (2026-08-14-velox-reinstall-gaps-1 +to 3). + +Fallback ordering if the test finds a real gap: +- Before partitioning starts: the old system is intact — the ZBM repair + route (efibootmgr entry pointing at the ZBM loader on the ESP, then + amd-ucode swap in a chroot) is still available. +- After partitioning: the floor is a manual Arch install; the backup makes + that survivable. + +* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk) + +- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates + the microcode vendor-detection fix (archsetup, 2026-08-08) and was built + without ARCHSETUP_DIR at all. + #+begin_src sh + cd ~/code/archangel && sudo ARCHSETUP_DIR="$HOME/code/archsetup" ./build.sh + #+end_src + Two traps in that one line, and either alone silently produces a bare ISO + with archsetup absent (archangel, 2026-08-20). The =VAR=value= form is + required because sudo's =env_reset= discards an exported variable. And + =$HOME= is required because zsh does not expand a tilde on the right-hand + side of an assignment — the earlier =ARCHSETUP_DIR=~/code/archsetup= here + passed the literal string. build.sh now warns and reports baked/not-baked + in its closing summary, so the failure is visible rather than silent. +- [ ] build.sh fixes before the final rebuild (archangel repo): + - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the + credential audit — off the portable USB stick). + - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile + is on the ISO at =/root/velox-zfs.conf=. +- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=), + =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs + spot-check. +- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy + drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16 + archangel ISOs removed. Boot the stick and pick + =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu. + +* Phase 1 — UEFI setup on velox (BIOS screen, before any boot) + +- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned; + the new board ships with it enforced by factory default. +- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01; + a wrong clock breaks TLS and pacman signature checks in the live env. + Rough accuracy is fine — NTP tightens it once networked. +- [ ] While you're in setup: check boot-order UI shows the USB. + +* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED + +NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is +DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke +around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio +and mybitch backed up today). Everything since Jul 6 exists only on the old +SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO +backup at all. This pass also keeps the repair fallback alive until +partitioning starts. + +- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=. +- [ ] Import the old pool read-only and unlock: + #+begin_src sh + zpool import -N -o readonly=on -R /mnt zroot + zfs load-key zroot # passphrase prompt + zfs mount zroot/ROOT/default + zfs mount -a 2>/dev/null # home datasets etc.; ignore failures + #+end_src +- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout + the backup job uses (etc + home), into a clearly-named one-off dir: + #+begin_src sh + rsync -aHAX --info=progress2 /mnt/etc /mnt/home \ + truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/ + #+end_src + (=/usr= is in the regular backups but is all reinstallable — skip unless + paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.) +- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest + files in =~/documents= and =~/downloads=. +- [ ] Check for uncommitted repo work and either push or note it: + =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge), + =~/.dotfiles=, anything under =~/code=. +- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=. + +* Phase 3 — Install (the actual DR test) + +- [ ] Review the profile, then run the installer: + #+begin_src sh + less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme + archangel --config-file /root/velox-zfs.conf + #+end_src + Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome= + placeholders — fine for install; both change post-install (=zfs change-key + zroot= for the pool, =passwd= for root). +- [ ] Record every rough edge as a DR-test finding — that's the point of + running it this way. Anything that needs a manual nudge gets a todo entry + in archangel or archsetup afterward. +- [ ] Reboot into ZBM → boot the new environment. + +* Phase 4 — archsetup (first boot of the installed system) + +- [ ] Log in as root, network up, then verify the clock synced. +- [ ] Get archsetup — two paths, test the offline one since this is a DR + drill (the online curl path is the everyday alternative): + #+begin_src sh + # offline: mount the install USB and copy the baked tree + mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt + cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup + ./archsetup + #+end_src +- [ ] Expected on the new board: =install_cpu_microcode= detects + AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests). + Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all + in the installer now — none need manual application afterward. +- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel + assumptions (swept 2026-08-13); maint's capability probe runtime-detects + amd-pstate. + +* Phase 5 — Post-install restore + verification + +- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The + secrets-bundle-in-ISO design is not built yet — manual restore is the + known gap, not a test failure. +- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=; + re-import the NM profile (autoconnect off, as before). +- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=. +- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known + idle instability (Panel Self Refresh hangs/reboots the machine; hit during + the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the + installed system's kernel command line — velox boots via ZBM, so set it on + the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default= + (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a + kernel that fixes PSR on Strix Point — track via the Framework issue + tracker (SoftwareFirmwareIssueTracker #110). +- [ ] New-hardware spot-checks: + - =journalctl -k | grep -i microcode= — amd-ucode applied. + - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect + amd-pstate(-epp). + - wifi + bluetooth up (new board radios), touchpad behavior, camera. + - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV. +- [ ] Fresh clones automatically carry the post-purge rewritten git history — + closes the clone-reconcile rider from 2026-08-11 without action. +- [ ] Fix and verify the backup timer on the fresh install — it was silently + broken since ~Jul 6. After the first manual run succeeds, confirm a new + DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it + broke (timer unit dead? mount failure? credential?) if the old journal + survives in the salvage copy. +- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate), + both daily drivers AMD. Fix the stale =intel_pstate= comment in + =airplane-mode= line 6 while at it (cosmetic). +- [ ] File every DR-test finding in the owning project's todo. + +* Timing + +Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or +Friday, leaving Saturday as pure buffer. If the install stalls past Friday +evening, cut losses to the manual-install floor. diff --git a/docs/2026-08-15-velox-uefi-boot-entry-reference.org b/docs/2026-08-15-velox-uefi-boot-entry-reference.org new file mode 100644 index 0000000..1eacfe5 --- /dev/null +++ b/docs/2026-08-15-velox-uefi-boot-entry-reference.org @@ -0,0 +1,108 @@ +#+TITLE: Velox UEFI Boot Entry — Recovery Reference +#+AUTHOR: Craig Jennings +#+DATE: 2026-08-15 + +Captured 2026-08-15 before a BIOS update (03.05 → 04.02) as insurance against +the update clearing NVRAM. Velox's mainboard swap on 2026-08-13 left exactly +this kind of empty NVRAM, which is what forced the reinstall — so a cleared +boot entry is the specific failure worth being able to undo in one command +rather than reconstruct. + +* Before recreating anything: check Secure Boot first + +The 04.02 update landed on 2026-09-12 (staged via fwupd, flashed on the next +reboot). It did NOT clear NVRAM: Boot0001 survived with its command line +intact. What it did was re-enable Secure Boot (Enforce Secure Boot = +Enabled), so the unsigned ZBM loader was rejected and the Framework BIOS +reported it as "Default Boot Device Missing / no bootable drive" rather than +a security violation. The symptom is indistinguishable from the NVRAM wipe +this document was written for. + +The tell: booting the Ventoy stick shows shim's MOK management screen. + +Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. The +machine then boots straight into ZBM. No efibootmgr needed. + +So when velox says no bootable device after a firmware update, check Secure +Boot before touching the boot entries. Only if Secure Boot is already off +and =efibootmgr -v= (from the stick) shows Boot0001 gone does the recreate +below apply. + +Before any future firmware update, record both =efibootmgr -v= and the +Secure Boot state so the post-reboot diagnosis is a comparison, not a guess. +The pre-firmware-update checklist in +[[file:workflows/system-health-check.org][docs/workflows/system-health-check.org]] +(Phase 3) carries the steps. + +Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT= +does not exist), so a real NVRAM wipe would still need the stick. Copying +=zfsbootmenu.efi= to =/efi/EFI/BOOT/BOOTX64.EFI= would let it boot unaided. +That belongs to archangel's ZBM install, where it is filed as [#C] +"Installed systems have no removable-media boot fallback on the ESP" +(2026-09-12) and ships with the next ISO rebuild after it lands. + +* State at capture + +- BIOS: 03.05 (2025-10-30) +- BootCurrent: 0001 +- BootOrder: 2001,0001,2002,2003 (USB ahead of ZBM — why the Ventoy stick + boots when it's inserted) +- Timeout: 0 seconds + +* The entry that matters + +=Boot0001* ZFSBootMenu= + +| field | value | +|----------------+----------------------------------------------| +| ESP part GUID | 8e51b680-f90a-444f-8da5-7e4f93625775 | +|----------------+----------------------------------------------| +| partition | 1 (GPT), start 0x800, size 0x100000 | +|----------------+----------------------------------------------| +| loader path | =\EFI\ZBM\zfsbootmenu.efi= | +|----------------+----------------------------------------------| +| cmdline (data) | =spl_hostid=0x22f8a7a1 zbm.timeout=3= | +| | =zbm.prefer=zroot zbm.import_policy=hostid= | +|----------------+----------------------------------------------| + +The =data= field is that command line in UTF-16LE, which is how efibootmgr +passes it as optional data. Recreate with =-u= and the plain string; efibootmgr +does the encoding. + +* Recreating it + +From a booted system (or the archangel ISO), with the ESP identified as +=/dev/nvme0n1p1= or whatever it enumerates as: + +#+begin_src bash +efibootmgr --create \ + --disk /dev/nvme0n1 --part 1 \ + --label "ZFSBootMenu" \ + --loader '\EFI\ZBM\zfsbootmenu.efi' \ + --unicode 'spl_hostid=0x22f8a7a1 zbm.timeout=3 zbm.prefer=zroot zbm.import_policy=hostid' +#+end_src + +Confirm the disk/part against =lsblk -o NAME,PARTUUID,PARTTYPENAME= first — +the partition GUID above is the authoritative identifier, not the device name, +which can enumerate differently. + +Then set the order so ZBM is reachable: + +#+begin_src bash +efibootmgr --bootorder 0001,2001,2002,2003 +#+end_src + +(The original order put USB first. Keep whichever you prefer; what matters is +that the ZBM entry exists and is in the list.) + +* Other entries (firmware-generated, recreate themselves) + +| Boot2001 | EFI USB Device | +|----------+----------------| +| Boot2002 | EFI DVD/CDROM | +|----------+----------------| +| Boot2003 | EFI Network | +|----------+----------------| + +These are stock firmware entries and come back on their own. Only Boot0001 +carries anything unique. diff --git a/docs/design/2026-07-15-velox-boot-failure-handoff.org b/docs/design/2026-07-15-velox-boot-failure-handoff.org new file mode 100644 index 0000000..5ec996f --- /dev/null +++ b/docs/design/2026-07-15-velox-boot-failure-handoff.org @@ -0,0 +1,61 @@ +#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-15 + +* Why this is coming to archsetup + +Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below. + +A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox. + +* Timeline + +- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50). +- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times). +- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel. +- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session. + +* Facts established so far (from the ZBM recovery shell) + +- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem. +- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective. +- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be. +- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone. + +* Working hypothesis + +A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent. + +* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell) + +1. Read pacman's log (on the zroot/var dataset): + #+begin_src sh + mkdir -p /mnt/var + mount -t zfs -o zfsutil zroot/var /mnt/var + tail -60 /mnt/var/log/pacman.log + #+end_src + Expect the failed/interrupted kernel transaction near the end; note its timestamp. +2. List recovery candidates: + #+begin_src sh + zfs list -t snapshot zroot/ROOT/default | tail -20 + #+end_src + Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox. + +* Recovery plan (agreed with Craig, pending the log read) + +1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction. +2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot. +3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched). +4. zpool export zroot, reboot — ZBM should now see the kernel. +5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown. + +* Retrospective candidates for archsetup + +- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion. +- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically. +- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above. +- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors. + +* Related loose end already in your inbox + +A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-1.org b/docs/design/2026-08-14-velox-reinstall-gaps-1.org new file mode 100644 index 0000000..cf0d723 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-1.org @@ -0,0 +1,137 @@ +#+TITLE: What the velox reinstall left behind — four gaps the install could close +#+AUTHOR: Craig Jennings + +* Heads-up: this was found from a .emacs.d session + +I opened a .emacs.d session on velox this morning, two days after the fresh +Arch install, and the first thing it did was fail: there was no =.ai/= +directory to read. Chasing that turned up four separate things the reinstall +did not restore. Three I repaired from the session; one needs me at my phone. + +None of this is a .emacs.d bug. They are all install-side gaps, which is why +they are landing in your inbox. Machine is velox; ratio was the reference for +every comparison below. + +* Gap 1 — the gitignored tooling layer does not survive a reinstall + +=~/.emacs.d= was re-cloned on 2026-08-13. Git brought back every tracked file +and none of the agent tooling, because =.gitignore= deliberately excludes it: +=.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, and =inbox/= were all simply +absent. That is the correct ignore policy — this repo relays to a public +mirror — but it means a reinstall silently drops the entire working state of +every gitignore-mode project. + +The damage on velox was total rather than partial: 374 files, 4.5 MB, +including =todo.org= (556 KB) and 184 archived session files. Nothing carries +it. Not git, not stow, not the bootstrap. + +I recovered it by rsyncing the set from ratio over the tailnet. Ratio was +authoritative and velox held nothing, so there was no merge to adjudicate — +which is luck, not design. Had velox held a few days of divergent state, this +would have been a hand reconciliation. It has been one before: 2026-07-31, when +the two machines' =.ai/= trees had forked to zero files in common. + +Worth knowing: this is fleet-general. Every project on the box that gitignores +its =.ai/= has the same hole, not just =.emacs.d=. + +What the install could do: after cloning a project, check whether a sibling +daily driver holds a =.ai/= for it, and offer to pull it across. Or at minimum, +list the projects whose tooling layer is missing so the gap is visible on day +one instead of at the first session that trips over it. + +* Gap 2 — stowed user timers come back linked but not enabled + +The unit files all arrived correctly through the dotfiles stow, symlinked into +=~/.config/systemd/user/= and resolving fine. But being present is not being +enabled, and the reinstall enabled only some of them: + +| unit | velox after reinstall | ratio | +|---------------------------+-----------------------+----------| +| calendar-sync.timer | enabled, active | enabled | +| agenda-render-cache.timer | enabled, active | enabled | +| roam-sync.timer | *linked, inactive* | enabled | +| signal-receive.timer | *linked, inactive* | enabled | +| emacs.service | linked, inactive | linked | + +=emacs.service= reads the same on both machines, so I take that one as +intentional and left it alone. The other two are real drift: =systemctl --user +enable= writes a =timers.target.wants= symlink into =~/.config/systemd/user/=, +and that symlink is not stow-managed, so nothing in the dotfiles repo carries +it. A stowed unit file is inert until something enables it. + +I enabled both with =systemctl --user enable --now=. Both fired immediately and +exited clean, and both now show a next elapse. + +What the install could do: enable the units it stows, explicitly, as a named +step. The inconsistency is the tell — two of four came back enabled, which +suggests something enables a subset and nothing enumerates the rest. + +* Gap 3 — the roam clone was stale, and held a diff that would have destroyed data + +This one has an ordering constraint, so it matters more than its size suggests. + +velox's =~/org/roam= was ten commits behind ratio, stuck at the 2026-08-04 +auto-sync while ratio was at 2026-08-14 — a direct consequence of gap 2, since +=roam-sync.timer= was never enabled here. + +The dangerous part: velox's clone also carried an *uncommitted* =inbox.org= +that had been emptied. Seventeen deletions, file down to zero bytes, holding a +pre-2026-08-04 state whose captures were long since processed on ratio. + +So the naive repair — enable =roam-sync.timer= and let it catch up — would have +committed that emptying and pushed it, deleting the four live inbox items on +ratio. The timer is the repo's only committer and it commits whatever it finds. + +I checked ratio's =inbox.org= first and confirmed it was a strict superset of +velox's HEAD version (same three items plus an 2026-08-09 capture), which made +the local change provably worthless. Then discarded it, fast-forwarded to +=a411b43=, and only then enabled the timer. Clone is clean and current, first +sync ran green. + +What the install could do: if it ever enables =roam-sync= on a rebuilt machine, +reconcile the clone *before* enabling, not after. An auto-committing timer +pointed at a stale dirty clone is a data-loss path, and the failure is silent +and remote — it lands on the *other* machine. + +* Gap 4 — signal-cli lost its registration, and that breaks the whole fleet + +=signal-receive.service= ran for the first time and reported: + +: signal-receive: +15045173983 not registered on this machine — nothing to do + +velox's signal-cli data dir holds a 39-byte empty =accounts.json=. Ratio still +has both numbers. So the reinstall wiped the registration, and per the design +notes velox was supposed to be the *primary* — ratio is the linked device. + +The effect is wider than velox, because of how =agent-text= dispatches: if the +local signal-cli holds the account it sends directly, otherwise it ssh-relays to +a hardcoded velox. Velox no longer holds it, so a send from here relays to +itself and fails; a send from any third machine relays to velox and fails the +same way. Only ratio still works, and only via the direct branch. The error text +blames "velox down or unreachable", which is misleading — velox is up and on the +tailnet, it just is not registered. + +This is the one I could not repair from the session: re-linking needs me at my +phone (Signal → Settings → Linked Devices, scanning the QR from =signal-cli +link -n velox=). Filed in .emacs.d's todo.org as [#B]. + +What the install could do: verify =signal-cli listAccounts= is non-empty after a +rebuild and say so loudly if it is not. Silent loss of the phone channel is +exactly the kind of thing nobody notices until the page that mattered never +arrives. + +* Summary of what I changed on velox + +- Restored =.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, =inbox/= to + =~/.emacs.d= by rsync from ratio. +- Discarded the stale local =inbox.org= diff in =~/org/roam= and fast-forwarded + the clone to current. +- Enabled and started =roam-sync.timer= and =signal-receive.timer=. + +Left alone, deliberately: =emacs.service= (matches ratio), and velox's Signal +registration (needs the phone). + +One unrelated thing I noticed while comparing the machines: ratio's signal-cli +warns its messages were last received twelve days ago, even though its +=signal-receive.timer= is enabled and active. That may be nothing, but the +receive cadence there is worth a look. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-2.org b/docs/design/2026-08-14-velox-reinstall-gaps-2.org new file mode 100644 index 0000000..95842ac --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-2.org @@ -0,0 +1,63 @@ +#+TITLE: Fifth reinstall gap — machine-local .local.el config, and a general shape +#+AUTHOR: Craig Jennings + +* Follow-up to this morning's handoff + +Sent you four gaps an hour ago +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]]). Here is a fifth, +found straight afterwards when I noticed calendar sync was dead on velox. + +* What was broken + +=calendar-sync.timer= was enabled and firing every fifteen minutes, and failing +every time with exit 255: + +: calendar-sync: No calendars configured (set calendar-sync-calendars) + +The three output files sat at zero bytes. The cause is that +=~/.emacs.d/calendar-sync.local.el= is gitignored, so the reinstall deleted it +along with everything else untracked, and the module's loader treats a missing +file as a *silent* no-op. So the config vanished quietly and the only symptom +was a failing unit nobody was watching. + +Cheap to fix once found: the repo tracks =calendar-sync.local.el.example=, and +that template already encodes the shape velox uses — feeds resolved by +=:secret-host= against =authinfo.gpg= rather than inlined. The authinfo entries +had survived, because =~/.authinfo.gpg= is a stow symlink into the dotfiles repo. +So rebuilding was one copy, and all three feeds now sync clean and land +byte-identical to ratio's. + +* The general shape, which is the part worth acting on + +This is the same failure as gap 1, one layer down, and it is worth stating +generally because the install can act on it: + +- A tracked =*.local.el.example= template plus a gitignored =*.local.el= is a + deliberate pattern in this config, not a one-off. =.gitignore= lines 56-58 + list three of them: =calendar-sync.local.el=, =signal-config.local.el=, + =google-keep.local.el=. Every one of those is gone on velox right now. I have + only repaired the calendar one. +- Secrets held *by reference* survive a rebuild; secrets held *inline* do not. + The calendar config came back for free because the tokens were in + =authinfo.gpg=, which is stow-managed and therefore travels. Ratio's copy of + the same file inlines its URLs, and had ratio been the machine rebuilt, those + three feed tokens would simply have been gone. +- The failure was silent by design. A missing local config is a no-op, which is + right for a machine that never configured the feature and wrong for one that + just lost it. + +* What the install could do + +- After a rebuild, enumerate every tracked =*.local.el.example= in a project and + report which have no corresponding =*.local.el=. That is a one-line find and it + turns a silent no-op into a visible checklist item. +- Same for any =*.local.*= convention elsewhere in the fleet — the pattern is not + specific to Emacs. +- Worth pairing with gap 2: a unit that is enabled and failing every fifteen + minutes for two days is its own signal. A post-rebuild pass over + =systemctl --user list-units --state=failed= would have caught this one + without knowing anything about calendars. + +That last one generalizes best. Of the five gaps I have sent you, three were +things that *looked* fine — a stowed unit file, an enabled timer, a present +clone — and were not. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-3.org b/docs/design/2026-08-14-velox-reinstall-gaps-3.org new file mode 100644 index 0000000..9675973 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-3.org @@ -0,0 +1,98 @@ +#+TITLE: Reinstall gaps, part three — per-install certs and credentials, and one failure that hid the others +#+AUTHOR: Craig Jennings + +* Third handoff today + +Two earlier notes covered five gaps +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]] and +[[file:2026-08-14-velox-reinstall-gaps-2.org][the follow-up]]). +Email was the last thing broken on velox after the 2026-08-13 rebuild, and it +turned up two more — both the same shape, and one of them with a property worth +generalizing. + +Email is fully working now: three accounts, 21,853 messages, 4.0 GB indexed. + +* Gap 6 — the Proton Bridge TLS cert is per-install, and its absence disabled every account + +=~/.mbsyncrc= carries =CertificateFile /home/cjennings/.config/protonbridge.pem=. +That file did not exist after the rebuild, and it cannot be restored from backup +or copied from the other machine: Proton Bridge generates a fresh self-signed +cert per installation. Velox's is issued 2026-08-13 23:44 with a different +fingerprint from ratio's 2026-01-30 one. + +*The part worth acting on is the blast radius.* mbsync parses its entire config +before doing any work, so a missing =CertificateFile= referenced by *one* account +aborts the run for *all* of them. Gmail and dmail need no bridge and no cert, and +both were dead anyway. The error names only the missing pem, so the symptom +("no mail at all") and the message ("this one file is missing") look unrelated. + +Recovery does not need the bridge GUI. The running bridge presents the cert on +its own IMAP port, so it can be pulled straight off the handshake: + +: openssl s_client -connect 127.0.0.1:1143 -starttls imap -showcerts </dev/null \ +: | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > ~/.config/protonbridge.pem + +That is a two-second, fully scriptable step, which makes it a good candidate for +the install rather than a runbook line. + +* Gap 7 — the bridge password is per-install too, and reports a stale value misleadingly + +=~/.mbsyncrc= resolves the cmail password with =cat ~/.config/.cmailpass=. That +file is plaintext and, unusually for my setup, a real file rather than a stow +symlink — so it is not in the dotfiles repo, not encrypted, and not carried to a +new machine. + +The file survived the rebuild but held the *previous* install's password, because +the bridge regenerates it per installation. Ratio's and velox's differ by sha256, +confirmed today. + +*The diagnostic trap:* Proton Bridge answers a wrong password with =no such +user=. I read that as "the bridge has no account signed in" and went looking for +a login problem. The account was configured the whole time. If the install ever +validates bridge connectivity, it should not treat =no such user= as evidence +about account state. + +* The generalization + +Gaps 6 and 7 are the same as 1 through 5, sharpened. Everything that broke in +this rebuild was *generated on the machine by an application* rather than carried +by git, stow, or the dotfiles repo: + +| gap | artifact | why it did not travel | +| 1 | =.ai/=, =todo.org=, =CLAUDE.md= | gitignored | +| 2 | =timers.target.wants= symlinks | written by systemctl enable | +| 3 | roam clone state | local working tree | +| 4 | signal-cli registration | per-device identity | +| 5 | =*.local.el= configs | gitignored | +| 6 | bridge TLS cert | per-install, regenerated | +| 7 | bridge password | per-install, regenerated | + +Gaps 6 and 7 add a distinction the earlier note missed. For 1, 3 and 5 the old +value is still correct, so *restoring* fixes them. For 4, 6 and 7 the old value is +*worthless* — the application has generated a new one, and only *re-deriving* +from the live system fixes them. An install that tries to restore these will +produce exactly what happened here: a file that exists, looks right, and +authenticates against nothing. + +So the install's post-rebuild checklist wants two columns, not one: what to +restore, and what to re-derive. + +* What the install could do + +- Re-derive the bridge cert from the running bridge with the =openssl s_client= + line above. Scriptable, no GUI, no secrets. +- Re-derive the bridge password from the bridge rather than expecting the file to + be right, and rewrite =.cmailpass=. (I have filed a task on my side to make + =PassCmd= ask the bridge directly, which would remove the file entirely.) +- Add a cheap post-rebuild validation that =mbsync --list= parses. Config-parse + failures disable every account at once and say nothing about mail, so they are + worth catching explicitly rather than via "no new mail" hours later. +- More generally: keep the restore list and the re-derive list separate, per the + table above. + +* Unrelated, but noticed while comparing the machines + +=~/.config/.gmailpass.gpg= and =~/.config/.dmailpass.gpg= resolve to mode 777 in +the dotfiles repo, on both machines. They are gpg-encrypted so the contents are +safe, but world-writable is wrong for a credential file. That is a dotfiles fix, +not an archsetup one — noting it here only because it surfaced in the same pass. diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org index 777e724..43aeff5 100644 --- a/docs/workflows/system-health-check.org +++ b/docs/workflows/system-health-check.org @@ -244,7 +244,7 @@ Firmware stays a manual step (=topgrade.toml= keeps =[firmware] upgrade = false= 1. =sudo efibootmgr -v= — every boot entry with its loader path and command line, pasted into the session's context file. 2. Secure Boot state — =bootctl status 2>/dev/null | grep -i 'secure boot'=. -After the flash, if the machine reports no bootable device, check Secure Boot *first*. The Framework 04.02 update on velox re-enabled it, which rejects the unsigned ZFSBootMenu loader and reads as "Default Boot Device Missing" rather than a security violation; the boot entries were untouched (see the Known Issues Log, 2026-09-12). Only when Secure Boot is off and =efibootmgr -v= from a stick shows the entry gone does the boot-entry recreate apply (velox: =working/velox-reinstall/velox-uefi-boot-entry-reference.org= in archsetup). +After the flash, if the machine reports no bootable device, check Secure Boot *first*. The Framework 04.02 update on velox re-enabled it, which rejects the unsigned ZFSBootMenu loader and reads as "Default Boot Device Missing" rather than a security violation; the boot entries were untouched (see the Known Issues Log, 2026-09-12). Only when Secure Boot is off and =efibootmgr -v= from a stick shows the entry gone does the boot-entry recreate apply (velox: =docs/2026-08-15-velox-uefi-boot-entry-reference.org= in archsetup). *** Two-Stage Reboot Pattern (MANDATORY if Phase 3 installed kernel / iproute2 / systemd / NetworkManager) @@ -1059,7 +1059,7 @@ Each entry is scoped to one host (or =any=). When Phase 1 cross-references findi ** 2026-09-12: any — fwupdmgr activates passim, a public LAN listener :host: any - Symptom: running =fwupdmgr= (refresh, update) D-Bus-activates =passim.service=, fwupd's LAN metadata-sharing daemon, which listens on =0.0.0.0:27500= and trips the maint listeners check to crit. -- The unit is static (no =[Install]= section), so =systemctl disable= is a no-op and it comes back on the next fwupdmgr run. Masking is what holds: =systemctl mask passim.service=. Ratio has been masked since 2026-07-21; velox was only stopped and disabled on 2026-09-12 and still needs the mask. The installer masks it as part of installing fwupd. =P2pPolicy=nothing= under =[fwupd]= in =/etc/fwupd/fwupd.conf= also works, but that file is pacman-owned and invites pacnew churn, so the mask is the form in use. +- The unit is static (no =[Install]= section), so =systemctl disable= is a no-op and it comes back on the next fwupdmgr run. Masking is what holds: =systemctl mask passim.service=. Ratio has been masked since 2026-07-21; velox was stopped and disabled on 2026-09-12 (the disable being the no-op) and got the mask on 2026-09-13. The installer masks it as part of installing fwupd. =P2pPolicy=nothing= under =[fwupd]= in =/etc/fwupd/fwupd.conf= also works, but that file is pacman-owned and invites pacnew churn, so the mask is the form in use. - Classification: KNOWN — a passim listener means a machine that predates the mask or lost it; mask it, don't allowlist it. ** 2026-09-12: velox — topgrade containers step fails on locally built images |