From 2f81174b1de13d9148113fb9f3d389ca6c61ac5c Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Sun, 13 Sep 2026 09:23:39 -0500 Subject: chore(tasks): close the velox reinstall drill and the lock-clock bug The reinstall drill finished on 2026-08-14 and every finding it surfaced already has its own task, so what remained was the record. I filed its working-dir artifacts into permanent homes. The runbook went to docs/ with an Outcome section, since the checklist was never ticked as it ran. The UEFI boot-entry reference went to docs/ too. The three reinstall-gap reports went to docs/design/. The rescued wttrin bundle moved into its own working dir under the task that owns it. Every inbound link is repointed and working/velox-reinstall/ is gone. The lock-screen clock bug is closed because it no longer happens on velox. I couldn't identify the commit that fixed it from the dotfiles or archsetup logs, and the note says so. Its manual-testing check retires with it. --- docs/2026-08-13-velox-reinstall-runbook.org | 184 +++++++++++++++++++++ .../2026-08-15-velox-uefi-boot-entry-reference.org | 108 ++++++++++++ docs/design/2026-08-14-velox-reinstall-gaps-1.org | 137 +++++++++++++++ docs/design/2026-08-14-velox-reinstall-gaps-2.org | 63 +++++++ docs/design/2026-08-14-velox-reinstall-gaps-3.org | 98 +++++++++++ docs/workflows/system-health-check.org | 2 +- todo.org | 82 ++++----- working/emacs-wttrin-rescue/wttrin-bf0457f.bundle | Bin 0 -> 1826602 bytes .../2026-08-14-emacsd-velox-reinstall-gaps-1.org | 137 --------------- .../2026-08-14-emacsd-velox-reinstall-gaps-2.org | 63 ------- .../2026-08-14-emacsd-velox-reinstall-gaps-3.org | 98 ----------- .../velox-reinstall/velox-reinstall-runbook.org | 161 ------------------ .../velox-uefi-boot-entry-reference.org | 108 ------------ working/velox-reinstall/wttrin-bf0457f.bundle | Bin 1826602 -> 0 bytes 14 files changed, 627 insertions(+), 614 deletions(-) create mode 100644 docs/2026-08-13-velox-reinstall-runbook.org create mode 100644 docs/2026-08-15-velox-uefi-boot-entry-reference.org create mode 100644 docs/design/2026-08-14-velox-reinstall-gaps-1.org create mode 100644 docs/design/2026-08-14-velox-reinstall-gaps-2.org create mode 100644 docs/design/2026-08-14-velox-reinstall-gaps-3.org create mode 100644 working/emacs-wttrin-rescue/wttrin-bf0457f.bundle delete mode 100644 working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-1.org delete mode 100644 working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-2.org delete mode 100644 working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-3.org delete mode 100644 working/velox-reinstall/velox-reinstall-runbook.org delete mode 100644 working/velox-reinstall/velox-uefi-boot-entry-reference.org delete mode 100644 working/velox-reinstall/wttrin-bf0457f.bundle diff --git a/docs/2026-08-13-velox-reinstall-runbook.org b/docs/2026-08-13-velox-reinstall-runbook.org new file mode 100644 index 0000000..2d99fc9 --- /dev/null +++ b/docs/2026-08-13-velox-reinstall-runbook.org @@ -0,0 +1,184 @@ +#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup +#+AUTHOR: Craig Jennings +#+DATE: 2026-08-13 + +Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon +890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry, +and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the +pool. Decision: full reinstall via archangel + archsetup, run deliberately as +a disaster-recovery test of the ISO and scripts before the Sunday flight. +Recent backup in hand; ratio available as the working machine. + +* Outcome (recorded 2026-09-13 Sun) + +The drill ran on 2026-08-13 and 14 and velox came back as a working daily +driver: fresh install from the archangel ISO, keys and data restored from +the salvage backup, 23 repos re-cloned, rsyncshot reinstalled, hibernate +proven end to end. The checklist below was the live plan; it was not ticked +as the phases ran, so read it as the plan, not a log of each step. + +What the drill found, each filed as its own task rather than fixed in place: +velox's truenas backups had silently stopped on 2026-07-06 (found on 08-13 +before partitioning, which is what made the salvage pass required); four phantom +reboots were a ribbon disturbed by the board swap; a fresh install never +clones rulesets, never links the .emacs.d systemd user units, ships no +brightness udev rule, and loses gcalcli and the signal-cli registration. +Those live in archsetup's todo as the post-rebuild verification pass and its +siblings. The one commit that existed only on the old disk (emacs-wttrin +bf0457f) was rescued as a bundle and has its own task. + +Companion documents: the UEFI boot-entry recovery reference +([[file:2026-08-15-velox-uefi-boot-entry-reference.org][2026-08-15-velox-uefi-boot-entry-reference.org]]) +and the three gap reports under docs/design (2026-08-14-velox-reinstall-gaps-1 +to 3). + +Fallback ordering if the test finds a real gap: +- Before partitioning starts: the old system is intact — the ZBM repair + route (efibootmgr entry pointing at the ZBM loader on the ESP, then + amd-ucode swap in a chroot) is still available. +- After partitioning: the floor is a manual Arch install; the backup makes + that survivable. + +* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk) + +- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates + the microcode vendor-detection fix (archsetup, 2026-08-08) and was built + without ARCHSETUP_DIR at all. + #+begin_src sh + cd ~/code/archangel && sudo ARCHSETUP_DIR="$HOME/code/archsetup" ./build.sh + #+end_src + Two traps in that one line, and either alone silently produces a bare ISO + with archsetup absent (archangel, 2026-08-20). The =VAR=value= form is + required because sudo's =env_reset= discards an exported variable. And + =$HOME= is required because zsh does not expand a tilde on the right-hand + side of an assignment — the earlier =ARCHSETUP_DIR=~/code/archsetup= here + passed the literal string. build.sh now warns and reports baked/not-baked + in its closing summary, so the failure is visible rather than silent. +- [ ] build.sh fixes before the final rebuild (archangel repo): + - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the + credential audit — off the portable USB stick). + - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile + is on the ISO at =/root/velox-zfs.conf=. +- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=), + =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs + spot-check. +- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy + drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16 + archangel ISOs removed. Boot the stick and pick + =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu. + +* Phase 1 — UEFI setup on velox (BIOS screen, before any boot) + +- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned; + the new board ships with it enforced by factory default. +- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01; + a wrong clock breaks TLS and pacman signature checks in the live env. + Rough accuracy is fine — NTP tightens it once networked. +- [ ] While you're in setup: check boot-order UI shows the USB. + +* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED + +NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is +DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke +around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio +and mybitch backed up today). Everything since Jul 6 exists only on the old +SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO +backup at all. This pass also keeps the repair fallback alive until +partitioning starts. + +- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=. +- [ ] Import the old pool read-only and unlock: + #+begin_src sh + zpool import -N -o readonly=on -R /mnt zroot + zfs load-key zroot # passphrase prompt + zfs mount zroot/ROOT/default + zfs mount -a 2>/dev/null # home datasets etc.; ignore failures + #+end_src +- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout + the backup job uses (etc + home), into a clearly-named one-off dir: + #+begin_src sh + rsync -aHAX --info=progress2 /mnt/etc /mnt/home \ + truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/ + #+end_src + (=/usr= is in the regular backups but is all reinstallable — skip unless + paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.) +- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest + files in =~/documents= and =~/downloads=. +- [ ] Check for uncommitted repo work and either push or note it: + =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge), + =~/.dotfiles=, anything under =~/code=. +- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=. + +* Phase 3 — Install (the actual DR test) + +- [ ] Review the profile, then run the installer: + #+begin_src sh + less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme + archangel --config-file /root/velox-zfs.conf + #+end_src + Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome= + placeholders — fine for install; both change post-install (=zfs change-key + zroot= for the pool, =passwd= for root). +- [ ] Record every rough edge as a DR-test finding — that's the point of + running it this way. Anything that needs a manual nudge gets a todo entry + in archangel or archsetup afterward. +- [ ] Reboot into ZBM → boot the new environment. + +* Phase 4 — archsetup (first boot of the installed system) + +- [ ] Log in as root, network up, then verify the clock synced. +- [ ] Get archsetup — two paths, test the offline one since this is a DR + drill (the online curl path is the everyday alternative): + #+begin_src sh + # offline: mount the install USB and copy the baked tree + mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt + cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup + ./archsetup + #+end_src +- [ ] Expected on the new board: =install_cpu_microcode= detects + AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests). + Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all + in the installer now — none need manual application afterward. +- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel + assumptions (swept 2026-08-13); maint's capability probe runtime-detects + amd-pstate. + +* Phase 5 — Post-install restore + verification + +- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The + secrets-bundle-in-ISO design is not built yet — manual restore is the + known gap, not a test failure. +- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=; + re-import the NM profile (autoconnect off, as before). +- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=. +- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known + idle instability (Panel Self Refresh hangs/reboots the machine; hit during + the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the + installed system's kernel command line — velox boots via ZBM, so set it on + the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default= + (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a + kernel that fixes PSR on Strix Point — track via the Framework issue + tracker (SoftwareFirmwareIssueTracker #110). +- [ ] New-hardware spot-checks: + - =journalctl -k | grep -i microcode= — amd-ucode applied. + - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect + amd-pstate(-epp). + - wifi + bluetooth up (new board radios), touchpad behavior, camera. + - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV. +- [ ] Fresh clones automatically carry the post-purge rewritten git history — + closes the clone-reconcile rider from 2026-08-11 without action. +- [ ] Fix and verify the backup timer on the fresh install — it was silently + broken since ~Jul 6. After the first manual run succeeds, confirm a new + DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it + broke (timer unit dead? mount failure? credential?) if the old journal + survives in the salvage copy. +- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate), + both daily drivers AMD. Fix the stale =intel_pstate= comment in + =airplane-mode= line 6 while at it (cosmetic). +- [ ] File every DR-test finding in the owning project's todo. + +* Timing + +Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or +Friday, leaving Saturday as pure buffer. If the install stalls past Friday +evening, cut losses to the manual-install floor. diff --git a/docs/2026-08-15-velox-uefi-boot-entry-reference.org b/docs/2026-08-15-velox-uefi-boot-entry-reference.org new file mode 100644 index 0000000..1eacfe5 --- /dev/null +++ b/docs/2026-08-15-velox-uefi-boot-entry-reference.org @@ -0,0 +1,108 @@ +#+TITLE: Velox UEFI Boot Entry — Recovery Reference +#+AUTHOR: Craig Jennings +#+DATE: 2026-08-15 + +Captured 2026-08-15 before a BIOS update (03.05 → 04.02) as insurance against +the update clearing NVRAM. Velox's mainboard swap on 2026-08-13 left exactly +this kind of empty NVRAM, which is what forced the reinstall — so a cleared +boot entry is the specific failure worth being able to undo in one command +rather than reconstruct. + +* Before recreating anything: check Secure Boot first + +The 04.02 update landed on 2026-09-12 (staged via fwupd, flashed on the next +reboot). It did NOT clear NVRAM: Boot0001 survived with its command line +intact. What it did was re-enable Secure Boot (Enforce Secure Boot = +Enabled), so the unsigned ZBM loader was rejected and the Framework BIOS +reported it as "Default Boot Device Missing / no bootable drive" rather than +a security violation. The symptom is indistinguishable from the NVRAM wipe +this document was written for. + +The tell: booting the Ventoy stick shows shim's MOK management screen. + +Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. The +machine then boots straight into ZBM. No efibootmgr needed. + +So when velox says no bootable device after a firmware update, check Secure +Boot before touching the boot entries. Only if Secure Boot is already off +and =efibootmgr -v= (from the stick) shows Boot0001 gone does the recreate +below apply. + +Before any future firmware update, record both =efibootmgr -v= and the +Secure Boot state so the post-reboot diagnosis is a comparison, not a guess. +The pre-firmware-update checklist in +[[file:workflows/system-health-check.org][docs/workflows/system-health-check.org]] +(Phase 3) carries the steps. + +Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT= +does not exist), so a real NVRAM wipe would still need the stick. Copying +=zfsbootmenu.efi= to =/efi/EFI/BOOT/BOOTX64.EFI= would let it boot unaided. +That belongs to archangel's ZBM install, where it is filed as [#C] +"Installed systems have no removable-media boot fallback on the ESP" +(2026-09-12) and ships with the next ISO rebuild after it lands. + +* State at capture + +- BIOS: 03.05 (2025-10-30) +- BootCurrent: 0001 +- BootOrder: 2001,0001,2002,2003 (USB ahead of ZBM — why the Ventoy stick + boots when it's inserted) +- Timeout: 0 seconds + +* The entry that matters + +=Boot0001* ZFSBootMenu= + +| field | value | +|----------------+----------------------------------------------| +| ESP part GUID | 8e51b680-f90a-444f-8da5-7e4f93625775 | +|----------------+----------------------------------------------| +| partition | 1 (GPT), start 0x800, size 0x100000 | +|----------------+----------------------------------------------| +| loader path | =\EFI\ZBM\zfsbootmenu.efi= | +|----------------+----------------------------------------------| +| cmdline (data) | =spl_hostid=0x22f8a7a1 zbm.timeout=3= | +| | =zbm.prefer=zroot zbm.import_policy=hostid= | +|----------------+----------------------------------------------| + +The =data= field is that command line in UTF-16LE, which is how efibootmgr +passes it as optional data. Recreate with =-u= and the plain string; efibootmgr +does the encoding. + +* Recreating it + +From a booted system (or the archangel ISO), with the ESP identified as +=/dev/nvme0n1p1= or whatever it enumerates as: + +#+begin_src bash +efibootmgr --create \ + --disk /dev/nvme0n1 --part 1 \ + --label "ZFSBootMenu" \ + --loader '\EFI\ZBM\zfsbootmenu.efi' \ + --unicode 'spl_hostid=0x22f8a7a1 zbm.timeout=3 zbm.prefer=zroot zbm.import_policy=hostid' +#+end_src + +Confirm the disk/part against =lsblk -o NAME,PARTUUID,PARTTYPENAME= first — +the partition GUID above is the authoritative identifier, not the device name, +which can enumerate differently. + +Then set the order so ZBM is reachable: + +#+begin_src bash +efibootmgr --bootorder 0001,2001,2002,2003 +#+end_src + +(The original order put USB first. Keep whichever you prefer; what matters is +that the ZBM entry exists and is in the list.) + +* Other entries (firmware-generated, recreate themselves) + +| Boot2001 | EFI USB Device | +|----------+----------------| +| Boot2002 | EFI DVD/CDROM | +|----------+----------------| +| Boot2003 | EFI Network | +|----------+----------------| + +These are stock firmware entries and come back on their own. Only Boot0001 +carries anything unique. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-1.org b/docs/design/2026-08-14-velox-reinstall-gaps-1.org new file mode 100644 index 0000000..cf0d723 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-1.org @@ -0,0 +1,137 @@ +#+TITLE: What the velox reinstall left behind — four gaps the install could close +#+AUTHOR: Craig Jennings + +* Heads-up: this was found from a .emacs.d session + +I opened a .emacs.d session on velox this morning, two days after the fresh +Arch install, and the first thing it did was fail: there was no =.ai/= +directory to read. Chasing that turned up four separate things the reinstall +did not restore. Three I repaired from the session; one needs me at my phone. + +None of this is a .emacs.d bug. They are all install-side gaps, which is why +they are landing in your inbox. Machine is velox; ratio was the reference for +every comparison below. + +* Gap 1 — the gitignored tooling layer does not survive a reinstall + +=~/.emacs.d= was re-cloned on 2026-08-13. Git brought back every tracked file +and none of the agent tooling, because =.gitignore= deliberately excludes it: +=.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, and =inbox/= were all simply +absent. That is the correct ignore policy — this repo relays to a public +mirror — but it means a reinstall silently drops the entire working state of +every gitignore-mode project. + +The damage on velox was total rather than partial: 374 files, 4.5 MB, +including =todo.org= (556 KB) and 184 archived session files. Nothing carries +it. Not git, not stow, not the bootstrap. + +I recovered it by rsyncing the set from ratio over the tailnet. Ratio was +authoritative and velox held nothing, so there was no merge to adjudicate — +which is luck, not design. Had velox held a few days of divergent state, this +would have been a hand reconciliation. It has been one before: 2026-07-31, when +the two machines' =.ai/= trees had forked to zero files in common. + +Worth knowing: this is fleet-general. Every project on the box that gitignores +its =.ai/= has the same hole, not just =.emacs.d=. + +What the install could do: after cloning a project, check whether a sibling +daily driver holds a =.ai/= for it, and offer to pull it across. Or at minimum, +list the projects whose tooling layer is missing so the gap is visible on day +one instead of at the first session that trips over it. + +* Gap 2 — stowed user timers come back linked but not enabled + +The unit files all arrived correctly through the dotfiles stow, symlinked into +=~/.config/systemd/user/= and resolving fine. But being present is not being +enabled, and the reinstall enabled only some of them: + +| unit | velox after reinstall | ratio | +|---------------------------+-----------------------+----------| +| calendar-sync.timer | enabled, active | enabled | +| agenda-render-cache.timer | enabled, active | enabled | +| roam-sync.timer | *linked, inactive* | enabled | +| signal-receive.timer | *linked, inactive* | enabled | +| emacs.service | linked, inactive | linked | + +=emacs.service= reads the same on both machines, so I take that one as +intentional and left it alone. The other two are real drift: =systemctl --user +enable= writes a =timers.target.wants= symlink into =~/.config/systemd/user/=, +and that symlink is not stow-managed, so nothing in the dotfiles repo carries +it. A stowed unit file is inert until something enables it. + +I enabled both with =systemctl --user enable --now=. Both fired immediately and +exited clean, and both now show a next elapse. + +What the install could do: enable the units it stows, explicitly, as a named +step. The inconsistency is the tell — two of four came back enabled, which +suggests something enables a subset and nothing enumerates the rest. + +* Gap 3 — the roam clone was stale, and held a diff that would have destroyed data + +This one has an ordering constraint, so it matters more than its size suggests. + +velox's =~/org/roam= was ten commits behind ratio, stuck at the 2026-08-04 +auto-sync while ratio was at 2026-08-14 — a direct consequence of gap 2, since +=roam-sync.timer= was never enabled here. + +The dangerous part: velox's clone also carried an *uncommitted* =inbox.org= +that had been emptied. Seventeen deletions, file down to zero bytes, holding a +pre-2026-08-04 state whose captures were long since processed on ratio. + +So the naive repair — enable =roam-sync.timer= and let it catch up — would have +committed that emptying and pushed it, deleting the four live inbox items on +ratio. The timer is the repo's only committer and it commits whatever it finds. + +I checked ratio's =inbox.org= first and confirmed it was a strict superset of +velox's HEAD version (same three items plus an 2026-08-09 capture), which made +the local change provably worthless. Then discarded it, fast-forwarded to +=a411b43=, and only then enabled the timer. Clone is clean and current, first +sync ran green. + +What the install could do: if it ever enables =roam-sync= on a rebuilt machine, +reconcile the clone *before* enabling, not after. An auto-committing timer +pointed at a stale dirty clone is a data-loss path, and the failure is silent +and remote — it lands on the *other* machine. + +* Gap 4 — signal-cli lost its registration, and that breaks the whole fleet + +=signal-receive.service= ran for the first time and reported: + +: signal-receive: +15045173983 not registered on this machine — nothing to do + +velox's signal-cli data dir holds a 39-byte empty =accounts.json=. Ratio still +has both numbers. So the reinstall wiped the registration, and per the design +notes velox was supposed to be the *primary* — ratio is the linked device. + +The effect is wider than velox, because of how =agent-text= dispatches: if the +local signal-cli holds the account it sends directly, otherwise it ssh-relays to +a hardcoded velox. Velox no longer holds it, so a send from here relays to +itself and fails; a send from any third machine relays to velox and fails the +same way. Only ratio still works, and only via the direct branch. The error text +blames "velox down or unreachable", which is misleading — velox is up and on the +tailnet, it just is not registered. + +This is the one I could not repair from the session: re-linking needs me at my +phone (Signal → Settings → Linked Devices, scanning the QR from =signal-cli +link -n velox=). Filed in .emacs.d's todo.org as [#B]. + +What the install could do: verify =signal-cli listAccounts= is non-empty after a +rebuild and say so loudly if it is not. Silent loss of the phone channel is +exactly the kind of thing nobody notices until the page that mattered never +arrives. + +* Summary of what I changed on velox + +- Restored =.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, =inbox/= to + =~/.emacs.d= by rsync from ratio. +- Discarded the stale local =inbox.org= diff in =~/org/roam= and fast-forwarded + the clone to current. +- Enabled and started =roam-sync.timer= and =signal-receive.timer=. + +Left alone, deliberately: =emacs.service= (matches ratio), and velox's Signal +registration (needs the phone). + +One unrelated thing I noticed while comparing the machines: ratio's signal-cli +warns its messages were last received twelve days ago, even though its +=signal-receive.timer= is enabled and active. That may be nothing, but the +receive cadence there is worth a look. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-2.org b/docs/design/2026-08-14-velox-reinstall-gaps-2.org new file mode 100644 index 0000000..95842ac --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-2.org @@ -0,0 +1,63 @@ +#+TITLE: Fifth reinstall gap — machine-local .local.el config, and a general shape +#+AUTHOR: Craig Jennings + +* Follow-up to this morning's handoff + +Sent you four gaps an hour ago +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]]). Here is a fifth, +found straight afterwards when I noticed calendar sync was dead on velox. + +* What was broken + +=calendar-sync.timer= was enabled and firing every fifteen minutes, and failing +every time with exit 255: + +: calendar-sync: No calendars configured (set calendar-sync-calendars) + +The three output files sat at zero bytes. The cause is that +=~/.emacs.d/calendar-sync.local.el= is gitignored, so the reinstall deleted it +along with everything else untracked, and the module's loader treats a missing +file as a *silent* no-op. So the config vanished quietly and the only symptom +was a failing unit nobody was watching. + +Cheap to fix once found: the repo tracks =calendar-sync.local.el.example=, and +that template already encodes the shape velox uses — feeds resolved by +=:secret-host= against =authinfo.gpg= rather than inlined. The authinfo entries +had survived, because =~/.authinfo.gpg= is a stow symlink into the dotfiles repo. +So rebuilding was one copy, and all three feeds now sync clean and land +byte-identical to ratio's. + +* The general shape, which is the part worth acting on + +This is the same failure as gap 1, one layer down, and it is worth stating +generally because the install can act on it: + +- A tracked =*.local.el.example= template plus a gitignored =*.local.el= is a + deliberate pattern in this config, not a one-off. =.gitignore= lines 56-58 + list three of them: =calendar-sync.local.el=, =signal-config.local.el=, + =google-keep.local.el=. Every one of those is gone on velox right now. I have + only repaired the calendar one. +- Secrets held *by reference* survive a rebuild; secrets held *inline* do not. + The calendar config came back for free because the tokens were in + =authinfo.gpg=, which is stow-managed and therefore travels. Ratio's copy of + the same file inlines its URLs, and had ratio been the machine rebuilt, those + three feed tokens would simply have been gone. +- The failure was silent by design. A missing local config is a no-op, which is + right for a machine that never configured the feature and wrong for one that + just lost it. + +* What the install could do + +- After a rebuild, enumerate every tracked =*.local.el.example= in a project and + report which have no corresponding =*.local.el=. That is a one-line find and it + turns a silent no-op into a visible checklist item. +- Same for any =*.local.*= convention elsewhere in the fleet — the pattern is not + specific to Emacs. +- Worth pairing with gap 2: a unit that is enabled and failing every fifteen + minutes for two days is its own signal. A post-rebuild pass over + =systemctl --user list-units --state=failed= would have caught this one + without knowing anything about calendars. + +That last one generalizes best. Of the five gaps I have sent you, three were +things that *looked* fine — a stowed unit file, an enabled timer, a present +clone — and were not. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-3.org b/docs/design/2026-08-14-velox-reinstall-gaps-3.org new file mode 100644 index 0000000..9675973 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-3.org @@ -0,0 +1,98 @@ +#+TITLE: Reinstall gaps, part three — per-install certs and credentials, and one failure that hid the others +#+AUTHOR: Craig Jennings + +* Third handoff today + +Two earlier notes covered five gaps +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]] and +[[file:2026-08-14-velox-reinstall-gaps-2.org][the follow-up]]). +Email was the last thing broken on velox after the 2026-08-13 rebuild, and it +turned up two more — both the same shape, and one of them with a property worth +generalizing. + +Email is fully working now: three accounts, 21,853 messages, 4.0 GB indexed. + +* Gap 6 — the Proton Bridge TLS cert is per-install, and its absence disabled every account + +=~/.mbsyncrc= carries =CertificateFile /home/cjennings/.config/protonbridge.pem=. +That file did not exist after the rebuild, and it cannot be restored from backup +or copied from the other machine: Proton Bridge generates a fresh self-signed +cert per installation. Velox's is issued 2026-08-13 23:44 with a different +fingerprint from ratio's 2026-01-30 one. + +*The part worth acting on is the blast radius.* mbsync parses its entire config +before doing any work, so a missing =CertificateFile= referenced by *one* account +aborts the run for *all* of them. Gmail and dmail need no bridge and no cert, and +both were dead anyway. The error names only the missing pem, so the symptom +("no mail at all") and the message ("this one file is missing") look unrelated. + +Recovery does not need the bridge GUI. The running bridge presents the cert on +its own IMAP port, so it can be pulled straight off the handshake: + +: openssl s_client -connect 127.0.0.1:1143 -starttls imap -showcerts ~/.config/protonbridge.pem + +That is a two-second, fully scriptable step, which makes it a good candidate for +the install rather than a runbook line. + +* Gap 7 — the bridge password is per-install too, and reports a stale value misleadingly + +=~/.mbsyncrc= resolves the cmail password with =cat ~/.config/.cmailpass=. That +file is plaintext and, unusually for my setup, a real file rather than a stow +symlink — so it is not in the dotfiles repo, not encrypted, and not carried to a +new machine. + +The file survived the rebuild but held the *previous* install's password, because +the bridge regenerates it per installation. Ratio's and velox's differ by sha256, +confirmed today. + +*The diagnostic trap:* Proton Bridge answers a wrong password with =no such +user=. I read that as "the bridge has no account signed in" and went looking for +a login problem. The account was configured the whole time. If the install ever +validates bridge connectivity, it should not treat =no such user= as evidence +about account state. + +* The generalization + +Gaps 6 and 7 are the same as 1 through 5, sharpened. Everything that broke in +this rebuild was *generated on the machine by an application* rather than carried +by git, stow, or the dotfiles repo: + +| gap | artifact | why it did not travel | +| 1 | =.ai/=, =todo.org=, =CLAUDE.md= | gitignored | +| 2 | =timers.target.wants= symlinks | written by systemctl enable | +| 3 | roam clone state | local working tree | +| 4 | signal-cli registration | per-device identity | +| 5 | =*.local.el= configs | gitignored | +| 6 | bridge TLS cert | per-install, regenerated | +| 7 | bridge password | per-install, regenerated | + +Gaps 6 and 7 add a distinction the earlier note missed. For 1, 3 and 5 the old +value is still correct, so *restoring* fixes them. For 4, 6 and 7 the old value is +*worthless* — the application has generated a new one, and only *re-deriving* +from the live system fixes them. An install that tries to restore these will +produce exactly what happened here: a file that exists, looks right, and +authenticates against nothing. + +So the install's post-rebuild checklist wants two columns, not one: what to +restore, and what to re-derive. + +* What the install could do + +- Re-derive the bridge cert from the running bridge with the =openssl s_client= + line above. Scriptable, no GUI, no secrets. +- Re-derive the bridge password from the bridge rather than expecting the file to + be right, and rewrite =.cmailpass=. (I have filed a task on my side to make + =PassCmd= ask the bridge directly, which would remove the file entirely.) +- Add a cheap post-rebuild validation that =mbsync --list= parses. Config-parse + failures disable every account at once and say nothing about mail, so they are + worth catching explicitly rather than via "no new mail" hours later. +- More generally: keep the restore list and the re-derive list separate, per the + table above. + +* Unrelated, but noticed while comparing the machines + +=~/.config/.gmailpass.gpg= and =~/.config/.dmailpass.gpg= resolve to mode 777 in +the dotfiles repo, on both machines. They are gpg-encrypted so the contents are +safe, but world-writable is wrong for a credential file. That is a dotfiles fix, +not an archsetup one — noting it here only because it surfaced in the same pass. diff --git a/docs/workflows/system-health-check.org b/docs/workflows/system-health-check.org index 37c1932..43aeff5 100644 --- a/docs/workflows/system-health-check.org +++ b/docs/workflows/system-health-check.org @@ -244,7 +244,7 @@ Firmware stays a manual step (=topgrade.toml= keeps =[firmware] upgrade = false= 1. =sudo efibootmgr -v= — every boot entry with its loader path and command line, pasted into the session's context file. 2. Secure Boot state — =bootctl status 2>/dev/null | grep -i 'secure boot'=. -After the flash, if the machine reports no bootable device, check Secure Boot *first*. The Framework 04.02 update on velox re-enabled it, which rejects the unsigned ZFSBootMenu loader and reads as "Default Boot Device Missing" rather than a security violation; the boot entries were untouched (see the Known Issues Log, 2026-09-12). Only when Secure Boot is off and =efibootmgr -v= from a stick shows the entry gone does the boot-entry recreate apply (velox: =working/velox-reinstall/velox-uefi-boot-entry-reference.org= in archsetup). +After the flash, if the machine reports no bootable device, check Secure Boot *first*. The Framework 04.02 update on velox re-enabled it, which rejects the unsigned ZFSBootMenu loader and reads as "Default Boot Device Missing" rather than a security violation; the boot entries were untouched (see the Known Issues Log, 2026-09-12). Only when Secure Boot is off and =efibootmgr -v= from a stick shows the entry gone does the boot-entry recreate apply (velox: =docs/2026-08-15-velox-uefi-boot-entry-reference.org= in archsetup). *** Two-Stage Reboot Pattern (MANDATORY if Phase 3 installed kernel / iproute2 / systemd / NetworkManager) diff --git a/todo.org b/todo.org index 598ed47..a54ff66 100644 --- a/todo.org +++ b/todo.org @@ -223,8 +223,8 @@ to read file" on stderr and nothing on stdout; piped into =grep -c=, that empty stdout reads as 0 and looks like a missing module. The gate has to run as root and check the exit status, not just the count. -** TODO [#A] Lock-screen clock stale after a real sleep :bug:hyprland:dotfiles:velox: -SCHEDULED: <2026-08-25 Tue> +** DONE [#A] Lock-screen clock stale after a real sleep :bug:hyprland:dotfiles:velox: +CLOSED: [2026-09-13 Sun] SCHEDULED: <2026-08-25 Tue> :PROPERTIES: :CREATED: [2026-08-25 Tue] :LAST_REVIEWED: 2026-08-25 @@ -248,6 +248,13 @@ different fix: stale-then-corrects → repaint interval; frozen with live input → clock rendering; frozen with dead input → hyprlock hung, a crash/hang recovery bug the =screen-lock= watchdog doesn't cover. +*** 2026-09-13 Sun @ 07:57:04 -0500 Closed: fixed, per Craig +Craig confirmed on 2026-09-13 Sun that the stale clock after a real sleep is fixed on +velox. I could not identify the commit from here (nothing in the dotfiles +or archsetup log since 2026-08-24 names hyprlock, the lock clock, sleep or +resume), so this closes on his report rather than a cited change; the +manual-testing check for it is retired with it. + ** TODO [#C] post-rebuild-check: probe that Emacs frames come up Wayland-native :feature:emacs:velox:solo:quick: :PROPERTIES: :CREATED: [2026-08-25 Tue] @@ -495,8 +502,8 @@ Alternatives if it drags on: change Signal's tray setting so it keeps a window (=~/.config/Signal/ephemeral.json= =system-tray-setting=), or run a waybar carrying the #5240 fallback. -** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: -DEADLINE: <2026-08-15 Sat> +** DONE [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: +CLOSED: [2026-09-13 Sun] DEADLINE: <2026-08-15 Sat> :PROPERTIES: :CREATED: [2026-08-13 Thu] :LAST_REVIEWED: 2026-08-13 @@ -504,7 +511,7 @@ DEADLINE: <2026-08-15 Sat> Mainboard swapped Intel→AMD (Ryzen AI 9 HX 370); new NVRAM has no boot entry. Decision: full reinstall via archangel+archsetup, run deliberately as a disaster-recovery drill before the Sunday flight. Runbook (live checklist): -[[file:working/velox-reinstall/velox-reinstall-runbook.org][working/velox-reinstall/velox-reinstall-runbook.org]] +[[file:docs/2026-08-13-velox-reinstall-runbook.org][docs/2026-08-13-velox-reinstall-runbook.org]] Done 2026-08-13: ISO rebuilt (archangel-2026-08-13, archsetup baked with AMD microcode detection, velox profiles at /root/, .ai/inbox excluded — build.sh edits pending commit in archangel), contents verified, dotfiles swept clean of @@ -526,6 +533,17 @@ listeners on 5357 and 27500. Same pass fast-forwarded velox's dotfiles to f56fd1a and archsetup to dc00a62, and confirmed the headless Proton Bridge service is still disabled there. +*** 2026-09-13 Sun @ 07:56:37 -0500 Closed the drill and filed its working-dir artifacts +The reinstall itself finished on 2026-08-14; every finding it surfaced is +its own task, and the live convergence steps landed on 2026-09-13, so nothing +was left in this task but the record. Filed per the working-files +convention: the runbook to docs/2026-08-13-velox-reinstall-runbook.org (with +an Outcome section, since the checklist was never ticked during the run), +the boot-entry reference to docs/2026-08-15-velox-uefi-boot-entry-reference.org, +the three gap reports to docs/design/2026-08-14-velox-reinstall-gaps-1 to 3, +and the wttrin bundle to working/emacs-wttrin-rescue/ under the task that +owns it. working/velox-reinstall/ is gone and every inbound link repointed. + ** TODO [#B] Truenas session-host VM for long-running agent sessions :feature:tooling: :PROPERTIES: :CREATED: [2026-08-13 Thu] @@ -791,9 +809,9 @@ paging, and because the flight is Sunday. *** 2026-09-13 Sun @ 07:14:54 -0500 Moved the three gap reports into the reinstall working dir The 2026-08-14 reports that define the five gaps moved out of inbox/ into -[[file:working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-1.org][gaps 1-4]], -[[file:working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-2.org][gap 5]] and -[[file:working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-3.org][the two email-side gaps]] (the Bridge cert and the Bridge password). +[[file:docs/design/2026-08-14-velox-reinstall-gaps-1.org][gaps 1-4]], +[[file:docs/design/2026-08-14-velox-reinstall-gaps-2.org][gap 5]] and +[[file:docs/design/2026-08-14-velox-reinstall-gaps-3.org][the two email-side gaps]] (the Bridge cert and the Bridge password). They file with the rest of the reinstall artifacts when that task closes. ** TODO [#B] Restoring a git repo from backup can resurrect a dangerous diff :bug: @@ -864,7 +882,7 @@ bf0457f "feat: add wttrin-hide-follow-line to hide the wttr.in follow line" (2026-06-24) was the only genuinely unpushed commit anywhere on the old velox — 3 files, 103 insertions, with a test file. Rescued as a verified git bundle before the disk was wiped: -[[file:working/velox-reinstall/wttrin-bf0457f.bundle][working/velox-reinstall/wttrin-bf0457f.bundle]] +[[file:working/emacs-wttrin-rescue/wttrin-bf0457f.bundle][working/emacs-wttrin-rescue/wttrin-bf0457f.bundle]] To land it: clone emacs-wttrin, =git fetch --branches=, review the commit, then push to git@cjennings.net:emacs-wttrin.git. Delete the bundle once it's on the remote. @@ -874,11 +892,11 @@ Cloned the remote bare and asked it for the object directly: =git cat-file -t bf0457f= returns "Not a valid object name", so the commit has never reached =git@cjennings.net:emacs-wttrin.git=. Remote =main= is =ee8fdeb=. -That makes =working/velox-reinstall/wttrin-bf0457f.bundle= the sole surviving +That makes =working/emacs-wttrin-rescue/wttrin-bf0457f.bundle= (moved there 2026-09-13 when the reinstall task filed its artifacts) the sole surviving copy of 103 insertions across three files, on one laptop that is travelling. -Worth doing sooner than its =[#B]= suggests for that reason alone, and it also -pins the working directory open — the reinstall task cannot file its artifacts -away while this bundle is still load-bearing. +Worth doing sooner than its =[#B]= suggests for that reason alone. It also +pinned the reinstall working directory open until 2026-09-13, when the bundle +moved into its own working dir here and the reinstall task closed. ** TODO [#B] archsetup doesn't clone rulesets :bug:velox: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -2375,38 +2393,10 @@ fi Expected: "stays gone". If it regenerates, Hyprland is still resolving its config to the .conf path and the port is not actually live — stop and investigate. -*** Lock screen after a real sleep: is the clock frozen, or is all of hyprlock frozen? -What we're verifying: which of two very different faults produces the stale -lock-screen clock Craig sees after waking the machine. Three isolated tests on -2026-08-24 all failed to reproduce it — hyprlock v0.9.6 repainted correctly -within one second of a display power-cycle, of a three-minute SIGSTOP, and of -both together with the =path = screenshot= background. So the cause needs a real -suspend on real AMD hardware, which only Craig can run. - -The distinguishing question is whether *anything* on the lock screen is alive. -A frozen clock with live input is a rendering bug in the clock. A frozen clock -with dead input means hyprlock is hung and the compositor is simply holding its -last pre-suspend frame — a different bug, in the crash/hang recovery layer. - -Run this the next time the machine has actually slept (30 min idle, or a lid -close), not on a screen you just locked by hand. - -- Wake the machine and do NOT type your password yet. -- Read the clock. Note the time it shows and the real time. -- Type one character into the password field. -Expected (record which): either the dots appear normally — input is alive, only -the clock is stale — or nothing appears at all, in which case hyprlock is hung. -- Keep watching for a full 90 seconds without touching anything else. -Expected (record which): the clock corrects itself to the current time within a -minute, or it never changes at all. -- Note whether the blurred background looks like your desktop or like a - previous lock screen (a lock screen photographed behind a lock screen means a - second hyprlock was started on top of the first). - -Expected: enough to pick one of three — stale-then-corrects (a repaint-interval -fix), frozen-with-live-input (a rendering fix), or frozen-with-dead-input -(hyprlock hung; the =screen-lock= watchdog only catches a hyprlock that -*exits*, and its own header says it is blind to one that hangs). +*** 2026-09-13 Sun @ 07:57:32 -0500 Retired the lock-screen clock check: fixed, per Craig +Craig reported on 2026-09-13 Sun that the stale clock after a real sleep no longer +happens on velox, so the three-outcome check never needed running. The +parent bug task is closed with the same note. *** Lock keybind has no crash or hang recovery What we're verifying: that a hand-lock is as recoverable as an idle lock. @@ -3659,7 +3649,7 @@ than it looked when it was filed sight-unseen; still worth doing (unlisted fixes happen, and ACPI tables change), just no longer the front-runner. Deliberately deferred past the flight — a cleared NVRAM is the failure that started this whole rebuild. Boot-entry recovery reference captured at -[[file:working/velox-reinstall/velox-uefi-boot-entry-reference.org][working/velox-reinstall/velox-uefi-boot-entry-reference.org]]. +[[file:docs/2026-08-15-velox-uefi-boot-entry-reference.org][docs/2026-08-15-velox-uefi-boot-entry-reference.org]]. Order of attack on return, cheapest first: reseat the touchpad's *own* press connector at the mainboard (free, untried) → BIOS 04.02 → fit the replacement diff --git a/working/emacs-wttrin-rescue/wttrin-bf0457f.bundle b/working/emacs-wttrin-rescue/wttrin-bf0457f.bundle new file mode 100644 index 0000000..5921866 Binary files /dev/null and b/working/emacs-wttrin-rescue/wttrin-bf0457f.bundle differ diff --git a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-1.org b/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-1.org deleted file mode 100644 index cf0d723..0000000 --- a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-1.org +++ /dev/null @@ -1,137 +0,0 @@ -#+TITLE: What the velox reinstall left behind — four gaps the install could close -#+AUTHOR: Craig Jennings - -* Heads-up: this was found from a .emacs.d session - -I opened a .emacs.d session on velox this morning, two days after the fresh -Arch install, and the first thing it did was fail: there was no =.ai/= -directory to read. Chasing that turned up four separate things the reinstall -did not restore. Three I repaired from the session; one needs me at my phone. - -None of this is a .emacs.d bug. They are all install-side gaps, which is why -they are landing in your inbox. Machine is velox; ratio was the reference for -every comparison below. - -* Gap 1 — the gitignored tooling layer does not survive a reinstall - -=~/.emacs.d= was re-cloned on 2026-08-13. Git brought back every tracked file -and none of the agent tooling, because =.gitignore= deliberately excludes it: -=.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, and =inbox/= were all simply -absent. That is the correct ignore policy — this repo relays to a public -mirror — but it means a reinstall silently drops the entire working state of -every gitignore-mode project. - -The damage on velox was total rather than partial: 374 files, 4.5 MB, -including =todo.org= (556 KB) and 184 archived session files. Nothing carries -it. Not git, not stow, not the bootstrap. - -I recovered it by rsyncing the set from ratio over the tailnet. Ratio was -authoritative and velox held nothing, so there was no merge to adjudicate — -which is luck, not design. Had velox held a few days of divergent state, this -would have been a hand reconciliation. It has been one before: 2026-07-31, when -the two machines' =.ai/= trees had forked to zero files in common. - -Worth knowing: this is fleet-general. Every project on the box that gitignores -its =.ai/= has the same hole, not just =.emacs.d=. - -What the install could do: after cloning a project, check whether a sibling -daily driver holds a =.ai/= for it, and offer to pull it across. Or at minimum, -list the projects whose tooling layer is missing so the gap is visible on day -one instead of at the first session that trips over it. - -* Gap 2 — stowed user timers come back linked but not enabled - -The unit files all arrived correctly through the dotfiles stow, symlinked into -=~/.config/systemd/user/= and resolving fine. But being present is not being -enabled, and the reinstall enabled only some of them: - -| unit | velox after reinstall | ratio | -|---------------------------+-----------------------+----------| -| calendar-sync.timer | enabled, active | enabled | -| agenda-render-cache.timer | enabled, active | enabled | -| roam-sync.timer | *linked, inactive* | enabled | -| signal-receive.timer | *linked, inactive* | enabled | -| emacs.service | linked, inactive | linked | - -=emacs.service= reads the same on both machines, so I take that one as -intentional and left it alone. The other two are real drift: =systemctl --user -enable= writes a =timers.target.wants= symlink into =~/.config/systemd/user/=, -and that symlink is not stow-managed, so nothing in the dotfiles repo carries -it. A stowed unit file is inert until something enables it. - -I enabled both with =systemctl --user enable --now=. Both fired immediately and -exited clean, and both now show a next elapse. - -What the install could do: enable the units it stows, explicitly, as a named -step. The inconsistency is the tell — two of four came back enabled, which -suggests something enables a subset and nothing enumerates the rest. - -* Gap 3 — the roam clone was stale, and held a diff that would have destroyed data - -This one has an ordering constraint, so it matters more than its size suggests. - -velox's =~/org/roam= was ten commits behind ratio, stuck at the 2026-08-04 -auto-sync while ratio was at 2026-08-14 — a direct consequence of gap 2, since -=roam-sync.timer= was never enabled here. - -The dangerous part: velox's clone also carried an *uncommitted* =inbox.org= -that had been emptied. Seventeen deletions, file down to zero bytes, holding a -pre-2026-08-04 state whose captures were long since processed on ratio. - -So the naive repair — enable =roam-sync.timer= and let it catch up — would have -committed that emptying and pushed it, deleting the four live inbox items on -ratio. The timer is the repo's only committer and it commits whatever it finds. - -I checked ratio's =inbox.org= first and confirmed it was a strict superset of -velox's HEAD version (same three items plus an 2026-08-09 capture), which made -the local change provably worthless. Then discarded it, fast-forwarded to -=a411b43=, and only then enabled the timer. Clone is clean and current, first -sync ran green. - -What the install could do: if it ever enables =roam-sync= on a rebuilt machine, -reconcile the clone *before* enabling, not after. An auto-committing timer -pointed at a stale dirty clone is a data-loss path, and the failure is silent -and remote — it lands on the *other* machine. - -* Gap 4 — signal-cli lost its registration, and that breaks the whole fleet - -=signal-receive.service= ran for the first time and reported: - -: signal-receive: +15045173983 not registered on this machine — nothing to do - -velox's signal-cli data dir holds a 39-byte empty =accounts.json=. Ratio still -has both numbers. So the reinstall wiped the registration, and per the design -notes velox was supposed to be the *primary* — ratio is the linked device. - -The effect is wider than velox, because of how =agent-text= dispatches: if the -local signal-cli holds the account it sends directly, otherwise it ssh-relays to -a hardcoded velox. Velox no longer holds it, so a send from here relays to -itself and fails; a send from any third machine relays to velox and fails the -same way. Only ratio still works, and only via the direct branch. The error text -blames "velox down or unreachable", which is misleading — velox is up and on the -tailnet, it just is not registered. - -This is the one I could not repair from the session: re-linking needs me at my -phone (Signal → Settings → Linked Devices, scanning the QR from =signal-cli -link -n velox=). Filed in .emacs.d's todo.org as [#B]. - -What the install could do: verify =signal-cli listAccounts= is non-empty after a -rebuild and say so loudly if it is not. Silent loss of the phone channel is -exactly the kind of thing nobody notices until the page that mattered never -arrives. - -* Summary of what I changed on velox - -- Restored =.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, =inbox/= to - =~/.emacs.d= by rsync from ratio. -- Discarded the stale local =inbox.org= diff in =~/org/roam= and fast-forwarded - the clone to current. -- Enabled and started =roam-sync.timer= and =signal-receive.timer=. - -Left alone, deliberately: =emacs.service= (matches ratio), and velox's Signal -registration (needs the phone). - -One unrelated thing I noticed while comparing the machines: ratio's signal-cli -warns its messages were last received twelve days ago, even though its -=signal-receive.timer= is enabled and active. That may be nothing, but the -receive cadence there is worth a look. diff --git a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-2.org b/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-2.org deleted file mode 100644 index 27f0da8..0000000 --- a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-2.org +++ /dev/null @@ -1,63 +0,0 @@ -#+TITLE: Fifth reinstall gap — machine-local .local.el config, and a general shape -#+AUTHOR: Craig Jennings - -* Follow-up to this morning's handoff - -Sent you four gaps an hour ago -([[file:2026-08-14-emacsd-velox-reinstall-gaps-1.org][the first report]]). Here is a fifth, -found straight afterwards when I noticed calendar sync was dead on velox. - -* What was broken - -=calendar-sync.timer= was enabled and firing every fifteen minutes, and failing -every time with exit 255: - -: calendar-sync: No calendars configured (set calendar-sync-calendars) - -The three output files sat at zero bytes. The cause is that -=~/.emacs.d/calendar-sync.local.el= is gitignored, so the reinstall deleted it -along with everything else untracked, and the module's loader treats a missing -file as a *silent* no-op. So the config vanished quietly and the only symptom -was a failing unit nobody was watching. - -Cheap to fix once found: the repo tracks =calendar-sync.local.el.example=, and -that template already encodes the shape velox uses — feeds resolved by -=:secret-host= against =authinfo.gpg= rather than inlined. The authinfo entries -had survived, because =~/.authinfo.gpg= is a stow symlink into the dotfiles repo. -So rebuilding was one copy, and all three feeds now sync clean and land -byte-identical to ratio's. - -* The general shape, which is the part worth acting on - -This is the same failure as gap 1, one layer down, and it is worth stating -generally because the install can act on it: - -- A tracked =*.local.el.example= template plus a gitignored =*.local.el= is a - deliberate pattern in this config, not a one-off. =.gitignore= lines 56-58 - list three of them: =calendar-sync.local.el=, =signal-config.local.el=, - =google-keep.local.el=. Every one of those is gone on velox right now. I have - only repaired the calendar one. -- Secrets held *by reference* survive a rebuild; secrets held *inline* do not. - The calendar config came back for free because the tokens were in - =authinfo.gpg=, which is stow-managed and therefore travels. Ratio's copy of - the same file inlines its URLs, and had ratio been the machine rebuilt, those - three feed tokens would simply have been gone. -- The failure was silent by design. A missing local config is a no-op, which is - right for a machine that never configured the feature and wrong for one that - just lost it. - -* What the install could do - -- After a rebuild, enumerate every tracked =*.local.el.example= in a project and - report which have no corresponding =*.local.el=. That is a one-line find and it - turns a silent no-op into a visible checklist item. -- Same for any =*.local.*= convention elsewhere in the fleet — the pattern is not - specific to Emacs. -- Worth pairing with gap 2: a unit that is enabled and failing every fifteen - minutes for two days is its own signal. A post-rebuild pass over - =systemctl --user list-units --state=failed= would have caught this one - without knowing anything about calendars. - -That last one generalizes best. Of the five gaps I have sent you, three were -things that *looked* fine — a stowed unit file, an enabled timer, a present -clone — and were not. diff --git a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-3.org b/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-3.org deleted file mode 100644 index 0df4d75..0000000 --- a/working/velox-reinstall/2026-08-14-emacsd-velox-reinstall-gaps-3.org +++ /dev/null @@ -1,98 +0,0 @@ -#+TITLE: Reinstall gaps, part three — per-install certs and credentials, and one failure that hid the others -#+AUTHOR: Craig Jennings - -* Third handoff today - -Two earlier notes covered five gaps -([[file:2026-08-14-emacsd-velox-reinstall-gaps-1.org][the first report]] and -[[file:2026-08-14-emacsd-velox-reinstall-gaps-2.org][the follow-up]]). -Email was the last thing broken on velox after the 2026-08-13 rebuild, and it -turned up two more — both the same shape, and one of them with a property worth -generalizing. - -Email is fully working now: three accounts, 21,853 messages, 4.0 GB indexed. - -* Gap 6 — the Proton Bridge TLS cert is per-install, and its absence disabled every account - -=~/.mbsyncrc= carries =CertificateFile /home/cjennings/.config/protonbridge.pem=. -That file did not exist after the rebuild, and it cannot be restored from backup -or copied from the other machine: Proton Bridge generates a fresh self-signed -cert per installation. Velox's is issued 2026-08-13 23:44 with a different -fingerprint from ratio's 2026-01-30 one. - -*The part worth acting on is the blast radius.* mbsync parses its entire config -before doing any work, so a missing =CertificateFile= referenced by *one* account -aborts the run for *all* of them. Gmail and dmail need no bridge and no cert, and -both were dead anyway. The error names only the missing pem, so the symptom -("no mail at all") and the message ("this one file is missing") look unrelated. - -Recovery does not need the bridge GUI. The running bridge presents the cert on -its own IMAP port, so it can be pulled straight off the handshake: - -: openssl s_client -connect 127.0.0.1:1143 -starttls imap -showcerts ~/.config/protonbridge.pem - -That is a two-second, fully scriptable step, which makes it a good candidate for -the install rather than a runbook line. - -* Gap 7 — the bridge password is per-install too, and reports a stale value misleadingly - -=~/.mbsyncrc= resolves the cmail password with =cat ~/.config/.cmailpass=. That -file is plaintext and, unusually for my setup, a real file rather than a stow -symlink — so it is not in the dotfiles repo, not encrypted, and not carried to a -new machine. - -The file survived the rebuild but held the *previous* install's password, because -the bridge regenerates it per installation. Ratio's and velox's differ by sha256, -confirmed today. - -*The diagnostic trap:* Proton Bridge answers a wrong password with =no such -user=. I read that as "the bridge has no account signed in" and went looking for -a login problem. The account was configured the whole time. If the install ever -validates bridge connectivity, it should not treat =no such user= as evidence -about account state. - -* The generalization - -Gaps 6 and 7 are the same as 1 through 5, sharpened. Everything that broke in -this rebuild was *generated on the machine by an application* rather than carried -by git, stow, or the dotfiles repo: - -| gap | artifact | why it did not travel | -| 1 | =.ai/=, =todo.org=, =CLAUDE.md= | gitignored | -| 2 | =timers.target.wants= symlinks | written by systemctl enable | -| 3 | roam clone state | local working tree | -| 4 | signal-cli registration | per-device identity | -| 5 | =*.local.el= configs | gitignored | -| 6 | bridge TLS cert | per-install, regenerated | -| 7 | bridge password | per-install, regenerated | - -Gaps 6 and 7 add a distinction the earlier note missed. For 1, 3 and 5 the old -value is still correct, so *restoring* fixes them. For 4, 6 and 7 the old value is -*worthless* — the application has generated a new one, and only *re-deriving* -from the live system fixes them. An install that tries to restore these will -produce exactly what happened here: a file that exists, looks right, and -authenticates against nothing. - -So the install's post-rebuild checklist wants two columns, not one: what to -restore, and what to re-derive. - -* What the install could do - -- Re-derive the bridge cert from the running bridge with the =openssl s_client= - line above. Scriptable, no GUI, no secrets. -- Re-derive the bridge password from the bridge rather than expecting the file to - be right, and rewrite =.cmailpass=. (I have filed a task on my side to make - =PassCmd= ask the bridge directly, which would remove the file entirely.) -- Add a cheap post-rebuild validation that =mbsync --list= parses. Config-parse - failures disable every account at once and say nothing about mail, so they are - worth catching explicitly rather than via "no new mail" hours later. -- More generally: keep the restore list and the re-derive list separate, per the - table above. - -* Unrelated, but noticed while comparing the machines - -=~/.config/.gmailpass.gpg= and =~/.config/.dmailpass.gpg= resolve to mode 777 in -the dotfiles repo, on both machines. They are gpg-encrypted so the contents are -safe, but world-writable is wrong for a credential file. That is a dotfiles fix, -not an archsetup one — noting it here only because it surfaced in the same pass. diff --git a/working/velox-reinstall/velox-reinstall-runbook.org b/working/velox-reinstall/velox-reinstall-runbook.org deleted file mode 100644 index 0c14946..0000000 --- a/working/velox-reinstall/velox-reinstall-runbook.org +++ /dev/null @@ -1,161 +0,0 @@ -#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup -#+AUTHOR: Craig Jennings -#+DATE: 2026-08-13 - -Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon -890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry, -and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the -pool. Decision: full reinstall via archangel + archsetup, run deliberately as -a disaster-recovery test of the ISO and scripts before the Sunday flight. -Recent backup in hand; ratio available as the working machine. - -Fallback ordering if the test finds a real gap: -- Before partitioning starts: the old system is intact — the ZBM repair - route (efibootmgr entry pointing at the ZBM loader on the ESP, then - amd-ucode swap in a chroot) is still available. -- After partitioning: the floor is a manual Arch install; the backup makes - that survivable. - -* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk) - -- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates - the microcode vendor-detection fix (archsetup, 2026-08-08) and was built - without ARCHSETUP_DIR at all. - #+begin_src sh - cd ~/code/archangel && sudo ARCHSETUP_DIR="$HOME/code/archsetup" ./build.sh - #+end_src - Two traps in that one line, and either alone silently produces a bare ISO - with archsetup absent (archangel, 2026-08-20). The =VAR=value= form is - required because sudo's =env_reset= discards an exported variable. And - =$HOME= is required because zsh does not expand a tilde on the right-hand - side of an assignment — the earlier =ARCHSETUP_DIR=~/code/archsetup= here - passed the literal string. build.sh now warns and reports baked/not-baked - in its closing summary, so the failure is visible rather than silent. -- [ ] build.sh fixes before the final rebuild (archangel repo): - - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the - credential audit — off the portable USB stick). - - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile - is on the ISO at =/root/velox-zfs.conf=. -- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=), - =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs - spot-check. -- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy - drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16 - archangel ISOs removed. Boot the stick and pick - =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu. - -* Phase 1 — UEFI setup on velox (BIOS screen, before any boot) - -- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned; - the new board ships with it enforced by factory default. -- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01; - a wrong clock breaks TLS and pacman signature checks in the live env. - Rough accuracy is fine — NTP tightens it once networked. -- [ ] While you're in setup: check boot-order UI shows the USB. - -* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED - -NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is -DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke -around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio -and mybitch backed up today). Everything since Jul 6 exists only on the old -SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO -backup at all. This pass also keeps the repair fallback alive until -partitioning starts. - -- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=. -- [ ] Import the old pool read-only and unlock: - #+begin_src sh - zpool import -N -o readonly=on -R /mnt zroot - zfs load-key zroot # passphrase prompt - zfs mount zroot/ROOT/default - zfs mount -a 2>/dev/null # home datasets etc.; ignore failures - #+end_src -- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout - the backup job uses (etc + home), into a clearly-named one-off dir: - #+begin_src sh - rsync -aHAX --info=progress2 /mnt/etc /mnt/home \ - truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/ - #+end_src - (=/usr= is in the regular backups but is all reinstallable — skip unless - paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.) -- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest - files in =~/documents= and =~/downloads=. -- [ ] Check for uncommitted repo work and either push or note it: - =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge), - =~/.dotfiles=, anything under =~/code=. -- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=. - -* Phase 3 — Install (the actual DR test) - -- [ ] Review the profile, then run the installer: - #+begin_src sh - less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme - archangel --config-file /root/velox-zfs.conf - #+end_src - Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome= - placeholders — fine for install; both change post-install (=zfs change-key - zroot= for the pool, =passwd= for root). -- [ ] Record every rough edge as a DR-test finding — that's the point of - running it this way. Anything that needs a manual nudge gets a todo entry - in archangel or archsetup afterward. -- [ ] Reboot into ZBM → boot the new environment. - -* Phase 4 — archsetup (first boot of the installed system) - -- [ ] Log in as root, network up, then verify the clock synced. -- [ ] Get archsetup — two paths, test the offline one since this is a DR - drill (the online curl path is the everyday alternative): - #+begin_src sh - # offline: mount the install USB and copy the baked tree - mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt - cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup - ./archsetup - #+end_src -- [ ] Expected on the new board: =install_cpu_microcode= detects - AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests). - Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all - in the installer now — none need manual application afterward. -- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel - assumptions (swept 2026-08-13); maint's capability probe runtime-detects - amd-pstate. - -* Phase 5 — Post-install restore + verification - -- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The - secrets-bundle-in-ISO design is not built yet — manual restore is the - known gap, not a test failure. -- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=; - re-import the NM profile (autoconnect off, as before). -- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=. -- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known - idle instability (Panel Self Refresh hangs/reboots the machine; hit during - the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the - installed system's kernel command line — velox boots via ZBM, so set it on - the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default= - (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a - kernel that fixes PSR on Strix Point — track via the Framework issue - tracker (SoftwareFirmwareIssueTracker #110). -- [ ] New-hardware spot-checks: - - =journalctl -k | grep -i microcode= — amd-ucode applied. - - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect - amd-pstate(-epp). - - wifi + bluetooth up (new board radios), touchpad behavior, camera. - - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV. -- [ ] Fresh clones automatically carry the post-purge rewritten git history — - closes the clone-reconcile rider from 2026-08-11 without action. -- [ ] Fix and verify the backup timer on the fresh install — it was silently - broken since ~Jul 6. After the first manual run succeeds, confirm a new - DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it - broke (timer unit dead? mount failure? credential?) if the old journal - survives in the salvage copy. -- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate), - both daily drivers AMD. Fix the stale =intel_pstate= comment in - =airplane-mode= line 6 while at it (cosmetic). -- [ ] File every DR-test finding in the owning project's todo. - -* Timing - -Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or -Friday, leaving Saturday as pure buffer. If the install stalls past Friday -evening, cut losses to the manual-install floor. diff --git a/working/velox-reinstall/velox-uefi-boot-entry-reference.org b/working/velox-reinstall/velox-uefi-boot-entry-reference.org deleted file mode 100644 index 53e6511..0000000 --- a/working/velox-reinstall/velox-uefi-boot-entry-reference.org +++ /dev/null @@ -1,108 +0,0 @@ -#+TITLE: Velox UEFI Boot Entry — Recovery Reference -#+AUTHOR: Craig Jennings -#+DATE: 2026-08-15 - -Captured 2026-08-15 before a BIOS update (03.05 → 04.02) as insurance against -the update clearing NVRAM. Velox's mainboard swap on 2026-08-13 left exactly -this kind of empty NVRAM, which is what forced the reinstall — so a cleared -boot entry is the specific failure worth being able to undo in one command -rather than reconstruct. - -* Before recreating anything: check Secure Boot first - -The 04.02 update landed on 2026-09-12 (staged via fwupd, flashed on the next -reboot). It did NOT clear NVRAM: Boot0001 survived with its command line -intact. What it did was re-enable Secure Boot (Enforce Secure Boot = -Enabled), so the unsigned ZBM loader was rejected and the Framework BIOS -reported it as "Default Boot Device Missing / no bootable drive" rather than -a security violation. The symptom is indistinguishable from the NVRAM wipe -this document was written for. - -The tell: booting the Ventoy stick shows shim's MOK management screen. - -Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. The -machine then boots straight into ZBM. No efibootmgr needed. - -So when velox says no bootable device after a firmware update, check Secure -Boot before touching the boot entries. Only if Secure Boot is already off -and =efibootmgr -v= (from the stick) shows Boot0001 gone does the recreate -below apply. - -Before any future firmware update, record both =efibootmgr -v= and the -Secure Boot state so the post-reboot diagnosis is a comparison, not a guess. -The pre-firmware-update checklist in -[[file:../../docs/workflows/system-health-check.org][docs/workflows/system-health-check.org]] -(Phase 3) carries the steps. - -Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT= -does not exist), so a real NVRAM wipe would still need the stick. Copying -=zfsbootmenu.efi= to =/efi/EFI/BOOT/BOOTX64.EFI= would let it boot unaided. -That belongs to archangel's ZBM install, where it is filed as [#C] -"Installed systems have no removable-media boot fallback on the ESP" -(2026-09-12) and ships with the next ISO rebuild after it lands. - -* State at capture - -- BIOS: 03.05 (2025-10-30) -- BootCurrent: 0001 -- BootOrder: 2001,0001,2002,2003 (USB ahead of ZBM — why the Ventoy stick - boots when it's inserted) -- Timeout: 0 seconds - -* The entry that matters - -=Boot0001* ZFSBootMenu= - -| field | value | -|----------------+----------------------------------------------| -| ESP part GUID | 8e51b680-f90a-444f-8da5-7e4f93625775 | -|----------------+----------------------------------------------| -| partition | 1 (GPT), start 0x800, size 0x100000 | -|----------------+----------------------------------------------| -| loader path | =\EFI\ZBM\zfsbootmenu.efi= | -|----------------+----------------------------------------------| -| cmdline (data) | =spl_hostid=0x22f8a7a1 zbm.timeout=3= | -| | =zbm.prefer=zroot zbm.import_policy=hostid= | -|----------------+----------------------------------------------| - -The =data= field is that command line in UTF-16LE, which is how efibootmgr -passes it as optional data. Recreate with =-u= and the plain string; efibootmgr -does the encoding. - -* Recreating it - -From a booted system (or the archangel ISO), with the ESP identified as -=/dev/nvme0n1p1= or whatever it enumerates as: - -#+begin_src bash -efibootmgr --create \ - --disk /dev/nvme0n1 --part 1 \ - --label "ZFSBootMenu" \ - --loader '\EFI\ZBM\zfsbootmenu.efi' \ - --unicode 'spl_hostid=0x22f8a7a1 zbm.timeout=3 zbm.prefer=zroot zbm.import_policy=hostid' -#+end_src - -Confirm the disk/part against =lsblk -o NAME,PARTUUID,PARTTYPENAME= first — -the partition GUID above is the authoritative identifier, not the device name, -which can enumerate differently. - -Then set the order so ZBM is reachable: - -#+begin_src bash -efibootmgr --bootorder 0001,2001,2002,2003 -#+end_src - -(The original order put USB first. Keep whichever you prefer; what matters is -that the ZBM entry exists and is in the list.) - -* Other entries (firmware-generated, recreate themselves) - -| Boot2001 | EFI USB Device | -|----------+----------------| -| Boot2002 | EFI DVD/CDROM | -|----------+----------------| -| Boot2003 | EFI Network | -|----------+----------------| - -These are stock firmware entries and come back on their own. Only Boot0001 -carries anything unique. diff --git a/working/velox-reinstall/wttrin-bf0457f.bundle b/working/velox-reinstall/wttrin-bf0457f.bundle deleted file mode 100644 index 5921866..0000000 Binary files a/working/velox-reinstall/wttrin-bf0457f.bundle and /dev/null differ -- cgit v1.2.3