diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/2026-07-15-velox-boot-failure-handoff.org | 61 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-1.org | 137 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-2.org | 63 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-3.org | 98 |
4 files changed, 359 insertions, 0 deletions
diff --git a/docs/design/2026-07-15-velox-boot-failure-handoff.org b/docs/design/2026-07-15-velox-boot-failure-handoff.org new file mode 100644 index 0000000..5ec996f --- /dev/null +++ b/docs/design/2026-07-15-velox-boot-failure-handoff.org @@ -0,0 +1,61 @@ +#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-15 + +* Why this is coming to archsetup + +Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below. + +A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox. + +* Timeline + +- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50). +- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times). +- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel. +- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session. + +* Facts established so far (from the ZBM recovery shell) + +- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem. +- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective. +- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be. +- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone. + +* Working hypothesis + +A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent. + +* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell) + +1. Read pacman's log (on the zroot/var dataset): + #+begin_src sh + mkdir -p /mnt/var + mount -t zfs -o zfsutil zroot/var /mnt/var + tail -60 /mnt/var/log/pacman.log + #+end_src + Expect the failed/interrupted kernel transaction near the end; note its timestamp. +2. List recovery candidates: + #+begin_src sh + zfs list -t snapshot zroot/ROOT/default | tail -20 + #+end_src + Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox. + +* Recovery plan (agreed with Craig, pending the log read) + +1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction. +2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot. +3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched). +4. zpool export zroot, reboot — ZBM should now see the kernel. +5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown. + +* Retrospective candidates for archsetup + +- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion. +- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically. +- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above. +- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors. + +* Related loose end already in your inbox + +A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-1.org b/docs/design/2026-08-14-velox-reinstall-gaps-1.org new file mode 100644 index 0000000..cf0d723 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-1.org @@ -0,0 +1,137 @@ +#+TITLE: What the velox reinstall left behind — four gaps the install could close +#+AUTHOR: Craig Jennings + +* Heads-up: this was found from a .emacs.d session + +I opened a .emacs.d session on velox this morning, two days after the fresh +Arch install, and the first thing it did was fail: there was no =.ai/= +directory to read. Chasing that turned up four separate things the reinstall +did not restore. Three I repaired from the session; one needs me at my phone. + +None of this is a .emacs.d bug. They are all install-side gaps, which is why +they are landing in your inbox. Machine is velox; ratio was the reference for +every comparison below. + +* Gap 1 — the gitignored tooling layer does not survive a reinstall + +=~/.emacs.d= was re-cloned on 2026-08-13. Git brought back every tracked file +and none of the agent tooling, because =.gitignore= deliberately excludes it: +=.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, and =inbox/= were all simply +absent. That is the correct ignore policy — this repo relays to a public +mirror — but it means a reinstall silently drops the entire working state of +every gitignore-mode project. + +The damage on velox was total rather than partial: 374 files, 4.5 MB, +including =todo.org= (556 KB) and 184 archived session files. Nothing carries +it. Not git, not stow, not the bootstrap. + +I recovered it by rsyncing the set from ratio over the tailnet. Ratio was +authoritative and velox held nothing, so there was no merge to adjudicate — +which is luck, not design. Had velox held a few days of divergent state, this +would have been a hand reconciliation. It has been one before: 2026-07-31, when +the two machines' =.ai/= trees had forked to zero files in common. + +Worth knowing: this is fleet-general. Every project on the box that gitignores +its =.ai/= has the same hole, not just =.emacs.d=. + +What the install could do: after cloning a project, check whether a sibling +daily driver holds a =.ai/= for it, and offer to pull it across. Or at minimum, +list the projects whose tooling layer is missing so the gap is visible on day +one instead of at the first session that trips over it. + +* Gap 2 — stowed user timers come back linked but not enabled + +The unit files all arrived correctly through the dotfiles stow, symlinked into +=~/.config/systemd/user/= and resolving fine. But being present is not being +enabled, and the reinstall enabled only some of them: + +| unit | velox after reinstall | ratio | +|---------------------------+-----------------------+----------| +| calendar-sync.timer | enabled, active | enabled | +| agenda-render-cache.timer | enabled, active | enabled | +| roam-sync.timer | *linked, inactive* | enabled | +| signal-receive.timer | *linked, inactive* | enabled | +| emacs.service | linked, inactive | linked | + +=emacs.service= reads the same on both machines, so I take that one as +intentional and left it alone. The other two are real drift: =systemctl --user +enable= writes a =timers.target.wants= symlink into =~/.config/systemd/user/=, +and that symlink is not stow-managed, so nothing in the dotfiles repo carries +it. A stowed unit file is inert until something enables it. + +I enabled both with =systemctl --user enable --now=. Both fired immediately and +exited clean, and both now show a next elapse. + +What the install could do: enable the units it stows, explicitly, as a named +step. The inconsistency is the tell — two of four came back enabled, which +suggests something enables a subset and nothing enumerates the rest. + +* Gap 3 — the roam clone was stale, and held a diff that would have destroyed data + +This one has an ordering constraint, so it matters more than its size suggests. + +velox's =~/org/roam= was ten commits behind ratio, stuck at the 2026-08-04 +auto-sync while ratio was at 2026-08-14 — a direct consequence of gap 2, since +=roam-sync.timer= was never enabled here. + +The dangerous part: velox's clone also carried an *uncommitted* =inbox.org= +that had been emptied. Seventeen deletions, file down to zero bytes, holding a +pre-2026-08-04 state whose captures were long since processed on ratio. + +So the naive repair — enable =roam-sync.timer= and let it catch up — would have +committed that emptying and pushed it, deleting the four live inbox items on +ratio. The timer is the repo's only committer and it commits whatever it finds. + +I checked ratio's =inbox.org= first and confirmed it was a strict superset of +velox's HEAD version (same three items plus an 2026-08-09 capture), which made +the local change provably worthless. Then discarded it, fast-forwarded to +=a411b43=, and only then enabled the timer. Clone is clean and current, first +sync ran green. + +What the install could do: if it ever enables =roam-sync= on a rebuilt machine, +reconcile the clone *before* enabling, not after. An auto-committing timer +pointed at a stale dirty clone is a data-loss path, and the failure is silent +and remote — it lands on the *other* machine. + +* Gap 4 — signal-cli lost its registration, and that breaks the whole fleet + +=signal-receive.service= ran for the first time and reported: + +: signal-receive: +15045173983 not registered on this machine — nothing to do + +velox's signal-cli data dir holds a 39-byte empty =accounts.json=. Ratio still +has both numbers. So the reinstall wiped the registration, and per the design +notes velox was supposed to be the *primary* — ratio is the linked device. + +The effect is wider than velox, because of how =agent-text= dispatches: if the +local signal-cli holds the account it sends directly, otherwise it ssh-relays to +a hardcoded velox. Velox no longer holds it, so a send from here relays to +itself and fails; a send from any third machine relays to velox and fails the +same way. Only ratio still works, and only via the direct branch. The error text +blames "velox down or unreachable", which is misleading — velox is up and on the +tailnet, it just is not registered. + +This is the one I could not repair from the session: re-linking needs me at my +phone (Signal → Settings → Linked Devices, scanning the QR from =signal-cli +link -n velox=). Filed in .emacs.d's todo.org as [#B]. + +What the install could do: verify =signal-cli listAccounts= is non-empty after a +rebuild and say so loudly if it is not. Silent loss of the phone channel is +exactly the kind of thing nobody notices until the page that mattered never +arrives. + +* Summary of what I changed on velox + +- Restored =.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, =inbox/= to + =~/.emacs.d= by rsync from ratio. +- Discarded the stale local =inbox.org= diff in =~/org/roam= and fast-forwarded + the clone to current. +- Enabled and started =roam-sync.timer= and =signal-receive.timer=. + +Left alone, deliberately: =emacs.service= (matches ratio), and velox's Signal +registration (needs the phone). + +One unrelated thing I noticed while comparing the machines: ratio's signal-cli +warns its messages were last received twelve days ago, even though its +=signal-receive.timer= is enabled and active. That may be nothing, but the +receive cadence there is worth a look. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-2.org b/docs/design/2026-08-14-velox-reinstall-gaps-2.org new file mode 100644 index 0000000..95842ac --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-2.org @@ -0,0 +1,63 @@ +#+TITLE: Fifth reinstall gap — machine-local .local.el config, and a general shape +#+AUTHOR: Craig Jennings + +* Follow-up to this morning's handoff + +Sent you four gaps an hour ago +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]]). Here is a fifth, +found straight afterwards when I noticed calendar sync was dead on velox. + +* What was broken + +=calendar-sync.timer= was enabled and firing every fifteen minutes, and failing +every time with exit 255: + +: calendar-sync: No calendars configured (set calendar-sync-calendars) + +The three output files sat at zero bytes. The cause is that +=~/.emacs.d/calendar-sync.local.el= is gitignored, so the reinstall deleted it +along with everything else untracked, and the module's loader treats a missing +file as a *silent* no-op. So the config vanished quietly and the only symptom +was a failing unit nobody was watching. + +Cheap to fix once found: the repo tracks =calendar-sync.local.el.example=, and +that template already encodes the shape velox uses — feeds resolved by +=:secret-host= against =authinfo.gpg= rather than inlined. The authinfo entries +had survived, because =~/.authinfo.gpg= is a stow symlink into the dotfiles repo. +So rebuilding was one copy, and all three feeds now sync clean and land +byte-identical to ratio's. + +* The general shape, which is the part worth acting on + +This is the same failure as gap 1, one layer down, and it is worth stating +generally because the install can act on it: + +- A tracked =*.local.el.example= template plus a gitignored =*.local.el= is a + deliberate pattern in this config, not a one-off. =.gitignore= lines 56-58 + list three of them: =calendar-sync.local.el=, =signal-config.local.el=, + =google-keep.local.el=. Every one of those is gone on velox right now. I have + only repaired the calendar one. +- Secrets held *by reference* survive a rebuild; secrets held *inline* do not. + The calendar config came back for free because the tokens were in + =authinfo.gpg=, which is stow-managed and therefore travels. Ratio's copy of + the same file inlines its URLs, and had ratio been the machine rebuilt, those + three feed tokens would simply have been gone. +- The failure was silent by design. A missing local config is a no-op, which is + right for a machine that never configured the feature and wrong for one that + just lost it. + +* What the install could do + +- After a rebuild, enumerate every tracked =*.local.el.example= in a project and + report which have no corresponding =*.local.el=. That is a one-line find and it + turns a silent no-op into a visible checklist item. +- Same for any =*.local.*= convention elsewhere in the fleet — the pattern is not + specific to Emacs. +- Worth pairing with gap 2: a unit that is enabled and failing every fifteen + minutes for two days is its own signal. A post-rebuild pass over + =systemctl --user list-units --state=failed= would have caught this one + without knowing anything about calendars. + +That last one generalizes best. Of the five gaps I have sent you, three were +things that *looked* fine — a stowed unit file, an enabled timer, a present +clone — and were not. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-3.org b/docs/design/2026-08-14-velox-reinstall-gaps-3.org new file mode 100644 index 0000000..9675973 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-3.org @@ -0,0 +1,98 @@ +#+TITLE: Reinstall gaps, part three — per-install certs and credentials, and one failure that hid the others +#+AUTHOR: Craig Jennings + +* Third handoff today + +Two earlier notes covered five gaps +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]] and +[[file:2026-08-14-velox-reinstall-gaps-2.org][the follow-up]]). +Email was the last thing broken on velox after the 2026-08-13 rebuild, and it +turned up two more — both the same shape, and one of them with a property worth +generalizing. + +Email is fully working now: three accounts, 21,853 messages, 4.0 GB indexed. + +* Gap 6 — the Proton Bridge TLS cert is per-install, and its absence disabled every account + +=~/.mbsyncrc= carries =CertificateFile /home/cjennings/.config/protonbridge.pem=. +That file did not exist after the rebuild, and it cannot be restored from backup +or copied from the other machine: Proton Bridge generates a fresh self-signed +cert per installation. Velox's is issued 2026-08-13 23:44 with a different +fingerprint from ratio's 2026-01-30 one. + +*The part worth acting on is the blast radius.* mbsync parses its entire config +before doing any work, so a missing =CertificateFile= referenced by *one* account +aborts the run for *all* of them. Gmail and dmail need no bridge and no cert, and +both were dead anyway. The error names only the missing pem, so the symptom +("no mail at all") and the message ("this one file is missing") look unrelated. + +Recovery does not need the bridge GUI. The running bridge presents the cert on +its own IMAP port, so it can be pulled straight off the handshake: + +: openssl s_client -connect 127.0.0.1:1143 -starttls imap -showcerts </dev/null \ +: | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > ~/.config/protonbridge.pem + +That is a two-second, fully scriptable step, which makes it a good candidate for +the install rather than a runbook line. + +* Gap 7 — the bridge password is per-install too, and reports a stale value misleadingly + +=~/.mbsyncrc= resolves the cmail password with =cat ~/.config/.cmailpass=. That +file is plaintext and, unusually for my setup, a real file rather than a stow +symlink — so it is not in the dotfiles repo, not encrypted, and not carried to a +new machine. + +The file survived the rebuild but held the *previous* install's password, because +the bridge regenerates it per installation. Ratio's and velox's differ by sha256, +confirmed today. + +*The diagnostic trap:* Proton Bridge answers a wrong password with =no such +user=. I read that as "the bridge has no account signed in" and went looking for +a login problem. The account was configured the whole time. If the install ever +validates bridge connectivity, it should not treat =no such user= as evidence +about account state. + +* The generalization + +Gaps 6 and 7 are the same as 1 through 5, sharpened. Everything that broke in +this rebuild was *generated on the machine by an application* rather than carried +by git, stow, or the dotfiles repo: + +| gap | artifact | why it did not travel | +| 1 | =.ai/=, =todo.org=, =CLAUDE.md= | gitignored | +| 2 | =timers.target.wants= symlinks | written by systemctl enable | +| 3 | roam clone state | local working tree | +| 4 | signal-cli registration | per-device identity | +| 5 | =*.local.el= configs | gitignored | +| 6 | bridge TLS cert | per-install, regenerated | +| 7 | bridge password | per-install, regenerated | + +Gaps 6 and 7 add a distinction the earlier note missed. For 1, 3 and 5 the old +value is still correct, so *restoring* fixes them. For 4, 6 and 7 the old value is +*worthless* — the application has generated a new one, and only *re-deriving* +from the live system fixes them. An install that tries to restore these will +produce exactly what happened here: a file that exists, looks right, and +authenticates against nothing. + +So the install's post-rebuild checklist wants two columns, not one: what to +restore, and what to re-derive. + +* What the install could do + +- Re-derive the bridge cert from the running bridge with the =openssl s_client= + line above. Scriptable, no GUI, no secrets. +- Re-derive the bridge password from the bridge rather than expecting the file to + be right, and rewrite =.cmailpass=. (I have filed a task on my side to make + =PassCmd= ask the bridge directly, which would remove the file entirely.) +- Add a cheap post-rebuild validation that =mbsync --list= parses. Config-parse + failures disable every account at once and say nothing about mail, so they are + worth catching explicitly rather than via "no new mail" hours later. +- More generally: keep the restore list and the re-derive list separate, per the + table above. + +* Unrelated, but noticed while comparing the machines + +=~/.config/.gmailpass.gpg= and =~/.config/.dmailpass.gpg= resolve to mode 777 in +the dotfiles repo, on both machines. They are gpg-encrypted so the contents are +safe, but world-writable is wrong for a credential file. That is a dotfiles fix, +not an archsetup one — noting it here only because it surfaced in the same pass. |