diff options
Diffstat (limited to 'docs/design')
| -rw-r--r-- | docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 | ||||
| -rw-r--r-- | docs/design/2026-07-15-velox-boot-failure-handoff.org | 61 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-1.org | 137 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-2.org | 63 | ||||
| -rw-r--r-- | docs/design/2026-08-14-velox-reinstall-gaps-3.org | 98 |
5 files changed, 362 insertions, 1 deletions
diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 74790c6..4d57b86 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,6 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSSEC=yes= validates RRSIG inception/expiry windows against the wall clock, so a clock weeks off fails every query before it leaves the machine. Measured on velox 2026-08-19 with the clock wound back 27 days: resolved logged =signature-expired= against the root DNSKEY and every DS beneath it, and resolution died outright. =DNSOverTLS=yes= is *not* what bites, despite being the obvious suspect — the DoT handshake to =1.1.1.1:853= verified clean at that same clock, because a resolver certificate is good for about a year while an RRSIG window is days to weeks. A skew large enough to break DNSSEC normally leaves the certificate valid. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. =DNSSEC=allow-downgrade= does *not* help here, which is worth knowing because it is the obvious reach: resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure rather than a support failure, so no downgrade fires. Measured on velox: six retries over eighteen seconds, plus =resolvectl reset-server-features=, all dead. The only cure is correcting the clock, which is why the NTP source has to be reachable without DNS. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -107,7 +108,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] - Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] - Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] -- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. This entry assumes DNS still works; when the resolver runs DoT or DNSSEC the same skew kills DNS first and =set-ntp true= cannot recover it — see the clock/DNS deadlock in the DNS layer. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] - Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] - VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] - IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify <con> ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] @@ -309,6 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- Clock skew breaks DNSSEC validation, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/docs/design/2026-07-15-velox-boot-failure-handoff.org b/docs/design/2026-07-15-velox-boot-failure-handoff.org new file mode 100644 index 0000000..5ec996f --- /dev/null +++ b/docs/design/2026-07-15-velox-boot-failure-handoff.org @@ -0,0 +1,61 @@ +#+TITLE: Velox boot failure — ZBM found no bootable kernel; diagnosis in progress, recovery plan attached +#+AUTHOR: Craig Jennings +#+DATE: 2026-07-15 + +* Why this is coming to archsetup + +Velox fails to boot: ZFSBootMenu reports it can't find a bootable environment with a kernel. Craig reports the last working velox session was an archsetup health-check run that included the pacman upgrade — so the breakage most likely happened inside archsetup's own workflow, and Craig wants the diagnosis + retrospective to continue here with full context. The .emacs.d session (where this was triaged, only because that's where Craig was sitting) hands off everything below. + +A phone photo of the zfs list output from velox's ZBM recovery shell accompanies this note in the inbox. + +* Timeline + +- 2026-07-13 ~23:50 CDT — velox last seen on the tailnet (per tailscale status read 2026-07-14 ~17:50). +- During that last session: archsetup health-check workflow ran, including a pacman upgrade (Craig's recollection — pacman.log will confirm exact times). +- 2026-07-14 late evening — Craig boots velox; ZBM: no bootable environment with a kernel. +- 2026-07-14/15 — triage from the ZBM recovery shell, Craig driving, guided from the .emacs.d session. + +* Facts established so far (from the ZBM recovery shell) + +- zroot imported, health ONLINE. Every dataset's keystatus is "available" — encryption unlocked, not a key problem. +- Layout confirmed from zfs list: zroot/ROOT/default (mountpoint /), separate datasets for home, home/root, media, var, var/cache, var/lib, var/lib/docker plus many docker layer children (legacy mountpoints). NOTE: no separate zroot/var/log dataset — /var/log lives inside zroot/var. That differs from the sanoid dataset list in archsetup's configure_zfs_snapshots (which configures zroot/var/log and zroot/var/lib/pacman as their own datasets) — worth reconciling in the retrospective. +- Mounted the BE read-only style: mkdir -p /mnt/be && mount -t zfs -o zfsutil zroot/ROOT/default /mnt/be. +- THE FINDING: /mnt/be/boot contains ONLY intel-ucode.img. vmlinuz-linux, initramfs-linux.img, and initramfs-linux-fallback.img are all gone. + +* Working hypothesis + +A kernel upgrade during the health-check run removed the old kernel files and never completed installing the new ones (interrupted transaction, mkinitcpio failure, or a /boot shadowing issue), and the machine was powered off with /boot empty. Arch's upgrade removes the running kernel's files at package-replace time, so a failure between "remove old" and "install new + mkinitcpio" leaves exactly this state: microcode present, kernel and initramfs absent. + +* Remaining diagnosis steps (not yet run — velox is sitting at the ZBM shell) + +1. Read pacman's log (on the zroot/var dataset): + #+begin_src sh + mkdir -p /mnt/var + mount -t zfs -o zfsutil zroot/var /mnt/var + tail -60 /mnt/var/log/pacman.log + #+end_src + Expect the failed/interrupted kernel transaction near the end; note its timestamp. +2. List recovery candidates: + #+begin_src sh + zfs list -t snapshot zroot/ROOT/default | tail -20 + #+end_src + Sanoid is configured for hourly=6/daily=7 on the ROOT dataset, so a pre-damage snapshot should exist. Check whether any pre-pacman_* snapshots appear — that tells us whether the 2026-06-29 pre-pacman hook design is actually installed on velox. + +* Recovery plan (agreed with Craig, pending the log read) + +1. Pick the newest zroot/ROOT/default snapshot that predates the failed transaction. +2. If the pool is imported read-only (zpool get readonly zroot): zpool export zroot && zpool import -f -N zroot. +3. zfs rollback -r zroot/ROOT/default@<snapshot> (the -r discards snapshots newer than the target; home/var/media are separate datasets and untouched). +4. zpool export zroot, reboot — ZBM should now see the kernel. +5. After first boot: re-run pacman -Syu attended, and confirm /boot holds vmlinuz-linux + initramfs-linux.img before any shutdown. + +* Retrospective candidates for archsetup + +- Does the health-check / upgrade flow verify /boot contents (kernel + initramfs present, mkinitcpio exit status) after a kernel upgrade? This failure would have been caught by a one-line post-upgrade assertion. +- Is the pre-pacman snapshot hook (2026-06-29 design, zroot/ROOT/default@pre-pacman_<ts>) installed on velox? The snapshot listing in step 2 above answers this empirically. +- The sanoid config vs actual dataset layout mismatch (var/log, var/lib/pacman) noted above. +- Whether the upgrade step should refuse to end the session (or page Craig) when a kernel transaction errors. + +* Related loose end already in your inbox + +A separate note (2026-07-14-1751) asks to add inetutils to the install base; velox also still needs that package installed once it boots again. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-1.org b/docs/design/2026-08-14-velox-reinstall-gaps-1.org new file mode 100644 index 0000000..cf0d723 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-1.org @@ -0,0 +1,137 @@ +#+TITLE: What the velox reinstall left behind — four gaps the install could close +#+AUTHOR: Craig Jennings + +* Heads-up: this was found from a .emacs.d session + +I opened a .emacs.d session on velox this morning, two days after the fresh +Arch install, and the first thing it did was fail: there was no =.ai/= +directory to read. Chasing that turned up four separate things the reinstall +did not restore. Three I repaired from the session; one needs me at my phone. + +None of this is a .emacs.d bug. They are all install-side gaps, which is why +they are landing in your inbox. Machine is velox; ratio was the reference for +every comparison below. + +* Gap 1 — the gitignored tooling layer does not survive a reinstall + +=~/.emacs.d= was re-cloned on 2026-08-13. Git brought back every tracked file +and none of the agent tooling, because =.gitignore= deliberately excludes it: +=.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, and =inbox/= were all simply +absent. That is the correct ignore policy — this repo relays to a public +mirror — but it means a reinstall silently drops the entire working state of +every gitignore-mode project. + +The damage on velox was total rather than partial: 374 files, 4.5 MB, +including =todo.org= (556 KB) and 184 archived session files. Nothing carries +it. Not git, not stow, not the bootstrap. + +I recovered it by rsyncing the set from ratio over the tailnet. Ratio was +authoritative and velox held nothing, so there was no merge to adjudicate — +which is luck, not design. Had velox held a few days of divergent state, this +would have been a hand reconciliation. It has been one before: 2026-07-31, when +the two machines' =.ai/= trees had forked to zero files in common. + +Worth knowing: this is fleet-general. Every project on the box that gitignores +its =.ai/= has the same hole, not just =.emacs.d=. + +What the install could do: after cloning a project, check whether a sibling +daily driver holds a =.ai/= for it, and offer to pull it across. Or at minimum, +list the projects whose tooling layer is missing so the gap is visible on day +one instead of at the first session that trips over it. + +* Gap 2 — stowed user timers come back linked but not enabled + +The unit files all arrived correctly through the dotfiles stow, symlinked into +=~/.config/systemd/user/= and resolving fine. But being present is not being +enabled, and the reinstall enabled only some of them: + +| unit | velox after reinstall | ratio | +|---------------------------+-----------------------+----------| +| calendar-sync.timer | enabled, active | enabled | +| agenda-render-cache.timer | enabled, active | enabled | +| roam-sync.timer | *linked, inactive* | enabled | +| signal-receive.timer | *linked, inactive* | enabled | +| emacs.service | linked, inactive | linked | + +=emacs.service= reads the same on both machines, so I take that one as +intentional and left it alone. The other two are real drift: =systemctl --user +enable= writes a =timers.target.wants= symlink into =~/.config/systemd/user/=, +and that symlink is not stow-managed, so nothing in the dotfiles repo carries +it. A stowed unit file is inert until something enables it. + +I enabled both with =systemctl --user enable --now=. Both fired immediately and +exited clean, and both now show a next elapse. + +What the install could do: enable the units it stows, explicitly, as a named +step. The inconsistency is the tell — two of four came back enabled, which +suggests something enables a subset and nothing enumerates the rest. + +* Gap 3 — the roam clone was stale, and held a diff that would have destroyed data + +This one has an ordering constraint, so it matters more than its size suggests. + +velox's =~/org/roam= was ten commits behind ratio, stuck at the 2026-08-04 +auto-sync while ratio was at 2026-08-14 — a direct consequence of gap 2, since +=roam-sync.timer= was never enabled here. + +The dangerous part: velox's clone also carried an *uncommitted* =inbox.org= +that had been emptied. Seventeen deletions, file down to zero bytes, holding a +pre-2026-08-04 state whose captures were long since processed on ratio. + +So the naive repair — enable =roam-sync.timer= and let it catch up — would have +committed that emptying and pushed it, deleting the four live inbox items on +ratio. The timer is the repo's only committer and it commits whatever it finds. + +I checked ratio's =inbox.org= first and confirmed it was a strict superset of +velox's HEAD version (same three items plus an 2026-08-09 capture), which made +the local change provably worthless. Then discarded it, fast-forwarded to +=a411b43=, and only then enabled the timer. Clone is clean and current, first +sync ran green. + +What the install could do: if it ever enables =roam-sync= on a rebuilt machine, +reconcile the clone *before* enabling, not after. An auto-committing timer +pointed at a stale dirty clone is a data-loss path, and the failure is silent +and remote — it lands on the *other* machine. + +* Gap 4 — signal-cli lost its registration, and that breaks the whole fleet + +=signal-receive.service= ran for the first time and reported: + +: signal-receive: +15045173983 not registered on this machine — nothing to do + +velox's signal-cli data dir holds a 39-byte empty =accounts.json=. Ratio still +has both numbers. So the reinstall wiped the registration, and per the design +notes velox was supposed to be the *primary* — ratio is the linked device. + +The effect is wider than velox, because of how =agent-text= dispatches: if the +local signal-cli holds the account it sends directly, otherwise it ssh-relays to +a hardcoded velox. Velox no longer holds it, so a send from here relays to +itself and fails; a send from any third machine relays to velox and fails the +same way. Only ratio still works, and only via the direct branch. The error text +blames "velox down or unreachable", which is misleading — velox is up and on the +tailnet, it just is not registered. + +This is the one I could not repair from the session: re-linking needs me at my +phone (Signal → Settings → Linked Devices, scanning the QR from =signal-cli +link -n velox=). Filed in .emacs.d's todo.org as [#B]. + +What the install could do: verify =signal-cli listAccounts= is non-empty after a +rebuild and say so loudly if it is not. Silent loss of the phone channel is +exactly the kind of thing nobody notices until the page that mattered never +arrives. + +* Summary of what I changed on velox + +- Restored =.ai/=, =.claude/=, =CLAUDE.md=, =todo.org=, =inbox/= to + =~/.emacs.d= by rsync from ratio. +- Discarded the stale local =inbox.org= diff in =~/org/roam= and fast-forwarded + the clone to current. +- Enabled and started =roam-sync.timer= and =signal-receive.timer=. + +Left alone, deliberately: =emacs.service= (matches ratio), and velox's Signal +registration (needs the phone). + +One unrelated thing I noticed while comparing the machines: ratio's signal-cli +warns its messages were last received twelve days ago, even though its +=signal-receive.timer= is enabled and active. That may be nothing, but the +receive cadence there is worth a look. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-2.org b/docs/design/2026-08-14-velox-reinstall-gaps-2.org new file mode 100644 index 0000000..95842ac --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-2.org @@ -0,0 +1,63 @@ +#+TITLE: Fifth reinstall gap — machine-local .local.el config, and a general shape +#+AUTHOR: Craig Jennings + +* Follow-up to this morning's handoff + +Sent you four gaps an hour ago +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]]). Here is a fifth, +found straight afterwards when I noticed calendar sync was dead on velox. + +* What was broken + +=calendar-sync.timer= was enabled and firing every fifteen minutes, and failing +every time with exit 255: + +: calendar-sync: No calendars configured (set calendar-sync-calendars) + +The three output files sat at zero bytes. The cause is that +=~/.emacs.d/calendar-sync.local.el= is gitignored, so the reinstall deleted it +along with everything else untracked, and the module's loader treats a missing +file as a *silent* no-op. So the config vanished quietly and the only symptom +was a failing unit nobody was watching. + +Cheap to fix once found: the repo tracks =calendar-sync.local.el.example=, and +that template already encodes the shape velox uses — feeds resolved by +=:secret-host= against =authinfo.gpg= rather than inlined. The authinfo entries +had survived, because =~/.authinfo.gpg= is a stow symlink into the dotfiles repo. +So rebuilding was one copy, and all three feeds now sync clean and land +byte-identical to ratio's. + +* The general shape, which is the part worth acting on + +This is the same failure as gap 1, one layer down, and it is worth stating +generally because the install can act on it: + +- A tracked =*.local.el.example= template plus a gitignored =*.local.el= is a + deliberate pattern in this config, not a one-off. =.gitignore= lines 56-58 + list three of them: =calendar-sync.local.el=, =signal-config.local.el=, + =google-keep.local.el=. Every one of those is gone on velox right now. I have + only repaired the calendar one. +- Secrets held *by reference* survive a rebuild; secrets held *inline* do not. + The calendar config came back for free because the tokens were in + =authinfo.gpg=, which is stow-managed and therefore travels. Ratio's copy of + the same file inlines its URLs, and had ratio been the machine rebuilt, those + three feed tokens would simply have been gone. +- The failure was silent by design. A missing local config is a no-op, which is + right for a machine that never configured the feature and wrong for one that + just lost it. + +* What the install could do + +- After a rebuild, enumerate every tracked =*.local.el.example= in a project and + report which have no corresponding =*.local.el=. That is a one-line find and it + turns a silent no-op into a visible checklist item. +- Same for any =*.local.*= convention elsewhere in the fleet — the pattern is not + specific to Emacs. +- Worth pairing with gap 2: a unit that is enabled and failing every fifteen + minutes for two days is its own signal. A post-rebuild pass over + =systemctl --user list-units --state=failed= would have caught this one + without knowing anything about calendars. + +That last one generalizes best. Of the five gaps I have sent you, three were +things that *looked* fine — a stowed unit file, an enabled timer, a present +clone — and were not. diff --git a/docs/design/2026-08-14-velox-reinstall-gaps-3.org b/docs/design/2026-08-14-velox-reinstall-gaps-3.org new file mode 100644 index 0000000..9675973 --- /dev/null +++ b/docs/design/2026-08-14-velox-reinstall-gaps-3.org @@ -0,0 +1,98 @@ +#+TITLE: Reinstall gaps, part three — per-install certs and credentials, and one failure that hid the others +#+AUTHOR: Craig Jennings + +* Third handoff today + +Two earlier notes covered five gaps +([[file:2026-08-14-velox-reinstall-gaps-1.org][the first report]] and +[[file:2026-08-14-velox-reinstall-gaps-2.org][the follow-up]]). +Email was the last thing broken on velox after the 2026-08-13 rebuild, and it +turned up two more — both the same shape, and one of them with a property worth +generalizing. + +Email is fully working now: three accounts, 21,853 messages, 4.0 GB indexed. + +* Gap 6 — the Proton Bridge TLS cert is per-install, and its absence disabled every account + +=~/.mbsyncrc= carries =CertificateFile /home/cjennings/.config/protonbridge.pem=. +That file did not exist after the rebuild, and it cannot be restored from backup +or copied from the other machine: Proton Bridge generates a fresh self-signed +cert per installation. Velox's is issued 2026-08-13 23:44 with a different +fingerprint from ratio's 2026-01-30 one. + +*The part worth acting on is the blast radius.* mbsync parses its entire config +before doing any work, so a missing =CertificateFile= referenced by *one* account +aborts the run for *all* of them. Gmail and dmail need no bridge and no cert, and +both were dead anyway. The error names only the missing pem, so the symptom +("no mail at all") and the message ("this one file is missing") look unrelated. + +Recovery does not need the bridge GUI. The running bridge presents the cert on +its own IMAP port, so it can be pulled straight off the handshake: + +: openssl s_client -connect 127.0.0.1:1143 -starttls imap -showcerts </dev/null \ +: | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > ~/.config/protonbridge.pem + +That is a two-second, fully scriptable step, which makes it a good candidate for +the install rather than a runbook line. + +* Gap 7 — the bridge password is per-install too, and reports a stale value misleadingly + +=~/.mbsyncrc= resolves the cmail password with =cat ~/.config/.cmailpass=. That +file is plaintext and, unusually for my setup, a real file rather than a stow +symlink — so it is not in the dotfiles repo, not encrypted, and not carried to a +new machine. + +The file survived the rebuild but held the *previous* install's password, because +the bridge regenerates it per installation. Ratio's and velox's differ by sha256, +confirmed today. + +*The diagnostic trap:* Proton Bridge answers a wrong password with =no such +user=. I read that as "the bridge has no account signed in" and went looking for +a login problem. The account was configured the whole time. If the install ever +validates bridge connectivity, it should not treat =no such user= as evidence +about account state. + +* The generalization + +Gaps 6 and 7 are the same as 1 through 5, sharpened. Everything that broke in +this rebuild was *generated on the machine by an application* rather than carried +by git, stow, or the dotfiles repo: + +| gap | artifact | why it did not travel | +| 1 | =.ai/=, =todo.org=, =CLAUDE.md= | gitignored | +| 2 | =timers.target.wants= symlinks | written by systemctl enable | +| 3 | roam clone state | local working tree | +| 4 | signal-cli registration | per-device identity | +| 5 | =*.local.el= configs | gitignored | +| 6 | bridge TLS cert | per-install, regenerated | +| 7 | bridge password | per-install, regenerated | + +Gaps 6 and 7 add a distinction the earlier note missed. For 1, 3 and 5 the old +value is still correct, so *restoring* fixes them. For 4, 6 and 7 the old value is +*worthless* — the application has generated a new one, and only *re-deriving* +from the live system fixes them. An install that tries to restore these will +produce exactly what happened here: a file that exists, looks right, and +authenticates against nothing. + +So the install's post-rebuild checklist wants two columns, not one: what to +restore, and what to re-derive. + +* What the install could do + +- Re-derive the bridge cert from the running bridge with the =openssl s_client= + line above. Scriptable, no GUI, no secrets. +- Re-derive the bridge password from the bridge rather than expecting the file to + be right, and rewrite =.cmailpass=. (I have filed a task on my side to make + =PassCmd= ask the bridge directly, which would remove the file entirely.) +- Add a cheap post-rebuild validation that =mbsync --list= parses. Config-parse + failures disable every account at once and say nothing about mail, so they are + worth catching explicitly rather than via "no new mail" hours later. +- More generally: keep the restore list and the re-derive list separate, per the + table above. + +* Unrelated, but noticed while comparing the machines + +=~/.config/.gmailpass.gpg= and =~/.config/.dmailpass.gpg= resolve to mode 777 in +the dotfiles repo, on both machines. They are gpg-encrypted so the contents are +safe, but world-writable is wrong for a credential file. That is a dotfiles fix, +not an archsetup one — noting it here only because it surfaced in the same pass. |