fix(installer): give NTP an IP source so a wrong clock can't kill DNS

The installer wrote both halves of a deadlock. configure_dns pins DNSOverTLS=yes with DNSSEC=yes, and both validate against the wall clock. The chrony step enables chronyd without writing a config, so the machine runs Arch's stock one, whose only source is a pool hostname.

Boot with a wrong clock and DoT validation fails, so nothing resolves. Chrony then can't resolve its pool, so the clock stays wrong. Neither side moves, and recovery takes a second device. Velox hit this on the road and I diagnosed it from a phone.

An address needs no DNS and no certificate, so two IP-addressed sources in a drop-in break the cycle whatever caused the skew. Stock chrony.conf reads no drop-in directory, so it gets a confdir line pointing at one.

post-rebuild-check grows a sixth check for the same property. It reads sources only from files chrony is told to read. A drop-in beside a chrony.conf that never names its directory is one chrony won't open, so counting it would pass the machine while describing a file nothing reads.

The failure taxonomy gains the mode in its DNS layer and a cluster 5 triage line. Its egress-layer clock entry assumed working DNS and offered set-ntp true, which can't recover this. That entry now says so.

1 files changed, 265 insertions, 17 deletions

diff --git a/todo.org b/todo.org
index 32eb9a2..1bb810d 100644
--- a/todo.org
+++ b/todo.org
@@ -45,6 +45,86 @@ below):
   input-side-spec.org (DRAFT, four decisions open).
 
 * Archsetup Open Work
+** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox:
+:PROPERTIES:
+:CREATED: [2026-08-19 Wed]
+:LAST_REVIEWED: 2026-08-19
+:END:
+
+The installer wrote both halves of a deadlock. =configure_dns= pins
+=DNSOverTLS=yes= with =DNSSEC=yes=, and both validate against the wall clock;
+the chrony step enables chronyd without writing a config, so the machine runs
+Arch's stock one whose only source is =pool 2.arch.pool.ntp.org= — a hostname.
+Boot with a wrong clock and DoT certificate validation fails, so nothing
+resolves; chrony then cannot resolve its pool, so the clock stays wrong.
+Neither side moves. It caught velox on the road 2026-08-19 and had to be
+diagnosed from a phone.
+
+Fixed at the root: the installer now writes
+=/etc/chrony.d/10-bootstrap-ip-ntp.conf= with two IP-addressed Cloudflare
+sources and points stock chrony.conf at the drop-in. An address needs no DNS
+and carries no certificate, so the escape hatch holds whatever broke the clock.
+velox has the same drop-in applied live, verified with =chronyc -n sources=
+(=162.159.200.1= selected) and =timedatectl= reporting synchronized.
+
+What is left here is the part I could not verify: the decisive test is a full
+power-down and cold boot, confirming the clock corrects itself untouched. See
+the manual-testing entry. Until that runs, the fix is sound by construction
+rather than demonstrated.
+
+Grading: Critical severity (total loss of network — no DNS means no egress, and
+recovery needs a second device) x some users sometimes (only machines that boot
+with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 =
+[#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in
+this state it is fully offline with no local path out.
+
+*** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed
+The installer carries the drop-in; =post-rebuild-check= grew a sixth check that
+fails a machine whose every NTP source is a hostname; the net failure taxonomy
+gained the mode in its DNS layer plus a cluster 5 triage line, and its existing
+egress-layer clock entry now says outright that its remedy does not apply when
+DoT or DNSSEC is on.
+
+The doctor half is in dotfiles: =classify.py= reached "DNS not resolving → net
+repair dns-test" here, which cannot help, because every public resolver fails
+the same clock-sensitive validation — so the doctor sent you round a loop. It
+now emits a =clock-dns= row ahead of the generic DNS verdict. Detection is
+deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass
+query addressed by IP over plain UDP/53 to tell "resolved is refusing to
+validate" apart from "DNS is genuinely dead".
+
+** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature:
+:PROPERTIES:
+:CREATED: [2026-08-19 Wed]
+:LAST_REVIEWED: 2026-08-19
+:END:
+
+The doctor now *names* the deadlock but hands the user two commands rather than
+running anything — the verdict is =needs-user-action=. That was the honest call
+at the time: correcting the clock needs the real time, and I could not exercise
+a repair against the actual failure state without deliberately wedging velox's
+network mid-session.
+
+An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a
+certificate. The shape would be =chronyc add server <ip> iburst= followed by
+=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to
+a running chronyd over its socket, so this works in exactly the state that
+blocks everything else. Two things to settle before building it: whether
+=makestep= actually steps on the first sample after a runtime =add server= or
+needs a poll first, and what the doctor does on a box running timesyncd rather
+than chrony.
+
+Worth less now than it looks: a machine built by the current installer carries
+the IP-addressed source and never reaches the deadlock. This is for machines
+built before the fix.
+
+Grading: Minor severity (the doctor already names the fault correctly and hands
+over a working remedy; only the automation is missing) x rare edge case (only
+pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this
+state has no working network and cannot look anything up, which makes the
+two-command handoff harder to follow than it reads. Minor x rare = P4, and I am
+leaving it at [#C] rather than [#D] because it sits one step from done.
+
 ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware:
 DEADLINE: <2026-08-14 Fri>
 :PROPERTIES:
@@ -79,6 +159,26 @@ grep the journal for new "Power key pressed" lines — zero means fixed.
 Must be done before the Sunday flight — a phantom press mid-travel with the
 shield on is survivable, but the connector should not be trusted at 30,000
 feet on the loose setting.
+
+*** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree
+The reseat did not happen before the flight, and velox is travelling. The
+deadline blew past on 08-14.
+
+The two symptoms have separated, which is worth recording because it changes
+what the evidence proves. The phantom presses have stopped: fifteen "Power key
+pressed" entries between 08-14 04:29 and 08-15 20:04, then nothing at all
+across five boots including today's. The touchpad has not — there is still no
+touchpad node under =/dev/input/by-path/=, which is the same dead interrupt
+line the body describes.
+
+So the quiet power button is not evidence the connector reseated itself. The
+interrupt line is the symptom that cannot be masked in software, and it is
+still dead, so the ribbon is still unseated. The most likely reason the
+presses stopped is that the machine has been sitting on hotel surfaces instead
+of being carried and flexed.
+
+The interim shield is still live (=HandlePowerKey=ignore=), and the escalation
+note stands: an EC-level glitch cuts power below systemd regardless of it.
 ** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore:
 DEADLINE: <2026-08-15 Sat>
 :PROPERTIES:
@@ -196,7 +296,29 @@ machine-level half is already correct.
 Grading: Major severity (a crash loop burning battery and filling the
 journal, silently) x every user every time on any laptop with the TLP fix
 applied = P1 = [#A].
-** TODO [#A] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox:
+
+*** 2026-08-17 Mon @ 19:57:42 -0700 The loop stopped at the reboot; the defect did not
+velox rebooted at 16:04 and there have been zero coredumps since, against 47
+in the twelve hours before it. So the loop is not currently burning anything.
+
+That is not a fix, and the distinction matters for whoever picks this up.
+=powerprofilesctl get= still fails exactly as recorded — =NameHasNoOwner ...
+unit is masked= — so every precondition for the loop is intact and it returns
+whenever the caller next polls. What the reboot cleared is the caller's state,
+not the bug.
+
+Narrowed the search the body asks for: =power.py= is the *only* file in
+dotfiles that shells out to =powerprofilesctl= (=SETTINGS_POWERPROFILESCTL=,
+line 14), so the caller is inside the settings module rather than waybar or a
+timer. Worth knowing that the coredumps are =powerprofilesctl= itself aborting
+— it is a python script, which is why they log as =/usr/bin/python3.14=
+SIGABRT rather than under its own name.
+
+Grade unchanged. The matrix inputs did not move: the severity is what happens
+while the machine is in that state, and the frequency row is every laptop
+carrying the TLP fix. A quiet interval since a reboot is not a frequency
+change.
+** TODO [#B] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox:
 :PROPERTIES:
 :CREATED: [2026-08-17 Mon]
 :LAST_REVIEWED: 2026-08-17
@@ -236,6 +358,24 @@ unless you look) x rare edge case (one machine, specific conditions) = P2 =
 [#B]... except that this is a live, ongoing drain on a travelling machine
 rather than a latent defect, so it takes [#A] until the machine is back to
 normal. Re-grade to [#B] once resolved and the question is only prevention.
+
+*** 2026-08-17 Mon @ 19:57:42 -0700 The reboot cleared it; re-graded [#A] to [#B] as the task instructed
+velox rebooted at 16:04. The wedge is gone: =systemctl --user is-enabled
+roam-sync.timer= now answers =enabled= in well under a second, where every
+unit-file call hung indefinitely before, and =list-timers= shows
+calendar-sync, roam-sync and agenda-render-cache all firing on schedule
+again. So the remedy the task named — a logout or reboot — was taken and
+worked.
+
+Nothing here was diagnosed further, which means the cause is still unproven
+and both candidates in the body stand. What is left is prevention, and the
+task's own grading says that is [#B]: the live-drain argument was the only
+thing holding it at [#A], and the drain has stopped. Re-graded per that
+instruction rather than by a fresh judgment.
+
+Reproducing it deliberately is the open question, and it is not obviously
+worth doing — it costs a wedged session to learn something the crash-loop fix
+may make moot.
 ** TODO [#A] The installer clones my two working repos shallow and read-only :bug:velox:
 :PROPERTIES:
 :CREATED: [2026-08-17 Mon]
@@ -446,7 +586,7 @@ users sometimes = P3 = [#C].
 ** TODO [#B] Land the rescued emacs-wttrin commit :chore:velox:
 :PROPERTIES:
 :CREATED: [2026-08-14 Fri]
-:LAST_REVIEWED: 2026-08-14
+:LAST_REVIEWED: 2026-08-17
 :END:
 bf0457f "feat: add wttrin-hide-follow-line to hide the wttr.in follow line"
 (2026-06-24) was the only genuinely unpushed commit anywhere on the old
@@ -456,6 +596,17 @@ git bundle before the disk was wiped:
 To land it: clone emacs-wttrin, =git fetch <bundle> --branches=, review the
 commit, then push to git@cjennings.net:emacs-wttrin.git. Delete the bundle
 once it's on the remote.
+
+*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: still unlanded, and the bundle is still the only copy
+Cloned the remote bare and asked it for the object directly: =git cat-file -t
+bf0457f= returns "Not a valid object name", so the commit has never reached
+=git@cjennings.net:emacs-wttrin.git=. Remote =main= is =ee8fdeb=.
+
+That makes =working/velox-reinstall/wttrin-bf0457f.bundle= the sole surviving
+copy of 103 insertions across three files, on one laptop that is travelling.
+Worth doing sooner than its =[#B]= suggests for that reason alone, and it also
+pins the working directory open — the reinstall task cannot file its artifacts
+away while this bundle is still load-bearing.
 ** TODO [#B] archsetup doesn't clone rulesets :bug:velox:
 DEADLINE: <2026-08-15 Sat>
 :PROPERTIES:
@@ -868,18 +1019,38 @@ doc above (not published, since they map the setup). Follow-ons: the rotation
 VERIFY above, velox reconcile on return, the secrets-repo split (top of Open
 Work), the wireguard =.gitignore= bug (line ~191), the cgit move (below), and a
 pre-receive secret-scan hook so this can't recur.
-*** TODO [#A] velox: reconcile its clones after the history rewrite
-velox was offline for repair during the 2026-08-09 purge, so its clones still
-hold the pre-rewrite history and are diverged from the rewritten remotes. On
-its return: force-fetch + rebase local work onto the rewritten main in both
-repos (or re-clone), force-update the local tag, local-gc, before its next
-push. Also on the velox riders on the sleep/suspend task.
+*** 2026-08-17 Mon @ 19:57:42 -0700 Moot — the 08-13 wipe re-cloned velox from the rewritten remotes
+This asked velox to reconcile clones that no longer exist. The machine was
+wiped and reinstalled on 2026-08-13, so every repo on it was cloned fresh
+*after* the purge and never held the pre-rewrite history at all. The runbook
+anticipated this ("fresh clones automatically carry the post-purge rewritten
+git history"); nobody closed the task once the reinstall took that route.
+
+Verified rather than assumed: both repos are level with =origin/main= today —
+archsetup at =6faa31c=, dotfiles at =65940f2=, both trees clean.
+
+One thing the reinstall did leave, and it is filed separately: the installer
+cloned both repos =--depth 1=, so the history was present-but-truncated until
+today's =git fetch --unshallow= (see the shallow-clone =[#A]=). A reconcile
+against the rewritten remote was still unnecessary — a shallow clone of the
+right history is not a diverged clone of the wrong one.
 ** TODO [#B] Move archsetup off cgit to cjennings@cjennings.net :chore:security:
 :PROPERTIES:
-:LAST_REVIEWED: 2026-07-21
+:LAST_REVIEWED: 2026-08-17
 :END:
 Decided (Craig, 2026-07-20): move the archsetup repo off the public cgit host (git@cjennings.net, scan-path /var/git) to Craig's private account remote cjennings@cjennings.net, so it is no longer world-cloneable. This is the archsetup-specific fix for the cgit-exposure finding above.
 Plan: create a bare repo under cjennings's control off the cgit scan-path (e.g. =~cjennings/git/archsetup.git=); push current main + tags there; migrate the post-receive hook that publishes the installer to =/var/www/cjennings/archsetup= so curl-install keeps working (the single published file stays public by design; only the repo goes private); update the origin remote on ratio and velox to =cjennings@cjennings.net:git/archsetup.git=; remove =/var/git/archsetup.git= so cgit no longer serves it. Verify: anonymous =git clone https://git.cjennings.net/archsetup.git= fails, the new private clone works from both machines, and the curl-install URL still returns the installer. Keep the two daily drivers' remotes in sync (daily-drivers rule).
+
+*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: unstarted, and the exposure is confirmed live
+Ran the task's own verification step as it stands today, which is the honest
+way to check an unstarted task rather than reading its body back. Anonymous
+=git ls-remote https://git.cjennings.net/archsetup.git= succeeded with no
+credentials and returned =6faa31c= — this afternoon's HEAD. So the repo is
+still world-cloneable and current to the commit, not a stale published
+snapshot.
+
+=origin= on this machine is still =git@cjennings.net:archsetup.git=, the cgit
+account, so nothing has moved. Everything in the plan stands unchanged.
 ** TODO [#B] Velox boot-failure retrospective — upgrade guard gaps :bug:zfs:maint:
 :PROPERTIES:
 :LAST_REVIEWED: 2026-07-21
@@ -1331,15 +1502,28 @@ Verify (manual, live): see Manual testing and validation.
 *** 2026-07-09 Thu @ 16:32:54 -0500 Audit reconcile: Phase 4 is filed on the dotfiles side, waiting on them
 The dotfiles project accepted the Phase 4 handoff and filed it as a =[#C]= task in their own =todo.org= (their note, 2026-07-08 16:56): the help-text audit + panel help affordance, the user-guide/README, and the ratio rollout doc. Not started there. They ping when it lands, and this task's Phase 4 child closes then. Nothing to do here meanwhile.
 
-*** TODO Phase 4 — docs + rollout :network:blocked:
-Deliverable: in-app help (=net --help= + per-command, panel help affordance);
-README/user-guide (commands, indicator states, panel, config keys, make targets,
-troubleshooting from the failure table, rollback); archsetup Hyprland dep install
+*** 2026-08-17 Mon @ 19:57:42 -0700 Landed on the dotfiles side; the block is cleared
+dotfiles shipped it as =138da7b= and closed its own task, so this one closes
+with it and the =:blocked:= tag comes off. Found by checking their =todo.org=
+rather than waiting for the ping — their close-out note says "archsetup pinged
+so its Phase 4 task can close", so the handoff worked and only this end was
+left open.
+
+All three acceptance criteria are met on their side: the help audit found and
+fixed a stale =net repair= action list (nine of nineteen actions were named;
+both the CLI help and =repair.py='s docstring now generate from the ACTIONS
+registry), =net/README.md= covers every command plus the recovery targets, and
+the ratio rollout is documented with both daily drivers verified current.
+
+They split the panel help affordance out rather than inventing it — no sibling
+panel has one, so its shape is a design call. It is tracked on their side, not
+here.
+
+Original deliverable, for the record: in-app help (=net --help= + per-command,
+panel help affordance); README/user-guide; archsetup Hyprland dep install
 (=gtk4-layer-shell=, =python-gobject=, =speedtest-go-bin=); ratio manual dep +
-stow step.
-Verify: =net --help= and each subcommand complete; user-guide covers every command
-+ the recovery targets.
-Build handed off to the dotfiles project 2026-07-04 (=~/.dotfiles/inbox/2026-07-04-1305-from-archsetup-phase4-handoff.md=): archsetup deps confirmed installed, the remaining help/user-guide/rollout-doc work is in the net package. dotfiles pings back when it lands.
+stow step. Handed off 2026-07-04 with the archsetup deps already confirmed
+installed.
 
 *** TODO Phase 5 — VPN / WireGuard CLI fold (vNext) :network:
 Rescoped 2026-07-04 (audit): the tunnels track already shipped most of the original Phase 5. Panel tunnel bring-up/down and detection landed (dotfiles 2d9d060 probes tailscale/NM-wireguard/Proton; 21db05a brings overlays up/down from the panel's Tunnels sub-view; 31ba056 diagnose/doctor understand tunnel routes; archsetup 2e40781 wireguard config import; the net-panel-other-interfaces spec is IMPLEMENTED). What remains for Phase 5 is only the =net vpn ...= CLI subcommand — cli.py still has no vpn/tunnel parser. Fold the panel's existing tunnel operations into a CLI surface; spec separately when picked up.
@@ -1601,6 +1785,29 @@ Add kernel parameter: ~rtc_cmos.use_acpi_alarm=1~ (will become systemd default)
 Consider: ~acpi_mask_gpe=0x1A~ for battery drain, suspend-then-hibernate config
 See Framework community notes on logind.conf and sleep.conf settings
 
+*** 2026-08-17 Mon @ 19:57:42 -0700 Four of the five riders are done; WireGuard is the one left
+The riders were written for "when velox returns from repair". It came back as
+a full reinstall instead, and the installer carried most of them, so I checked
+each on the live machine rather than reading the list back:
+
+- tlp radio-enable — done. =/etc/tlp.d/01-custom.conf:10= carries
+  =DEVICES_TO_ENABLE_ON_STARTUP="bluetooth wifi"=, written by the installer.
+- touchpad auto-detection — the dotfiles half is done: =touchpad-auto
+  --detect= prints =pixa3854:00-093a:0274-touchpad=. Read that carefully
+  though — it names the device the config expects, not a device delivering
+  events. The touchpad is still dead on the ribbon fault, so this rider is
+  satisfied and the hardware still is not.
+- podman socket — done, =podman.socket= is enabled.
+- camera udev — done, =72-usb-passthrough-cameras.rules= is installed.
+- *wolf WireGuard — not done, and it is the one that was time-critical.* No
+  =~/.config/wireguard/wolf.conf.gpg= and no WireGuard profile in
+  NetworkManager. The 08-08 decision set this up specifically so velox could
+  reach home from the road, on the argument that it is cheap at home and
+  expensive from a hotel. velox is now in the hotel.
+
+The suspend work itself is untouched — no kernel parameter, no drain
+measurement. Only the riders moved.
+
 ** TODO [#B] Manual testing and validation :test:
 :PROPERTIES:
 :LAST_REVIEWED: 2026-07-09
@@ -1609,6 +1816,47 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child
 
 Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely.
 
+*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched?
+What we're verifying: that the IP-addressed NTP drop-in actually breaks the
+bootstrap deadlock on a real cold start. This is the one test no agent can run —
+it needs a full power-down, which is exactly the event that empties a failing
+RTC. Everything else about the fix is verified; this is the part that rests on
+construction (an address needs no DNS, NTP carries no certificate) rather than
+on having been seen work.
+
+Do this before relying on it away from home — the failure mode strands the
+machine with no network and no way to look anything up.
+- Confirm the drop-in is in place and chrony is using it (block below).
+- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only
+  loses time when the machine is actually off.
+- Leave it off long enough to matter if the coin cell is the culprit (overnight
+  is the honest test; a few minutes may not drain anything).
+- Power on. Do not touch the clock, do not run anything. Just log in and wait
+  about a minute.
+- Run the verification block below.
+#+begin_src sh :results output
+echo "--- drop-in present? ---"
+cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING"
+echo "--- is chrony reading it? ---"
+grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read"
+echo "--- sources (the IP literal should be selected, marked ^*) ---"
+chronyc -n sources
+echo "--- clock ---"
+timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service'
+echo "--- did DNS come back on its own? ---"
+getent hosts gnu.org || echo "DNS STILL DEAD"
+#+end_src
+Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=,
+=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and
+one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org
+resolves — all without you having set the time.
+
+If the RTC came up wrong and the clock corrected itself anyway, the fix works
+and the coin cell question is answered separately (a wrong RTC time in that
+output means the cell is dying). If the clock is still wrong or DNS is still
+dead, the fix did not hold: capture that whole block and promote this to a
+top-level TODO.
+
 *** Floating layout: freeze positions, border flash, glyph, exit to master
 What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation.
 - Go to a workspace with 2-3 tiled windows in master.