From 4dfecc5f705c9df0d1f8e8245276146f826e16cc Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Mon, 17 Aug 2026 11:28:55 -0700 Subject: feat(scripts): add post-rebuild-check for the gaps a reinstall leaves A rebuilt machine looks finished and isn't. Five gaps surfaced on velox within two days of its reinstall. Three looked fine on inspection: a stowed unit file, an enabled-looking timer, a present git clone. The script runs those five checks and prints a line for each whether or not it finds anything. Every probe fails closed. A check that cannot run reports a finding rather than a pass, which matters more here than anywhere: a silent no-op in the checker is the exact failure it exists to catch. `systemctl --user` exits 1 with empty output when there is no user bus, so reading that as "no failed units" would call a machine healthy at the moment nothing was checked. Calls are bounded for the same reason. A check that hangs reports nothing at all, and the machine most in need of checking is the one it hangs on. I suppressed three classes of finding, each because the live run produced them and reality disagreed. A timer-activated service is supposed to sit linked and not enabled. One expected tooling file is seed-only, so most projects legitimately never have one. Vendored package trees ship their own example files. Left in, those were 19 of the first run's 27 findings, and a check nobody reads is a check that isn't run. The post-install checklist points at it, and 58 tests cover it. --- docs/post-install-checklist.org | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) (limited to 'docs') diff --git a/docs/post-install-checklist.org b/docs/post-install-checklist.org index 97fc0d5..f0545a7 100644 --- a/docs/post-install-checklist.org +++ b/docs/post-install-checklist.org @@ -18,6 +18,32 @@ bluetooth pairing landed below. * Checklist +** Run the post-rebuild check first + +Before working through the manual steps below, run: + +#+begin_src sh +~/code/archsetup/scripts/post-rebuild-check +#+end_src + +It runs the five checks a rebuilt machine actually needs — failed units, +user units that are present but never enabled, =*.example= configs whose +real sibling is missing, gitignore-mode projects missing the working state +their own =.gitignore= names, and the signal-cli registration. Each prints +a line whether or not it finds anything; exit 1 means something needs +attention. + +These are the gaps velox hit within two days of its 2026-08-13 reinstall, +and three of the five looked fine on casual inspection: a stowed unit file, +an enabled-looking timer, a present git clone. Run it again a day or two +after the install, once timers have had a chance to fail. + +It normally finishes in a second or two. On a machine whose user systemd is +wedged it takes a couple of minutes instead, because every =systemctl= call +is bounded at five seconds and check 2 makes one per unit. That is the slow +case working as intended: it reports what it could not read rather than +hanging. Set =PRC_SYSTEMCTL_TIMEOUT= lower to cut the wait. + ** Pair bluetooth peripherals Pairing is inherently interactive (scan, pick the device, confirm), so it -- cgit v1.2.3 From afbf011aa0937b5702b6d8c1bfca0809ed809425 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:16:46 -0700 Subject: fix(installer): give NTP an IP source so a wrong clock can't kill DNS The installer wrote both halves of a deadlock. configure_dns pins DNSOverTLS=yes with DNSSEC=yes, and both validate against the wall clock. The chrony step enables chronyd without writing a config, so the machine runs Arch's stock one, whose only source is a pool hostname. Boot with a wrong clock and DoT validation fails, so nothing resolves. Chrony then can't resolve its pool, so the clock stays wrong. Neither side moves, and recovery takes a second device. Velox hit this on the road and I diagnosed it from a phone. An address needs no DNS and no certificate, so two IP-addressed sources in a drop-in break the cycle whatever caused the skew. Stock chrony.conf reads no drop-in directory, so it gets a confdir line pointing at one. post-rebuild-check grows a sixth check for the same property. It reads sources only from files chrony is told to read. A drop-in beside a chrony.conf that never names its directory is one chrony won't open, so counting it would pass the machine while describing a file nothing reads. The failure taxonomy gains the mode in its DNS layer and a cluster 5 triage line. Its egress-layer clock entry assumed working DNS and offered set-ntp true, which can't recover this. That entry now says so. --- archsetup | 30 +++ docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +- scripts/post-rebuild-check | 85 ++++++- .../post-rebuild-check/test_post_rebuild_check.py | 90 ++++++- todo.org | 282 +++++++++++++++++++-- 5 files changed, 461 insertions(+), 30 deletions(-) (limited to 'docs') diff --git a/archsetup b/archsetup index edd4062..078408c 100755 --- a/archsetup +++ b/archsetup @@ -1200,6 +1200,36 @@ configure_build_environment() { echo 'OPTIONS=""' > /etc/sysconfig/chronyd systemctl enable chronyd.service >> "$logfile" 2>&1 || error_warn "$action" "$?" + # Bootstrap NTP sources addressed by IP, never by hostname. + # + # Arch's stock chrony.conf names its pool by hostname, and the DNS this + # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both + # validate against the wall clock, so a machine that boots with a wrong + # clock resolves nothing: chrony cannot reach the pool, so the clock stays + # wrong, so DNS stays dead. Neither side moves, and recovery needs a second + # device to look up an NTP address by hand. An IP-addressed source needs no + # DNS and no certificate, so it breaks the deadlock unattended. I would + # rather carry two extra server lines than lose a laptop's network to any + # RTC fault. See the clock/DNS deadlock entry in the net failure taxonomy + # under docs/design/. + action="adding IP-addressed NTP bootstrap sources" && display "task" "$action" + mkdir -p /etc/chrony.d + cat << 'EOF' > /etc/chrony.d/10-bootstrap-ip-ntp.conf +# Reachable without DNS, so a wrong clock can always correct itself. +server 162.159.200.1 iburst +server 162.159.200.123 iburst +EOF + # Stock chrony.conf reads no drop-in directory, so point it at one. + if [ -f /etc/chrony.conf ]; then + backup_system_file /etc/chrony.conf + if ! grep -qE '^[[:space:]]*confdir[[:space:]]+/etc/chrony\.d' /etc/chrony.conf; then + printf '\n# Read drop-ins (archsetup owns /etc/chrony.d).\nconfdir /etc/chrony.d\n' \ + >> /etc/chrony.conf || error_warn "$action" "$?" + fi + else + error_warn "$action (no /etc/chrony.conf to point at /etc/chrony.d)" 1 + fi + action="configuring compiler to use all processor cores" && display "task" "$action" backup_system_file /etc/makepkg.conf sed -i "s/-j2/-j$(nproc)/;s/^#MAKEFLAGS/MAKEFLAGS/" /etc/makepkg.conf >> "$logfile" 2>&1 diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 74790c6..70421d0 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,6 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -107,7 +108,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] - Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] - Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] -- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. This entry assumes DNS still works; when the resolver runs DoT or DNSSEC the same skew kills DNS first and =set-ntp true= cannot recover it — see the clock/DNS deadlock in the DNS layer. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] - Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] - VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] - IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] @@ -309,6 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/scripts/post-rebuild-check b/scripts/post-rebuild-check index 8807f85..c18ae8f 100755 --- a/scripts/post-rebuild-check +++ b/scripts/post-rebuild-check @@ -21,6 +21,10 @@ # 5. signal-cli holds no registered account (velox lost its # registration, and because agent-text relays into this machine, # that silently broke paging for the WHOLE fleet) +# 6. every NTP source is named by hostname (a wrong clock fails the +# DoT/DNSSEC validation this machine's DNS runs on, so nothing +# resolves -- including the NTP pool that would fix the clock; velox +# deadlocked exactly this way 2026-08-19 and needed a second device) # # The .gitignore rule in check 4 is what scopes it: a tooling path is only # expected where the project's own .gitignore names it, so a project that @@ -50,6 +54,8 @@ # ~/.dotfiles) # PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output; "" = no account, # the special value MISSING = binary absent +# PRC_NTP_SOURCES newline list of configured NTP server addresses; +# the special value MISSING = no NTP daemon active # PRC_SYSTEMCTL path to the systemctl binary (a fake, under test) # PRC_SYSTEMCTL_TIMEOUT seconds to allow each systemctl call (default 5) # @@ -61,9 +67,10 @@ usage() { cat <<'EOF' post-rebuild-check - verify a rebuilt machine is actually finished -Runs the five checks that caught velox's 2026-08 reinstall gaps: failed +Runs the six checks that caught velox's 2026-08 reinstall gaps: failed units, present-but-inert user units, orphaned *.example configs, missing -per-project tooling state, and the signal-cli registration. +per-project tooling state, the signal-cli registration, and whether time +sync can recover from a wrong clock without DNS. Usage: post-rebuild-check [--help] @@ -84,6 +91,7 @@ TOTAL_FINDINGS=0 CHECK_FINDINGS=0 FINDING_LINES="" signal_missing="" +ntp_missing="" # Every systemctl call is bounded. A wedged user manager spins and answers # nothing -- seen live on velox 2026-08-17, where `is-enabled`, `cat`, and @@ -179,7 +187,7 @@ while IFS= read -r line; do unit=${line#*:} finding "$scope unit failed: $unit" done < "$STAGE" -report "check 1/5: failed units" +report "check 1/6: failed units" # --- 2. user unit files present but not enabled --------------------------- @@ -269,7 +277,7 @@ while read -r name state; do esac finding "unit file present but not enabled: $name ($state)" done < "$STAGE" -report "check 2/5: unit files" +report "check 2/6: unit files" # --- 3. *.example files whose real sibling is missing --------------------- @@ -314,7 +322,7 @@ while IFS= read -r root; do [ -e "${ex%.example}" ] || finding "example without its real file: $ex" done < "$WORK/examples" done < "$WORK/roots" -report "check 3/5: local files" +report "check 3/6: local files" # --- 4. gitignore-mode projects missing their tooling --------------------- @@ -368,7 +376,7 @@ todo.org todo\.org inbox inbox EOF done < "$WORK/projects" -report "check 4/5: project tooling" +report "check 4/6: project tooling" # --- 5. signal-cli registration ------------------------------------------- @@ -395,7 +403,68 @@ if [ "$signal_missing" = 1 ]; then elif [ -z "$signal_missing" ] && [ -z "$accounts" ]; then finding "no signal account registered — agent-text relays into this machine, so paging breaks for the whole fleet" fi -report "check 5/5: signal registration" +report "check 5/6: signal registration" + +# --- 6. NTP can recover a wrong clock without DNS ------------------------- +# +# The clock/DNS bootstrap deadlock. This machine resolves through DNSOverTLS +# with DNSSEC, and both validate against the wall clock, so a boot with a +# wrong clock resolves nothing at all. If every configured NTP source is named +# by hostname, the daemon that would correct the clock needs the DNS the clock +# is breaking, and the machine cannot recover without a second device -- +# which is exactly what happened on velox 2026-08-19. One source addressed by +# IP breaks the cycle, so that is what this check looks for. + +# True when the argument is an address rather than a name. An address needs no +# resolver, which is the whole property being checked. +is_ip_literal() { + case "$1" in + "") return 1 ;; + *:*) case "$1" in *[!0-9A-Fa-f:]*) return 1 ;; esac + return 0 ;; + *[!0-9.]*) return 1 ;; + *.*) return 0 ;; + esac + return 1 +} + +if [ -n "${PRC_NTP_SOURCES+set}" ]; then + ntp_sources=$PRC_NTP_SOURCES + if [ "$ntp_sources" = "MISSING" ]; then + ntp_sources="" + ntp_missing=1 + fi +elif sctl is-active chronyd >/dev/null 2>&1; then + # Both the main file and any drop-in: the IP-addressed source belongs in a + # drop-in, so reading only chrony.conf would miss every correct machine. + ntp_sources=$(cat /etc/chrony.conf /etc/chrony.d/*.conf 2>/dev/null \ + | awk '$1 == "server" || $1 == "pool" { print $2 }') +elif sctl is-active systemd-timesyncd >/dev/null 2>&1; then + ntp_sources=$(awk -F= '/^[[:space:]]*NTP=/ { print $2 }' \ + /etc/systemd/timesyncd.conf 2>/dev/null | tr ' ' '\n') +else + ntp_sources="" + ntp_missing=1 +fi + +if [ "$ntp_missing" = 1 ]; then + finding "no NTP implementation is active — nothing corrects the clock, and a wrong clock takes DNS down with it" +elif [ -z "$ntp_sources" ]; then + finding "no NTP sources are configured — nothing was checked, and nothing corrects the clock" +else + ntp_has_literal="" + stage "$ntp_sources" + while IFS= read -r src_addr; do + [ -z "$src_addr" ] && continue + if is_ip_literal "$src_addr"; then + ntp_has_literal=1 + fi + done < "$STAGE" + if [ -z "$ntp_has_literal" ]; then + finding "every NTP source is named by hostname — a wrong clock breaks DNS, so nothing can resolve them and the clock stays wrong" + fi +fi +report "check 6/6: NTP bootstrap" # --- summary -------------------------------------------------------------- @@ -403,5 +472,5 @@ if [ "$TOTAL_FINDINGS" -eq 0 ]; then echo "all checks clean" exit 0 fi -echo "$TOTAL_FINDINGS finding(s) across 5 checks" +echo "$TOTAL_FINDINGS finding(s) across 6 checks" exit 1 diff --git a/tests/post-rebuild-check/test_post_rebuild_check.py b/tests/post-rebuild-check/test_post_rebuild_check.py index 4894451..757039b 100644 --- a/tests/post-rebuild-check/test_post_rebuild_check.py +++ b/tests/post-rebuild-check/test_post_rebuild_check.py @@ -21,6 +21,8 @@ probe"): PRC_LOCAL_SCAN_ROOTS newline-separated roots to scan for *.example orphans PRC_PROJECT_ROOTS newline-separated project dirs for the tooling check PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output ("" = no accounts); + PRC_NTP_SOURCES newline list of configured NTP server addresses + ("MISSING" = no NTP daemon active) the special value MISSING means the binary is absent Run from repo root: @@ -39,7 +41,8 @@ CHECK = os.path.join(REPO_ROOT, "scripts", "post-rebuild-check") def run_check(failed_units="", unit_states="", local_roots="", - project_roots="", signal_accounts="+15045551234"): + project_roots="", signal_accounts="+15045551234", + ntp_sources="162.159.200.1\npool.ntp.org"): """Run the script with every probe stubbed; defaults are all-clean. Roots are newline-separated. Empty means "the seam is set and names no @@ -52,11 +55,88 @@ def run_check(failed_units="", unit_states="", local_roots="", env["PRC_LOCAL_SCAN_ROOTS"] = local_roots env["PRC_PROJECT_ROOTS"] = project_roots env["PRC_SIGNAL_ACCOUNTS"] = signal_accounts + env["PRC_NTP_SOURCES"] = ntp_sources return subprocess.run( ["sh", CHECK], capture_output=True, text=True, timeout=30, env=env, ) +class NtpBootstrap(unittest.TestCase): + """Check 6 — the clock/DNS bootstrap deadlock. + + A wrong clock fails the DoT certificate and DNSSEC signature checks this + machine's DNS runs on, so nothing resolves; and an NTP daemon whose every + source is a hostname then cannot resolve the servers that would correct + the clock. One source addressed by IP is what makes the machine able to + recover on its own. + """ + + # --- Normal cases --------------------------------------------------- + + def test_an_ip_addressed_source_is_clean(self): + r = run_check(ntp_sources="162.159.200.1\npool.ntp.org") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + self.assertEqual(r.returncode, 0, r.stdout) + + def test_all_hostname_sources_is_a_finding(self): + # The velox 2026-08-19 shape exactly: stock Arch chrony.conf, whose + # only source is a pool hostname. + r = run_check(ntp_sources="2.arch.pool.ntp.org") + self.assertIn("every NTP source is named by hostname", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_an_ipv6_addressed_source_counts(self): + r = run_check(ntp_sources="2606:4700:f1::1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + # --- Boundary cases ------------------------------------------------- + + def test_the_literal_may_sit_anywhere_in_the_list(self): + # Order must not matter; the property is "at least one", and the + # drop-in that carries it is read after the main config. + r = run_check(ntp_sources="a.pool.ntp.org\nb.pool.ntp.org\n162.159.200.1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_blank_lines_between_sources_are_ignored(self): + r = run_check(ntp_sources="\n\n162.159.200.1\n\n") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_a_hostname_containing_digits_and_dots_is_not_an_address(self): + # The trap in any naive "looks like an IP" test: these resolve through + # DNS like any other name, so counting one as an address would hand a + # deadlocked machine a clean bill. + for host in ("0.arch.pool.ntp.org", "3.us.pool.ntp.org", "time1.google.com"): + with self.subTest(host=host): + r = run_check(ntp_sources=host) + self.assertIn("every NTP source is named by hostname", r.stdout) + + # --- Error cases ---------------------------------------------------- + + def test_no_ntp_daemon_is_a_finding(self): + r = run_check(ntp_sources="MISSING") + self.assertIn("no NTP implementation is active", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_no_sources_configured_is_a_finding(self): + # Fails closed: an empty list proves nothing about the machine, and + # reporting ok would be a false pass on a box with no time sync at all. + r = run_check(ntp_sources="") + self.assertIn("no NTP sources are configured", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_unset_seam_falls_through_to_the_real_probe(self): + # Same contract as every other seam: unset means "really look", so a + # caller who forgets the variable cannot silently skip the check. + env = dict(os.environ) + env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", + "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) + env.pop("PRC_NTP_SOURCES", None) + r = subprocess.run(["sh", CHECK], capture_output=True, text=True, + timeout=30, env=env) + self.assertIn("check 6/6: NTP bootstrap", r.stdout) + + class AllClean(unittest.TestCase): # --- Normal cases --------------------------------------------------- @@ -482,6 +562,7 @@ class SignalAccount(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234", "signal_missing": "1"}) r = subprocess.run(["sh", CHECK], capture_output=True, text=True, @@ -504,6 +585,7 @@ class ProbeFailure(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) for n in names: env.pop(n, None) @@ -624,7 +706,7 @@ class WedgedSystemctl(unittest.TestCase): self.assertEqual(r.returncode, 1) self.assertIn("could not query user units", r.stdout) # The run must reach the end rather than stopping at the first call. - self.assertIn("check 5/5", r.stdout) + self.assertIn("check 6/6", r.stdout) def test_a_hanging_systemctl_does_not_stall_the_whole_run(self): # The fake sleeps 8s against a 1s bound, so a bounded run lands near @@ -648,7 +730,7 @@ class Reporting(unittest.TestCase): unit_states="c.timer disabled") self.assertEqual(r.returncode, 1) summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "3 finding(s) across 5 checks") + self.assertEqual(summary, "3 finding(s) across 6 checks") def test_the_summary_count_tracks_every_check(self): # One finding from each of the five, so a counter that drops or @@ -664,7 +746,7 @@ class Reporting(unittest.TestCase): local_roots=scan, project_roots=proj, signal_accounts="") summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "5 finding(s) across 5 checks") + self.assertEqual(summary, "5 finding(s) across 6 checks") def test_help_exits_zero(self): r = subprocess.run(["sh", CHECK, "--help"], diff --git a/todo.org b/todo.org index 32eb9a2..1bb810d 100644 --- a/todo.org +++ b/todo.org @@ -45,6 +45,86 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work +** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The installer wrote both halves of a deadlock. =configure_dns= pins +=DNSOverTLS=yes= with =DNSSEC=yes=, and both validate against the wall clock; +the chrony step enables chronyd without writing a config, so the machine runs +Arch's stock one whose only source is =pool 2.arch.pool.ntp.org= — a hostname. +Boot with a wrong clock and DoT certificate validation fails, so nothing +resolves; chrony then cannot resolve its pool, so the clock stays wrong. +Neither side moves. It caught velox on the road 2026-08-19 and had to be +diagnosed from a phone. + +Fixed at the root: the installer now writes +=/etc/chrony.d/10-bootstrap-ip-ntp.conf= with two IP-addressed Cloudflare +sources and points stock chrony.conf at the drop-in. An address needs no DNS +and carries no certificate, so the escape hatch holds whatever broke the clock. +velox has the same drop-in applied live, verified with =chronyc -n sources= +(=162.159.200.1= selected) and =timedatectl= reporting synchronized. + +What is left here is the part I could not verify: the decisive test is a full +power-down and cold boot, confirming the clock corrects itself untouched. See +the manual-testing entry. Until that runs, the fix is sound by construction +rather than demonstrated. + +Grading: Critical severity (total loss of network — no DNS means no egress, and +recovery needs a second device) x some users sometimes (only machines that boot +with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = +[#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in +this state it is fully offline with no local path out. + +*** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed +The installer carries the drop-in; =post-rebuild-check= grew a sixth check that +fails a machine whose every NTP source is a hostname; the net failure taxonomy +gained the mode in its DNS layer plus a cluster 5 triage line, and its existing +egress-layer clock entry now says outright that its remedy does not apply when +DoT or DNSSEC is on. + +The doctor half is in dotfiles: =classify.py= reached "DNS not resolving → net +repair dns-test" here, which cannot help, because every public resolver fails +the same clock-sensitive validation — so the doctor sent you round a loop. It +now emits a =clock-dns= row ahead of the generic DNS verdict. Detection is +deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass +query addressed by IP over plain UDP/53 to tell "resolved is refusing to +validate" apart from "DNS is genuinely dead". + +** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The doctor now *names* the deadlock but hands the user two commands rather than +running anything — the verdict is =needs-user-action=. That was the honest call +at the time: correcting the clock needs the real time, and I could not exercise +a repair against the actual failure state without deliberately wedging velox's +network mid-session. + +An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a +certificate. The shape would be =chronyc add server iburst= followed by +=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to +a running chronyd over its socket, so this works in exactly the state that +blocks everything else. Two things to settle before building it: whether +=makestep= actually steps on the first sample after a runtime =add server= or +needs a poll first, and what the doctor does on a box running timesyncd rather +than chrony. + +Worth less now than it looks: a machine built by the current installer carries +the IP-addressed source and never reaches the deadlock. This is for machines +built before the fix. + +Grading: Minor severity (the doctor already names the fault correctly and hands +over a working remedy; only the automation is missing) x rare edge case (only +pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this +state has no working network and cannot look anything up, which makes the +two-command handoff harder to follow than it reads. Minor x rare = P4, and I am +leaving it at [#C] rather than [#D] because it sits one step from done. + ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> :PROPERTIES: @@ -79,6 +159,26 @@ grep the journal for new "Power key pressed" lines — zero means fixed. Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree +The reseat did not happen before the flight, and velox is travelling. The +deadline blew past on 08-14. + +The two symptoms have separated, which is worth recording because it changes +what the evidence proves. The phantom presses have stopped: fifteen "Power key +pressed" entries between 08-14 04:29 and 08-15 20:04, then nothing at all +across five boots including today's. The touchpad has not — there is still no +touchpad node under =/dev/input/by-path/=, which is the same dead interrupt +line the body describes. + +So the quiet power button is not evidence the connector reseated itself. The +interrupt line is the symptom that cannot be masked in software, and it is +still dead, so the ribbon is still unseated. The most likely reason the +presses stopped is that the machine has been sitting on hotel surfaces instead +of being carried and flexed. + +The interim shield is still live (=HandlePowerKey=ignore=), and the escalation +note stands: an EC-level glitch cuts power below systemd regardless of it. ** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -196,7 +296,29 @@ machine-level half is already correct. Grading: Major severity (a crash loop burning battery and filling the journal, silently) x every user every time on any laptop with the TLP fix applied = P1 = [#A]. -** TODO [#A] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: + +*** 2026-08-17 Mon @ 19:57:42 -0700 The loop stopped at the reboot; the defect did not +velox rebooted at 16:04 and there have been zero coredumps since, against 47 +in the twelve hours before it. So the loop is not currently burning anything. + +That is not a fix, and the distinction matters for whoever picks this up. +=powerprofilesctl get= still fails exactly as recorded — =NameHasNoOwner ... +unit is masked= — so every precondition for the loop is intact and it returns +whenever the caller next polls. What the reboot cleared is the caller's state, +not the bug. + +Narrowed the search the body asks for: =power.py= is the *only* file in +dotfiles that shells out to =powerprofilesctl= (=SETTINGS_POWERPROFILESCTL=, +line 14), so the caller is inside the settings module rather than waybar or a +timer. Worth knowing that the coredumps are =powerprofilesctl= itself aborting +— it is a python script, which is why they log as =/usr/bin/python3.14= +SIGABRT rather than under its own name. + +Grade unchanged. The matrix inputs did not move: the severity is what happens +while the machine is in that state, and the frequency row is every laptop +carrying the TLP fix. A quiet interval since a reboot is not a frequency +change. +** TODO [#B] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] :LAST_REVIEWED: 2026-08-17 @@ -236,6 +358,24 @@ unless you look) x rare edge case (one machine, specific conditions) = P2 = [#B]... except that this is a live, ongoing drain on a travelling machine rather than a latent defect, so it takes [#A] until the machine is back to normal. Re-grade to [#B] once resolved and the question is only prevention. + +*** 2026-08-17 Mon @ 19:57:42 -0700 The reboot cleared it; re-graded [#A] to [#B] as the task instructed +velox rebooted at 16:04. The wedge is gone: =systemctl --user is-enabled +roam-sync.timer= now answers =enabled= in well under a second, where every +unit-file call hung indefinitely before, and =list-timers= shows +calendar-sync, roam-sync and agenda-render-cache all firing on schedule +again. So the remedy the task named — a logout or reboot — was taken and +worked. + +Nothing here was diagnosed further, which means the cause is still unproven +and both candidates in the body stand. What is left is prevention, and the +task's own grading says that is [#B]: the live-drain argument was the only +thing holding it at [#A], and the drain has stopped. Re-graded per that +instruction rather than by a fresh judgment. + +Reproducing it deliberately is the open question, and it is not obviously +worth doing — it costs a wedged session to learn something the crash-loop fix +may make moot. ** TODO [#A] The installer clones my two working repos shallow and read-only :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] @@ -446,7 +586,7 @@ users sometimes = P3 = [#C]. ** TODO [#B] Land the rescued emacs-wttrin commit :chore:velox: :PROPERTIES: :CREATED: [2026-08-14 Fri] -:LAST_REVIEWED: 2026-08-14 +:LAST_REVIEWED: 2026-08-17 :END: bf0457f "feat: add wttrin-hide-follow-line to hide the wttr.in follow line" (2026-06-24) was the only genuinely unpushed commit anywhere on the old @@ -456,6 +596,17 @@ git bundle before the disk was wiped: To land it: clone emacs-wttrin, =git fetch --branches=, review the commit, then push to git@cjennings.net:emacs-wttrin.git. Delete the bundle once it's on the remote. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: still unlanded, and the bundle is still the only copy +Cloned the remote bare and asked it for the object directly: =git cat-file -t +bf0457f= returns "Not a valid object name", so the commit has never reached +=git@cjennings.net:emacs-wttrin.git=. Remote =main= is =ee8fdeb=. + +That makes =working/velox-reinstall/wttrin-bf0457f.bundle= the sole surviving +copy of 103 insertions across three files, on one laptop that is travelling. +Worth doing sooner than its =[#B]= suggests for that reason alone, and it also +pins the working directory open — the reinstall task cannot file its artifacts +away while this bundle is still load-bearing. ** TODO [#B] archsetup doesn't clone rulesets :bug:velox: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -868,18 +1019,38 @@ doc above (not published, since they map the setup). Follow-ons: the rotation VERIFY above, velox reconcile on return, the secrets-repo split (top of Open Work), the wireguard =.gitignore= bug (line ~191), the cgit move (below), and a pre-receive secret-scan hook so this can't recur. -*** TODO [#A] velox: reconcile its clones after the history rewrite -velox was offline for repair during the 2026-08-09 purge, so its clones still -hold the pre-rewrite history and are diverged from the rewritten remotes. On -its return: force-fetch + rebase local work onto the rewritten main in both -repos (or re-clone), force-update the local tag, local-gc, before its next -push. Also on the velox riders on the sleep/suspend task. +*** 2026-08-17 Mon @ 19:57:42 -0700 Moot — the 08-13 wipe re-cloned velox from the rewritten remotes +This asked velox to reconcile clones that no longer exist. The machine was +wiped and reinstalled on 2026-08-13, so every repo on it was cloned fresh +*after* the purge and never held the pre-rewrite history at all. The runbook +anticipated this ("fresh clones automatically carry the post-purge rewritten +git history"); nobody closed the task once the reinstall took that route. + +Verified rather than assumed: both repos are level with =origin/main= today — +archsetup at =6faa31c=, dotfiles at =65940f2=, both trees clean. + +One thing the reinstall did leave, and it is filed separately: the installer +cloned both repos =--depth 1=, so the history was present-but-truncated until +today's =git fetch --unshallow= (see the shallow-clone =[#A]=). A reconcile +against the rewritten remote was still unnecessary — a shallow clone of the +right history is not a diverged clone of the wrong one. ** TODO [#B] Move archsetup off cgit to cjennings@cjennings.net :chore:security: :PROPERTIES: -:LAST_REVIEWED: 2026-07-21 +:LAST_REVIEWED: 2026-08-17 :END: Decided (Craig, 2026-07-20): move the archsetup repo off the public cgit host (git@cjennings.net, scan-path /var/git) to Craig's private account remote cjennings@cjennings.net, so it is no longer world-cloneable. This is the archsetup-specific fix for the cgit-exposure finding above. Plan: create a bare repo under cjennings's control off the cgit scan-path (e.g. =~cjennings/git/archsetup.git=); push current main + tags there; migrate the post-receive hook that publishes the installer to =/var/www/cjennings/archsetup= so curl-install keeps working (the single published file stays public by design; only the repo goes private); update the origin remote on ratio and velox to =cjennings@cjennings.net:git/archsetup.git=; remove =/var/git/archsetup.git= so cgit no longer serves it. Verify: anonymous =git clone https://git.cjennings.net/archsetup.git= fails, the new private clone works from both machines, and the curl-install URL still returns the installer. Keep the two daily drivers' remotes in sync (daily-drivers rule). + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: unstarted, and the exposure is confirmed live +Ran the task's own verification step as it stands today, which is the honest +way to check an unstarted task rather than reading its body back. Anonymous +=git ls-remote https://git.cjennings.net/archsetup.git= succeeded with no +credentials and returned =6faa31c= — this afternoon's HEAD. So the repo is +still world-cloneable and current to the commit, not a stale published +snapshot. + +=origin= on this machine is still =git@cjennings.net:archsetup.git=, the cgit +account, so nothing has moved. Everything in the plan stands unchanged. ** TODO [#B] Velox boot-failure retrospective — upgrade guard gaps :bug:zfs:maint: :PROPERTIES: :LAST_REVIEWED: 2026-07-21 @@ -1331,15 +1502,28 @@ Verify (manual, live): see Manual testing and validation. *** 2026-07-09 Thu @ 16:32:54 -0500 Audit reconcile: Phase 4 is filed on the dotfiles side, waiting on them The dotfiles project accepted the Phase 4 handoff and filed it as a =[#C]= task in their own =todo.org= (their note, 2026-07-08 16:56): the help-text audit + panel help affordance, the user-guide/README, and the ratio rollout doc. Not started there. They ping when it lands, and this task's Phase 4 child closes then. Nothing to do here meanwhile. -*** TODO Phase 4 — docs + rollout :network:blocked: -Deliverable: in-app help (=net --help= + per-command, panel help affordance); -README/user-guide (commands, indicator states, panel, config keys, make targets, -troubleshooting from the failure table, rollback); archsetup Hyprland dep install +*** 2026-08-17 Mon @ 19:57:42 -0700 Landed on the dotfiles side; the block is cleared +dotfiles shipped it as =138da7b= and closed its own task, so this one closes +with it and the =:blocked:= tag comes off. Found by checking their =todo.org= +rather than waiting for the ping — their close-out note says "archsetup pinged +so its Phase 4 task can close", so the handoff worked and only this end was +left open. + +All three acceptance criteria are met on their side: the help audit found and +fixed a stale =net repair= action list (nine of nineteen actions were named; +both the CLI help and =repair.py='s docstring now generate from the ACTIONS +registry), =net/README.md= covers every command plus the recovery targets, and +the ratio rollout is documented with both daily drivers verified current. + +They split the panel help affordance out rather than inventing it — no sibling +panel has one, so its shape is a design call. It is tracked on their side, not +here. + +Original deliverable, for the record: in-app help (=net --help= + per-command, +panel help affordance); README/user-guide; archsetup Hyprland dep install (=gtk4-layer-shell=, =python-gobject=, =speedtest-go-bin=); ratio manual dep + -stow step. -Verify: =net --help= and each subcommand complete; user-guide covers every command -+ the recovery targets. -Build handed off to the dotfiles project 2026-07-04 (=~/.dotfiles/inbox/2026-07-04-1305-from-archsetup-phase4-handoff.md=): archsetup deps confirmed installed, the remaining help/user-guide/rollout-doc work is in the net package. dotfiles pings back when it lands. +stow step. Handed off 2026-07-04 with the archsetup deps already confirmed +installed. *** TODO Phase 5 — VPN / WireGuard CLI fold (vNext) :network: Rescoped 2026-07-04 (audit): the tunnels track already shipped most of the original Phase 5. Panel tunnel bring-up/down and detection landed (dotfiles 2d9d060 probes tailscale/NM-wireguard/Proton; 21db05a brings overlays up/down from the panel's Tunnels sub-view; 31ba056 diagnose/doctor understand tunnel routes; archsetup 2e40781 wireguard config import; the net-panel-other-interfaces spec is IMPLEMENTED). What remains for Phase 5 is only the =net vpn ...= CLI subcommand — cli.py still has no vpn/tunnel parser. Fold the panel's existing tunnel operations into a CLI surface; spec separately when picked up. @@ -1601,6 +1785,29 @@ Add kernel parameter: ~rtc_cmos.use_acpi_alarm=1~ (will become systemd default) Consider: ~acpi_mask_gpe=0x1A~ for battery drain, suspend-then-hibernate config See Framework community notes on logind.conf and sleep.conf settings +*** 2026-08-17 Mon @ 19:57:42 -0700 Four of the five riders are done; WireGuard is the one left +The riders were written for "when velox returns from repair". It came back as +a full reinstall instead, and the installer carried most of them, so I checked +each on the live machine rather than reading the list back: + +- tlp radio-enable — done. =/etc/tlp.d/01-custom.conf:10= carries + =DEVICES_TO_ENABLE_ON_STARTUP="bluetooth wifi"=, written by the installer. +- touchpad auto-detection — the dotfiles half is done: =touchpad-auto + --detect= prints =pixa3854:00-093a:0274-touchpad=. Read that carefully + though — it names the device the config expects, not a device delivering + events. The touchpad is still dead on the ribbon fault, so this rider is + satisfied and the hardware still is not. +- podman socket — done, =podman.socket= is enabled. +- camera udev — done, =72-usb-passthrough-cameras.rules= is installed. +- *wolf WireGuard — not done, and it is the one that was time-critical.* No + =~/.config/wireguard/wolf.conf.gpg= and no WireGuard profile in + NetworkManager. The 08-08 decision set this up specifically so velox could + reach home from the road, on the argument that it is cheap at home and + expensive from a hotel. velox is now in the hotel. + +The suspend work itself is untouched — no kernel parameter, no drain +measurement. Only the riders moved. + ** TODO [#B] Manual testing and validation :test: :PROPERTIES: :LAST_REVIEWED: 2026-07-09 @@ -1609,6 +1816,47 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. +*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? +What we're verifying: that the IP-addressed NTP drop-in actually breaks the +bootstrap deadlock on a real cold start. This is the one test no agent can run — +it needs a full power-down, which is exactly the event that empties a failing +RTC. Everything else about the fix is verified; this is the part that rests on +construction (an address needs no DNS, NTP carries no certificate) rather than +on having been seen work. + +Do this before relying on it away from home — the failure mode strands the +machine with no network and no way to look anything up. +- Confirm the drop-in is in place and chrony is using it (block below). +- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only + loses time when the machine is actually off. +- Leave it off long enough to matter if the coin cell is the culprit (overnight + is the honest test; a few minutes may not drain anything). +- Power on. Do not touch the clock, do not run anything. Just log in and wait + about a minute. +- Run the verification block below. +#+begin_src sh :results output +echo "--- drop-in present? ---" +cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" +echo "--- is chrony reading it? ---" +grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" +echo "--- sources (the IP literal should be selected, marked ^*) ---" +chronyc -n sources +echo "--- clock ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' +echo "--- did DNS come back on its own? ---" +getent hosts gnu.org || echo "DNS STILL DEAD" +#+end_src +Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, +=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and +one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org +resolves — all without you having set the time. + +If the RTC came up wrong and the clock corrected itself anyway, the fix works +and the coin cell question is answered separately (a wrong RTC time in that +output means the cell is dying). If the clock is still wrong or DNS is still +dead, the fix did not hold: capture that whole block and promote this to a +top-level TODO. + *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. - Go to a workspace with 2-3 tiled windows in master. -- cgit v1.2.3 From 7b68f77a9b99e5400472986cb80bae5fb92e2320 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:32:07 -0700 Subject: docs: correct the clock/DNS deadlock mechanism to DNSSEC I reproduced the failure by winding velox's clock back 27 days with chronyd stopped, and the cause is not what I recorded. Resolved logged signature-expired against the root DNSKEY and every DS beneath it. The DoT handshake to 1.1.1.1:853 verified clean at that same clock, and the Cloudflare certificate runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window is days to weeks while a certificate is good for a year, so a skew that breaks DNSSEC normally leaves DoT untouched. DNSSEC=allow-downgrade does not rescue it either. Resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure, so no downgrade fires. Six retries over eighteen seconds plus a reset-server-features, all dead. I briefly believed otherwise off a test whose success was a cache hit. The fix itself is verified end to end. With the clock wound back and no DNS at all, chronyd reached the IP-addressed source and stepped the clock straight back. Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's build date to the minute, and systemd advances a garbage RTC to its own build epoch at boot. --- archsetup | 4 +- docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +- todo.org | 99 ++++++++++++++++++++-- 3 files changed, 94 insertions(+), 13 deletions(-) (limited to 'docs') diff --git a/archsetup b/archsetup index 078408c..4bceda5 100755 --- a/archsetup +++ b/archsetup @@ -1203,8 +1203,8 @@ configure_build_environment() { # Bootstrap NTP sources addressed by IP, never by hostname. # # Arch's stock chrony.conf names its pool by hostname, and the DNS this - # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both - # validate against the wall clock, so a machine that boots with a wrong + # installer configures later runs DNSSEC=yes, which validates signature + # windows against the wall clock, so a machine that boots with a wrong # clock resolves nothing: chrony cannot reach the pool, so the clock stays # wrong, so DNS stays dead. Neither side moves, and recovery needs a second # device to look up an NTP address by hand. An IP-addressed source needs no diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 70421d0..4d57b86 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,7 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] -- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSSEC=yes= validates RRSIG inception/expiry windows against the wall clock, so a clock weeks off fails every query before it leaves the machine. Measured on velox 2026-08-19 with the clock wound back 27 days: resolved logged =signature-expired= against the root DNSKEY and every DS beneath it, and resolution died outright. =DNSOverTLS=yes= is *not* what bites, despite being the obvious suspect — the DoT handshake to =1.1.1.1:853= verified clean at that same clock, because a resolver certificate is good for about a year while an RRSIG window is days to weeks. A skew large enough to break DNSSEC normally leaves the certificate valid. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. =DNSSEC=allow-downgrade= does *not* help here, which is worth knowing because it is the obvious reach: resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure rather than a support failure, so no downgrade fires. Measured on velox: six retries over eighteen seconds, plus =resolvectl reset-server-features=, all dead. The only cure is correcting the clock, which is why the NTP source has to be reachable without DNS. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -310,7 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. -- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. +- Clock skew breaks DNSSEC validation, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/todo.org b/todo.org index 1bb810d..fa3252d 100644 --- a/todo.org +++ b/todo.org @@ -78,6 +78,41 @@ with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = [#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in this state it is fully offline with no local path out. +*** 2026-08-19 Wed @ 12:25:00 -0700 Reproduced it, and the mechanism was not what either of us said +I wound velox's clock back 27 days with chronyd stopped and watched it fail. +Resolution died outright, and plain UDP/53 to 1.1.1.1 kept answering throughout +— the discriminator the doctor keys on, confirmed live rather than reasoned. + +The cause is DNSSEC, not DNS-over-TLS. resolved logged =signature-expired= +against the root DNSKEY and every DS beneath it. The DoT handshake to +=1.1.1.1:853= verified clean at that same clock, and the Cloudflare certificate +runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window +is days to weeks and a certificate is good for a year, so a skew that breaks +DNSSEC normally leaves DoT untouched. The phone session blamed the certificate +and I carried that forward into the first commit; both were wrong. + +=DNSSEC=allow-downgrade= does not rescue it either, which matters because it is +the obvious reach and it is what ratio runs. resolved downgrades when a server +lacks DNSSEC support, and a signature-window failure is a validation failure, so +no downgrade fires. Six retries over eighteen seconds plus +=resolvectl reset-server-features=, all dead. I briefly believed otherwise off a +test whose success was a cache hit (=Data from: cache network=). + +So ratio was exposed after all, and I have given it the same drop-in. Its +=162.159.200.1= is selected and its clock is synchronized. + +The fix itself is verified end to end: with the clock wound back and no DNS at +all, chronyd reached the IP-addressed source and stepped the clock from +2026-07-23 straight back to 2026-08-19. That is the whole claim, demonstrated +rather than argued. + +Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's +build date to the minute (=/usr/lib/systemd/systemd=, 10:43:59), and systemd +advances a garbage RTC to its own build epoch at boot. Not timesyncd's +last-good-sync timestamp, which cannot be it — timesyncd is disabled here. That +also confirms the RTC really was reading earlier than that, so the coin cell +stays the prime suspect. + *** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed The installer carries the drop-in; =post-rebuild-check= grew a sixth check that fails a machine whose every NTP source is a hostname; the net failure taxonomy @@ -93,6 +128,51 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". +** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, +because the reason I changed it turned out to be false. It does not prevent the +clock deadlock. The IP-addressed NTP source does, and that is already in place +on both machines. + +What remains is a different question the taxonomy already documents: =DNSSEC=yes= +hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel +and airport problem and therefore velox's problem more than ratio's. +=allow-downgrade= trades authenticated answers for staying online. ratio already +runs =allow-downgrade=, so the fleet disagrees with itself and with the +installer, and I do not know whether ratio's setting was a deliberate policy or +a forgotten workaround for one bad network. + +I have not made this call. It is a security posture change and it should be made +knowingly rather than as a side effect of a theory I disproved an hour later. + +** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +Three defaults were chosen for a desktop and then applied to the machine that +travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per +connection (every hotel reconnect looks like a new device, so the portal login +starts over), and hostname-only NTP (the deadlock). Two are now fixed for every +machine, and the third is the VERIFY above. + +The installer already branches on battery presence in three places: +=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config +itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and +network policy simply does not use it. Wiring the same test around +=configure_networking= would let laptop and desktop defaults diverge deliberately +instead of by drift, and would stop the next instance of this from happening. + +Grading: Minor severity (nothing is broken today; this prevents a recurrence) x +some users sometimes (bites when a new default suits one machine class and not +the other) = P3 = [#C]. + ** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: :PROPERTIES: :CREATED: [2026-08-19 Wed] @@ -1817,15 +1897,16 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. *** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: that the IP-addressed NTP drop-in actually breaks the -bootstrap deadlock on a real cold start. This is the one test no agent can run — -it needs a full power-down, which is exactly the event that empties a failing -RTC. Everything else about the fix is verified; this is the part that rests on -construction (an address needs no DNS, NTP carries no certificate) rather than -on having been seen work. - -Do this before relying on it away from home — the failure mode strands the -machine with no network and no way to look anything up. +What we're verifying: the coin cell, not the fix. The fix itself is already +demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, +and chronyd reached the IP-addressed source and stepped it straight back. What a +cold boot adds is the hardware question: whether the RTC actually loses time +across a full power-down, which is the thing that started this. + +Read the outcome carefully, because only one branch is informative. An RTC time +that comes up wrong and then self-corrects tells you the cell is dying and the +fix is holding. An RTC that comes up correct tells you nothing about the +deadlock at all, only that the cell survived this particular night. - Confirm the drop-in is in place and chrony is using it (block below). - Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only loses time when the machine is actually off. -- cgit v1.2.3