From afbf011aa0937b5702b6d8c1bfca0809ed809425 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:16:46 -0700 Subject: fix(installer): give NTP an IP source so a wrong clock can't kill DNS The installer wrote both halves of a deadlock. configure_dns pins DNSOverTLS=yes with DNSSEC=yes, and both validate against the wall clock. The chrony step enables chronyd without writing a config, so the machine runs Arch's stock one, whose only source is a pool hostname. Boot with a wrong clock and DoT validation fails, so nothing resolves. Chrony then can't resolve its pool, so the clock stays wrong. Neither side moves, and recovery takes a second device. Velox hit this on the road and I diagnosed it from a phone. An address needs no DNS and no certificate, so two IP-addressed sources in a drop-in break the cycle whatever caused the skew. Stock chrony.conf reads no drop-in directory, so it gets a confdir line pointing at one. post-rebuild-check grows a sixth check for the same property. It reads sources only from files chrony is told to read. A drop-in beside a chrony.conf that never names its directory is one chrony won't open, so counting it would pass the machine while describing a file nothing reads. The failure taxonomy gains the mode in its DNS layer and a cluster 5 triage line. Its egress-layer clock entry assumed working DNS and offered set-ntp true, which can't recover this. That entry now says so. --- docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'docs/design') diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 74790c6..70421d0 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,6 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -107,7 +108,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] - Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] - Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] -- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. This entry assumes DNS still works; when the resolver runs DoT or DNSSEC the same skew kills DNS first and =set-ntp true= cannot recover it — see the clock/DNS deadlock in the DNS layer. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] - Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] - VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] - IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] @@ -309,6 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked -- cgit v1.2.3 From 7b68f77a9b99e5400472986cb80bae5fb92e2320 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:32:07 -0700 Subject: docs: correct the clock/DNS deadlock mechanism to DNSSEC I reproduced the failure by winding velox's clock back 27 days with chronyd stopped, and the cause is not what I recorded. Resolved logged signature-expired against the root DNSKEY and every DS beneath it. The DoT handshake to 1.1.1.1:853 verified clean at that same clock, and the Cloudflare certificate runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window is days to weeks while a certificate is good for a year, so a skew that breaks DNSSEC normally leaves DoT untouched. DNSSEC=allow-downgrade does not rescue it either. Resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure, so no downgrade fires. Six retries over eighteen seconds plus a reset-server-features, all dead. I briefly believed otherwise off a test whose success was a cache hit. The fix itself is verified end to end. With the clock wound back and no DNS at all, chronyd reached the IP-addressed source and stepped the clock straight back. Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's build date to the minute, and systemd advances a garbage RTC to its own build epoch at boot. --- archsetup | 4 +- docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +- todo.org | 99 ++++++++++++++++++++-- 3 files changed, 94 insertions(+), 13 deletions(-) (limited to 'docs/design') diff --git a/archsetup b/archsetup index 078408c..4bceda5 100755 --- a/archsetup +++ b/archsetup @@ -1203,8 +1203,8 @@ configure_build_environment() { # Bootstrap NTP sources addressed by IP, never by hostname. # # Arch's stock chrony.conf names its pool by hostname, and the DNS this - # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both - # validate against the wall clock, so a machine that boots with a wrong + # installer configures later runs DNSSEC=yes, which validates signature + # windows against the wall clock, so a machine that boots with a wrong # clock resolves nothing: chrony cannot reach the pool, so the clock stays # wrong, so DNS stays dead. Neither side moves, and recovery needs a second # device to look up an NTP address by hand. An IP-addressed source needs no diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 70421d0..4d57b86 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,7 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] -- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSSEC=yes= validates RRSIG inception/expiry windows against the wall clock, so a clock weeks off fails every query before it leaves the machine. Measured on velox 2026-08-19 with the clock wound back 27 days: resolved logged =signature-expired= against the root DNSKEY and every DS beneath it, and resolution died outright. =DNSOverTLS=yes= is *not* what bites, despite being the obvious suspect — the DoT handshake to =1.1.1.1:853= verified clean at that same clock, because a resolver certificate is good for about a year while an RRSIG window is days to weeks. A skew large enough to break DNSSEC normally leaves the certificate valid. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. =DNSSEC=allow-downgrade= does *not* help here, which is worth knowing because it is the obvious reach: resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure rather than a support failure, so no downgrade fires. Measured on velox: six retries over eighteen seconds, plus =resolvectl reset-server-features=, all dead. The only cure is correcting the clock, which is why the NTP source has to be reachable without DNS. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -310,7 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. -- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. +- Clock skew breaks DNSSEC validation, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/todo.org b/todo.org index 1bb810d..fa3252d 100644 --- a/todo.org +++ b/todo.org @@ -78,6 +78,41 @@ with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = [#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in this state it is fully offline with no local path out. +*** 2026-08-19 Wed @ 12:25:00 -0700 Reproduced it, and the mechanism was not what either of us said +I wound velox's clock back 27 days with chronyd stopped and watched it fail. +Resolution died outright, and plain UDP/53 to 1.1.1.1 kept answering throughout +— the discriminator the doctor keys on, confirmed live rather than reasoned. + +The cause is DNSSEC, not DNS-over-TLS. resolved logged =signature-expired= +against the root DNSKEY and every DS beneath it. The DoT handshake to +=1.1.1.1:853= verified clean at that same clock, and the Cloudflare certificate +runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window +is days to weeks and a certificate is good for a year, so a skew that breaks +DNSSEC normally leaves DoT untouched. The phone session blamed the certificate +and I carried that forward into the first commit; both were wrong. + +=DNSSEC=allow-downgrade= does not rescue it either, which matters because it is +the obvious reach and it is what ratio runs. resolved downgrades when a server +lacks DNSSEC support, and a signature-window failure is a validation failure, so +no downgrade fires. Six retries over eighteen seconds plus +=resolvectl reset-server-features=, all dead. I briefly believed otherwise off a +test whose success was a cache hit (=Data from: cache network=). + +So ratio was exposed after all, and I have given it the same drop-in. Its +=162.159.200.1= is selected and its clock is synchronized. + +The fix itself is verified end to end: with the clock wound back and no DNS at +all, chronyd reached the IP-addressed source and stepped the clock from +2026-07-23 straight back to 2026-08-19. That is the whole claim, demonstrated +rather than argued. + +Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's +build date to the minute (=/usr/lib/systemd/systemd=, 10:43:59), and systemd +advances a garbage RTC to its own build epoch at boot. Not timesyncd's +last-good-sync timestamp, which cannot be it — timesyncd is disabled here. That +also confirms the RTC really was reading earlier than that, so the coin cell +stays the prime suspect. + *** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed The installer carries the drop-in; =post-rebuild-check= grew a sixth check that fails a machine whose every NTP source is a hostname; the net failure taxonomy @@ -93,6 +128,51 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". +** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, +because the reason I changed it turned out to be false. It does not prevent the +clock deadlock. The IP-addressed NTP source does, and that is already in place +on both machines. + +What remains is a different question the taxonomy already documents: =DNSSEC=yes= +hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel +and airport problem and therefore velox's problem more than ratio's. +=allow-downgrade= trades authenticated answers for staying online. ratio already +runs =allow-downgrade=, so the fleet disagrees with itself and with the +installer, and I do not know whether ratio's setting was a deliberate policy or +a forgotten workaround for one bad network. + +I have not made this call. It is a security posture change and it should be made +knowingly rather than as a side effect of a theory I disproved an hour later. + +** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +Three defaults were chosen for a desktop and then applied to the machine that +travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per +connection (every hotel reconnect looks like a new device, so the portal login +starts over), and hostname-only NTP (the deadlock). Two are now fixed for every +machine, and the third is the VERIFY above. + +The installer already branches on battery presence in three places: +=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config +itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and +network policy simply does not use it. Wiring the same test around +=configure_networking= would let laptop and desktop defaults diverge deliberately +instead of by drift, and would stop the next instance of this from happening. + +Grading: Minor severity (nothing is broken today; this prevents a recurrence) x +some users sometimes (bites when a new default suits one machine class and not +the other) = P3 = [#C]. + ** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: :PROPERTIES: :CREATED: [2026-08-19 Wed] @@ -1817,15 +1897,16 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. *** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: that the IP-addressed NTP drop-in actually breaks the -bootstrap deadlock on a real cold start. This is the one test no agent can run — -it needs a full power-down, which is exactly the event that empties a failing -RTC. Everything else about the fix is verified; this is the part that rests on -construction (an address needs no DNS, NTP carries no certificate) rather than -on having been seen work. - -Do this before relying on it away from home — the failure mode strands the -machine with no network and no way to look anything up. +What we're verifying: the coin cell, not the fix. The fix itself is already +demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, +and chronyd reached the IP-addressed source and stepped it straight back. What a +cold boot adds is the hardware question: whether the RTC actually loses time +across a full power-down, which is the thing that started this. + +Read the outcome carefully, because only one branch is informative. An RTC time +that comes up wrong and then self-corrects tells you the cell is dying and the +fix is holding. An RTC that comes up correct tells you nothing about the +deadlock at all, only that the cell survived this particular night. - Confirm the drop-in is in place and chrony is using it (block below). - Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only loses time when the machine is actually off. -- cgit v1.2.3