diff options
| -rwxr-xr-x | archsetup | 30 | ||||
| -rw-r--r-- | docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 | ||||
| -rwxr-xr-x | scripts/post-rebuild-check | 85 | ||||
| -rw-r--r-- | tests/post-rebuild-check/test_post_rebuild_check.py | 90 | ||||
| -rw-r--r-- | todo.org | 282 |
5 files changed, 461 insertions, 30 deletions
@@ -1200,6 +1200,36 @@ configure_build_environment() { echo 'OPTIONS=""' > /etc/sysconfig/chronyd systemctl enable chronyd.service >> "$logfile" 2>&1 || error_warn "$action" "$?" + # Bootstrap NTP sources addressed by IP, never by hostname. + # + # Arch's stock chrony.conf names its pool by hostname, and the DNS this + # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both + # validate against the wall clock, so a machine that boots with a wrong + # clock resolves nothing: chrony cannot reach the pool, so the clock stays + # wrong, so DNS stays dead. Neither side moves, and recovery needs a second + # device to look up an NTP address by hand. An IP-addressed source needs no + # DNS and no certificate, so it breaks the deadlock unattended. I would + # rather carry two extra server lines than lose a laptop's network to any + # RTC fault. See the clock/DNS deadlock entry in the net failure taxonomy + # under docs/design/. + action="adding IP-addressed NTP bootstrap sources" && display "task" "$action" + mkdir -p /etc/chrony.d + cat << 'EOF' > /etc/chrony.d/10-bootstrap-ip-ntp.conf +# Reachable without DNS, so a wrong clock can always correct itself. +server 162.159.200.1 iburst +server 162.159.200.123 iburst +EOF + # Stock chrony.conf reads no drop-in directory, so point it at one. + if [ -f /etc/chrony.conf ]; then + backup_system_file /etc/chrony.conf + if ! grep -qE '^[[:space:]]*confdir[[:space:]]+/etc/chrony\.d' /etc/chrony.conf; then + printf '\n# Read drop-ins (archsetup owns /etc/chrony.d).\nconfdir /etc/chrony.d\n' \ + >> /etc/chrony.conf || error_warn "$action" "$?" + fi + else + error_warn "$action (no /etc/chrony.conf to point at /etc/chrony.d)" 1 + fi + action="configuring compiler to use all processor cores" && display "task" "$action" backup_system_file /etc/makepkg.conf sed -i "s/-j2/-j$(nproc)/;s/^#MAKEFLAGS/MAKEFLAGS/" /etc/makepkg.conf >> "$logfile" 2>&1 diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 74790c6..70421d0 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,6 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -107,7 +108,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] - Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] - Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] -- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. This entry assumes DNS still works; when the resolver runs DoT or DNSSEC the same skew kills DNS first and =set-ntp true= cannot recover it — see the clock/DNS deadlock in the DNS layer. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] - Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] - VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] - IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify <con> ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] @@ -309,6 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/scripts/post-rebuild-check b/scripts/post-rebuild-check index 8807f85..c18ae8f 100755 --- a/scripts/post-rebuild-check +++ b/scripts/post-rebuild-check @@ -21,6 +21,10 @@ # 5. signal-cli holds no registered account (velox lost its # registration, and because agent-text relays into this machine, # that silently broke paging for the WHOLE fleet) +# 6. every NTP source is named by hostname (a wrong clock fails the +# DoT/DNSSEC validation this machine's DNS runs on, so nothing +# resolves -- including the NTP pool that would fix the clock; velox +# deadlocked exactly this way 2026-08-19 and needed a second device) # # The .gitignore rule in check 4 is what scopes it: a tooling path is only # expected where the project's own .gitignore names it, so a project that @@ -50,6 +54,8 @@ # ~/.dotfiles) # PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output; "" = no account, # the special value MISSING = binary absent +# PRC_NTP_SOURCES newline list of configured NTP server addresses; +# the special value MISSING = no NTP daemon active # PRC_SYSTEMCTL path to the systemctl binary (a fake, under test) # PRC_SYSTEMCTL_TIMEOUT seconds to allow each systemctl call (default 5) # @@ -61,9 +67,10 @@ usage() { cat <<'EOF' post-rebuild-check - verify a rebuilt machine is actually finished -Runs the five checks that caught velox's 2026-08 reinstall gaps: failed +Runs the six checks that caught velox's 2026-08 reinstall gaps: failed units, present-but-inert user units, orphaned *.example configs, missing -per-project tooling state, and the signal-cli registration. +per-project tooling state, the signal-cli registration, and whether time +sync can recover from a wrong clock without DNS. Usage: post-rebuild-check [--help] @@ -84,6 +91,7 @@ TOTAL_FINDINGS=0 CHECK_FINDINGS=0 FINDING_LINES="" signal_missing="" +ntp_missing="" # Every systemctl call is bounded. A wedged user manager spins and answers # nothing -- seen live on velox 2026-08-17, where `is-enabled`, `cat`, and @@ -179,7 +187,7 @@ while IFS= read -r line; do unit=${line#*:} finding "$scope unit failed: $unit" done < "$STAGE" -report "check 1/5: failed units" +report "check 1/6: failed units" # --- 2. user unit files present but not enabled --------------------------- @@ -269,7 +277,7 @@ while read -r name state; do esac finding "unit file present but not enabled: $name ($state)" done < "$STAGE" -report "check 2/5: unit files" +report "check 2/6: unit files" # --- 3. *.example files whose real sibling is missing --------------------- @@ -314,7 +322,7 @@ while IFS= read -r root; do [ -e "${ex%.example}" ] || finding "example without its real file: $ex" done < "$WORK/examples" done < "$WORK/roots" -report "check 3/5: local files" +report "check 3/6: local files" # --- 4. gitignore-mode projects missing their tooling --------------------- @@ -368,7 +376,7 @@ todo.org todo\.org inbox inbox EOF done < "$WORK/projects" -report "check 4/5: project tooling" +report "check 4/6: project tooling" # --- 5. signal-cli registration ------------------------------------------- @@ -395,7 +403,68 @@ if [ "$signal_missing" = 1 ]; then elif [ -z "$signal_missing" ] && [ -z "$accounts" ]; then finding "no signal account registered — agent-text relays into this machine, so paging breaks for the whole fleet" fi -report "check 5/5: signal registration" +report "check 5/6: signal registration" + +# --- 6. NTP can recover a wrong clock without DNS ------------------------- +# +# The clock/DNS bootstrap deadlock. This machine resolves through DNSOverTLS +# with DNSSEC, and both validate against the wall clock, so a boot with a +# wrong clock resolves nothing at all. If every configured NTP source is named +# by hostname, the daemon that would correct the clock needs the DNS the clock +# is breaking, and the machine cannot recover without a second device -- +# which is exactly what happened on velox 2026-08-19. One source addressed by +# IP breaks the cycle, so that is what this check looks for. + +# True when the argument is an address rather than a name. An address needs no +# resolver, which is the whole property being checked. +is_ip_literal() { + case "$1" in + "") return 1 ;; + *:*) case "$1" in *[!0-9A-Fa-f:]*) return 1 ;; esac + return 0 ;; + *[!0-9.]*) return 1 ;; + *.*) return 0 ;; + esac + return 1 +} + +if [ -n "${PRC_NTP_SOURCES+set}" ]; then + ntp_sources=$PRC_NTP_SOURCES + if [ "$ntp_sources" = "MISSING" ]; then + ntp_sources="" + ntp_missing=1 + fi +elif sctl is-active chronyd >/dev/null 2>&1; then + # Both the main file and any drop-in: the IP-addressed source belongs in a + # drop-in, so reading only chrony.conf would miss every correct machine. + ntp_sources=$(cat /etc/chrony.conf /etc/chrony.d/*.conf 2>/dev/null \ + | awk '$1 == "server" || $1 == "pool" { print $2 }') +elif sctl is-active systemd-timesyncd >/dev/null 2>&1; then + ntp_sources=$(awk -F= '/^[[:space:]]*NTP=/ { print $2 }' \ + /etc/systemd/timesyncd.conf 2>/dev/null | tr ' ' '\n') +else + ntp_sources="" + ntp_missing=1 +fi + +if [ "$ntp_missing" = 1 ]; then + finding "no NTP implementation is active — nothing corrects the clock, and a wrong clock takes DNS down with it" +elif [ -z "$ntp_sources" ]; then + finding "no NTP sources are configured — nothing was checked, and nothing corrects the clock" +else + ntp_has_literal="" + stage "$ntp_sources" + while IFS= read -r src_addr; do + [ -z "$src_addr" ] && continue + if is_ip_literal "$src_addr"; then + ntp_has_literal=1 + fi + done < "$STAGE" + if [ -z "$ntp_has_literal" ]; then + finding "every NTP source is named by hostname — a wrong clock breaks DNS, so nothing can resolve them and the clock stays wrong" + fi +fi +report "check 6/6: NTP bootstrap" # --- summary -------------------------------------------------------------- @@ -403,5 +472,5 @@ if [ "$TOTAL_FINDINGS" -eq 0 ]; then echo "all checks clean" exit 0 fi -echo "$TOTAL_FINDINGS finding(s) across 5 checks" +echo "$TOTAL_FINDINGS finding(s) across 6 checks" exit 1 diff --git a/tests/post-rebuild-check/test_post_rebuild_check.py b/tests/post-rebuild-check/test_post_rebuild_check.py index 4894451..757039b 100644 --- a/tests/post-rebuild-check/test_post_rebuild_check.py +++ b/tests/post-rebuild-check/test_post_rebuild_check.py @@ -21,6 +21,8 @@ probe"): PRC_LOCAL_SCAN_ROOTS newline-separated roots to scan for *.example orphans PRC_PROJECT_ROOTS newline-separated project dirs for the tooling check PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output ("" = no accounts); + PRC_NTP_SOURCES newline list of configured NTP server addresses + ("MISSING" = no NTP daemon active) the special value MISSING means the binary is absent Run from repo root: @@ -39,7 +41,8 @@ CHECK = os.path.join(REPO_ROOT, "scripts", "post-rebuild-check") def run_check(failed_units="", unit_states="", local_roots="", - project_roots="", signal_accounts="+15045551234"): + project_roots="", signal_accounts="+15045551234", + ntp_sources="162.159.200.1\npool.ntp.org"): """Run the script with every probe stubbed; defaults are all-clean. Roots are newline-separated. Empty means "the seam is set and names no @@ -52,11 +55,88 @@ def run_check(failed_units="", unit_states="", local_roots="", env["PRC_LOCAL_SCAN_ROOTS"] = local_roots env["PRC_PROJECT_ROOTS"] = project_roots env["PRC_SIGNAL_ACCOUNTS"] = signal_accounts + env["PRC_NTP_SOURCES"] = ntp_sources return subprocess.run( ["sh", CHECK], capture_output=True, text=True, timeout=30, env=env, ) +class NtpBootstrap(unittest.TestCase): + """Check 6 — the clock/DNS bootstrap deadlock. + + A wrong clock fails the DoT certificate and DNSSEC signature checks this + machine's DNS runs on, so nothing resolves; and an NTP daemon whose every + source is a hostname then cannot resolve the servers that would correct + the clock. One source addressed by IP is what makes the machine able to + recover on its own. + """ + + # --- Normal cases --------------------------------------------------- + + def test_an_ip_addressed_source_is_clean(self): + r = run_check(ntp_sources="162.159.200.1\npool.ntp.org") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + self.assertEqual(r.returncode, 0, r.stdout) + + def test_all_hostname_sources_is_a_finding(self): + # The velox 2026-08-19 shape exactly: stock Arch chrony.conf, whose + # only source is a pool hostname. + r = run_check(ntp_sources="2.arch.pool.ntp.org") + self.assertIn("every NTP source is named by hostname", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_an_ipv6_addressed_source_counts(self): + r = run_check(ntp_sources="2606:4700:f1::1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + # --- Boundary cases ------------------------------------------------- + + def test_the_literal_may_sit_anywhere_in_the_list(self): + # Order must not matter; the property is "at least one", and the + # drop-in that carries it is read after the main config. + r = run_check(ntp_sources="a.pool.ntp.org\nb.pool.ntp.org\n162.159.200.1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_blank_lines_between_sources_are_ignored(self): + r = run_check(ntp_sources="\n\n162.159.200.1\n\n") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_a_hostname_containing_digits_and_dots_is_not_an_address(self): + # The trap in any naive "looks like an IP" test: these resolve through + # DNS like any other name, so counting one as an address would hand a + # deadlocked machine a clean bill. + for host in ("0.arch.pool.ntp.org", "3.us.pool.ntp.org", "time1.google.com"): + with self.subTest(host=host): + r = run_check(ntp_sources=host) + self.assertIn("every NTP source is named by hostname", r.stdout) + + # --- Error cases ---------------------------------------------------- + + def test_no_ntp_daemon_is_a_finding(self): + r = run_check(ntp_sources="MISSING") + self.assertIn("no NTP implementation is active", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_no_sources_configured_is_a_finding(self): + # Fails closed: an empty list proves nothing about the machine, and + # reporting ok would be a false pass on a box with no time sync at all. + r = run_check(ntp_sources="") + self.assertIn("no NTP sources are configured", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_unset_seam_falls_through_to_the_real_probe(self): + # Same contract as every other seam: unset means "really look", so a + # caller who forgets the variable cannot silently skip the check. + env = dict(os.environ) + env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", + "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) + env.pop("PRC_NTP_SOURCES", None) + r = subprocess.run(["sh", CHECK], capture_output=True, text=True, + timeout=30, env=env) + self.assertIn("check 6/6: NTP bootstrap", r.stdout) + + class AllClean(unittest.TestCase): # --- Normal cases --------------------------------------------------- @@ -482,6 +562,7 @@ class SignalAccount(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234", "signal_missing": "1"}) r = subprocess.run(["sh", CHECK], capture_output=True, text=True, @@ -504,6 +585,7 @@ class ProbeFailure(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) for n in names: env.pop(n, None) @@ -624,7 +706,7 @@ class WedgedSystemctl(unittest.TestCase): self.assertEqual(r.returncode, 1) self.assertIn("could not query user units", r.stdout) # The run must reach the end rather than stopping at the first call. - self.assertIn("check 5/5", r.stdout) + self.assertIn("check 6/6", r.stdout) def test_a_hanging_systemctl_does_not_stall_the_whole_run(self): # The fake sleeps 8s against a 1s bound, so a bounded run lands near @@ -648,7 +730,7 @@ class Reporting(unittest.TestCase): unit_states="c.timer disabled") self.assertEqual(r.returncode, 1) summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "3 finding(s) across 5 checks") + self.assertEqual(summary, "3 finding(s) across 6 checks") def test_the_summary_count_tracks_every_check(self): # One finding from each of the five, so a counter that drops or @@ -664,7 +746,7 @@ class Reporting(unittest.TestCase): local_roots=scan, project_roots=proj, signal_accounts="") summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "5 finding(s) across 5 checks") + self.assertEqual(summary, "5 finding(s) across 6 checks") def test_help_exits_zero(self): r = subprocess.run(["sh", CHECK, "--help"], @@ -45,6 +45,86 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work +** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The installer wrote both halves of a deadlock. =configure_dns= pins +=DNSOverTLS=yes= with =DNSSEC=yes=, and both validate against the wall clock; +the chrony step enables chronyd without writing a config, so the machine runs +Arch's stock one whose only source is =pool 2.arch.pool.ntp.org= — a hostname. +Boot with a wrong clock and DoT certificate validation fails, so nothing +resolves; chrony then cannot resolve its pool, so the clock stays wrong. +Neither side moves. It caught velox on the road 2026-08-19 and had to be +diagnosed from a phone. + +Fixed at the root: the installer now writes +=/etc/chrony.d/10-bootstrap-ip-ntp.conf= with two IP-addressed Cloudflare +sources and points stock chrony.conf at the drop-in. An address needs no DNS +and carries no certificate, so the escape hatch holds whatever broke the clock. +velox has the same drop-in applied live, verified with =chronyc -n sources= +(=162.159.200.1= selected) and =timedatectl= reporting synchronized. + +What is left here is the part I could not verify: the decisive test is a full +power-down and cold boot, confirming the clock corrects itself untouched. See +the manual-testing entry. Until that runs, the fix is sound by construction +rather than demonstrated. + +Grading: Critical severity (total loss of network — no DNS means no egress, and +recovery needs a second device) x some users sometimes (only machines that boot +with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = +[#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in +this state it is fully offline with no local path out. + +*** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed +The installer carries the drop-in; =post-rebuild-check= grew a sixth check that +fails a machine whose every NTP source is a hostname; the net failure taxonomy +gained the mode in its DNS layer plus a cluster 5 triage line, and its existing +egress-layer clock entry now says outright that its remedy does not apply when +DoT or DNSSEC is on. + +The doctor half is in dotfiles: =classify.py= reached "DNS not resolving → net +repair dns-test" here, which cannot help, because every public resolver fails +the same clock-sensitive validation — so the doctor sent you round a loop. It +now emits a =clock-dns= row ahead of the generic DNS verdict. Detection is +deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass +query addressed by IP over plain UDP/53 to tell "resolved is refusing to +validate" apart from "DNS is genuinely dead". + +** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The doctor now *names* the deadlock but hands the user two commands rather than +running anything — the verdict is =needs-user-action=. That was the honest call +at the time: correcting the clock needs the real time, and I could not exercise +a repair against the actual failure state without deliberately wedging velox's +network mid-session. + +An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a +certificate. The shape would be =chronyc add server <ip> iburst= followed by +=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to +a running chronyd over its socket, so this works in exactly the state that +blocks everything else. Two things to settle before building it: whether +=makestep= actually steps on the first sample after a runtime =add server= or +needs a poll first, and what the doctor does on a box running timesyncd rather +than chrony. + +Worth less now than it looks: a machine built by the current installer carries +the IP-addressed source and never reaches the deadlock. This is for machines +built before the fix. + +Grading: Minor severity (the doctor already names the fault correctly and hands +over a working remedy; only the automation is missing) x rare edge case (only +pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this +state has no working network and cannot look anything up, which makes the +two-command handoff harder to follow than it reads. Minor x rare = P4, and I am +leaving it at [#C] rather than [#D] because it sits one step from done. + ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> :PROPERTIES: @@ -79,6 +159,26 @@ grep the journal for new "Power key pressed" lines — zero means fixed. Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree +The reseat did not happen before the flight, and velox is travelling. The +deadline blew past on 08-14. + +The two symptoms have separated, which is worth recording because it changes +what the evidence proves. The phantom presses have stopped: fifteen "Power key +pressed" entries between 08-14 04:29 and 08-15 20:04, then nothing at all +across five boots including today's. The touchpad has not — there is still no +touchpad node under =/dev/input/by-path/=, which is the same dead interrupt +line the body describes. + +So the quiet power button is not evidence the connector reseated itself. The +interrupt line is the symptom that cannot be masked in software, and it is +still dead, so the ribbon is still unseated. The most likely reason the +presses stopped is that the machine has been sitting on hotel surfaces instead +of being carried and flexed. + +The interim shield is still live (=HandlePowerKey=ignore=), and the escalation +note stands: an EC-level glitch cuts power below systemd regardless of it. ** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -196,7 +296,29 @@ machine-level half is already correct. Grading: Major severity (a crash loop burning battery and filling the journal, silently) x every user every time on any laptop with the TLP fix applied = P1 = [#A]. -** TODO [#A] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: + +*** 2026-08-17 Mon @ 19:57:42 -0700 The loop stopped at the reboot; the defect did not +velox rebooted at 16:04 and there have been zero coredumps since, against 47 +in the twelve hours before it. So the loop is not currently burning anything. + +That is not a fix, and the distinction matters for whoever picks this up. +=powerprofilesctl get= still fails exactly as recorded — =NameHasNoOwner ... +unit is masked= — so every precondition for the loop is intact and it returns +whenever the caller next polls. What the reboot cleared is the caller's state, +not the bug. + +Narrowed the search the body asks for: =power.py= is the *only* file in +dotfiles that shells out to =powerprofilesctl= (=SETTINGS_POWERPROFILESCTL=, +line 14), so the caller is inside the settings module rather than waybar or a +timer. Worth knowing that the coredumps are =powerprofilesctl= itself aborting +— it is a python script, which is why they log as =/usr/bin/python3.14= +SIGABRT rather than under its own name. + +Grade unchanged. The matrix inputs did not move: the severity is what happens +while the machine is in that state, and the frequency row is every laptop +carrying the TLP fix. A quiet interval since a reboot is not a frequency +change. +** TODO [#B] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] :LAST_REVIEWED: 2026-08-17 @@ -236,6 +358,24 @@ unless you look) x rare edge case (one machine, specific conditions) = P2 = [#B]... except that this is a live, ongoing drain on a travelling machine rather than a latent defect, so it takes [#A] until the machine is back to normal. Re-grade to [#B] once resolved and the question is only prevention. + +*** 2026-08-17 Mon @ 19:57:42 -0700 The reboot cleared it; re-graded [#A] to [#B] as the task instructed +velox rebooted at 16:04. The wedge is gone: =systemctl --user is-enabled +roam-sync.timer= now answers =enabled= in well under a second, where every +unit-file call hung indefinitely before, and =list-timers= shows +calendar-sync, roam-sync and agenda-render-cache all firing on schedule +again. So the remedy the task named — a logout or reboot — was taken and +worked. + +Nothing here was diagnosed further, which means the cause is still unproven +and both candidates in the body stand. What is left is prevention, and the +task's own grading says that is [#B]: the live-drain argument was the only +thing holding it at [#A], and the drain has stopped. Re-graded per that +instruction rather than by a fresh judgment. + +Reproducing it deliberately is the open question, and it is not obviously +worth doing — it costs a wedged session to learn something the crash-loop fix +may make moot. ** TODO [#A] The installer clones my two working repos shallow and read-only :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] @@ -446,7 +586,7 @@ users sometimes = P3 = [#C]. ** TODO [#B] Land the rescued emacs-wttrin commit :chore:velox: :PROPERTIES: :CREATED: [2026-08-14 Fri] -:LAST_REVIEWED: 2026-08-14 +:LAST_REVIEWED: 2026-08-17 :END: bf0457f "feat: add wttrin-hide-follow-line to hide the wttr.in follow line" (2026-06-24) was the only genuinely unpushed commit anywhere on the old @@ -456,6 +596,17 @@ git bundle before the disk was wiped: To land it: clone emacs-wttrin, =git fetch <bundle> --branches=, review the commit, then push to git@cjennings.net:emacs-wttrin.git. Delete the bundle once it's on the remote. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: still unlanded, and the bundle is still the only copy +Cloned the remote bare and asked it for the object directly: =git cat-file -t +bf0457f= returns "Not a valid object name", so the commit has never reached +=git@cjennings.net:emacs-wttrin.git=. Remote =main= is =ee8fdeb=. + +That makes =working/velox-reinstall/wttrin-bf0457f.bundle= the sole surviving +copy of 103 insertions across three files, on one laptop that is travelling. +Worth doing sooner than its =[#B]= suggests for that reason alone, and it also +pins the working directory open — the reinstall task cannot file its artifacts +away while this bundle is still load-bearing. ** TODO [#B] archsetup doesn't clone rulesets :bug:velox: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -868,18 +1019,38 @@ doc above (not published, since they map the setup). Follow-ons: the rotation VERIFY above, velox reconcile on return, the secrets-repo split (top of Open Work), the wireguard =.gitignore= bug (line ~191), the cgit move (below), and a pre-receive secret-scan hook so this can't recur. -*** TODO [#A] velox: reconcile its clones after the history rewrite -velox was offline for repair during the 2026-08-09 purge, so its clones still -hold the pre-rewrite history and are diverged from the rewritten remotes. On -its return: force-fetch + rebase local work onto the rewritten main in both -repos (or re-clone), force-update the local tag, local-gc, before its next -push. Also on the velox riders on the sleep/suspend task. +*** 2026-08-17 Mon @ 19:57:42 -0700 Moot — the 08-13 wipe re-cloned velox from the rewritten remotes +This asked velox to reconcile clones that no longer exist. The machine was +wiped and reinstalled on 2026-08-13, so every repo on it was cloned fresh +*after* the purge and never held the pre-rewrite history at all. The runbook +anticipated this ("fresh clones automatically carry the post-purge rewritten +git history"); nobody closed the task once the reinstall took that route. + +Verified rather than assumed: both repos are level with =origin/main= today — +archsetup at =6faa31c=, dotfiles at =65940f2=, both trees clean. + +One thing the reinstall did leave, and it is filed separately: the installer +cloned both repos =--depth 1=, so the history was present-but-truncated until +today's =git fetch --unshallow= (see the shallow-clone =[#A]=). A reconcile +against the rewritten remote was still unnecessary — a shallow clone of the +right history is not a diverged clone of the wrong one. ** TODO [#B] Move archsetup off cgit to cjennings@cjennings.net :chore:security: :PROPERTIES: -:LAST_REVIEWED: 2026-07-21 +:LAST_REVIEWED: 2026-08-17 :END: Decided (Craig, 2026-07-20): move the archsetup repo off the public cgit host (git@cjennings.net, scan-path /var/git) to Craig's private account remote cjennings@cjennings.net, so it is no longer world-cloneable. This is the archsetup-specific fix for the cgit-exposure finding above. Plan: create a bare repo under cjennings's control off the cgit scan-path (e.g. =~cjennings/git/archsetup.git=); push current main + tags there; migrate the post-receive hook that publishes the installer to =/var/www/cjennings/archsetup= so curl-install keeps working (the single published file stays public by design; only the repo goes private); update the origin remote on ratio and velox to =cjennings@cjennings.net:git/archsetup.git=; remove =/var/git/archsetup.git= so cgit no longer serves it. Verify: anonymous =git clone https://git.cjennings.net/archsetup.git= fails, the new private clone works from both machines, and the curl-install URL still returns the installer. Keep the two daily drivers' remotes in sync (daily-drivers rule). + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: unstarted, and the exposure is confirmed live +Ran the task's own verification step as it stands today, which is the honest +way to check an unstarted task rather than reading its body back. Anonymous +=git ls-remote https://git.cjennings.net/archsetup.git= succeeded with no +credentials and returned =6faa31c= — this afternoon's HEAD. So the repo is +still world-cloneable and current to the commit, not a stale published +snapshot. + +=origin= on this machine is still =git@cjennings.net:archsetup.git=, the cgit +account, so nothing has moved. Everything in the plan stands unchanged. ** TODO [#B] Velox boot-failure retrospective — upgrade guard gaps :bug:zfs:maint: :PROPERTIES: :LAST_REVIEWED: 2026-07-21 @@ -1331,15 +1502,28 @@ Verify (manual, live): see Manual testing and validation. *** 2026-07-09 Thu @ 16:32:54 -0500 Audit reconcile: Phase 4 is filed on the dotfiles side, waiting on them The dotfiles project accepted the Phase 4 handoff and filed it as a =[#C]= task in their own =todo.org= (their note, 2026-07-08 16:56): the help-text audit + panel help affordance, the user-guide/README, and the ratio rollout doc. Not started there. They ping when it lands, and this task's Phase 4 child closes then. Nothing to do here meanwhile. -*** TODO Phase 4 — docs + rollout :network:blocked: -Deliverable: in-app help (=net --help= + per-command, panel help affordance); -README/user-guide (commands, indicator states, panel, config keys, make targets, -troubleshooting from the failure table, rollback); archsetup Hyprland dep install +*** 2026-08-17 Mon @ 19:57:42 -0700 Landed on the dotfiles side; the block is cleared +dotfiles shipped it as =138da7b= and closed its own task, so this one closes +with it and the =:blocked:= tag comes off. Found by checking their =todo.org= +rather than waiting for the ping — their close-out note says "archsetup pinged +so its Phase 4 task can close", so the handoff worked and only this end was +left open. + +All three acceptance criteria are met on their side: the help audit found and +fixed a stale =net repair= action list (nine of nineteen actions were named; +both the CLI help and =repair.py='s docstring now generate from the ACTIONS +registry), =net/README.md= covers every command plus the recovery targets, and +the ratio rollout is documented with both daily drivers verified current. + +They split the panel help affordance out rather than inventing it — no sibling +panel has one, so its shape is a design call. It is tracked on their side, not +here. + +Original deliverable, for the record: in-app help (=net --help= + per-command, +panel help affordance); README/user-guide; archsetup Hyprland dep install (=gtk4-layer-shell=, =python-gobject=, =speedtest-go-bin=); ratio manual dep + -stow step. -Verify: =net --help= and each subcommand complete; user-guide covers every command -+ the recovery targets. -Build handed off to the dotfiles project 2026-07-04 (=~/.dotfiles/inbox/2026-07-04-1305-from-archsetup-phase4-handoff.md=): archsetup deps confirmed installed, the remaining help/user-guide/rollout-doc work is in the net package. dotfiles pings back when it lands. +stow step. Handed off 2026-07-04 with the archsetup deps already confirmed +installed. *** TODO Phase 5 — VPN / WireGuard CLI fold (vNext) :network: Rescoped 2026-07-04 (audit): the tunnels track already shipped most of the original Phase 5. Panel tunnel bring-up/down and detection landed (dotfiles 2d9d060 probes tailscale/NM-wireguard/Proton; 21db05a brings overlays up/down from the panel's Tunnels sub-view; 31ba056 diagnose/doctor understand tunnel routes; archsetup 2e40781 wireguard config import; the net-panel-other-interfaces spec is IMPLEMENTED). What remains for Phase 5 is only the =net vpn ...= CLI subcommand — cli.py still has no vpn/tunnel parser. Fold the panel's existing tunnel operations into a CLI surface; spec separately when picked up. @@ -1601,6 +1785,29 @@ Add kernel parameter: ~rtc_cmos.use_acpi_alarm=1~ (will become systemd default) Consider: ~acpi_mask_gpe=0x1A~ for battery drain, suspend-then-hibernate config See Framework community notes on logind.conf and sleep.conf settings +*** 2026-08-17 Mon @ 19:57:42 -0700 Four of the five riders are done; WireGuard is the one left +The riders were written for "when velox returns from repair". It came back as +a full reinstall instead, and the installer carried most of them, so I checked +each on the live machine rather than reading the list back: + +- tlp radio-enable — done. =/etc/tlp.d/01-custom.conf:10= carries + =DEVICES_TO_ENABLE_ON_STARTUP="bluetooth wifi"=, written by the installer. +- touchpad auto-detection — the dotfiles half is done: =touchpad-auto + --detect= prints =pixa3854:00-093a:0274-touchpad=. Read that carefully + though — it names the device the config expects, not a device delivering + events. The touchpad is still dead on the ribbon fault, so this rider is + satisfied and the hardware still is not. +- podman socket — done, =podman.socket= is enabled. +- camera udev — done, =72-usb-passthrough-cameras.rules= is installed. +- *wolf WireGuard — not done, and it is the one that was time-critical.* No + =~/.config/wireguard/wolf.conf.gpg= and no WireGuard profile in + NetworkManager. The 08-08 decision set this up specifically so velox could + reach home from the road, on the argument that it is cheap at home and + expensive from a hotel. velox is now in the hotel. + +The suspend work itself is untouched — no kernel parameter, no drain +measurement. Only the riders moved. + ** TODO [#B] Manual testing and validation :test: :PROPERTIES: :LAST_REVIEWED: 2026-07-09 @@ -1609,6 +1816,47 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. +*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? +What we're verifying: that the IP-addressed NTP drop-in actually breaks the +bootstrap deadlock on a real cold start. This is the one test no agent can run — +it needs a full power-down, which is exactly the event that empties a failing +RTC. Everything else about the fix is verified; this is the part that rests on +construction (an address needs no DNS, NTP carries no certificate) rather than +on having been seen work. + +Do this before relying on it away from home — the failure mode strands the +machine with no network and no way to look anything up. +- Confirm the drop-in is in place and chrony is using it (block below). +- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only + loses time when the machine is actually off. +- Leave it off long enough to matter if the coin cell is the culprit (overnight + is the honest test; a few minutes may not drain anything). +- Power on. Do not touch the clock, do not run anything. Just log in and wait + about a minute. +- Run the verification block below. +#+begin_src sh :results output +echo "--- drop-in present? ---" +cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" +echo "--- is chrony reading it? ---" +grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" +echo "--- sources (the IP literal should be selected, marked ^*) ---" +chronyc -n sources +echo "--- clock ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' +echo "--- did DNS come back on its own? ---" +getent hosts gnu.org || echo "DNS STILL DEAD" +#+end_src +Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, +=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and +one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org +resolves — all without you having set the time. + +If the RTC came up wrong and the clock corrected itself anyway, the fix works +and the coin cell question is answered separately (a wrong RTC time in that +output means the cell is dying). If the clock is still wrong or DNS is still +dead, the fix did not hold: capture that whole block and promote this to a +top-level TODO. + *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. - Go to a workspace with 2-3 tiled windows in master. |