From c409dd1644ed621f78a275809f791ec376e68779 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 14:32:42 -0700 Subject: chore(tasks): close the clock-deadlock work and file what it turned up Closes the deadlock task and its two follow-ups. The laptop-branch one is cancelled rather than done. Choosing allow-downgrade fleet-wide made all three motivating defaults identical on every machine, so the branch had nothing to put on either side. Three things filed from today. The RTC reset traced to an abrupt power loss at 01:33:18, recorded against the input-cover ribbon task as the cause rather than as a separate bug. The agent-text relay reports success for a page that never arrived, blocked on the shared-asset change being applied. A failed hostname lookup takes seven seconds, measured and set aside. --- todo.org | 248 ++++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 152 insertions(+), 96 deletions(-) diff --git a/todo.org b/todo.org index fa3252d..4a34798 100644 --- a/todo.org +++ b/todo.org @@ -45,7 +45,69 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work -** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +** TODO [#B] agent-text relay reports success for a message that went nowhere :bug:blocked: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +rulesets accepted the relay-fallback change I sent on 2026-08-16 and parked it +for Craig's approval, since =agent-text= is a synced shared asset and does not +self-apply. Their review named a case neither side tested, and it is a real +defect in the shape I proposed: the relay loop breaks on rc 0, so a host that is +reachable but has no registered signal-cli account reports success for a message +that was never delivered. Paging is the one thing that must not fail silently, +because the whole point is reaching Craig when he is not watching. + +The fix is to verify delivery rather than trust the exit code, most likely by +checking =signal-cli listAccounts= on the relay host before sending, or by +treating an empty account store as a failure and continuing down the list. + +=:blocked:= on rulesets: the change is parked awaiting Craig's approval there, +and this defect should be corrected in the same apply rather than landing broken +and being patched after. Their prepared diff and review are in rulesets +=working/agent-text-relay-fallback/=. + +Two conditions from their review are theirs to carry, not mine: +=agent-text.bats= test 2 is written for the old single-host behavior and needs a +self-relay-skip case, and =protocols.org= "Reaching Craig" repeats the stale +primary-on-velox claim. + +Grading: Major severity (a page that silently does not arrive is worse than one +that fails loudly, and it is the channel used when Craig is away) x some users +sometimes (only when a relay host is up but unregistered, which is exactly +velox's current state) = P2 = [#B]. + +** TODO [#C] A failed hostname lookup takes seven seconds :bug: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +=getent hosts fake-vm= takes about 7.2 seconds to return not-found on velox. +Measured repeatedly with the cache flushed between runs. Anything that looks up +a name that does not exist pays it: an ssh typo, shell completion, a script +probing for a host. + +Not caused by the DNSSEC change. A/B measured today, cache flushed each time: +7691ms and 7232ms on =allow-downgrade= against 6804ms and 7482ms on =yes=, so +the setting makes no difference and this predates it. The likely shape is the +tailnet search domain (=search tailf3bb8c.ts.net=) being tried first, then the +two DoT upstreams, each with its own timeout, before NXDOMAIN comes back. + +Found because it blew a 20-second timeout in +=tests.net-scenarios.test_run_net_scenarios=, which shells out to ssh a +deliberately-bogus =root@fake-vm=. That suite passes on its own and the failure +did not recur, so the timeout needed this latency plus the DNS disruption from +the clock testing running alongside it. Worth knowing that the suite sits close +enough to the edge for a slow resolver to tip it. + +Grading: Minor severity (nothing behaves wrong, it just waits) x some users +sometimes (every failed lookup, which is occasional rather than constant) = P3 = +[#C]. + +** DONE [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 @@ -128,82 +190,69 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". -** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +** DONE [#C] DNSSEC strictness on the travelling laptop :velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, -because the reason I changed it turned out to be false. It does not prevent the -clock deadlock. The IP-addressed NTP source does, and that is already in place -on both machines. - -What remains is a different question the taxonomy already documents: =DNSSEC=yes= -hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel -and airport problem and therefore velox's problem more than ratio's. -=allow-downgrade= trades authenticated answers for staying online. ratio already -runs =allow-downgrade=, so the fleet disagrees with itself and with the -installer, and I do not know whether ratio's setting was a deliberate policy or -a forgotten workaround for one bad network. +Craig chose =allow-downgrade= everywhere. Applied to velox, ratio, and the +installer, and ratio's =DNSOverTLS= tightened from =opportunistic= to =yes= in +the same pass, so all three now agree: encrypted DNS always, validation +best-effort. -I have not made this call. It is a security posture change and it should be made -knowingly rather than as a side effect of a theory I disproved an hour later. +The reasoning that settled it: the deadlock is fixed by the IP-addressed NTP +source, and =allow-downgrade= was measured not to help with it at all. What +=allow-downgrade= does buy is the venue-resolver case the taxonomy documents, +where =yes= turns a resolver that mangles DNSSEC records into no answer at all. +That is a hotel and airport problem, so it is velox's problem, and the +encryption is the half worth being strict about. -** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +** CANCELLED [#C] Branch network policy on laptop vs desktop in the installer :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -Three defaults were chosen for a desktop and then applied to the machine that -travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per -connection (every hotel reconnect looks like a new device, so the portal login -starts over), and hostname-only NTP (the deadlock). Two are now fixed for every -machine, and the third is the VERIFY above. +Cancelled because the decision above emptied it. All three motivating cases now +want the same value on every machine: =DNSSEC=allow-downgrade=, a stable +per-network wifi MAC, and an IP-addressed NTP source. A branch with nothing to +put on either side is machinery built for a divergence that does not exist, and +it would be the kind of scaffolding that rots unread. -The installer already branches on battery presence in three places: -=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config -itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and -network policy simply does not use it. Wiring the same test around -=configure_networking= would let laptop and desktop defaults diverge deliberately -instead of by drift, and would stop the next instance of this from happening. +Worth keeping the observation, which is the part with a shelf life: when a +network default does need to differ by machine class, the test already exists. +=ls /sys/class/power_supply/BAT*= is what =prune_waybar_battery=, the ppd mask, +and the TLP config all key on. Reopen this then rather than building it now. -Grading: Minor severity (nothing is broken today; this prevents a recurrence) x -some users sometimes (bites when a new default suits one machine class and not -the other) = P3 = [#C]. - -** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +** DONE [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -The doctor now *names* the deadlock but hands the user two commands rather than -running anything — the verdict is =needs-user-action=. That was the honest call -at the time: correcting the clock needs the real time, and I could not exercise -a repair against the actual failure state without deliberately wedging velox's -network mid-session. - -An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a -certificate. The shape would be =chronyc add server iburst= followed by -=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to -a running chronyd over its socket, so this works in exactly the state that -blocks everything else. Two things to settle before building it: whether -=makestep= actually steps on the first sample after a runtime =add server= or -needs a poll first, and what the doctor does on a box running timesyncd rather -than chrony. - -Worth less now than it looks: a machine built by the current installer carries -the IP-addressed source and never reaches the deadlock. This is for machines -built before the fix. - -Grading: Minor severity (the doctor already names the fault correctly and hands -over a working remedy; only the automation is missing) x rare edge case (only -pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this -state has no working network and cannot look anything up, which makes the -two-command handoff harder to follow than it reads. Minor x rare = P4, and I am -leaving it at [#C] rather than [#D] because it sits one step from done. +Shipped as the =clock-ip-ntp= repair, and the verdict is =fixable= rather than +terminal. Both open questions got answered by driving a real deadlock instead of +reasoning about it: =chronyc add server= returns =200 OK= against a running +chronyd, and =makestep= needs a sample to land, so it took four calls and about +eight seconds rather than working on the first. The repair retries accordingly. + +Verified end to end on velox against a genuine deadlock (wrong clock, chronyd +running with only an unresolvable hostname source, DNS dead): the repair +corrected the clock in 6.1 seconds and DNS came back. + +The live run also caught a defect no unit test would have. The doctor reported +"Saved password for SpectrumSetup-3C was rejected" — because +=_recent_auth_failure= greps =journalctl --since -5min=, and a clock weeks off +windows onto a different incident's entries. It would have sent Craig to +re-enter a password that was never wrong. Fixed twice over: the journal half is +now skipped when the clock is untrustworthy, and the clock verdict is ordered +above the auth verdict, since everything below it reasons over timestamps that +only mean something once the clock is right. Airplane mode and hard rfkill stay +above, being physical states the clock has no bearing on. ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> @@ -240,6 +289,28 @@ Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. +*** 2026-08-19 Wed @ 12:45:00 -0700 This is what stranded the machine, and the failure got worse +The 2026-08-19 network outage traces back here. Boot -3 ended at 01:33:18 with +no shutdown sequence at all: no power-off target, no unmounting, just a +coredump backtrace and then nothing. The next boot's kernel line reads +=rtc_cmos 00:01: setting system clock to 2025-01-01T00:00:16 UTC=. The RTC was +reset to a firmware default, not drifted, and no firmware update was applied +(=fwupdmgr get-history= is empty). + +That is a different and worse symptom than the phantom short-press logged +before. Those produced =Power key pressed short= and an orderly poweroff. This +one logged no button event and no shutdown, which reads as the board losing +power outright rather than being asked to stop. Whether it is the same +connector, I can't prove from the journal, but it is the same machine with a +known loose input-cover ribbon and a battery that was not flat (100% now). + +Consequence, which is the part that makes this worth the [#A]: the RTC reset +put the clock at 2025-01-01, systemd advanced it to its own build epoch of +2026-07-23, DNSSEC rejected every signature as out-of-window, and the machine +lost DNS entirely. Craig had to diagnose it from his phone. The NTP drop-in +landed today makes a recurrence self-healing, but it treats the symptom. This +task is the cause, and it is still open. + *** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree The reseat did not happen before the flight, and velox is travelling. The deadline blew past on 08-14. @@ -1896,47 +1967,32 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. -*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: the coin cell, not the fix. The fix itself is already -demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, -and chronyd reached the IP-addressed source and stepped it straight back. What a -cold boot adds is the hardware question: whether the RTC actually loses time -across a full power-down, which is the thing that started this. - -Read the outcome carefully, because only one branch is informative. An RTC time -that comes up wrong and then self-corrects tells you the cell is dying and the -fix is holding. An RTC that comes up correct tells you nothing about the -deadlock at all, only that the cell survived this particular night. -- Confirm the drop-in is in place and chrony is using it (block below). -- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only - loses time when the machine is actually off. -- Leave it off long enough to matter if the coin cell is the culprit (overnight - is the honest test; a few minutes may not drain anything). -- Power on. Do not touch the clock, do not run anything. Just log in and wait - about a minute. -- Run the verification block below. +*** Clock/DNS deadlock: does the next abrupt power loss strand velox again? +What we're verifying: that the machine survives an RTC reset unattended. Not the +coin cell, which is new with the 2026-08-13 mainboard and is ruled out. The RTC +did not drift on 2026-08-19, it was reset to exactly 2025-01-01T00:00:16 by an +abrupt power loss at 01:33:18 that left no shutdown sequence in the journal. + +This one can't be scheduled. Run the block the next time velox comes up after an +unexpected power loss, before touching the clock. #+begin_src sh :results output -echo "--- drop-in present? ---" -cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" -echo "--- is chrony reading it? ---" -grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" -echo "--- sources (the IP literal should be selected, marked ^*) ---" +echo "--- what did the RTC read at this boot? ---" +journalctl -b 0 | grep -m1 'rtc_cmos.*setting system clock' +echo "--- did systemd have to advance the clock to its build epoch? ---" +journalctl --list-boots | tail -3 +echo "--- sources: is an IP-addressed one selected? ---" chronyc -n sources -echo "--- clock ---" -timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' -echo "--- did DNS come back on its own? ---" -getent hosts gnu.org || echo "DNS STILL DEAD" +echo "--- clock + DNS ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized' +getent hosts gnu.org || echo "DNS DEAD" #+end_src -Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, -=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and -one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org -resolves — all without you having set the time. - -If the RTC came up wrong and the clock corrected itself anyway, the fix works -and the coin cell question is answered separately (a wrong RTC time in that -output means the cell is dying). If the clock is still wrong or DNS is still -dead, the fix did not hold: capture that whole block and promote this to a -top-level TODO. +Expected: even if the RTC came up at 2025-01-01 and systemd advanced the clock +to 2026-07-23, chrony reached 162.159.200.1 without DNS, stepped the clock to +now, and names resolve. You did nothing. + +If instead the clock is still wrong or DNS is dead, the fix did not hold in the +field despite holding under a simulated skew. Capture that whole block and +promote this to a top-level TODO. *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. -- cgit v1.2.3