diff options
| -rw-r--r-- | todo.org | 248 |
1 files changed, 152 insertions, 96 deletions
@@ -45,7 +45,69 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work -** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +** TODO [#B] agent-text relay reports success for a message that went nowhere :bug:blocked: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +rulesets accepted the relay-fallback change I sent on 2026-08-16 and parked it +for Craig's approval, since =agent-text= is a synced shared asset and does not +self-apply. Their review named a case neither side tested, and it is a real +defect in the shape I proposed: the relay loop breaks on rc 0, so a host that is +reachable but has no registered signal-cli account reports success for a message +that was never delivered. Paging is the one thing that must not fail silently, +because the whole point is reaching Craig when he is not watching. + +The fix is to verify delivery rather than trust the exit code, most likely by +checking =signal-cli listAccounts= on the relay host before sending, or by +treating an empty account store as a failure and continuing down the list. + +=:blocked:= on rulesets: the change is parked awaiting Craig's approval there, +and this defect should be corrected in the same apply rather than landing broken +and being patched after. Their prepared diff and review are in rulesets +=working/agent-text-relay-fallback/=. + +Two conditions from their review are theirs to carry, not mine: +=agent-text.bats= test 2 is written for the old single-host behavior and needs a +self-relay-skip case, and =protocols.org= "Reaching Craig" repeats the stale +primary-on-velox claim. + +Grading: Major severity (a page that silently does not arrive is worse than one +that fails loudly, and it is the channel used when Craig is away) x some users +sometimes (only when a relay host is up but unregistered, which is exactly +velox's current state) = P2 = [#B]. + +** TODO [#C] A failed hostname lookup takes seven seconds :bug: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +=getent hosts fake-vm= takes about 7.2 seconds to return not-found on velox. +Measured repeatedly with the cache flushed between runs. Anything that looks up +a name that does not exist pays it: an ssh typo, shell completion, a script +probing for a host. + +Not caused by the DNSSEC change. A/B measured today, cache flushed each time: +7691ms and 7232ms on =allow-downgrade= against 6804ms and 7482ms on =yes=, so +the setting makes no difference and this predates it. The likely shape is the +tailnet search domain (=search tailf3bb8c.ts.net=) being tried first, then the +two DoT upstreams, each with its own timeout, before NXDOMAIN comes back. + +Found because it blew a 20-second timeout in +=tests.net-scenarios.test_run_net_scenarios=, which shells out to ssh a +deliberately-bogus =root@fake-vm=. That suite passes on its own and the failure +did not recur, so the timeout needed this latency plus the DNS disruption from +the clock testing running alongside it. Worth knowing that the suite sits close +enough to the edge for a slow resolver to tip it. + +Grading: Minor severity (nothing behaves wrong, it just waits) x some users +sometimes (every failed lookup, which is occasional rather than constant) = P3 = +[#C]. + +** DONE [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 @@ -128,82 +190,69 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". -** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +** DONE [#C] DNSSEC strictness on the travelling laptop :velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, -because the reason I changed it turned out to be false. It does not prevent the -clock deadlock. The IP-addressed NTP source does, and that is already in place -on both machines. - -What remains is a different question the taxonomy already documents: =DNSSEC=yes= -hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel -and airport problem and therefore velox's problem more than ratio's. -=allow-downgrade= trades authenticated answers for staying online. ratio already -runs =allow-downgrade=, so the fleet disagrees with itself and with the -installer, and I do not know whether ratio's setting was a deliberate policy or -a forgotten workaround for one bad network. +Craig chose =allow-downgrade= everywhere. Applied to velox, ratio, and the +installer, and ratio's =DNSOverTLS= tightened from =opportunistic= to =yes= in +the same pass, so all three now agree: encrypted DNS always, validation +best-effort. -I have not made this call. It is a security posture change and it should be made -knowingly rather than as a side effect of a theory I disproved an hour later. +The reasoning that settled it: the deadlock is fixed by the IP-addressed NTP +source, and =allow-downgrade= was measured not to help with it at all. What +=allow-downgrade= does buy is the venue-resolver case the taxonomy documents, +where =yes= turns a resolver that mangles DNSSEC records into no answer at all. +That is a hotel and airport problem, so it is velox's problem, and the +encryption is the half worth being strict about. -** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +** CANCELLED [#C] Branch network policy on laptop vs desktop in the installer :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -Three defaults were chosen for a desktop and then applied to the machine that -travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per -connection (every hotel reconnect looks like a new device, so the portal login -starts over), and hostname-only NTP (the deadlock). Two are now fixed for every -machine, and the third is the VERIFY above. +Cancelled because the decision above emptied it. All three motivating cases now +want the same value on every machine: =DNSSEC=allow-downgrade=, a stable +per-network wifi MAC, and an IP-addressed NTP source. A branch with nothing to +put on either side is machinery built for a divergence that does not exist, and +it would be the kind of scaffolding that rots unread. -The installer already branches on battery presence in three places: -=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config -itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and -network policy simply does not use it. Wiring the same test around -=configure_networking= would let laptop and desktop defaults diverge deliberately -instead of by drift, and would stop the next instance of this from happening. +Worth keeping the observation, which is the part with a shelf life: when a +network default does need to differ by machine class, the test already exists. +=ls /sys/class/power_supply/BAT*= is what =prune_waybar_battery=, the ppd mask, +and the TLP config all key on. Reopen this then rather than building it now. -Grading: Minor severity (nothing is broken today; this prevents a recurrence) x -some users sometimes (bites when a new default suits one machine class and not -the other) = P3 = [#C]. - -** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +** DONE [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -The doctor now *names* the deadlock but hands the user two commands rather than -running anything — the verdict is =needs-user-action=. That was the honest call -at the time: correcting the clock needs the real time, and I could not exercise -a repair against the actual failure state without deliberately wedging velox's -network mid-session. - -An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a -certificate. The shape would be =chronyc add server <ip> iburst= followed by -=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to -a running chronyd over its socket, so this works in exactly the state that -blocks everything else. Two things to settle before building it: whether -=makestep= actually steps on the first sample after a runtime =add server= or -needs a poll first, and what the doctor does on a box running timesyncd rather -than chrony. - -Worth less now than it looks: a machine built by the current installer carries -the IP-addressed source and never reaches the deadlock. This is for machines -built before the fix. - -Grading: Minor severity (the doctor already names the fault correctly and hands -over a working remedy; only the automation is missing) x rare edge case (only -pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this -state has no working network and cannot look anything up, which makes the -two-command handoff harder to follow than it reads. Minor x rare = P4, and I am -leaving it at [#C] rather than [#D] because it sits one step from done. +Shipped as the =clock-ip-ntp= repair, and the verdict is =fixable= rather than +terminal. Both open questions got answered by driving a real deadlock instead of +reasoning about it: =chronyc add server= returns =200 OK= against a running +chronyd, and =makestep= needs a sample to land, so it took four calls and about +eight seconds rather than working on the first. The repair retries accordingly. + +Verified end to end on velox against a genuine deadlock (wrong clock, chronyd +running with only an unresolvable hostname source, DNS dead): the repair +corrected the clock in 6.1 seconds and DNS came back. + +The live run also caught a defect no unit test would have. The doctor reported +"Saved password for SpectrumSetup-3C was rejected" — because +=_recent_auth_failure= greps =journalctl --since -5min=, and a clock weeks off +windows onto a different incident's entries. It would have sent Craig to +re-enter a password that was never wrong. Fixed twice over: the journal half is +now skipped when the clock is untrustworthy, and the clock verdict is ordered +above the auth verdict, since everything below it reasons over timestamps that +only mean something once the clock is right. Airplane mode and hard rfkill stay +above, being physical states the clock has no bearing on. ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> @@ -240,6 +289,28 @@ Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. +*** 2026-08-19 Wed @ 12:45:00 -0700 This is what stranded the machine, and the failure got worse +The 2026-08-19 network outage traces back here. Boot -3 ended at 01:33:18 with +no shutdown sequence at all: no power-off target, no unmounting, just a +coredump backtrace and then nothing. The next boot's kernel line reads +=rtc_cmos 00:01: setting system clock to 2025-01-01T00:00:16 UTC=. The RTC was +reset to a firmware default, not drifted, and no firmware update was applied +(=fwupdmgr get-history= is empty). + +That is a different and worse symptom than the phantom short-press logged +before. Those produced =Power key pressed short= and an orderly poweroff. This +one logged no button event and no shutdown, which reads as the board losing +power outright rather than being asked to stop. Whether it is the same +connector, I can't prove from the journal, but it is the same machine with a +known loose input-cover ribbon and a battery that was not flat (100% now). + +Consequence, which is the part that makes this worth the [#A]: the RTC reset +put the clock at 2025-01-01, systemd advanced it to its own build epoch of +2026-07-23, DNSSEC rejected every signature as out-of-window, and the machine +lost DNS entirely. Craig had to diagnose it from his phone. The NTP drop-in +landed today makes a recurrence self-healing, but it treats the symptom. This +task is the cause, and it is still open. + *** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree The reseat did not happen before the flight, and velox is travelling. The deadline blew past on 08-14. @@ -1896,47 +1967,32 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. -*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: the coin cell, not the fix. The fix itself is already -demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, -and chronyd reached the IP-addressed source and stepped it straight back. What a -cold boot adds is the hardware question: whether the RTC actually loses time -across a full power-down, which is the thing that started this. - -Read the outcome carefully, because only one branch is informative. An RTC time -that comes up wrong and then self-corrects tells you the cell is dying and the -fix is holding. An RTC that comes up correct tells you nothing about the -deadlock at all, only that the cell survived this particular night. -- Confirm the drop-in is in place and chrony is using it (block below). -- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only - loses time when the machine is actually off. -- Leave it off long enough to matter if the coin cell is the culprit (overnight - is the honest test; a few minutes may not drain anything). -- Power on. Do not touch the clock, do not run anything. Just log in and wait - about a minute. -- Run the verification block below. +*** Clock/DNS deadlock: does the next abrupt power loss strand velox again? +What we're verifying: that the machine survives an RTC reset unattended. Not the +coin cell, which is new with the 2026-08-13 mainboard and is ruled out. The RTC +did not drift on 2026-08-19, it was reset to exactly 2025-01-01T00:00:16 by an +abrupt power loss at 01:33:18 that left no shutdown sequence in the journal. + +This one can't be scheduled. Run the block the next time velox comes up after an +unexpected power loss, before touching the clock. #+begin_src sh :results output -echo "--- drop-in present? ---" -cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" -echo "--- is chrony reading it? ---" -grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" -echo "--- sources (the IP literal should be selected, marked ^*) ---" +echo "--- what did the RTC read at this boot? ---" +journalctl -b 0 | grep -m1 'rtc_cmos.*setting system clock' +echo "--- did systemd have to advance the clock to its build epoch? ---" +journalctl --list-boots | tail -3 +echo "--- sources: is an IP-addressed one selected? ---" chronyc -n sources -echo "--- clock ---" -timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' -echo "--- did DNS come back on its own? ---" -getent hosts gnu.org || echo "DNS STILL DEAD" +echo "--- clock + DNS ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized' +getent hosts gnu.org || echo "DNS DEAD" #+end_src -Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, -=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and -one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org -resolves — all without you having set the time. - -If the RTC came up wrong and the clock corrected itself anyway, the fix works -and the coin cell question is answered separately (a wrong RTC time in that -output means the cell is dying). If the clock is still wrong or DNS is still -dead, the fix did not hold: capture that whole block and promote this to a -top-level TODO. +Expected: even if the RTC came up at 2025-01-01 and systemd advanced the clock +to 2026-07-23, chrony reached 162.159.200.1 without DNS, stepped the clock to +now, and names resolve. You did nothing. + +If instead the clock is still wrong or DNS is dead, the fix did not hold in the +field despite holding under a simulated skew. Capture that whole block and +promote this to a top-level TODO. *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. |