From 1d486a7b43e3fdd2e194b579285d979dc9beb788 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Sun, 16 Aug 2026 05:38:05 -0500 Subject: chore(todo): file the obsbot guard poll bug and archive resolved work obsbot-wb-guard runs from the shared stow tier, so it starts on machines with no OBSBOT and polls every two seconds forever doing nothing. The task carries the ConditionPathExists fix and the grading, plus the trap that disabling a linked unit by hand deletes its stow symlink. Three resolved subtrees moved to the archive. --- todo.org | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index 8da5c7b..e549f80 100644 --- a/todo.org +++ b/todo.org @@ -542,6 +542,21 @@ Found by sentry (2026-07-25), verified by exercising. =hyprland/.local/bin/wayba Repro: a conf with =America/Chicago|Home=, =Not/AZone|Bad=, =Europe/London|London= renders =tooltip: ""= (Home and London gone too). Grade: minor severity (one module's tooltip blanks, no data loss) x rare edge case (a malformed conf row) = P4 = [#D]. Fix: wrap the per-row =ZoneInfo=/=datetime= in a try/except and =continue=, so a typo drops only that row and the valid zones still render. Solo + quick: the script already has an env-override test harness (=WAYBAR_TIME_EPOCH=, =WAYBAR_WORLDCLOCK_CONF=), so a red-first test is cheap. +** TODO [#C] obsbot-wb-guard polls forever on machines with no OBSBOT :bug:dotfiles:quick:solo: +:PROPERTIES: +:LAST_REVIEWED: 2026-08-16 +:END: +=obsbot-wb-guard.service= is =WantedBy=graphical-session.target= and lives in the shared =common/= stow tier, so it starts on every machine. Its main path is =while :; do check_once; sleep 2; done=, and =check_once= returns early when the camera node is absent. On a machine with no OBSBOT attached that is a process waking every two seconds forever to do nothing, which on a laptop is battery spend for zero benefit. No restart loop, though: the loop never exits, so =Restart=on-failure= never fires. + +Found 2026-08-16 on velox, after enabling it to match ratio and then having to disable it again by hand. A per-machine disable is the wrong shape, because it drifts velox from ratio permanently and a re-stow or a future audit will just put it back. + +Fix: give the unit =ConditionPathExists= on the camera node (=/dev/v4l/by-id/usb-Remo_Tech_Co.__Ltd._OBSBOT_PW106-video-index0=, the same default the script uses) so systemd skips it on any machine without the camera and starts it normally on ratio. Then re-enable it on velox, where it will simply be skipped. Note the limit: a camera plugged in later will not start it until the next login, which is the right trade against a permanent poll. + +Careful when disabling by hand in the meantime: =systemctl --user disable= on a *linked* unit deletes the unit symlink, and that symlink is stow-managed, so a bare disable silently removes a file from the dotfiles stow tree. Restore the link afterward or re-stow. + +Grade: minor severity (wasted wakeups and battery, no data loss, no failure) x every boot on any machine without the camera = P3 = [#C]. + +Solo: buildable here (archsetup owns dotfiles end-to-end), verifiable by the agent (assert the unit is skipped on velox and still active on ratio), and no design call left open. ** TODO [#C] Auto-dim status forgotten on layout change :bug:dotfiles: :PROPERTIES: :LAST_REVIEWED: 2026-07-25 @@ -2282,26 +2297,6 @@ The maintenance console's coredump metric flagged telega-server on ratio (8 core * Archsetup Resolved -** DONE [#C] Org-capture float popup grows too large :bug:hyprland:quick:solo: -CLOSED: [2026-07-14 Tue] -:PROPERTIES: -:LAST_REVIEWED: 2026-07-13 -:END: -Craig answered the pre-flight (2026-07-14): cap at 120 wide, height proportional. Applied as 120 Emacs columns (11 px/col measured from the live daemon) = 1320 px wide, height 653 px from the old rule's aspect. Ratio's size rule shrank from the 1892x936 scratchpad match and both hosts gained a max_size growth cap (the field is max_size — bare "maxsize" is invalid and hyprctl reload won't say so; check hyprctl configerrors). Verified live: config clean, a probe window floats at exactly 1320x653. Dotfiles 9c4dc2f. -** DONE [#C] Panel smoke: faceplate state-word assertion fails on the live compositor :bug:dotfiles:test: -CLOSED: [2026-07-14 Tue] -Diagnosed and fixed within the session: not a race — test drift. Dotfiles b581d5d (2026-07-05) made the faceplate word the static subsystem identity (NETWORKING / BLUETOOTH / AUDIO) and updated the audio smoke, but the net and bt smokes kept asserting the retired live-state words and had failed on every run since. Both now assert the identity word like audio's (dotfiles 32cd99f); both smokes run RESULT: OK end to end, which also green-gates the doctor-streaming change. -** DONE [#C] Realtime lamp output for the net + bt doctors :feature:solo: -CLOSED: [2026-07-14 Tue] -:PROPERTIES: -:LAST_REVIEWED: 2026-07-09 -:END: -Shipped in dotfiles 0318a91. Both doctors stream: diagnose() emits each step as it completes (bt streams the first diagnosis only — the fix loop's re-diagnoses would replay the chain), and a repair's row goes up amber at attempt start and settles green/red with narration + evidence at completion, so the lamp blinks for the repair's real duration. The 3.5-entry height cap turned out to already be in both wells (it landed with the doctor expansions), so only the streaming half needed building. 5 new tests across net + bt; both suites green; AT-SPI smokes at parity with HEAD (one pre-existing state-word failure, filed separately). -Retrofit the net doctor (=~/.dotfiles/net/src/net/doctor.py=) and bluetooth doctor (=~/.dotfiles/bluetooth/src/bt/doctor.py=) to stream results as a live output wall — one lamp per escalation step, amber while running, green on success, red on failure — instead of a final summary. Matches the maintenance-console doctor design (see [[file:docs/design/maintenance-console-design-ideas.org][maintenance-console-design-ideas.org]], "Doctor = live output wall"). Goal: every doctor in the system reads the same way. Both doctors already step through an escalation chain re-probing after each, so the steps are natural lamp boundaries. - -Scope note (Craig, 2026-07-07): realtime lamp *behavior* only. The maintenance console's wider results-wall layout (date+time stamp column, COPY, persistent history) does NOT backport — the net/bt panels are ~400px wide and lack the horizontal real estate. Their existing output wells keep their compact layout; this task just makes them stream live. - -Addendum (Craig, 2026-07-07): DO backport the 3.5-entry height convention — every panel's output well caps at 3.5 visible entries, the half-visible entry being the scroll cue, with the dark slate-on-black scrollbar. Layout stays compact per above; only the height cap + scroll affordance carries over. ** DONE [#B] Absorb the clock-panel project into the dotfiles :feature:waybar:dotfiles: CLOSED: [2026-07-18 Sat] Absorbed into =~/.dotfiles= (commit 3fab11d): package =clock/src/clock/= (renamed from clock_panel), the six PNG watchface layers packaged inside the module at =clock/src/clock/assets/=, a stowed =clock-panel= shell shim (LD_PRELOADs gtk4-layer-shell), waybar left-click now =clock-panel toggle= with the absolute path dropped, tests converted pytest→unittest into =tests/clock/= plus an asset-load guard. Kept the layer-shell overlay and the socket toggle. The standalone repo is archived (ARCHIVED.md), kept for its design history. Verified live: the bar click renders the polished watchface. -- cgit v1.2.3 From 5c7a5888ed2a68476500b00d4fe3df5d3bde2f0f Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Mon, 17 Aug 2026 11:30:06 -0700 Subject: chore(todo): file what today's work turned up, and fix two checks that lied Four tasks are new. The installer clones my own archsetup and dotfiles repos with --depth 1, so both came back from the rebuild with seven commits of history instead of hundreds. Power profiles crash on a loop against the daemon this repo now masks. The user manager on that machine is spinning and cannot resolve unit files. The new post-rebuild-check wants its probes routed through one guarded helper, because four separate "the probe failed and the check said ok" bugs is a shape rather than four mistakes. I rechecked two recurring items against their own unblock conditions rather than re-stamping them. The zfs DKMS build is still waiting on an upstream release, and the lyricsgenius integrity workaround still fails for the same structural reason. The credential-history entry has been asking a question git could not answer. The filenames recorded there are not the paths those files live at, and a path git has never seen returns no commits rather than an error, so every check since June came back falsely clean. I corrected them to the real paths, which turn out to be six files rather than five. --- todo.org | 277 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 241 insertions(+), 36 deletions(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index e549f80..1ec7cf1 100644 --- a/todo.org +++ b/todo.org @@ -120,6 +120,192 @@ hand-copied key sprawl; a firm RAM carve-out so builds don't fight the ZFS ARC; headless only — desktop-coupled sessions stay on ratio/velox. Build deliberately AFTER the vacation, not before Sunday. Companion idea (cheaper, complementary): put ratio on the UPS. +** TODO [#B] post-rebuild-check: route every probe through one guarded helper :refactor:solo: +:PROPERTIES: +:CREATED: [2026-08-17 Mon] +:LAST_REVIEWED: 2026-08-17 +:END: +The script works and is well tested, but its shape keeps producing the same +bug. Across three review rounds the reviewer found FOUR separate instances of +"the probe failed and the check reported ok", each in a different place: +=systemctl= in check 1, the enablement read in check 2, =find= in check 3, and +=grep= in check 4. A fifth was latent in an unguarded staged write. Every one +was individually fixed, and I only stopped finding more because someone kept +looking. + +That is a design problem rather than four bugs. The script has five +hand-written probes, and each one has to remember to branch on its own exit +status. Nothing enforces it, nothing fails a review that forgets it, and the +failure is invisible because the wrong behaviour is a clean "ok". + +Shape: one helper every probe must go through, which cannot return a value +without an explicit success, so that "I could not read this" is +unrepresentable as "nothing to report". Roughly: + +: probe "" # sets a value on success, records a finding otherwise + +Then each check consumes the helper's result rather than a raw command +substitution, and a new check written later inherits the discipline instead of +having to re-derive it. Worth pairing with a test that asserts no check can +report ok when its probe exits non-zero, generically, so the fifth instance is +caught by the suite rather than by a reviewer. + +Not urgent: the current version is correct as far as anyone has found, ships +with 58 tests, and proved itself on a genuinely wedged machine. This is +prevention. + +Grading: Minor severity (no known live defect, the risk is future) x +most-users-frequently (every future edit to this script) = P3 = [#C]... except +the failure mode is silent and the script's whole job is catching silent +failures, so a regression here is uniquely undetectable. P2 = [#B]. + +:solo: — the surface is one script and its suite, the refactor is +behaviour-preserving, and the existing 58 tests plus a mutation battery are +the objective check that it stayed so. +** TODO [#A] powerprofilesctl crashes on a loop since ppd was masked :bug:velox:dotfiles: +:PROPERTIES: +:CREATED: [2026-08-17 Mon] +:LAST_REVIEWED: 2026-08-17 +:END: +Something polls power state every 10-30 seconds, and each poll runs +=powerprofilesctl get=, which SIGABRTs. 47 coredumps on velox on 2026-08-17 +alone, the earliest at 08:34, four in one minute while I was watching. + +Cause is the 2026-08-16 fix that masked =power-profiles-daemon= so TLP +survives on laptops. That fix is right and stays. What it did not account for +is the settings module's power backing +(=~/.dotfiles/settings/src/settings/power.py=), which shells out to +=powerprofilesctl=. Against a masked unit the D-Bus activation fails with +=NameHasNoOwner ... unit is masked=, and the caller aborts rather than +degrading. + +Run by hand the same command exits 0 and prints the error, so the abort is +context-dependent and the caller needs finding before the fix is written. +Ratio does not mask ppd, which is why this is velox-only and why it appeared +the day after the masking. + +Costs: journal spam, coredump disk churn, and repeated failed D-Bus +activations on a travelling laptop's battery. It is also the leading suspect +for the wedged user manager filed below. + +Fix shape: =power.py= should treat a masked or unavailable ppd as a +first-class "no profile control here" state rather than an error path, and +the poller should stop retrying a unit it has been told is masked. The +machine-level half is already correct. + +Grading: Major severity (a crash loop burning battery and filling the +journal, silently) x every user every time on any laptop with the TLP fix +applied = P1 = [#A]. +** TODO [#A] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: +:PROPERTIES: +:CREATED: [2026-08-17 Mon] +:LAST_REVIEWED: 2026-08-17 +:END: +Live on velox 2026-08-17 from about 10:29. =systemd --user= (pid 2235) sits +in state R at 96% CPU, measured over a 3-second sample rather than taken from +the lifetime average. It stopped logging at 10:29, so its timers appear to +have stopped firing too. + +The split is the diagnostic: =systemctl --user list-units= still returns +instantly, while =is-enabled=, =cat=, =show=, and =list-unit-files= all hang +indefinitely. So the manager answers from its in-memory unit list and wedges +on anything that has to resolve unit files. It is spinning in userspace, not +blocked on I/O (=/proc/2235/wchan= is 0, no syscall pending). + +Remedies tried, neither worked: =systemctl --user daemon-reexec= hangs like +every other unit-file call, and the signal form (=kill -59=, SIGRTMIN+25) +was accepted but changed nothing. The next step is a logout/login or reboot, +which is Craig's call because it closes his running session. I deliberately +did not kill the manager: that would tear down the graphical session and +everything under it. + +Suspected cause is the powerprofilesctl crash loop filed above, whose +repeated activation attempts against a masked unit are the only new load on +this machine. I cannot prove it, and I have to name the other candidate +honestly: my own =post-rebuild-check= runs called =systemctl --user +is-enabled= roughly thirty times per run over several runs, and the wedge +appeared during that window. The crash loop predates those runs by an hour +and a half, which is why it is the leading suspect rather than the certain +one. + +What it costs: unit-file operations are unavailable, user timers appear +stopped, and a core is pinned on a laptop running on battery. + +Grading: Major severity (a pinned core and stopped user timers, invisible +unless you look) x rare edge case (one machine, specific conditions) = P2 = +[#B]... except that this is a live, ongoing drain on a travelling machine +rather than a latent defect, so it takes [#A] until the machine is back to +normal. Re-grade to [#B] once resolved and the question is only prevention. +** TODO [#A] The installer shallow-clones the two repos I develop in :bug:velox:solo: +:PROPERTIES: +:CREATED: [2026-08-17 Mon] +:LAST_REVIEWED: 2026-08-17 +:END: +=archsetup:1432= clones the user's archsetup repo and =archsetup:1445= clones +dotfiles, both with =--depth 1=. Those are not build directories. They are the +two repos I actively develop in, and on velox they came back from the +2026-08-13 rebuild with 7 commits of history each instead of 851. + +Found 2026-08-17, and found the worst way: I ran the credential-file history +check that the GitHub-release task asks for, and it reported all five files +absent from history with a clean exit. The real answer is that this clone +cannot see the history those files live in. A shallow clone does not error on +=git log -- =, it answers "no commits" — so a security question came back +falsely clean, and nothing about the output said otherwise. + +Everything else it breaks is quieter: =git log=, =blame=, =bisect=, and any +archaeology past the boundary. The tree looks completely normal, which is why +this survived four days on the machine. + +The right shape is already in the codebase. =scripts/post-install.sh:42-51= +takes depth as a per-repo argument and defaults to a full clone, so wallpaper +gets =--depth 1= and org does not. The AUR build clones (=archsetup:855=, +=:1673=, =:1677=) are correctly shallow and stay that way. Only the two +user-repo sites change. + +Repair on a machine already built: =git fetch --unshallow= in each repo. + +Grading: Major severity (two working repos silently missing their history on +the machine I develop on, and it returns confidently wrong answers to history +questions rather than failing) x every user every time (every fresh install, +both daily drivers) = P1 = [#A]. + +:solo: — the change is two lines plus tests in the existing +=tests/installer-steps/= shape, and it is verifiable locally by asserting the +clone command carries no =--depth= for these two repos. +** TODO [#B] post-rebuild-check needs a reference-host mode :feature:velox:solo: +:PROPERTIES: +:CREATED: [2026-08-17 Mon] +:LAST_REVIEWED: 2026-08-17 +:END: +=scripts/post-rebuild-check= ships and works, but its first live run on velox +2026-08-17 showed the output is mostly steady state rather than drift. Of the +8 findings that survived three rounds of false-positive removal, comparing +against ratio says exactly ONE is real: =obsbot-wb-guard= is enabled on ratio +and merely linked on velox, which is the deliberate deferral recorded +2026-08-16. The other three unit findings (=emacs=, =geoclue-agent=, +=obs-record-watchdog.timer=) are linked on ratio too, and the three =.claude= +absences are absent on ratio too. + +So the signal-to-noise is about 1:7, and the thing that separates them is a +comparison against the other daily driver — the same discipline that kept the +2026-08-16 session honest when check 4 read as nine projects missing +=CLAUDE.md= and ratio turned out to be missing the identical files. + +Shape: =--reference-host = runs the same five checks on the far machine +over tailscale (ssh, read-only) and reports only the *differences*. Findings +present on both machines are steady state and get summarized as a count rather +than listed. Falls back to the current standalone behavior when the reference +host is unreachable, and says so. + +Grading: Minor severity (the tool works and its findings are accurate; they +are just buried) x every use = P3 = [#C]... except that a check nobody reads +is a check that isn't run, which is the failure mode the whole task existed to +close. Most-users-frequently x Major = P2 = [#B]. + +:solo: — the checks exist, the ssh path is proven (the 2026-08-17 session ran +exactly this comparison by hand), and correctness is verifiable locally by +diffing the two reports. ** TODO [#C] screen-lock test suite red on ratio :bug:test:dotfiles: :PROPERTIES: :CREATED: [2026-08-13 Thu] @@ -291,6 +477,7 @@ dial placement 2026-08-13): Note: ratio has no swap partition, so hibernate stays velox-only until ratio gets one; the dial entry should degrade gracefully where there's no resume target. +** TODO [#A] Move secrets out of public dotfiles → private repo + combined personal ISO :feature:security:dotfiles: :PROPERTIES: :CREATED: [2026-08-11 Tue] :LAST_REVIEWED: 2026-08-11 @@ -310,6 +497,40 @@ archangel+archsetup ISO that's already ~80% built. Two ISO modes: generic don't start the migration until the credentials are rotated. Not started. Not :solo: — repo standup and history rewrite are Craig's calls; promote to a real spec (spec-create) when work resumes. +** TODO [#B] Settings toggles reset silently at session start :bug:dotfiles: +:PROPERTIES: +:CREATED: [2026-07-28 Tue] +:LAST_REVIEWED: 2026-07-28 +:END: +Craig, from the roam inbox 2026-07-28: "launching into wayland doesn't honor previous caffeine settings ...or I expect any other settings in the desktop settings module." Captured right after the 08:59 reboot. + +Confirmed, and it generalizes past caffeine. The settings module splits cleanly into two halves, and only one of them persists. + +Persisted, in =~/.config/desktop-settings/state.json= (=store.py= =DEFAULTS=): program slots, idle-tripper stages, wallpaper. These come back correctly. + +Not persisted — every one is derived live from a process or a compositor runtime option, so a session restart resets it to whatever =hyprland.conf= establishes: +- Caffeine — =caffeine_state()= is =pgrep -x hypridle= inverted, and =hyprland.conf:73= runs =exec-once = pkill -x hypridle; hypridle=. So every launch unconditionally starts hypridle, which means caffeine is *always* OFF after login. There is no code path that could restore it ON. +- Auto-dim — =dim_state()= reads =hyprctl getoption decoration:dim_inactive=, a compositor runtime value that resets to the config default on restart. +- Night light — =state()= is =pgrep -x gammastep=; the process dies with the session. +- DND — =dunstctl=; dunst restarts fresh from =exec-once=. +- Power profile / brightness — owned by powerprofilesctl and systemd-backlight, outside this module's scope. + +Verified live 10 minutes after the reboot: hypridle running (caffeine OFF), dim =false=, gammastep not running, dnd =false=, power =balanced=. Every toggle sat at its factory position. + +The failure is silent, which is what makes it bite: nothing tells you the value you set was discarded. That is the mechanism behind the 2026-07-27 lockout, where Craig believed caffeine was on and the screen locked anyway. + +Grading: Major severity (the panel's core promise is holding these values, and the reset is silent and total across all four toggles) x most users frequently (every session start resets them, though it only harms when a deliberate non-default was set) = P2 = [#B]. + +Not :solo: — the fix needs Craig's call on *which* toggles should persist and whether persistence is per-toggle opt-in. Restoring night light at 3pm or caffeine on a laptop are both plausibly wrong, so this is a preference question, not a derivable one. The mechanism itself (extend =store.py= with a =toggles= block, restore on session start) is mechanical once that's settled. + +Related: =[#B] Caffeine state is unreadable on both surfaces= covers display accuracy — whether the surfaces report the truth. This covers whether the value survives at all. Distinct bugs, same subsystem. + +*** Side finding — gammastep loses a startup race and nothing relaunches it +=hyprland.conf:75= runs =exec-once = gammastep=, but no gammastep process is alive. Today's three launch logs tell the story: =gammastep-2026-07-28-090034.log= carries "Wayland connection experienced a fatal error: -1 / Temperature adjustment failed", and the other two are empty. + +Launched by hand afterward it runs fine and survives, so gammastep is not broken — it loses a race against compositor readiness at session start. Nothing relaunches it, so night light is simply off for the whole session, silently. (An earlier read of this said night light "has likely never worked from the config". That was wrong: the failure is a startup race, not a permanent break.) + +Worth its own task — the fix is a readiness wait or a retry around that exec-once, not a persistence change. Filed here for now because it surfaced during this investigation. ** VERIFY [#A] Pre-vacation fix list — morning review SCHEDULED: <2026-08-08 Sat> :PROPERTIES: @@ -1961,7 +2182,7 @@ NOTE (2026-07-04 audit): the "four-tab panel" framing predates the instrument-co ** DOING [#B] Prepare for GitHub open-source release :PROPERTIES: -:LAST_REVIEWED: 2026-07-09 +:LAST_REVIEWED: 2026-08-17 :END: Remove personal info, credentials, and code quality issues before publishing. *** 2026-07-21 Tue @ 08:00:00 -0500 Audit reconcile: the four assets/ "& Claude" author lines are fixed @@ -2018,6 +2239,19 @@ Recommend: fresh repo for GitHub (keep cjennings.net remote with full history). History is now 589 commits (the 2026-05-11 note's "275" is stale). Only the calendar-feed file has been filter-repo'd so far (2026-05-20). The five credential files remain in history at their pre-=b10cba5= paths: =.tidal-dl.token.json= (5 commits), =calibre/smtp.py.json= (6), =transmission/settings.json= (5), =.msmtprc= (8), =.mbsyncrc= (9). None are tracked in the current tree. The scrub-or-fresh-repo decision still stands. ***** 2026-07-04 Sat @ 11:48:24 -0500 Count refresh — history now 565 commits; re-verify the 5-file claim before scrubbing The 2026-07-04 audit found the history is now 565 commits, down from the 589 recorded above. Because the count dropped, re-verify that the five credential files are still present in history (re-run the per-file =git log --all -- = check) before relying on the scrub scope — the earlier count is stale and the file set may have moved. +***** 2026-08-17 Mon @ 10:20:00 -0700 Corrected the paths — every prior check has been querying paths that never existed +The five filenames recorded above are not the paths these files live at, and =git log -- = answers "no commits" for a path it has never seen rather than erroring. So the checks return a clean result and mean nothing. The real paths, from =git log --all --name-only --diff-filter=A= over the full history, all sit under the pre-migration =dotfiles/= tree: + +- =dotfiles/system/.msmtprc= (3 commits) +- =dotfiles/system/.mbsyncrc= (2) +- =dotfiles/system/.config/calibre/smtp.py.json= (2) +- =dotfiles/system/.config/transmission/settings.json= (2) +- =dotfiles/system/.config/.tidal-dl.token.json= (2) +- =dotfiles/system/.config/.tidal-dl.json= (2) — a *sixth* file, never recorded here + +Use those paths for any future check, not the bare filenames. History is 891 commits; none of the six are in the current tree. The scrub-or-fresh-repo decision still stands and its scope is six files, not five. + +This surfaced while re-verifying on velox, where the check ALSO returned a false clean for a second, unrelated reason: the clone was shallow (7 commits), so it could not see the history either way. Both failures produce the same confident zero. Filed as =[#A] The installer shallow-clones the two repos I develop in=. ***** 2026-07-21 Tue @ 08:00:00 -0500 Re-verified: history now 851 commits; five files still present, per-file counts dropped 2026-07-21 audit re-verification. History is now 851 commits (=git rev-list --all --count=). The five credential files are still in history but at fewer commits each than the 2026-06-28 record: =.tidal-dl.token.json= 3 (was 5), =calibre/smtp.py.json= 4 (was 6), =transmission/settings.json= 3 (was 5), =.msmtprc= 5 (was 8), =.mbsyncrc= 6 (was 9). None are in the current tree. The scrub-or-fresh-repo decision still stands; the scope is smaller than recorded. @@ -2149,9 +2383,11 @@ From the roam inbox (routed 2026-07-13): the networking panel should track speed ** TODO [#C] zfs base VM image build failure: ZFS DKMS module missing :bug:zfs: :PROPERTIES: -:LAST_REVIEWED: 2026-07-09 +:LAST_REVIEWED: 2026-08-17 :END: =FS_PROFILE=zfs make test-vm-base= fails inside the VM at initramfs time: archangel reports "ZFS module not found! DKMS build may have failed" against the installed kernel (linux-lts 6.18.38 at the 2026-07-08 attempt). Consequences: the maint scenario harness's zfs lane (Phase 12) is filtered but unexercised, and a real zfs bare-metal install via archangel would plausibly hit the same wall. Priority per the bug matrix: Major severity (zfs install path broken) × some-users-sometimes = P3. When fixed, run =FS_PROFILE=zfs bash scripts/testing/run-maint-scenarios.sh --list= and add zfs scenario files (zpool scrub / autotrim / snapshot destroy) to the harness. +*** 2026-08-17 Mon @ 10:08:51 -0700 Rechecked: archzfs still on 2.3.3, still blocked +Ran the unblock check from the diagnosis below: archzfs' x86_64 index still serves only =zfs-dkms-2.3.3=. The first release supporting 6.18 is 2.4.0, so the blocking condition is unchanged and there is still nothing on our side to fix. Recheck again with the same one-liner. *** 2026-07-14 Tue @ 01:40:48 -0500 Diagnosed: OpenZFS/kernel version skew, blocked on archzfs Reproduced in ~1 minute of install: =dkms install zfs/2.3.3 -k 6.18.38-2-lts= exits 1 during pacstrap. Root cause confirmed: OpenZFS 2.3.3's META declares Linux-Maximum 6.15, and the VM installs linux-lts 6.18.38. The first release supporting 6.18 is 2.4.0 (2.4.1 covers 6.19), and archzfs currently serves only zfs-dkms 2.3.3-1 — nothing on our side to fix. Unblock condition: archzfs publishes zfs-dkms ≥2.4.0; recheck with =curl -s https://archzfs.com/archzfs/x86_64/ | grep -o 'zfs-dkms-[0-9.]*'=, then rerun =FS_PROFILE=zfs make test-vm-base=. @@ -2191,10 +2427,12 @@ Craig's roam capture 2026-07-20, routed via .emacs.d sentry inbox-zero as archse Craig: the camera works (bought for being Linux-friendly), it just needs configuring. There IS a config panel — =cameractrls= 0.6.10 is installed (a GTK GUI for camera controls: exposure, white balance, PTZ, focus, framing) plus =v4l-utils= for the CLI path. Caveat found 2026-07-21: no =/dev/video*= device is present right now, so the camera isn't currently plugged in / its UVC node isn't enumerated. Task: with the camera connected, confirm it enumerates as a /dev/video node, then set defaults in cameractrls. Small, mostly a live-hardware step. ** TODO [#C] Re-check python-lyricsgenius --skipinteg workaround :chore:solo: :PROPERTIES: -:LAST_REVIEWED: 2026-07-09 +:LAST_REVIEWED: 2026-08-17 :END: archsetup installs =python-lyricsgenius= with =--mflags --skipinteg=, skipping makepkg integrity + PGP checks — a workaround originally for an expired-signature issue upstream (surfaced by the 2026-06-23 --noconfirm audit). Periodically test whether the cause has cleared: if a plain =aur_install python-lyricsgenius= builds without complaint, drop the =--skipinteg= workaround. Removal needs a real AUR build to confirm, so it isn't a blind change. +*** 2026-08-17 Mon @ 10:08:51 -0700 Rechecked: still needed, cause unchanged +Fresh AUR clone, =makepkg --verifysource= on 3.7.0-1: the PyPI tarball passes, =LICENSE.txt= still FAILS its b2sum. The PKGBUILD still pins the license at github master, so its checksum drifts whenever upstream touches the file. =--skipinteg= stays. *** 2026-07-23 Thu @ 02:20:00 -0500 Rechecked: still needed, unchanged Fresh AUR clone, =makepkg --verifysource= on 3.7.0-1 (PKGBUILD still unchanged since the last check): the PyPI tarball passes, =LICENSE.txt= still FAILS its b2sum. Same structural cause — the source pins the license at github master, so its checksum drifts whenever upstream touches the file. =--skipinteg= stays. Nothing to change in the installer. @@ -3506,39 +3744,6 @@ ratio and absent on velox; full evidence and rationale in Scope: installer step + rule file + tests per existing shapes, and apply both live to velox over tailscale (daily-driver sync — neither exists there today). -:PROPERTIES: -:CREATED: [2026-07-28 Tue] -:LAST_REVIEWED: 2026-07-28 -:END: -Craig, from the roam inbox 2026-07-28: "launching into wayland doesn't honor previous caffeine settings ...or I expect any other settings in the desktop settings module." Captured right after the 08:59 reboot. - -Confirmed, and it generalizes past caffeine. The settings module splits cleanly into two halves, and only one of them persists. - -Persisted, in =~/.config/desktop-settings/state.json= (=store.py= =DEFAULTS=): program slots, idle-tripper stages, wallpaper. These come back correctly. - -Not persisted — every one is derived live from a process or a compositor runtime option, so a session restart resets it to whatever =hyprland.conf= establishes: -- Caffeine — =caffeine_state()= is =pgrep -x hypridle= inverted, and =hyprland.conf:73= runs =exec-once = pkill -x hypridle; hypridle=. So every launch unconditionally starts hypridle, which means caffeine is *always* OFF after login. There is no code path that could restore it ON. -- Auto-dim — =dim_state()= reads =hyprctl getoption decoration:dim_inactive=, a compositor runtime value that resets to the config default on restart. -- Night light — =state()= is =pgrep -x gammastep=; the process dies with the session. -- DND — =dunstctl=; dunst restarts fresh from =exec-once=. -- Power profile / brightness — owned by powerprofilesctl and systemd-backlight, outside this module's scope. - -Verified live 10 minutes after the reboot: hypridle running (caffeine OFF), dim =false=, gammastep not running, dnd =false=, power =balanced=. Every toggle sat at its factory position. - -The failure is silent, which is what makes it bite: nothing tells you the value you set was discarded. That is the mechanism behind the 2026-07-27 lockout, where Craig believed caffeine was on and the screen locked anyway. - -Grading: Major severity (the panel's core promise is holding these values, and the reset is silent and total across all four toggles) x most users frequently (every session start resets them, though it only harms when a deliberate non-default was set) = P2 = [#B]. - -Not :solo: — the fix needs Craig's call on *which* toggles should persist and whether persistence is per-toggle opt-in. Restoring night light at 3pm or caffeine on a laptop are both plausibly wrong, so this is a preference question, not a derivable one. The mechanism itself (extend =store.py= with a =toggles= block, restore on session start) is mechanical once that's settled. - -Related: =[#B] Caffeine state is unreadable on both surfaces= covers display accuracy — whether the surfaces report the truth. This covers whether the value survives at all. Distinct bugs, same subsystem. - -*** Side finding — gammastep loses a startup race and nothing relaunches it -=hyprland.conf:75= runs =exec-once = gammastep=, but no gammastep process is alive. Today's three launch logs tell the story: =gammastep-2026-07-28-090034.log= carries "Wayland connection experienced a fatal error: -1 / Temperature adjustment failed", and the other two are empty. - -Launched by hand afterward it runs fine and survives, so gammastep is not broken — it loses a race against compositor readiness at session start. Nothing relaunches it, so night light is simply off for the whole session, silently. (An earlier read of this said night light "has likely never worked from the config". That was wrong: the failure is a startup race, not a permanent break.) - -Worth its own task — the fix is a readiness wait or a retry around that exec-once, not a persistence change. Filed here for now because it surfaced during this investigation. ** DONE [#A] Comet KVM setup for truenas :feature:infra:truenas: CLOSED: [2026-08-08 Sat] :PROPERTIES: -- cgit v1.2.3 From 6faa31c1890e931c92d18a3c42ec57e0fd2503a8 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Mon, 17 Aug 2026 11:45:37 -0700 Subject: chore(todo): the installer also leaves the dotfiles clone unable to push Same two lines as the shallow-clone defect, found while pushing today. The dotfiles remote defaults to the public read-only endpoint, so a machine the installer built can fetch but never push. My other machine uses the ssh form, so the rebuilt one was the odd one out purely because the installer made it. That half is a decision rather than a fix, so the task loses its solo tag. The read-only default is right for anyone else installing this, who has no key on the server, and wrong for my own machines. The override already exists, so the only question is where my value lives. --- todo.org | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index 1ec7cf1..32eb9a2 100644 --- a/todo.org +++ b/todo.org @@ -236,7 +236,7 @@ unless you look) x rare edge case (one machine, specific conditions) = P2 = [#B]... except that this is a live, ongoing drain on a travelling machine rather than a latent defect, so it takes [#A] until the machine is back to normal. Re-grade to [#B] once resolved and the question is only prevention. -** TODO [#A] The installer shallow-clones the two repos I develop in :bug:velox:solo: +** TODO [#A] The installer clones my two working repos shallow and read-only :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] :LAST_REVIEWED: 2026-08-17 @@ -263,16 +263,36 @@ gets =--depth 1= and org does not. The AUR build clones (=archsetup:855=, =:1673=, =:1677=) are correctly shallow and stay that way. Only the two user-repo sites change. -Repair on a machine already built: =git fetch --unshallow= in each repo. +*Second defect, same two lines, found 2026-08-17 while pushing:* the dotfiles +clone could not push at all. =archsetup:245= defaults =dotfiles_repo= to +=https://git.cjennings.net/dotfiles.git=, the public read-only endpoint, so +=git push= returned 403. Ratio uses =git@cjennings.net:dotfiles.git= and +archsetup's own clone uses the matching ssh form, so velox was the odd one out +purely because it was the machine rebuilt by the installer. Repointed velox's +remote and pushed. + +That half needs a decision rather than a fix, which is why this task is no +longer =:solo:=. The https default is *correct for a stranger* installing +archsetup, who has no ssh key on the server, and this repo is being prepared +for public release. It is wrong for my own machines, which need to push. The +override already exists (=DOTFILES_REPO=, documented in +=archsetup.conf.example=), so the question is only where my personal value +lives: a config the personal ISO bakes in, a post-install step, or a detection +that prefers ssh when a key is present. Craig's call. + +Repair on a machine already built: =git fetch --unshallow= in each repo, and +=git remote set-url origin git@cjennings.net:.git= for dotfiles. Grading: Major severity (two working repos silently missing their history on the machine I develop on, and it returns confidently wrong answers to history questions rather than failing) x every user every time (every fresh install, both daily drivers) = P1 = [#A]. -:solo: — the change is two lines plus tests in the existing -=tests/installer-steps/= shape, and it is verifiable locally by asserting the -clone command carries no =--depth= for these two repos. +Not :solo:. The depth half is (two lines plus tests in the existing +=tests/installer-steps/= shape, verifiable by asserting the clone command +carries no =--depth= for these two repos). The remote-URL half needs the +decision above, so the task as a whole waits on it. Split it in two if the +depth fix is wanted sooner. ** TODO [#B] post-rebuild-check needs a reference-host mode :feature:velox:solo: :PROPERTIES: :CREATED: [2026-08-17 Mon] -- cgit v1.2.3 From afbf011aa0937b5702b6d8c1bfca0809ed809425 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:16:46 -0700 Subject: fix(installer): give NTP an IP source so a wrong clock can't kill DNS The installer wrote both halves of a deadlock. configure_dns pins DNSOverTLS=yes with DNSSEC=yes, and both validate against the wall clock. The chrony step enables chronyd without writing a config, so the machine runs Arch's stock one, whose only source is a pool hostname. Boot with a wrong clock and DoT validation fails, so nothing resolves. Chrony then can't resolve its pool, so the clock stays wrong. Neither side moves, and recovery takes a second device. Velox hit this on the road and I diagnosed it from a phone. An address needs no DNS and no certificate, so two IP-addressed sources in a drop-in break the cycle whatever caused the skew. Stock chrony.conf reads no drop-in directory, so it gets a confdir line pointing at one. post-rebuild-check grows a sixth check for the same property. It reads sources only from files chrony is told to read. A drop-in beside a chrony.conf that never names its directory is one chrony won't open, so counting it would pass the machine while describing a file nothing reads. The failure taxonomy gains the mode in its DNS layer and a cluster 5 triage line. Its egress-layer clock entry assumed working DNS and offered set-ntp true, which can't recover this. That entry now says so. --- archsetup | 30 +++ docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +- scripts/post-rebuild-check | 85 ++++++- .../post-rebuild-check/test_post_rebuild_check.py | 90 ++++++- todo.org | 282 +++++++++++++++++++-- 5 files changed, 461 insertions(+), 30 deletions(-) (limited to 'todo.org') diff --git a/archsetup b/archsetup index edd4062..078408c 100755 --- a/archsetup +++ b/archsetup @@ -1200,6 +1200,36 @@ configure_build_environment() { echo 'OPTIONS=""' > /etc/sysconfig/chronyd systemctl enable chronyd.service >> "$logfile" 2>&1 || error_warn "$action" "$?" + # Bootstrap NTP sources addressed by IP, never by hostname. + # + # Arch's stock chrony.conf names its pool by hostname, and the DNS this + # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both + # validate against the wall clock, so a machine that boots with a wrong + # clock resolves nothing: chrony cannot reach the pool, so the clock stays + # wrong, so DNS stays dead. Neither side moves, and recovery needs a second + # device to look up an NTP address by hand. An IP-addressed source needs no + # DNS and no certificate, so it breaks the deadlock unattended. I would + # rather carry two extra server lines than lose a laptop's network to any + # RTC fault. See the clock/DNS deadlock entry in the net failure taxonomy + # under docs/design/. + action="adding IP-addressed NTP bootstrap sources" && display "task" "$action" + mkdir -p /etc/chrony.d + cat << 'EOF' > /etc/chrony.d/10-bootstrap-ip-ntp.conf +# Reachable without DNS, so a wrong clock can always correct itself. +server 162.159.200.1 iburst +server 162.159.200.123 iburst +EOF + # Stock chrony.conf reads no drop-in directory, so point it at one. + if [ -f /etc/chrony.conf ]; then + backup_system_file /etc/chrony.conf + if ! grep -qE '^[[:space:]]*confdir[[:space:]]+/etc/chrony\.d' /etc/chrony.conf; then + printf '\n# Read drop-ins (archsetup owns /etc/chrony.d).\nconfdir /etc/chrony.d\n' \ + >> /etc/chrony.conf || error_warn "$action" "$?" + fi + else + error_warn "$action (no /etc/chrony.conf to point at /etc/chrony.d)" 1 + fi + action="configuring compiler to use all processor cores" && display "task" "$action" backup_system_file /etc/makepkg.conf sed -i "s/-j2/-j$(nproc)/;s/^#MAKEFLAGS/MAKEFLAGS/" /etc/makepkg.conf >> "$logfile" 2>&1 diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 74790c6..70421d0 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,6 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -107,7 +108,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - PPPoE / VPN link with a lower MTU not clamped (no). Browsing works but big transfers / some HTTPS hang. A PPPoE (1492) or VPN path has a smaller MTU and the too-large segments get dropped. Fix: set the tunnel/link MTU down (=.mtu 1420= for VPN, 1492 for PPPoE) or MSS-clamp on the gateway. [[https://thelineman.ca/articles/article-8-mtu-vpn-mss][vpn mtu/mss]] - Stale http_proxy env var points at a dead proxy (no). Every curl/wget/pacman fails though the network is fine; browsers may work. A leftover =http_proxy= points at an offline/off-network proxy. Fix: unset the vars, remove the export from =~/.profile= / =/etc/environment=. [[https://everything.curl.dev/usingcurl/proxies/env.html][curl proxy env]] - Unreachable PAC file off the corporate network hangs everything (no). Away from the office the browser stalls with no error. A system proxy set to "automatic" with a PAC URL that only resolves on the corporate LAN blocks waiting instead of falling back to DIRECT. Fix: switch system proxy to None (=gsettings … org.gnome.system.proxy mode 'none'=) or clear the PAC URL. [[https://bugzilla.mozilla.org/show_bug.cgi?id=1121800][ff pac hang]] -- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] +- Clock skew breaks every TLS handshake (yes). "Your connection is not private" on every HTTPS site though ping/DNS work; the clock is hours/years off. A dual-boot Windows RTC-localtime, unsynced NTP, or a dead CMOS battery leaves the clock wrong. Fix: =timedatectl set-ntp true= (=set-local-rtc 0= on dual-boot), replace the CMOS battery if it recurs. This entry assumes DNS still works; when the resolver runs DoT or DNSSEC the same skew kills DNS first and =set-ntp true= cannot recover it — see the clock/DNS deadlock in the DNS layer. [[https://wiki.archlinux.org/title/System_time][archwiki system time]] - Firewall default-deny drops all egress (yes). No traffic leaves right after enabling a firewall, or after both ufw and firewalld are on; even DNS fails. A default outgoing-deny policy, or two firewalls fighting over nftables. Fix: allow egress (=ufw default allow outgoing=) and run only one firewall. [[https://wiki.archlinux.org/title/Uncomplicated_Firewall][archwiki ufw]] - VPN kill-switch / leftover iptables rule strangles egress after VPN drops (yes; distinct from the route-capture case). Internet dies the moment the VPN disconnects and never returns until reboot. A kill-switch rule pinned traffic to tun0 and the leftover rule keeps dropping everything on the real interface. Fix: flush the stale rules (=iptables -F; iptables -P OUTPUT ACCEPT=, or restart the firewall), reconnect. [[https://bbs.archlinux.org/viewtopic.php?id=300104][arch ufw killswitch]] - IPv6 egress broken while IPv4 works (no; the egress angle of the broken-v6 family). Pages load slowly/intermittently; IPv4-only hosts are fine. The network advertises IPv6 with no working route and Happy Eyeballs keeps trying the dead AAAA path. Fix: =nmcli con modify ipv6.method disabled= until the network's IPv6 is fixed. [[https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4][ubuntu slow ipv6]] @@ -309,6 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. +- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/scripts/post-rebuild-check b/scripts/post-rebuild-check index 8807f85..c18ae8f 100755 --- a/scripts/post-rebuild-check +++ b/scripts/post-rebuild-check @@ -21,6 +21,10 @@ # 5. signal-cli holds no registered account (velox lost its # registration, and because agent-text relays into this machine, # that silently broke paging for the WHOLE fleet) +# 6. every NTP source is named by hostname (a wrong clock fails the +# DoT/DNSSEC validation this machine's DNS runs on, so nothing +# resolves -- including the NTP pool that would fix the clock; velox +# deadlocked exactly this way 2026-08-19 and needed a second device) # # The .gitignore rule in check 4 is what scopes it: a tooling path is only # expected where the project's own .gitignore names it, so a project that @@ -50,6 +54,8 @@ # ~/.dotfiles) # PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output; "" = no account, # the special value MISSING = binary absent +# PRC_NTP_SOURCES newline list of configured NTP server addresses; +# the special value MISSING = no NTP daemon active # PRC_SYSTEMCTL path to the systemctl binary (a fake, under test) # PRC_SYSTEMCTL_TIMEOUT seconds to allow each systemctl call (default 5) # @@ -61,9 +67,10 @@ usage() { cat <<'EOF' post-rebuild-check - verify a rebuilt machine is actually finished -Runs the five checks that caught velox's 2026-08 reinstall gaps: failed +Runs the six checks that caught velox's 2026-08 reinstall gaps: failed units, present-but-inert user units, orphaned *.example configs, missing -per-project tooling state, and the signal-cli registration. +per-project tooling state, the signal-cli registration, and whether time +sync can recover from a wrong clock without DNS. Usage: post-rebuild-check [--help] @@ -84,6 +91,7 @@ TOTAL_FINDINGS=0 CHECK_FINDINGS=0 FINDING_LINES="" signal_missing="" +ntp_missing="" # Every systemctl call is bounded. A wedged user manager spins and answers # nothing -- seen live on velox 2026-08-17, where `is-enabled`, `cat`, and @@ -179,7 +187,7 @@ while IFS= read -r line; do unit=${line#*:} finding "$scope unit failed: $unit" done < "$STAGE" -report "check 1/5: failed units" +report "check 1/6: failed units" # --- 2. user unit files present but not enabled --------------------------- @@ -269,7 +277,7 @@ while read -r name state; do esac finding "unit file present but not enabled: $name ($state)" done < "$STAGE" -report "check 2/5: unit files" +report "check 2/6: unit files" # --- 3. *.example files whose real sibling is missing --------------------- @@ -314,7 +322,7 @@ while IFS= read -r root; do [ -e "${ex%.example}" ] || finding "example without its real file: $ex" done < "$WORK/examples" done < "$WORK/roots" -report "check 3/5: local files" +report "check 3/6: local files" # --- 4. gitignore-mode projects missing their tooling --------------------- @@ -368,7 +376,7 @@ todo.org todo\.org inbox inbox EOF done < "$WORK/projects" -report "check 4/5: project tooling" +report "check 4/6: project tooling" # --- 5. signal-cli registration ------------------------------------------- @@ -395,7 +403,68 @@ if [ "$signal_missing" = 1 ]; then elif [ -z "$signal_missing" ] && [ -z "$accounts" ]; then finding "no signal account registered — agent-text relays into this machine, so paging breaks for the whole fleet" fi -report "check 5/5: signal registration" +report "check 5/6: signal registration" + +# --- 6. NTP can recover a wrong clock without DNS ------------------------- +# +# The clock/DNS bootstrap deadlock. This machine resolves through DNSOverTLS +# with DNSSEC, and both validate against the wall clock, so a boot with a +# wrong clock resolves nothing at all. If every configured NTP source is named +# by hostname, the daemon that would correct the clock needs the DNS the clock +# is breaking, and the machine cannot recover without a second device -- +# which is exactly what happened on velox 2026-08-19. One source addressed by +# IP breaks the cycle, so that is what this check looks for. + +# True when the argument is an address rather than a name. An address needs no +# resolver, which is the whole property being checked. +is_ip_literal() { + case "$1" in + "") return 1 ;; + *:*) case "$1" in *[!0-9A-Fa-f:]*) return 1 ;; esac + return 0 ;; + *[!0-9.]*) return 1 ;; + *.*) return 0 ;; + esac + return 1 +} + +if [ -n "${PRC_NTP_SOURCES+set}" ]; then + ntp_sources=$PRC_NTP_SOURCES + if [ "$ntp_sources" = "MISSING" ]; then + ntp_sources="" + ntp_missing=1 + fi +elif sctl is-active chronyd >/dev/null 2>&1; then + # Both the main file and any drop-in: the IP-addressed source belongs in a + # drop-in, so reading only chrony.conf would miss every correct machine. + ntp_sources=$(cat /etc/chrony.conf /etc/chrony.d/*.conf 2>/dev/null \ + | awk '$1 == "server" || $1 == "pool" { print $2 }') +elif sctl is-active systemd-timesyncd >/dev/null 2>&1; then + ntp_sources=$(awk -F= '/^[[:space:]]*NTP=/ { print $2 }' \ + /etc/systemd/timesyncd.conf 2>/dev/null | tr ' ' '\n') +else + ntp_sources="" + ntp_missing=1 +fi + +if [ "$ntp_missing" = 1 ]; then + finding "no NTP implementation is active — nothing corrects the clock, and a wrong clock takes DNS down with it" +elif [ -z "$ntp_sources" ]; then + finding "no NTP sources are configured — nothing was checked, and nothing corrects the clock" +else + ntp_has_literal="" + stage "$ntp_sources" + while IFS= read -r src_addr; do + [ -z "$src_addr" ] && continue + if is_ip_literal "$src_addr"; then + ntp_has_literal=1 + fi + done < "$STAGE" + if [ -z "$ntp_has_literal" ]; then + finding "every NTP source is named by hostname — a wrong clock breaks DNS, so nothing can resolve them and the clock stays wrong" + fi +fi +report "check 6/6: NTP bootstrap" # --- summary -------------------------------------------------------------- @@ -403,5 +472,5 @@ if [ "$TOTAL_FINDINGS" -eq 0 ]; then echo "all checks clean" exit 0 fi -echo "$TOTAL_FINDINGS finding(s) across 5 checks" +echo "$TOTAL_FINDINGS finding(s) across 6 checks" exit 1 diff --git a/tests/post-rebuild-check/test_post_rebuild_check.py b/tests/post-rebuild-check/test_post_rebuild_check.py index 4894451..757039b 100644 --- a/tests/post-rebuild-check/test_post_rebuild_check.py +++ b/tests/post-rebuild-check/test_post_rebuild_check.py @@ -21,6 +21,8 @@ probe"): PRC_LOCAL_SCAN_ROOTS newline-separated roots to scan for *.example orphans PRC_PROJECT_ROOTS newline-separated project dirs for the tooling check PRC_SIGNAL_ACCOUNTS signal-cli listAccounts output ("" = no accounts); + PRC_NTP_SOURCES newline list of configured NTP server addresses + ("MISSING" = no NTP daemon active) the special value MISSING means the binary is absent Run from repo root: @@ -39,7 +41,8 @@ CHECK = os.path.join(REPO_ROOT, "scripts", "post-rebuild-check") def run_check(failed_units="", unit_states="", local_roots="", - project_roots="", signal_accounts="+15045551234"): + project_roots="", signal_accounts="+15045551234", + ntp_sources="162.159.200.1\npool.ntp.org"): """Run the script with every probe stubbed; defaults are all-clean. Roots are newline-separated. Empty means "the seam is set and names no @@ -52,11 +55,88 @@ def run_check(failed_units="", unit_states="", local_roots="", env["PRC_LOCAL_SCAN_ROOTS"] = local_roots env["PRC_PROJECT_ROOTS"] = project_roots env["PRC_SIGNAL_ACCOUNTS"] = signal_accounts + env["PRC_NTP_SOURCES"] = ntp_sources return subprocess.run( ["sh", CHECK], capture_output=True, text=True, timeout=30, env=env, ) +class NtpBootstrap(unittest.TestCase): + """Check 6 — the clock/DNS bootstrap deadlock. + + A wrong clock fails the DoT certificate and DNSSEC signature checks this + machine's DNS runs on, so nothing resolves; and an NTP daemon whose every + source is a hostname then cannot resolve the servers that would correct + the clock. One source addressed by IP is what makes the machine able to + recover on its own. + """ + + # --- Normal cases --------------------------------------------------- + + def test_an_ip_addressed_source_is_clean(self): + r = run_check(ntp_sources="162.159.200.1\npool.ntp.org") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + self.assertEqual(r.returncode, 0, r.stdout) + + def test_all_hostname_sources_is_a_finding(self): + # The velox 2026-08-19 shape exactly: stock Arch chrony.conf, whose + # only source is a pool hostname. + r = run_check(ntp_sources="2.arch.pool.ntp.org") + self.assertIn("every NTP source is named by hostname", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_an_ipv6_addressed_source_counts(self): + r = run_check(ntp_sources="2606:4700:f1::1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + # --- Boundary cases ------------------------------------------------- + + def test_the_literal_may_sit_anywhere_in_the_list(self): + # Order must not matter; the property is "at least one", and the + # drop-in that carries it is read after the main config. + r = run_check(ntp_sources="a.pool.ntp.org\nb.pool.ntp.org\n162.159.200.1") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_blank_lines_between_sources_are_ignored(self): + r = run_check(ntp_sources="\n\n162.159.200.1\n\n") + self.assertIn("check 6/6: NTP bootstrap — ok", r.stdout) + + def test_a_hostname_containing_digits_and_dots_is_not_an_address(self): + # The trap in any naive "looks like an IP" test: these resolve through + # DNS like any other name, so counting one as an address would hand a + # deadlocked machine a clean bill. + for host in ("0.arch.pool.ntp.org", "3.us.pool.ntp.org", "time1.google.com"): + with self.subTest(host=host): + r = run_check(ntp_sources=host) + self.assertIn("every NTP source is named by hostname", r.stdout) + + # --- Error cases ---------------------------------------------------- + + def test_no_ntp_daemon_is_a_finding(self): + r = run_check(ntp_sources="MISSING") + self.assertIn("no NTP implementation is active", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_no_sources_configured_is_a_finding(self): + # Fails closed: an empty list proves nothing about the machine, and + # reporting ok would be a false pass on a box with no time sync at all. + r = run_check(ntp_sources="") + self.assertIn("no NTP sources are configured", r.stdout) + self.assertEqual(r.returncode, 1) + + def test_unset_seam_falls_through_to_the_real_probe(self): + # Same contract as every other seam: unset means "really look", so a + # caller who forgets the variable cannot silently skip the check. + env = dict(os.environ) + env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", + "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) + env.pop("PRC_NTP_SOURCES", None) + r = subprocess.run(["sh", CHECK], capture_output=True, text=True, + timeout=30, env=env) + self.assertIn("check 6/6: NTP bootstrap", r.stdout) + + class AllClean(unittest.TestCase): # --- Normal cases --------------------------------------------------- @@ -482,6 +562,7 @@ class SignalAccount(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234", "signal_missing": "1"}) r = subprocess.run(["sh", CHECK], capture_output=True, text=True, @@ -504,6 +585,7 @@ class ProbeFailure(unittest.TestCase): env = dict(os.environ) env.update({"PRC_FAILED_UNITS": "", "PRC_UNIT_STATES": "", "PRC_LOCAL_SCAN_ROOTS": "", "PRC_PROJECT_ROOTS": "", + "PRC_NTP_SOURCES": "162.159.200.1", "PRC_SIGNAL_ACCOUNTS": "+15045551234"}) for n in names: env.pop(n, None) @@ -624,7 +706,7 @@ class WedgedSystemctl(unittest.TestCase): self.assertEqual(r.returncode, 1) self.assertIn("could not query user units", r.stdout) # The run must reach the end rather than stopping at the first call. - self.assertIn("check 5/5", r.stdout) + self.assertIn("check 6/6", r.stdout) def test_a_hanging_systemctl_does_not_stall_the_whole_run(self): # The fake sleeps 8s against a 1s bound, so a bounded run lands near @@ -648,7 +730,7 @@ class Reporting(unittest.TestCase): unit_states="c.timer disabled") self.assertEqual(r.returncode, 1) summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "3 finding(s) across 5 checks") + self.assertEqual(summary, "3 finding(s) across 6 checks") def test_the_summary_count_tracks_every_check(self): # One finding from each of the five, so a counter that drops or @@ -664,7 +746,7 @@ class Reporting(unittest.TestCase): local_roots=scan, project_roots=proj, signal_accounts="") summary = r.stdout.strip().splitlines()[-1] - self.assertEqual(summary, "5 finding(s) across 5 checks") + self.assertEqual(summary, "5 finding(s) across 6 checks") def test_help_exits_zero(self): r = subprocess.run(["sh", CHECK, "--help"], diff --git a/todo.org b/todo.org index 32eb9a2..1bb810d 100644 --- a/todo.org +++ b/todo.org @@ -45,6 +45,86 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work +** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The installer wrote both halves of a deadlock. =configure_dns= pins +=DNSOverTLS=yes= with =DNSSEC=yes=, and both validate against the wall clock; +the chrony step enables chronyd without writing a config, so the machine runs +Arch's stock one whose only source is =pool 2.arch.pool.ntp.org= — a hostname. +Boot with a wrong clock and DoT certificate validation fails, so nothing +resolves; chrony then cannot resolve its pool, so the clock stays wrong. +Neither side moves. It caught velox on the road 2026-08-19 and had to be +diagnosed from a phone. + +Fixed at the root: the installer now writes +=/etc/chrony.d/10-bootstrap-ip-ntp.conf= with two IP-addressed Cloudflare +sources and points stock chrony.conf at the drop-in. An address needs no DNS +and carries no certificate, so the escape hatch holds whatever broke the clock. +velox has the same drop-in applied live, verified with =chronyc -n sources= +(=162.159.200.1= selected) and =timedatectl= reporting synchronized. + +What is left here is the part I could not verify: the decisive test is a full +power-down and cold boot, confirming the clock corrects itself untouched. See +the manual-testing entry. Until that runs, the fix is sound by construction +rather than demonstrated. + +Grading: Critical severity (total loss of network — no DNS means no egress, and +recovery needs a second device) x some users sometimes (only machines that boot +with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = +[#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in +this state it is fully offline with no local path out. + +*** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed +The installer carries the drop-in; =post-rebuild-check= grew a sixth check that +fails a machine whose every NTP source is a hostname; the net failure taxonomy +gained the mode in its DNS layer plus a cluster 5 triage line, and its existing +egress-layer clock entry now says outright that its remedy does not apply when +DoT or DNSSEC is on. + +The doctor half is in dotfiles: =classify.py= reached "DNS not resolving → net +repair dns-test" here, which cannot help, because every public resolver fails +the same clock-sensitive validation — so the doctor sent you round a loop. It +now emits a =clock-dns= row ahead of the generic DNS verdict. Detection is +deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass +query addressed by IP over plain UDP/53 to tell "resolved is refusing to +validate" apart from "DNS is genuinely dead". + +** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +The doctor now *names* the deadlock but hands the user two commands rather than +running anything — the verdict is =needs-user-action=. That was the honest call +at the time: correcting the clock needs the real time, and I could not exercise +a repair against the actual failure state without deliberately wedging velox's +network mid-session. + +An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a +certificate. The shape would be =chronyc add server iburst= followed by +=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to +a running chronyd over its socket, so this works in exactly the state that +blocks everything else. Two things to settle before building it: whether +=makestep= actually steps on the first sample after a runtime =add server= or +needs a poll first, and what the doctor does on a box running timesyncd rather +than chrony. + +Worth less now than it looks: a machine built by the current installer carries +the IP-addressed source and never reaches the deadlock. This is for machines +built before the fix. + +Grading: Minor severity (the doctor already names the fault correctly and hands +over a working remedy; only the automation is missing) x rare edge case (only +pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this +state has no working network and cannot look anything up, which makes the +two-command handoff harder to follow than it reads. Minor x rare = P4, and I am +leaving it at [#C] rather than [#D] because it sits one step from done. + ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> :PROPERTIES: @@ -79,6 +159,26 @@ grep the journal for new "Power key pressed" lines — zero means fixed. Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree +The reseat did not happen before the flight, and velox is travelling. The +deadline blew past on 08-14. + +The two symptoms have separated, which is worth recording because it changes +what the evidence proves. The phantom presses have stopped: fifteen "Power key +pressed" entries between 08-14 04:29 and 08-15 20:04, then nothing at all +across five boots including today's. The touchpad has not — there is still no +touchpad node under =/dev/input/by-path/=, which is the same dead interrupt +line the body describes. + +So the quiet power button is not evidence the connector reseated itself. The +interrupt line is the symptom that cannot be masked in software, and it is +still dead, so the ribbon is still unseated. The most likely reason the +presses stopped is that the machine has been sitting on hotel surfaces instead +of being carried and flexed. + +The interim shield is still live (=HandlePowerKey=ignore=), and the escalation +note stands: an EC-level glitch cuts power below systemd regardless of it. ** DOING [#A] Velox reinstall — DR test of archangel + archsetup :velox:chore: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -196,7 +296,29 @@ machine-level half is already correct. Grading: Major severity (a crash loop burning battery and filling the journal, silently) x every user every time on any laptop with the TLP fix applied = P1 = [#A]. -** TODO [#A] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: + +*** 2026-08-17 Mon @ 19:57:42 -0700 The loop stopped at the reboot; the defect did not +velox rebooted at 16:04 and there have been zero coredumps since, against 47 +in the twelve hours before it. So the loop is not currently burning anything. + +That is not a fix, and the distinction matters for whoever picks this up. +=powerprofilesctl get= still fails exactly as recorded — =NameHasNoOwner ... +unit is masked= — so every precondition for the loop is intact and it returns +whenever the caller next polls. What the reboot cleared is the caller's state, +not the bug. + +Narrowed the search the body asks for: =power.py= is the *only* file in +dotfiles that shells out to =powerprofilesctl= (=SETTINGS_POWERPROFILESCTL=, +line 14), so the caller is inside the settings module rather than waybar or a +timer. Worth knowing that the coredumps are =powerprofilesctl= itself aborting +— it is a python script, which is why they log as =/usr/bin/python3.14= +SIGABRT rather than under its own name. + +Grade unchanged. The matrix inputs did not move: the severity is what happens +while the machine is in that state, and the frequency row is every laptop +carrying the TLP fix. A quiet interval since a reboot is not a frequency +change. +** TODO [#B] velox's systemd --user spins at 96% and cannot resolve unit files :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] :LAST_REVIEWED: 2026-08-17 @@ -236,6 +358,24 @@ unless you look) x rare edge case (one machine, specific conditions) = P2 = [#B]... except that this is a live, ongoing drain on a travelling machine rather than a latent defect, so it takes [#A] until the machine is back to normal. Re-grade to [#B] once resolved and the question is only prevention. + +*** 2026-08-17 Mon @ 19:57:42 -0700 The reboot cleared it; re-graded [#A] to [#B] as the task instructed +velox rebooted at 16:04. The wedge is gone: =systemctl --user is-enabled +roam-sync.timer= now answers =enabled= in well under a second, where every +unit-file call hung indefinitely before, and =list-timers= shows +calendar-sync, roam-sync and agenda-render-cache all firing on schedule +again. So the remedy the task named — a logout or reboot — was taken and +worked. + +Nothing here was diagnosed further, which means the cause is still unproven +and both candidates in the body stand. What is left is prevention, and the +task's own grading says that is [#B]: the live-drain argument was the only +thing holding it at [#A], and the drain has stopped. Re-graded per that +instruction rather than by a fresh judgment. + +Reproducing it deliberately is the open question, and it is not obviously +worth doing — it costs a wedged session to learn something the crash-loop fix +may make moot. ** TODO [#A] The installer clones my two working repos shallow and read-only :bug:velox: :PROPERTIES: :CREATED: [2026-08-17 Mon] @@ -446,7 +586,7 @@ users sometimes = P3 = [#C]. ** TODO [#B] Land the rescued emacs-wttrin commit :chore:velox: :PROPERTIES: :CREATED: [2026-08-14 Fri] -:LAST_REVIEWED: 2026-08-14 +:LAST_REVIEWED: 2026-08-17 :END: bf0457f "feat: add wttrin-hide-follow-line to hide the wttr.in follow line" (2026-06-24) was the only genuinely unpushed commit anywhere on the old @@ -456,6 +596,17 @@ git bundle before the disk was wiped: To land it: clone emacs-wttrin, =git fetch --branches=, review the commit, then push to git@cjennings.net:emacs-wttrin.git. Delete the bundle once it's on the remote. + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: still unlanded, and the bundle is still the only copy +Cloned the remote bare and asked it for the object directly: =git cat-file -t +bf0457f= returns "Not a valid object name", so the commit has never reached +=git@cjennings.net:emacs-wttrin.git=. Remote =main= is =ee8fdeb=. + +That makes =working/velox-reinstall/wttrin-bf0457f.bundle= the sole surviving +copy of 103 insertions across three files, on one laptop that is travelling. +Worth doing sooner than its =[#B]= suggests for that reason alone, and it also +pins the working directory open — the reinstall task cannot file its artifacts +away while this bundle is still load-bearing. ** TODO [#B] archsetup doesn't clone rulesets :bug:velox: DEADLINE: <2026-08-15 Sat> :PROPERTIES: @@ -868,18 +1019,38 @@ doc above (not published, since they map the setup). Follow-ons: the rotation VERIFY above, velox reconcile on return, the secrets-repo split (top of Open Work), the wireguard =.gitignore= bug (line ~191), the cgit move (below), and a pre-receive secret-scan hook so this can't recur. -*** TODO [#A] velox: reconcile its clones after the history rewrite -velox was offline for repair during the 2026-08-09 purge, so its clones still -hold the pre-rewrite history and are diverged from the rewritten remotes. On -its return: force-fetch + rebase local work onto the rewritten main in both -repos (or re-clone), force-update the local tag, local-gc, before its next -push. Also on the velox riders on the sleep/suspend task. +*** 2026-08-17 Mon @ 19:57:42 -0700 Moot — the 08-13 wipe re-cloned velox from the rewritten remotes +This asked velox to reconcile clones that no longer exist. The machine was +wiped and reinstalled on 2026-08-13, so every repo on it was cloned fresh +*after* the purge and never held the pre-rewrite history at all. The runbook +anticipated this ("fresh clones automatically carry the post-purge rewritten +git history"); nobody closed the task once the reinstall took that route. + +Verified rather than assumed: both repos are level with =origin/main= today — +archsetup at =6faa31c=, dotfiles at =65940f2=, both trees clean. + +One thing the reinstall did leave, and it is filed separately: the installer +cloned both repos =--depth 1=, so the history was present-but-truncated until +today's =git fetch --unshallow= (see the shallow-clone =[#A]=). A reconcile +against the rewritten remote was still unnecessary — a shallow clone of the +right history is not a diverged clone of the wrong one. ** TODO [#B] Move archsetup off cgit to cjennings@cjennings.net :chore:security: :PROPERTIES: -:LAST_REVIEWED: 2026-07-21 +:LAST_REVIEWED: 2026-08-17 :END: Decided (Craig, 2026-07-20): move the archsetup repo off the public cgit host (git@cjennings.net, scan-path /var/git) to Craig's private account remote cjennings@cjennings.net, so it is no longer world-cloneable. This is the archsetup-specific fix for the cgit-exposure finding above. Plan: create a bare repo under cjennings's control off the cgit scan-path (e.g. =~cjennings/git/archsetup.git=); push current main + tags there; migrate the post-receive hook that publishes the installer to =/var/www/cjennings/archsetup= so curl-install keeps working (the single published file stays public by design; only the repo goes private); update the origin remote on ratio and velox to =cjennings@cjennings.net:git/archsetup.git=; remove =/var/git/archsetup.git= so cgit no longer serves it. Verify: anonymous =git clone https://git.cjennings.net/archsetup.git= fails, the new private clone works from both machines, and the curl-install URL still returns the installer. Keep the two daily drivers' remotes in sync (daily-drivers rule). + +*** 2026-08-17 Mon @ 19:57:42 -0700 Re-checked: unstarted, and the exposure is confirmed live +Ran the task's own verification step as it stands today, which is the honest +way to check an unstarted task rather than reading its body back. Anonymous +=git ls-remote https://git.cjennings.net/archsetup.git= succeeded with no +credentials and returned =6faa31c= — this afternoon's HEAD. So the repo is +still world-cloneable and current to the commit, not a stale published +snapshot. + +=origin= on this machine is still =git@cjennings.net:archsetup.git=, the cgit +account, so nothing has moved. Everything in the plan stands unchanged. ** TODO [#B] Velox boot-failure retrospective — upgrade guard gaps :bug:zfs:maint: :PROPERTIES: :LAST_REVIEWED: 2026-07-21 @@ -1331,15 +1502,28 @@ Verify (manual, live): see Manual testing and validation. *** 2026-07-09 Thu @ 16:32:54 -0500 Audit reconcile: Phase 4 is filed on the dotfiles side, waiting on them The dotfiles project accepted the Phase 4 handoff and filed it as a =[#C]= task in their own =todo.org= (their note, 2026-07-08 16:56): the help-text audit + panel help affordance, the user-guide/README, and the ratio rollout doc. Not started there. They ping when it lands, and this task's Phase 4 child closes then. Nothing to do here meanwhile. -*** TODO Phase 4 — docs + rollout :network:blocked: -Deliverable: in-app help (=net --help= + per-command, panel help affordance); -README/user-guide (commands, indicator states, panel, config keys, make targets, -troubleshooting from the failure table, rollback); archsetup Hyprland dep install +*** 2026-08-17 Mon @ 19:57:42 -0700 Landed on the dotfiles side; the block is cleared +dotfiles shipped it as =138da7b= and closed its own task, so this one closes +with it and the =:blocked:= tag comes off. Found by checking their =todo.org= +rather than waiting for the ping — their close-out note says "archsetup pinged +so its Phase 4 task can close", so the handoff worked and only this end was +left open. + +All three acceptance criteria are met on their side: the help audit found and +fixed a stale =net repair= action list (nine of nineteen actions were named; +both the CLI help and =repair.py='s docstring now generate from the ACTIONS +registry), =net/README.md= covers every command plus the recovery targets, and +the ratio rollout is documented with both daily drivers verified current. + +They split the panel help affordance out rather than inventing it — no sibling +panel has one, so its shape is a design call. It is tracked on their side, not +here. + +Original deliverable, for the record: in-app help (=net --help= + per-command, +panel help affordance); README/user-guide; archsetup Hyprland dep install (=gtk4-layer-shell=, =python-gobject=, =speedtest-go-bin=); ratio manual dep + -stow step. -Verify: =net --help= and each subcommand complete; user-guide covers every command -+ the recovery targets. -Build handed off to the dotfiles project 2026-07-04 (=~/.dotfiles/inbox/2026-07-04-1305-from-archsetup-phase4-handoff.md=): archsetup deps confirmed installed, the remaining help/user-guide/rollout-doc work is in the net package. dotfiles pings back when it lands. +stow step. Handed off 2026-07-04 with the archsetup deps already confirmed +installed. *** TODO Phase 5 — VPN / WireGuard CLI fold (vNext) :network: Rescoped 2026-07-04 (audit): the tunnels track already shipped most of the original Phase 5. Panel tunnel bring-up/down and detection landed (dotfiles 2d9d060 probes tailscale/NM-wireguard/Proton; 21db05a brings overlays up/down from the panel's Tunnels sub-view; 31ba056 diagnose/doctor understand tunnel routes; archsetup 2e40781 wireguard config import; the net-panel-other-interfaces spec is IMPLEMENTED). What remains for Phase 5 is only the =net vpn ...= CLI subcommand — cli.py still has no vpn/tunnel parser. Fold the panel's existing tunnel operations into a CLI surface; spec separately when picked up. @@ -1601,6 +1785,29 @@ Add kernel parameter: ~rtc_cmos.use_acpi_alarm=1~ (will become systemd default) Consider: ~acpi_mask_gpe=0x1A~ for battery drain, suspend-then-hibernate config See Framework community notes on logind.conf and sleep.conf settings +*** 2026-08-17 Mon @ 19:57:42 -0700 Four of the five riders are done; WireGuard is the one left +The riders were written for "when velox returns from repair". It came back as +a full reinstall instead, and the installer carried most of them, so I checked +each on the live machine rather than reading the list back: + +- tlp radio-enable — done. =/etc/tlp.d/01-custom.conf:10= carries + =DEVICES_TO_ENABLE_ON_STARTUP="bluetooth wifi"=, written by the installer. +- touchpad auto-detection — the dotfiles half is done: =touchpad-auto + --detect= prints =pixa3854:00-093a:0274-touchpad=. Read that carefully + though — it names the device the config expects, not a device delivering + events. The touchpad is still dead on the ribbon fault, so this rider is + satisfied and the hardware still is not. +- podman socket — done, =podman.socket= is enabled. +- camera udev — done, =72-usb-passthrough-cameras.rules= is installed. +- *wolf WireGuard — not done, and it is the one that was time-critical.* No + =~/.config/wireguard/wolf.conf.gpg= and no WireGuard profile in + NetworkManager. The 08-08 decision set this up specifically so velox could + reach home from the road, on the argument that it is cheap at home and + expensive from a hotel. velox is now in the hotel. + +The suspend work itself is untouched — no kernel parameter, no drain +measurement. Only the riders moved. + ** TODO [#B] Manual testing and validation :test: :PROPERTIES: :LAST_REVIEWED: 2026-07-09 @@ -1609,6 +1816,47 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. +*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? +What we're verifying: that the IP-addressed NTP drop-in actually breaks the +bootstrap deadlock on a real cold start. This is the one test no agent can run — +it needs a full power-down, which is exactly the event that empties a failing +RTC. Everything else about the fix is verified; this is the part that rests on +construction (an address needs no DNS, NTP carries no certificate) rather than +on having been seen work. + +Do this before relying on it away from home — the failure mode strands the +machine with no network and no way to look anything up. +- Confirm the drop-in is in place and chrony is using it (block below). +- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only + loses time when the machine is actually off. +- Leave it off long enough to matter if the coin cell is the culprit (overnight + is the honest test; a few minutes may not drain anything). +- Power on. Do not touch the clock, do not run anything. Just log in and wait + about a minute. +- Run the verification block below. +#+begin_src sh :results output +echo "--- drop-in present? ---" +cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" +echo "--- is chrony reading it? ---" +grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" +echo "--- sources (the IP literal should be selected, marked ^*) ---" +chronyc -n sources +echo "--- clock ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' +echo "--- did DNS come back on its own? ---" +getent hosts gnu.org || echo "DNS STILL DEAD" +#+end_src +Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, +=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and +one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org +resolves — all without you having set the time. + +If the RTC came up wrong and the clock corrected itself anyway, the fix works +and the coin cell question is answered separately (a wrong RTC time in that +output means the cell is dying). If the clock is still wrong or DNS is still +dead, the fix did not hold: capture that whole block and promote this to a +top-level TODO. + *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. - Go to a workspace with 2-3 tiled windows in master. -- cgit v1.2.3 From 7b68f77a9b99e5400472986cb80bae5fb92e2320 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 12:32:07 -0700 Subject: docs: correct the clock/DNS deadlock mechanism to DNSSEC I reproduced the failure by winding velox's clock back 27 days with chronyd stopped, and the cause is not what I recorded. Resolved logged signature-expired against the root DNSKEY and every DS beneath it. The DoT handshake to 1.1.1.1:853 verified clean at that same clock, and the Cloudflare certificate runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window is days to weeks while a certificate is good for a year, so a skew that breaks DNSSEC normally leaves DoT untouched. DNSSEC=allow-downgrade does not rescue it either. Resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure, so no downgrade fires. Six retries over eighteen seconds plus a reset-server-features, all dead. I briefly believed otherwise off a test whose success was a cache hit. The fix itself is verified end to end. With the clock wound back and no DNS at all, chronyd reached the IP-addressed source and stepped the clock straight back. Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's build date to the minute, and systemd advances a garbage RTC to its own build epoch at boot. --- archsetup | 4 +- docs/design/2026-07-10-net-bt-failure-taxonomy.org | 4 +- todo.org | 99 ++++++++++++++++++++-- 3 files changed, 94 insertions(+), 13 deletions(-) (limited to 'todo.org') diff --git a/archsetup b/archsetup index 078408c..4bceda5 100755 --- a/archsetup +++ b/archsetup @@ -1203,8 +1203,8 @@ configure_build_environment() { # Bootstrap NTP sources addressed by IP, never by hostname. # # Arch's stock chrony.conf names its pool by hostname, and the DNS this - # installer configures later runs DNSOverTLS=yes with DNSSEC=yes. Both - # validate against the wall clock, so a machine that boots with a wrong + # installer configures later runs DNSSEC=yes, which validates signature + # windows against the wall clock, so a machine that boots with a wrong # clock resolves nothing: chrony cannot reach the pool, so the clock stays # wrong, so DNS stays dead. Neither side moves, and recovery needs a second # device to look up an NTP address by hand. An IP-addressed source needs no diff --git a/docs/design/2026-07-10-net-bt-failure-taxonomy.org b/docs/design/2026-07-10-net-bt-failure-taxonomy.org index 70421d0..4d57b86 100644 --- a/docs/design/2026-07-10-net-bt-failure-taxonomy.org +++ b/docs/design/2026-07-10-net-bt-failure-taxonomy.org @@ -96,7 +96,7 @@ Six layers, mirroring the net doctor's probe ladder (link → IP/DHCP → gatewa - Another daemon overwrites resolv.conf (yes). DNS works then breaks (or breaks after VPN up/down) as dhcpcd/openvpn/openresolv rewrites resolv.conf. Multiple tools claim it with no coordination. Fix: pick one manager (openresolv =resolvconf=NO=, dhcpcd =nohook resolv.conf=), point resolv.conf at the stub, restart resolved. [[https://github.com/adrienverge/openfortivpn/issues/674][openfortivpn 674]] - nsswitch.conf hosts line broken (yes). All resolution fails, or LAN/mDNS names never resolve; the hosts line lacks =resolve=/=dns= in the right order or references an uninstalled nss module. Fix: set =hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns=. [[https://man.archlinux.org/man/nss-resolve.8.en][nss-resolve]] - Avahi/.local mDNS not resolving (yes). *.local names don't resolve though unicast DNS works. nss-mdns not wired in, or resolved's built-in mDNS collides with avahi. Fix: install nss-mdns, add =mdns_minimal [NOTFOUND=return]= before =resolve=, enable avahi-daemon, disable resolved MulticastDNS if both run. [[https://wiki.archlinux.org/title/Avahi][archwiki avahi]] -- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSOverTLS=yes= validates the resolver's certificate and =DNSSEC=yes= validates RRSIG inception/expiry windows; both are wall-clock checks, so a clock weeks in the past fails every query before it leaves the machine. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. +- Clock skew breaks DNS itself, and NTP cannot recover it (yes; field-observed 2026-08-19, velox, not from the 2026-07-10 sweep). Nothing resolves at all — not a slow lookup, a dead one — after a boot with a wrong clock. =DNSSEC=yes= validates RRSIG inception/expiry windows against the wall clock, so a clock weeks off fails every query before it leaves the machine. Measured on velox 2026-08-19 with the clock wound back 27 days: resolved logged =signature-expired= against the root DNSKEY and every DS beneath it, and resolution died outright. =DNSOverTLS=yes= is *not* what bites, despite being the obvious suspect — the DoT handshake to =1.1.1.1:853= verified clean at that same clock, because a resolver certificate is good for about a year while an RRSIG window is days to weeks. A skew large enough to break DNSSEC normally leaves the certificate valid. The trap is the recovery path: NTP daemons name their servers by hostname (=pool 2.arch.pool.ntp.org=, =NTP=time.cloudflare.com=), so the daemon that would fix the clock needs the DNS that the clock is breaking. Neither side moves and the machine cannot self-heal — diagnosis needs a second device. Distinguish from the plain clock-skew entry in the egress layer by where it bites: that one has working DNS and failing HTTPS, this one has no DNS at all. Confirm with =dig @1.1.1.1 example.com +short=, which goes out plain UDP/53 and bypasses resolved entirely; an answer there with resolved still failing puts the fault in the validation layer, not the network. Fix: set the clock by hand (=timedatectl set-time=), then =resolvectl flush-caches=. =DNSSEC=allow-downgrade= does *not* help here, which is worth knowing because it is the obvious reach: resolved downgrades when a server lacks DNSSEC support, and a signature-window failure is a validation failure rather than a support failure, so no downgrade fires. Measured on velox: six retries over eighteen seconds, plus =resolvectl reset-server-features=, all dead. The only cure is correcting the clock, which is why the NTP source has to be reachable without DNS. Prevent by giving the NTP daemon at least one source addressed by IP, which needs neither DNS nor a certificate — =server 162.159.200.1 iburst= in a chrony drop-in. Note =timedatectl set-ntp true= is *not* a fix here: it starts a daemon that still cannot resolve its pool. ** Egress / captive portal / MTU / proxy / clock / upstream @@ -310,7 +310,7 @@ Probe: dns-config + resolver-health + dns-resolve + the doctor's dns-test (which - VPN split-DNS not applied :: AUTO — =resolvectl domain/default-route= on the VPN link. - IPv6 AAAA lookups stall :: AUTO — disable IPv6 on the link (or the single-request option). Also cluster 8. - Another daemon overwrites resolv.conf :: PRIV — pick one manager, point resolv.conf at the stub. -- Clock skew breaks DoT/DNSSEC, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. +- Clock skew breaks DNSSEC validation, NTP deadlocked behind it :: PRIV — set the clock by hand, flush caches; prevent with an IP-addressed NTP source. The doctor must reach this verdict *before* any resolved restart, which cannot help and reads as a loop. - nsswitch.conf hosts line / avahi mDNS broken :: PRIV — fix the hosts line, install nss-mdns. ** Cluster 6 — names resolve, egress blocked diff --git a/todo.org b/todo.org index 1bb810d..fa3252d 100644 --- a/todo.org +++ b/todo.org @@ -78,6 +78,41 @@ with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = [#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in this state it is fully offline with no local path out. +*** 2026-08-19 Wed @ 12:25:00 -0700 Reproduced it, and the mechanism was not what either of us said +I wound velox's clock back 27 days with chronyd stopped and watched it fail. +Resolution died outright, and plain UDP/53 to 1.1.1.1 kept answering throughout +— the discriminator the doctor keys on, confirmed live rather than reasoned. + +The cause is DNSSEC, not DNS-over-TLS. resolved logged =signature-expired= +against the root DNSKEY and every DS beneath it. The DoT handshake to +=1.1.1.1:853= verified clean at that same clock, and the Cloudflare certificate +runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window +is days to weeks and a certificate is good for a year, so a skew that breaks +DNSSEC normally leaves DoT untouched. The phone session blamed the certificate +and I carried that forward into the first commit; both were wrong. + +=DNSSEC=allow-downgrade= does not rescue it either, which matters because it is +the obvious reach and it is what ratio runs. resolved downgrades when a server +lacks DNSSEC support, and a signature-window failure is a validation failure, so +no downgrade fires. Six retries over eighteen seconds plus +=resolvectl reset-server-features=, all dead. I briefly believed otherwise off a +test whose success was a cache hit (=Data from: cache network=). + +So ratio was exposed after all, and I have given it the same drop-in. Its +=162.159.200.1= is selected and its clock is synchronized. + +The fix itself is verified end to end: with the clock wound back and no DNS at +all, chronyd reached the IP-addressed source and stepped the clock from +2026-07-23 straight back to 2026-08-19. That is the whole claim, demonstrated +rather than argued. + +Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's +build date to the minute (=/usr/lib/systemd/systemd=, 10:43:59), and systemd +advances a garbage RTC to its own build epoch at boot. Not timesyncd's +last-good-sync timestamp, which cannot be it — timesyncd is disabled here. That +also confirms the RTC really was reading earlier than that, so the coin cell +stays the prime suspect. + *** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed The installer carries the drop-in; =post-rebuild-check= grew a sixth check that fails a machine whose every NTP source is a hostname; the net failure taxonomy @@ -93,6 +128,51 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". +** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, +because the reason I changed it turned out to be false. It does not prevent the +clock deadlock. The IP-addressed NTP source does, and that is already in place +on both machines. + +What remains is a different question the taxonomy already documents: =DNSSEC=yes= +hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel +and airport problem and therefore velox's problem more than ratio's. +=allow-downgrade= trades authenticated answers for staying online. ratio already +runs =allow-downgrade=, so the fleet disagrees with itself and with the +installer, and I do not know whether ratio's setting was a deliberate policy or +a forgotten workaround for one bad network. + +I have not made this call. It is a security posture change and it should be made +knowingly rather than as a side effect of a theory I disproved an hour later. + +** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +Three defaults were chosen for a desktop and then applied to the machine that +travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per +connection (every hotel reconnect looks like a new device, so the portal login +starts over), and hostname-only NTP (the deadlock). Two are now fixed for every +machine, and the third is the VERIFY above. + +The installer already branches on battery presence in three places: +=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config +itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and +network policy simply does not use it. Wiring the same test around +=configure_networking= would let laptop and desktop defaults diverge deliberately +instead of by drift, and would stop the next instance of this from happening. + +Grading: Minor severity (nothing is broken today; this prevents a recurrence) x +some users sometimes (bites when a new default suits one machine class and not +the other) = P3 = [#C]. + ** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: :PROPERTIES: :CREATED: [2026-08-19 Wed] @@ -1817,15 +1897,16 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. *** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: that the IP-addressed NTP drop-in actually breaks the -bootstrap deadlock on a real cold start. This is the one test no agent can run — -it needs a full power-down, which is exactly the event that empties a failing -RTC. Everything else about the fix is verified; this is the part that rests on -construction (an address needs no DNS, NTP carries no certificate) rather than -on having been seen work. - -Do this before relying on it away from home — the failure mode strands the -machine with no network and no way to look anything up. +What we're verifying: the coin cell, not the fix. The fix itself is already +demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, +and chronyd reached the IP-addressed source and stepped it straight back. What a +cold boot adds is the hardware question: whether the RTC actually loses time +across a full power-down, which is the thing that started this. + +Read the outcome carefully, because only one branch is informative. An RTC time +that comes up wrong and then self-corrects tells you the cell is dying and the +fix is holding. An RTC that comes up correct tells you nothing about the +deadlock at all, only that the cell survived this particular night. - Confirm the drop-in is in place and chrony is using it (block below). - Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only loses time when the machine is actually off. -- cgit v1.2.3 From c409dd1644ed621f78a275809f791ec376e68779 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 14:32:42 -0700 Subject: chore(tasks): close the clock-deadlock work and file what it turned up Closes the deadlock task and its two follow-ups. The laptop-branch one is cancelled rather than done. Choosing allow-downgrade fleet-wide made all three motivating defaults identical on every machine, so the branch had nothing to put on either side. Three things filed from today. The RTC reset traced to an abrupt power loss at 01:33:18, recorded against the input-cover ribbon task as the cause rather than as a separate bug. The agent-text relay reports success for a page that never arrived, blocked on the shared-asset change being applied. A failed hostname lookup takes seven seconds, measured and set aside. --- todo.org | 248 ++++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 152 insertions(+), 96 deletions(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index fa3252d..4a34798 100644 --- a/todo.org +++ b/todo.org @@ -45,7 +45,69 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work -** TODO [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +** TODO [#B] agent-text relay reports success for a message that went nowhere :bug:blocked: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +rulesets accepted the relay-fallback change I sent on 2026-08-16 and parked it +for Craig's approval, since =agent-text= is a synced shared asset and does not +self-apply. Their review named a case neither side tested, and it is a real +defect in the shape I proposed: the relay loop breaks on rc 0, so a host that is +reachable but has no registered signal-cli account reports success for a message +that was never delivered. Paging is the one thing that must not fail silently, +because the whole point is reaching Craig when he is not watching. + +The fix is to verify delivery rather than trust the exit code, most likely by +checking =signal-cli listAccounts= on the relay host before sending, or by +treating an empty account store as a failure and continuing down the list. + +=:blocked:= on rulesets: the change is parked awaiting Craig's approval there, +and this defect should be corrected in the same apply rather than landing broken +and being patched after. Their prepared diff and review are in rulesets +=working/agent-text-relay-fallback/=. + +Two conditions from their review are theirs to carry, not mine: +=agent-text.bats= test 2 is written for the old single-host behavior and needs a +self-relay-skip case, and =protocols.org= "Reaching Craig" repeats the stale +primary-on-velox claim. + +Grading: Major severity (a page that silently does not arrive is worse than one +that fails loudly, and it is the channel used when Craig is away) x some users +sometimes (only when a relay host is up but unregistered, which is exactly +velox's current state) = P2 = [#B]. + +** TODO [#C] A failed hostname lookup takes seven seconds :bug: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +=getent hosts fake-vm= takes about 7.2 seconds to return not-found on velox. +Measured repeatedly with the cache flushed between runs. Anything that looks up +a name that does not exist pays it: an ssh typo, shell completion, a script +probing for a host. + +Not caused by the DNSSEC change. A/B measured today, cache flushed each time: +7691ms and 7232ms on =allow-downgrade= against 6804ms and 7482ms on =yes=, so +the setting makes no difference and this predates it. The likely shape is the +tailnet search domain (=search tailf3bb8c.ts.net=) being tried first, then the +two DoT upstreams, each with its own timeout, before NXDOMAIN comes back. + +Found because it blew a 20-second timeout in +=tests.net-scenarios.test_run_net_scenarios=, which shells out to ssh a +deliberately-bogus =root@fake-vm=. That suite passes on its own and the failure +did not recur, so the timeout needed this latency plus the DNS disruption from +the clock testing running alongside it. Worth knowing that the suite sits close +enough to the edge for a slow resolver to tip it. + +Grading: Minor severity (nothing behaves wrong, it just waits) x some users +sometimes (every failed lookup, which is occasional rather than constant) = P3 = +[#C]. + +** DONE [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 @@ -128,82 +190,69 @@ deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass query addressed by IP over plain UDP/53 to tell "resolved is refusing to validate" apart from "DNS is genuinely dead". -** VERIFY [#C] DNSSEC strictness on the travelling laptop :velox: +** DONE [#C] DNSSEC strictness on the travelling laptop :velox: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -I changed velox to =DNSSEC=allow-downgrade= today and then put it back to =yes=, -because the reason I changed it turned out to be false. It does not prevent the -clock deadlock. The IP-addressed NTP source does, and that is already in place -on both machines. - -What remains is a different question the taxonomy already documents: =DNSSEC=yes= -hard-fails against venue resolvers that mangle DNSSEC records, which is a hotel -and airport problem and therefore velox's problem more than ratio's. -=allow-downgrade= trades authenticated answers for staying online. ratio already -runs =allow-downgrade=, so the fleet disagrees with itself and with the -installer, and I do not know whether ratio's setting was a deliberate policy or -a forgotten workaround for one bad network. +Craig chose =allow-downgrade= everywhere. Applied to velox, ratio, and the +installer, and ratio's =DNSOverTLS= tightened from =opportunistic= to =yes= in +the same pass, so all three now agree: encrypted DNS always, validation +best-effort. -I have not made this call. It is a security posture change and it should be made -knowingly rather than as a side effect of a theory I disproved an hour later. +The reasoning that settled it: the deadlock is fixed by the IP-addressed NTP +source, and =allow-downgrade= was measured not to help with it at all. What +=allow-downgrade= does buy is the venue-resolver case the taxonomy documents, +where =yes= turns a resolver that mangles DNSSEC records into no answer at all. +That is a hotel and airport problem, so it is velox's problem, and the +encryption is the half worth being strict about. -** TODO [#C] Branch network policy on laptop vs desktop in the installer :feature: +** CANCELLED [#C] Branch network policy on laptop vs desktop in the installer :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -Three defaults were chosen for a desktop and then applied to the machine that -travels: =DNSSEC=yes= (hard-fails on venue resolvers), a fresh wifi MAC per -connection (every hotel reconnect looks like a new device, so the portal login -starts over), and hostname-only NTP (the deadlock). Two are now fixed for every -machine, and the third is the VERIFY above. +Cancelled because the decision above emptied it. All three motivating cases now +want the same value on every machine: =DNSSEC=allow-downgrade=, a stable +per-network wifi MAC, and an IP-addressed NTP source. A branch with nothing to +put on either side is machinery built for a divergence that does not exist, and +it would be the kind of scaffolding that rots unread. -The installer already branches on battery presence in three places: -=prune_waybar_battery=, the ppd mask in =configure_tlp_power=, and the TLP config -itself, all keyed on =ls /sys/class/power_supply/BAT*=. The precedent exists and -network policy simply does not use it. Wiring the same test around -=configure_networking= would let laptop and desktop defaults diverge deliberately -instead of by drift, and would stop the next instance of this from happening. +Worth keeping the observation, which is the part with a shelf life: when a +network default does need to differ by machine class, the test already exists. +=ls /sys/class/power_supply/BAT*= is what =prune_waybar_battery=, the ppd mask, +and the TLP config all key on. Reopen this then rather than building it now. -Grading: Minor severity (nothing is broken today; this prevents a recurrence) x -some users sometimes (bites when a new default suits one machine class and not -the other) = P3 = [#C]. - -** TODO [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +** DONE [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -The doctor now *names* the deadlock but hands the user two commands rather than -running anything — the verdict is =needs-user-action=. That was the honest call -at the time: correcting the clock needs the real time, and I could not exercise -a repair against the actual failure state without deliberately wedging velox's -network mid-session. - -An automated fix is possible, because NTP over UDP/123 needs neither DNS nor a -certificate. The shape would be =chronyc add server iburst= followed by -=chronyc makestep=, as two new privileged verbs in =priv.py= — chronyc talks to -a running chronyd over its socket, so this works in exactly the state that -blocks everything else. Two things to settle before building it: whether -=makestep= actually steps on the first sample after a runtime =add server= or -needs a poll first, and what the doctor does on a box running timesyncd rather -than chrony. - -Worth less now than it looks: a machine built by the current installer carries -the IP-addressed source and never reaches the deadlock. This is for machines -built before the fix. - -Grading: Minor severity (the doctor already names the fault correctly and hands -over a working remedy; only the automation is missing) x rare edge case (only -pre-fix machines with a broken clock) = P4 = [#D]... except that a user in this -state has no working network and cannot look anything up, which makes the -two-command handoff harder to follow than it reads. Minor x rare = P4, and I am -leaving it at [#C] rather than [#D] because it sits one step from done. +Shipped as the =clock-ip-ntp= repair, and the verdict is =fixable= rather than +terminal. Both open questions got answered by driving a real deadlock instead of +reasoning about it: =chronyc add server= returns =200 OK= against a running +chronyd, and =makestep= needs a sample to land, so it took four calls and about +eight seconds rather than working on the first. The repair retries accordingly. + +Verified end to end on velox against a genuine deadlock (wrong clock, chronyd +running with only an unresolvable hostname source, DNS dead): the repair +corrected the clock in 6.1 seconds and DNS came back. + +The live run also caught a defect no unit test would have. The doctor reported +"Saved password for SpectrumSetup-3C was rejected" — because +=_recent_auth_failure= greps =journalctl --since -5min=, and a clock weeks off +windows onto a different incident's entries. It would have sent Craig to +re-enter a password that was never wrong. Fixed twice over: the journal half is +now skipped when the clock is untrustworthy, and the clock verdict is ordered +above the auth verdict, since everything below it reasons over timestamps that +only mean something once the clock is right. Airplane mode and hard rfkill stay +above, being physical states the clock has no bearing on. ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> @@ -240,6 +289,28 @@ Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. +*** 2026-08-19 Wed @ 12:45:00 -0700 This is what stranded the machine, and the failure got worse +The 2026-08-19 network outage traces back here. Boot -3 ended at 01:33:18 with +no shutdown sequence at all: no power-off target, no unmounting, just a +coredump backtrace and then nothing. The next boot's kernel line reads +=rtc_cmos 00:01: setting system clock to 2025-01-01T00:00:16 UTC=. The RTC was +reset to a firmware default, not drifted, and no firmware update was applied +(=fwupdmgr get-history= is empty). + +That is a different and worse symptom than the phantom short-press logged +before. Those produced =Power key pressed short= and an orderly poweroff. This +one logged no button event and no shutdown, which reads as the board losing +power outright rather than being asked to stop. Whether it is the same +connector, I can't prove from the journal, but it is the same machine with a +known loose input-cover ribbon and a battery that was not flat (100% now). + +Consequence, which is the part that makes this worth the [#A]: the RTC reset +put the clock at 2025-01-01, systemd advanced it to its own build epoch of +2026-07-23, DNSSEC rejected every signature as out-of-window, and the machine +lost DNS entirely. Craig had to diagnose it from his phone. The NTP drop-in +landed today makes a recurrence self-healing, but it treats the symptom. This +task is the cause, and it is still open. + *** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree The reseat did not happen before the flight, and velox is travelling. The deadline blew past on 08-14. @@ -1896,47 +1967,32 @@ Craig's standing checklist of everything that isn't agent-verifiable. Each child Priority and type tag added by that audit: the task carried neither, which kept the project's largest live container out of the agenda entirely. -*** Clock/DNS deadlock: does velox recover its clock from a cold boot, untouched? -What we're verifying: the coin cell, not the fix. The fix itself is already -demonstrated — on 2026-08-19 I wound the clock back 27 days with no DNS at all, -and chronyd reached the IP-addressed source and stepped it straight back. What a -cold boot adds is the hardware question: whether the RTC actually loses time -across a full power-down, which is the thing that started this. - -Read the outcome carefully, because only one branch is informative. An RTC time -that comes up wrong and then self-corrects tells you the cell is dying and the -fix is holding. An RTC that comes up correct tells you nothing about the -deadlock at all, only that the cell survived this particular night. -- Confirm the drop-in is in place and chrony is using it (block below). -- Shut all the way down — =poweroff=, not suspend, not reboot. The RTC only - loses time when the machine is actually off. -- Leave it off long enough to matter if the coin cell is the culprit (overnight - is the honest test; a few minutes may not drain anything). -- Power on. Do not touch the clock, do not run anything. Just log in and wait - about a minute. -- Run the verification block below. +*** Clock/DNS deadlock: does the next abrupt power loss strand velox again? +What we're verifying: that the machine survives an RTC reset unattended. Not the +coin cell, which is new with the 2026-08-13 mainboard and is ruled out. The RTC +did not drift on 2026-08-19, it was reset to exactly 2025-01-01T00:00:16 by an +abrupt power loss at 01:33:18 that left no shutdown sequence in the journal. + +This one can't be scheduled. Run the block the next time velox comes up after an +unexpected power loss, before touching the clock. #+begin_src sh :results output -echo "--- drop-in present? ---" -cat /etc/chrony.d/10-bootstrap-ip-ntp.conf 2>/dev/null || echo "MISSING" -echo "--- is chrony reading it? ---" -grep -n 'confdir' /etc/chrony.conf || echo "no confdir — drop-in is NOT being read" -echo "--- sources (the IP literal should be selected, marked ^*) ---" +echo "--- what did the RTC read at this boot? ---" +journalctl -b 0 | grep -m1 'rtc_cmos.*setting system clock' +echo "--- did systemd have to advance the clock to its build epoch? ---" +journalctl --list-boots | tail -3 +echo "--- sources: is an IP-addressed one selected? ---" chronyc -n sources -echo "--- clock ---" -timedatectl | grep -iE 'Local time|RTC time|synchronized|NTP service' -echo "--- did DNS come back on its own? ---" -getent hosts gnu.org || echo "DNS STILL DEAD" +echo "--- clock + DNS ---" +timedatectl | grep -iE 'Local time|RTC time|synchronized' +getent hosts gnu.org || echo "DNS DEAD" #+end_src -Expected: the drop-in is present, chrony.conf carries =confdir /etc/chrony.d=, -=chronyc -n sources= shows =162.159.200.1= or =162.159.200.123= reachable and -one of them selected (=^*=), =System clock synchronized: yes=, and gnu.org -resolves — all without you having set the time. - -If the RTC came up wrong and the clock corrected itself anyway, the fix works -and the coin cell question is answered separately (a wrong RTC time in that -output means the cell is dying). If the clock is still wrong or DNS is still -dead, the fix did not hold: capture that whole block and promote this to a -top-level TODO. +Expected: even if the RTC came up at 2025-01-01 and systemd advanced the clock +to 2026-07-23, chrony reached 162.159.200.1 without DNS, stepped the clock to +now, and names resolve. You did nothing. + +If instead the clock is still wrong or DNS is dead, the fix did not hold in the +field despite holding under a simulated skew. Capture that whole block and +promote this to a top-level TODO. *** Floating layout: freeze positions, border flash, glyph, exit to master What we're verifying: the rebuilt floating mode (Super+Shift+F) floats every window on the workspace via per-window setfloating (the old workspaceopt allfloat was deprecated and no-op'd, which is why nothing floated), freezes each in place, flashes the border gold on entry and exit, flips the waybar glyph to the floating icon, and exits to master. Live-verified on a headless output already (windows floated in place, dragged to overlap, glyph read Floating, toggled back clean); this is the on-your-own-monitor confirmation. -- cgit v1.2.3 From afd1f386c2769a1e8b3bc8f388a0dfaec0fd27e7 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Wed, 19 Aug 2026 14:50:12 -0700 Subject: chore(tasks): claim the roam captures and record the scaling diagnosis Four items claimed from the shared roam inbox: - Qt apps oversized on velox - function keys issuing media actions instead of F-keys - waybar panels launching expanded - a question about the analog clock avoiding being dragged The last is a VERIFY, since the capture asks for a conversation rather than describing a defect. Two unowned product links stay in the shared inbox. The Qt one is diagnosed and fixed in dotfiles ec8e9a8, recorded here with the measurement. A VERIFY rides along on whether the calibre DPI pin is still needed, because it looks like the same double-scaling worked around per-app. The cursor comment in that file records it a third time. Also closes the net-scenarios timeout, which predates today by a month. It guessed at a larger timeout or a real hang in the script and it was neither. The stubbed ssh drains stdin, so the tests hung whenever the runner had a live one. --- todo.org | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 174 insertions(+), 45 deletions(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index 4a34798..6f16129 100644 --- a/todo.org +++ b/todo.org @@ -45,38 +45,133 @@ below): input-side-spec.org (DRAFT, four decisions open). * Archsetup Open Work -** TODO [#B] agent-text relay reports success for a message that went nowhere :bug:blocked: +** TODO [#B] Qt apps render oversized on velox :bug:velox:solo: :PROPERTIES: :CREATED: [2026-08-19 Wed] :LAST_REVIEWED: 2026-08-19 :END: -rulesets accepted the relay-fallback change I sent on 2026-08-16 and parked it -for Craig's approval, since =agent-text= is a synced shared asset and does not -self-apply. Their review named a case neither side tested, and it is a real -defect in the shape I proposed: the relay loop breaks on rc 0, so a host that is -reachable but has no registered signal-cli account reports success for a message -that was never delivered. Paging is the one thing that must not fail silently, -because the whole point is reaching Craig when he is not watching. +From the roam inbox, Craig's words: "qt apps look huge on velox. how do we make +it look better on this particular machine, and not change ratio. it seems they +should have different QT configs." + +The shape is per-machine Qt scaling. velox is a high-DPI Framework panel and +ratio drives ordinary-DPI monitors, so one global Qt scale factor cannot suit +both. The fix has to be host-scoped rather than a value written into the shared +config, which is the same tier split the dotfiles already use. + +Grading: Minor severity (apps work, they are just the wrong size) x every user +every time (every Qt app launch on velox) = P2 = [#B]. + +*** 2026-08-19 Wed @ 15:05:00 -0700 Root cause found and fixed; needs a logout to take effect +velox's =conf.d/local.conf= scaled the panel twice. The monitor line sets +=1.566667= and the same file exported =QT_SCALE_FACTOR,1.5= and =GDK_SCALE,1.5=, +and Qt 6 on Wayland already takes its scale from the compositor, so the two +multiplied. Measured rather than reasoned: with the override Qt reports a +960x640 logical screen, without it 1440x960, and 2256/1.566667 is exactly 1440. +That is 1.5x too large, which matches "huge" precisely. + +Those env lines were not careless. The comment above them explains they existed +to compensate for =xwayland:force_zero_scaling = true= in the shared +hyprland.conf, which makes XWayland clients render unscaled and tiny. The +approach was what failed: an env var reaches every app, so fixing XWayland broke +every native Wayland client. Removing the vars alone would have traded "Qt huge" +for "Zoom tiny", so velox now turns =force_zero_scaling= off for itself instead. +XWayland scales through the compositor there, coming out correctly sized and +slightly soft. ratio is untouched and needs nothing, its monitor being scale 1. + +=force_zero_scaling= took effect on =hyprctl reload=. The env removal will not: +Hyprland applies =env== lines with setenv at parse time and never unsets them, +so the running compositor still hands 1.5 to everything it spawns. Craig has to +log out and back in. + +*** VERIFY Is CALIBRE_OVERRIDE_DPI still needed after the scaling fix? +The same file pins =CALIBRE_OVERRIDE_DPI,96= with the comment "calibre renders +oversized at the 1.57 compositor scale". Calibre is a Qt app, so that was almost +certainly this same double-scaling seen through one application and worked +around per-app rather than at the root. With the multiplier gone, the pin is +probably redundant and may now render calibre too small. + +Left in place rather than removed on a guess, since it was validated at 96 on +2026-06-27 and calibre has its own DPI handling. Worth opening calibre after the +next login and deciding by eye. + +The cursor entry in the same file records this identical failure a third time: +"Pre-scaling it (the old 36 = 24 x 1.5) double-applied on top of the +compositor's scale." Three instances of one mistake in one file, two previously +fixed in isolation without anyone naming the pattern. + +** TODO [#B] Function keys issue media actions instead of F-keys :bug:velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +From the roam inbox, Craig's words: "function keys should issue F+number +functionality rather than their media functionality when the button is hit. +currently it's reversed and I have to hit function and the f button for F+number +functionality." + +Check first whether this belongs to archsetup at all. On a Framework the Fn-lock +is a firmware-level toggle held in the keyboard itself (Fn+Esc on most +revisions), not something the OS sets, in which case this is one keystroke +rather than a change here. If it is instead a hid/keyboard-module quirk, it is +ours. + +Grading: Minor severity (the keys work, they are on the wrong layer, and there +is a workaround) x every user every time (every F-key press) = P2 = [#B]. + +** TODO [#C] Waybar panels launch expanded instead of collapsed :bug:dotfiles:waybar: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: -The fix is to verify delivery rather than trust the exit code, most likely by -checking =signal-cli listAccounts= on the relay host before sending, or by -treating an empty account store as a failure and continuing down the list. +From the roam inbox, Craig's words: "waybar panels should start up collapsed. +currently both the left and the right waybar panels launch expanded." -=:blocked:= on rulesets: the change is parked awaiting Craig's approval there, -and this defect should be corrected in the same apply rather than landing broken -and being patched after. Their prepared diff and review are in rulesets -=working/agent-text-relay-fallback/=. +Panel source is =~/.dotfiles=. Its heading in the roam inbox read "archsetup." +with a period rather than a colon, so the routing prefix did not match cleanly; +claimed on the plain reading of the text. -Two conditions from their review are theirs to carry, not mine: -=agent-text.bats= test 2 is written for the old single-host behavior and needs a -self-relay-skip case, and =protocols.org= "Reaching Craig" repeats the stale -primary-on-velox claim. +Grading: Cosmetic severity (presentation only, nothing is lost) x every user +every time (every session start) = P3 = [#C]. -Grading: Major severity (a page that silently does not arrive is worse than one -that fails loudly, and it is the channel used when Craig is away) x some users -sometimes (only when a relay host is up but unregistered, which is exactly -velox's current state) = P2 = [#B]. +** VERIFY [#C] The visible analog clock avoids being dragged :velox: +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +From the roam inbox, captured verbatim: "the visible analog clock avoids being +dragged. ask me about this." + +Filed as a VERIFY because the capture asks for a conversation rather than +describing a defect. What is the clock avoiding being dragged by, and is the +avoidance the bug or the intended behaviour? + +** CANCELLED [#B] agent-text relay reports success for a message that went nowhere :bug: +CLOSED: [2026-08-19 Wed] +:PROPERTIES: +:CREATED: [2026-08-19 Wed] +:LAST_REVIEWED: 2026-08-19 +:END: + +Not a defect. rulesets refuted it with measurements and I reproduced theirs +before accepting: on velox, whose account store is empty, +=signal-cli -a +15550000000 send= exits 1 with "User +15550000000 is not +registered", and =ssh 100.71.182.1 'exit 7'= returns 7, so a non-zero code +propagates faithfully back through the relay. The loop's +=[ "$rc" -eq 0 ] && break= therefore advances to the next host exactly as +intended. signal-cli fails closed. + +I filed this off a conditional in their handoff — ".emacs.d raised a case +neither of you tested ... *if* signal-cli send exits zero against an empty +account store" — and turned the "if" into a graded [#B] with a =:blocked:= tag +on another project, without running the one command that settles it. The +machine that proves it was in front of me the whole time. Their ask is fair and +I am recording it rather than the outcome alone: verify before filing a defect +against someone else's work, especially one carrying a blocking tag. ** TODO [#C] A failed hostname lookup takes seven seconds :bug: :PROPERTIES: @@ -289,27 +384,42 @@ Must be done before the Sunday flight — a phantom press mid-travel with the shield on is survivable, but the connector should not be trusted at 30,000 feet on the loose setting. -*** 2026-08-19 Wed @ 12:45:00 -0700 This is what stranded the machine, and the failure got worse -The 2026-08-19 network outage traces back here. Boot -3 ended at 01:33:18 with -no shutdown sequence at all: no power-off target, no unmounting, just a -coredump backtrace and then nothing. The next boot's kernel line reads -=rtc_cmos 00:01: setting system clock to 2025-01-01T00:00:16 UTC=. The RTC was -reset to a firmware default, not drifted, and no firmware update was applied -(=fwupdmgr get-history= is empty). - -That is a different and worse symptom than the phantom short-press logged -before. Those produced =Power key pressed short= and an orderly poweroff. This -one logged no button event and no shutdown, which reads as the board losing -power outright rather than being asked to stop. Whether it is the same -connector, I can't prove from the journal, but it is the same machine with a -known loose input-cover ribbon and a battery that was not flat (100% now). - -Consequence, which is the part that makes this worth the [#A]: the RTC reset -put the clock at 2025-01-01, systemd advanced it to its own build epoch of -2026-07-23, DNSSEC rejected every signature as out-of-window, and the machine -lost DNS entirely. Craig had to diagnose it from his phone. The NTP drop-in -landed today makes a recurrence self-healing, but it treats the symptom. This -task is the cause, and it is still open. +*** 2026-08-19 Wed @ 14:40:00 -0700 Retracted: the RTC reset is not this task's, and I should not have filed it here +I attributed the 2026-08-19 network outage to this ribbon earlier today. Craig +pushed back — he reseated it before the trip to get the touchpad working — and +he is right. The evidence does not support the attribution and some of it points +the other way. + +What actually holds. Boot -3 ended at 01:33:18 with no shutdown sequence: no +power-off target, no unmounting. The next boot's kernel line reads =rtc_cmos +00:01: setting system clock to 2025-01-01T00:00:16 UTC=, a firmware default, so +the RTC was reset rather than drifted. No firmware update was applied +(=fwupdmgr get-history= is empty) and the battery is fine. + +What refutes the ribbon. This boot logged *zero* =Power key pressed= events, and +so did the four boots before it. The phantom-press symptom had genuinely stopped +after 08-15, exactly as the 08-16 session recorded. The earlier events logged a +power-key press and an orderly poweroff; this logged neither, which makes it a +different signature, not a worse version of the same one. + +What I got wrong methodologically: I anchored on the most salient open hardware +task and read association as evidence. I even wrote "I can't prove it is the +same connector" and then filed it here anyway, which is the tell. + +Two things I checked and can rule out. There were no OOM kills — the 3,433 +matching lines are a systemd unit named "Periodically re-score Claude Code +processes for the OOM-killer" firing on a timer, not memory pressure, and there +is not a single "Killed process" line. Thermal is clean; the only mentions are +boot-time zone registration at 34C and 45C. + +One real thing the same window did surface, tracked separately: a python3 crash +loop, 251 core dumps in the final ten minutes, SIGABRT with =XFreeThreads= and +=PyEval_RestoreThread= in the trace. It does not explain the RTC, because +software cannot clear it, but it is its own problem. + +The open question that would settle the RTC is for Craig, not the journal: a +long power-button hold on a Framework triggers an EC-level reset that clears the +RTC, which fits a wedged machine being forced off. A 4-second hold would not. *** 2026-08-17 Mon @ 19:57:42 -0700 Not done, and the two symptoms now disagree The reseat did not happen before the flight, and velox is travelling. The @@ -1029,7 +1139,8 @@ Handoff from home (2026-07-25), originally combining the 2026-06-07 stale-compos 2. On =Upgrade= of =fontconfig=, =freetype2=, or =harfbuzz=, run =/usr/bin/fc-cache -f= after the transaction. The fontconfig 2.17→2.18 cache-format change left stale cache-9 files that crashed Qt6 apps in =FcCharSetHasChar= until the system font cache was rebuilt. Acceptance: hook files are source-controlled and installed by archsetup; package/operation/action fields are asserted from the generated hook text; the reminder is print-only and exits successfully; the font hook runs only after successful matching upgrades and invokes the absolute =fc-cache= path. Validate with the fast installer tests plus a disposable pacman-hook parser/install check when practical. -** TODO [#D] net-scenarios harness times out under back-to-back suite runs :test:tooling: +** DONE [#D] net-scenarios harness times out under back-to-back suite runs :test:tooling: +CLOSED: [2026-08-19 Wed] :PROPERTIES: :LAST_REVIEWED: 2026-07-24 :END: @@ -1046,6 +1157,24 @@ Standalone immediately after: 0.095s, all pass; the following quiet-machine full run was clean. Confirms the load-sensitivity read — reproduce under deliberate load before choosing between a bigger budget and a real hang. + +*** 2026-08-19 Wed @ 14:50:00 -0700 Root-caused and fixed: inherited stdin, not load +Not load, and not the network. The harness stubs ssh as =cat >/dev/null=, which +drains stdin to EOF. With no explicit stdin the stub inherits whatever the test +runner had, so it returned instantly when stdin was redirected and blocked +forever when it was a terminal or a live pipe. All five tests then burned their +20-second budget. + +That is why it looked like a load effect: a run launched immediately after +another inherited a different stdin than a standalone invocation. A/B measured +today — =make test-unit Date: Wed, 19 Aug 2026 14:56:37 -0700 Subject: chore(tasks): archive the closed work and convert finished sub-tasks Routine cleanup. The net-scenarios timeout subtree moves to the archive now that it is closed, an aged clock-panel subtree goes with it, and finished sub-tasks are rewritten as dated log entries under their parents. --- archive/task-archive.org | 3 + todo.org | 411 +++++++++++++++++++++++------------------------ 2 files changed, 204 insertions(+), 210 deletions(-) (limited to 'todo.org') diff --git a/archive/task-archive.org b/archive/task-archive.org index 42d9653..2925180 100644 --- a/archive/task-archive.org +++ b/archive/task-archive.org @@ -1624,3 +1624,6 @@ Retrofit the net doctor (=~/.dotfiles/net/src/net/doctor.py=) and bluetooth doct Scope note (Craig, 2026-07-07): realtime lamp *behavior* only. The maintenance console's wider results-wall layout (date+time stamp column, COPY, persistent history) does NOT backport — the net/bt panels are ~400px wide and lack the horizontal real estate. Their existing output wells keep their compact layout; this task just makes them stream live. Addendum (Craig, 2026-07-07): DO backport the 3.5-entry height convention — every panel's output well caps at 3.5 visible entries, the half-visible entry being the scroll cue, with the dark slate-on-black scrollbar. Layout stays compact per above; only the height cap + scroll affordance carries over. +** DONE [#B] Absorb the clock-panel project into the dotfiles :feature:waybar:dotfiles: +CLOSED: [2026-07-18 Sat] +Absorbed into =~/.dotfiles= (commit 3fab11d): package =clock/src/clock/= (renamed from clock_panel), the six PNG watchface layers packaged inside the module at =clock/src/clock/assets/=, a stowed =clock-panel= shell shim (LD_PRELOADs gtk4-layer-shell), waybar left-click now =clock-panel toggle= with the absolute path dropped, tests converted pytest→unittest into =tests/clock/= plus an asset-load guard. Kept the layer-shell overlay and the socket toggle. The standalone repo is archived (ARCHIVED.md), kept for its design history. Verified live: the bar click renders the polished watchface. diff --git a/todo.org b/todo.org index 6f16129..af8d75a 100644 --- a/todo.org +++ b/todo.org @@ -150,29 +150,6 @@ Filed as a VERIFY because the capture asks for a conversation rather than describing a defect. What is the clock avoiding being dragged by, and is the avoidance the bug or the intended behaviour? -** CANCELLED [#B] agent-text relay reports success for a message that went nowhere :bug: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:CREATED: [2026-08-19 Wed] -:LAST_REVIEWED: 2026-08-19 -:END: - -Not a defect. rulesets refuted it with measurements and I reproduced theirs -before accepting: on velox, whose account store is empty, -=signal-cli -a +15550000000 send= exits 1 with "User +15550000000 is not -registered", and =ssh 100.71.182.1 'exit 7'= returns 7, so a non-zero code -propagates faithfully back through the relay. The loop's -=[ "$rc" -eq 0 ] && break= therefore advances to the next host exactly as -intended. signal-cli fails closed. - -I filed this off a conditional in their handoff — ".emacs.d raised a case -neither of you tested ... *if* signal-cli send exits zero against an empty -account store" — and turned the "if" into a graded [#B] with a =:blocked:= tag -on another project, without running the one command that settles it. The -machine that proves it was in front of me the whole time. Their ask is fair and -I am recording it rather than the outcome alone: verify before filing a defect -against someone else's work, especially one carrying a blocking tag. - ** TODO [#C] A failed hostname lookup takes seven seconds :bug: :PROPERTIES: :CREATED: [2026-08-19 Wed] @@ -201,154 +178,6 @@ Grading: Minor severity (nothing behaves wrong, it just waits) x some users sometimes (every failed lookup, which is occasional rather than constant) = P3 = [#C]. -** DONE [#B] Clock/DNS bootstrap deadlock — recovery needs a second device :bug:velox: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:CREATED: [2026-08-19 Wed] -:LAST_REVIEWED: 2026-08-19 -:END: - -The installer wrote both halves of a deadlock. =configure_dns= pins -=DNSOverTLS=yes= with =DNSSEC=yes=, and both validate against the wall clock; -the chrony step enables chronyd without writing a config, so the machine runs -Arch's stock one whose only source is =pool 2.arch.pool.ntp.org= — a hostname. -Boot with a wrong clock and DoT certificate validation fails, so nothing -resolves; chrony then cannot resolve its pool, so the clock stays wrong. -Neither side moves. It caught velox on the road 2026-08-19 and had to be -diagnosed from a phone. - -Fixed at the root: the installer now writes -=/etc/chrony.d/10-bootstrap-ip-ntp.conf= with two IP-addressed Cloudflare -sources and points stock chrony.conf at the drop-in. An address needs no DNS -and carries no certificate, so the escape hatch holds whatever broke the clock. -velox has the same drop-in applied live, verified with =chronyc -n sources= -(=162.159.200.1= selected) and =timedatectl= reporting synchronized. - -What is left here is the part I could not verify: the decisive test is a full -power-down and cold boot, confirming the clock corrects itself untouched. See -the manual-testing entry. Until that runs, the fix is sound by construction -rather than demonstrated. - -Grading: Critical severity (total loss of network — no DNS means no egress, and -recovery needs a second device) x some users sometimes (only machines that boot -with a wrong clock, which is any RTC fault, BIOS reset, or drained cell) = P2 = -[#B]. Graded on the being-in-it, not the getting-into-it: once the machine is in -this state it is fully offline with no local path out. - -*** 2026-08-19 Wed @ 12:25:00 -0700 Reproduced it, and the mechanism was not what either of us said -I wound velox's clock back 27 days with chronyd stopped and watched it fail. -Resolution died outright, and plain UDP/53 to 1.1.1.1 kept answering throughout -— the discriminator the doctor keys on, confirmed live rather than reasoned. - -The cause is DNSSEC, not DNS-over-TLS. resolved logged =signature-expired= -against the root DNSKEY and every DS beneath it. The DoT handshake to -=1.1.1.1:853= verified clean at that same clock, and the Cloudflare certificate -runs Dec 2025 to Dec 2026, so it was never outside its window. An RRSIG window -is days to weeks and a certificate is good for a year, so a skew that breaks -DNSSEC normally leaves DoT untouched. The phone session blamed the certificate -and I carried that forward into the first commit; both were wrong. - -=DNSSEC=allow-downgrade= does not rescue it either, which matters because it is -the obvious reach and it is what ratio runs. resolved downgrades when a server -lacks DNSSEC support, and a signature-window failure is a validation failure, so -no downgrade fires. Six retries over eighteen seconds plus -=resolvectl reset-server-features=, all dead. I briefly believed otherwise off a -test whose success was a cache hit (=Data from: cache network=). - -So ratio was exposed after all, and I have given it the same drop-in. Its -=162.159.200.1= is selected and its clock is synchronized. - -The fix itself is verified end to end: with the clock wound back and no DNS at -all, chronyd reached the IP-addressed source and stepped the clock from -2026-07-23 straight back to 2026-08-19. That is the whole claim, demonstrated -rather than argued. - -Also settled: the clock landed on 2026-07-23 because that is systemd 261.2's -build date to the minute (=/usr/lib/systemd/systemd=, 10:43:59), and systemd -advances a garbage RTC to its own build epoch at boot. Not timesyncd's -last-good-sync timestamp, which cannot be it — timesyncd is disabled here. That -also confirms the RTC really was reading earlier than that, so the coin cell -stays the prime suspect. - -*** 2026-08-19 Wed @ 10:12:00 -0700 Root fix, doctor verdict, and taxonomy entry landed -The installer carries the drop-in; =post-rebuild-check= grew a sixth check that -fails a machine whose every NTP source is a hostname; the net failure taxonomy -gained the mode in its DNS layer plus a cluster 5 triage line, and its existing -egress-layer clock entry now says outright that its remedy does not apply when -DoT or DNSSEC is on. - -The doctor half is in dotfiles: =classify.py= reached "DNS not resolving → net -repair dns-test" here, which cannot help, because every public resolver fails -the same clock-sensitive validation — so the doctor sent you round a loop. It -now emits a =clock-dns= row ahead of the generic DNS verdict. Detection is -deliberately DNS-free: a local =timedatectl= read for sync state, and a bypass -query addressed by IP over plain UDP/53 to tell "resolved is refusing to -validate" apart from "DNS is genuinely dead". - -** DONE [#C] DNSSEC strictness on the travelling laptop :velox: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:CREATED: [2026-08-19 Wed] -:LAST_REVIEWED: 2026-08-19 -:END: - -Craig chose =allow-downgrade= everywhere. Applied to velox, ratio, and the -installer, and ratio's =DNSOverTLS= tightened from =opportunistic= to =yes= in -the same pass, so all three now agree: encrypted DNS always, validation -best-effort. - -The reasoning that settled it: the deadlock is fixed by the IP-addressed NTP -source, and =allow-downgrade= was measured not to help with it at all. What -=allow-downgrade= does buy is the venue-resolver case the taxonomy documents, -where =yes= turns a resolver that mangles DNSSEC records into no answer at all. -That is a hotel and airport problem, so it is velox's problem, and the -encryption is the half worth being strict about. - -** CANCELLED [#C] Branch network policy on laptop vs desktop in the installer :feature: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:CREATED: [2026-08-19 Wed] -:LAST_REVIEWED: 2026-08-19 -:END: - -Cancelled because the decision above emptied it. All three motivating cases now -want the same value on every machine: =DNSSEC=allow-downgrade=, a stable -per-network wifi MAC, and an IP-addressed NTP source. A branch with nothing to -put on either side is machinery built for a divergence that does not exist, and -it would be the kind of scaffolding that rots unread. - -Worth keeping the observation, which is the part with a shelf life: when a -network default does need to differ by machine class, the test already exists. -=ls /sys/class/power_supply/BAT*= is what =prune_waybar_battery=, the ppd mask, -and the TLP config all key on. Reopen this then rather than building it now. - -** DONE [#C] Automate the clock/DNS deadlock repair in the net doctor :feature: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:CREATED: [2026-08-19 Wed] -:LAST_REVIEWED: 2026-08-19 -:END: - -Shipped as the =clock-ip-ntp= repair, and the verdict is =fixable= rather than -terminal. Both open questions got answered by driving a real deadlock instead of -reasoning about it: =chronyc add server= returns =200 OK= against a running -chronyd, and =makestep= needs a sample to land, so it took four calls and about -eight seconds rather than working on the first. The repair retries accordingly. - -Verified end to end on velox against a genuine deadlock (wrong clock, chronyd -running with only an unresolvable hostname source, DNS dead): the repair -corrected the clock in 6.1 seconds and DNS came back. - -The live run also caught a defect no unit test would have. The doctor reported -"Saved password for SpectrumSetup-3C was rejected" — because -=_recent_auth_failure= greps =journalctl --since -5min=, and a clock weeks off -windows onto a different incident's entries. It would have sent Craig to -re-enter a password that was never wrong. Fixed twice over: the journal half is -now skipped when the clock is untrustworthy, and the clock verdict is ordered -above the auth verdict, since everything below it reasons over timestamps that -only mean something once the clock is right. Airplane mode and hard rfkill stay -above, being physical states the clock has no bearing on. - ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: DEADLINE: <2026-08-14 Fri> :PROPERTIES: @@ -1139,42 +968,6 @@ Handoff from home (2026-07-25), originally combining the 2026-06-07 stale-compos 2. On =Upgrade= of =fontconfig=, =freetype2=, or =harfbuzz=, run =/usr/bin/fc-cache -f= after the transaction. The fontconfig 2.17→2.18 cache-format change left stale cache-9 files that crashed Qt6 apps in =FcCharSetHasChar= until the system font cache was rebuilt. Acceptance: hook files are source-controlled and installed by archsetup; package/operation/action fields are asserted from the generated hook text; the reminder is print-only and exits successfully; the font hook runs only after successful matching upgrades and invokes the absolute =fc-cache= path. Validate with the fast installer tests plus a disposable pacman-hook parser/install check when practical. -** DONE [#D] net-scenarios harness times out under back-to-back suite runs :test:tooling: -CLOSED: [2026-08-19 Wed] -:PROPERTIES: -:LAST_REVIEWED: 2026-07-24 -:END: -=tests/net-scenarios/test_run_net_scenarios.py= errored on all 5 tests twice during round 12, each time =subprocess.TimeoutExpired= after its 20s budget on =scripts/testing/run-net-scenarios.sh --target root@fake-vm=. Both occurrences were in =make test-unit= runs launched immediately after a previous full run. It then passed 6 runs in a row (3 on a pristine tree, 3 with the round-12 change), and standalone it finishes in 0.09s, so this is not a regression from any code change. - -The harness stubs =ssh=, =rsync= and =jq= onto =PATH=, so nothing should touch the network at all — which is what makes a 20s timeout suspicious rather than merely slow. Worth reproducing under load before deciding whether the fix is a larger timeout or a real hang in the script. Evidence logs from the round: =/tmp/tu.log= and =/tmp/tu2.log= (tmpfs, gone after reboot). - -Not graded on the bug matrix: it is test infrastructure, not the shipped codebase. - -*** 2026-08-08 Sat @ 05:05:00 -0500 Recurred under concurrent load, same signature -All 5 tests hit the 20s TimeoutExpired again during a =make test-unit= run -that overlapped two review subagents running their own suites on the box. -Standalone immediately after: 0.095s, all pass; the following quiet-machine -full run was clean. Confirms the load-sensitivity read — reproduce under -deliberate load before choosing between a bigger budget and a real hang. - - -*** 2026-08-19 Wed @ 14:50:00 -0700 Root-caused and fixed: inherited stdin, not load -Not load, and not the network. The harness stubs ssh as =cat >/dev/null=, which -drains stdin to EOF. With no explicit stdin the stub inherits whatever the test -runner had, so it returned instantly when stdin was redirected and blocked -forever when it was a terminal or a live pipe. All five tests then burned their -20-second budget. - -That is why it looked like a load effect: a run launched immediately after -another inherited a different stdin than a standalone invocation. A/B measured -today — =make test-unit /dev/null=, which +drains stdin to EOF. With no explicit stdin the stub inherits whatever the test +runner had, so it returned instantly when stdin was redirected and blocked +forever when it was a terminal or a live pipe. All five tests then burned their +20-second budget. + +That is why it looked like a load effect: a run launched immediately after +another inherited a different stdin than a standalone invocation. A/B measured +today — =make test-unit Date: Thu, 20 Aug 2026 00:41:11 -0700 Subject: chore(tasks): record the clone decision and move the ribbon deadline The shallow-clone task now records why I settled the remote-URL half as I did, and the two options I turned down. Without that written down I'll relitigate it the next time I look at the https default and think detection would be tidier. The personal-ISO half is noted on the secrets task that will carry it. The input-cover ribbon reseat moves to 2026-08-26. It needs the machine open on a bench and I'm away until then. --- todo.org | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) (limited to 'todo.org') diff --git a/todo.org b/todo.org index af8d75a..d43dc3b 100644 --- a/todo.org +++ b/todo.org @@ -179,7 +179,7 @@ sometimes (every failed lookup, which is occasional rather than constant) = P3 = [#C]. ** TODO [#A] Reseat velox input-cover ribbon — phantom power button :bug:velox:hardware: -DEADLINE: <2026-08-14 Fri> +DEADLINE: <2026-08-26 Wed> :PROPERTIES: :CREATED: [2026-08-13 Thu] :LAST_REVIEWED: 2026-08-13 @@ -510,6 +510,18 @@ override already exists (=DOTFILES_REPO=, documented in lives: a config the personal ISO bakes in, a post-install step, or a detection that prefers ssh when a key is present. Craig's call. +*Decided 2026-08-19: the ISO bakes the value, and a check nets the rest.* +=archsetup:240= has the identical default for =archsetup_repo=, so this was +always two repos rather than one. I ruled out detection — archsetup never +restores =~/.ssh=, so key-presence at clone time depends on ordering it +doesn't control, and "any key means ssh" would break a stranger who has an +unrelated one. I ruled out a bare post-install step for the reason this whole +class of bug exists: manual steps don't get run, which is why this sat four +days. So the personal ISO carries =ARCHSETUP_REPO= / =DOTFILES_REPO= in the +ssh form (noted on the secrets/ISO task), and =post-rebuild-check= check 8 +flags any working repo still on the read-only endpoint — covering curl|bash +and stock-ISO installs, which the ISO value cannot reach. + Repair on a machine already built: =git fetch --unshallow= in each repo, and =git remote set-url origin git@cjennings.net:.git= for dotfiles. @@ -523,6 +535,16 @@ Not :solo:. The depth half is (two lines plus tests in the existing carries no =--depth= for these two repos). The remote-URL half needs the decision above, so the task as a whole waits on it. Split it in two if the depth fix is wanted sooner. +*** 2026-08-19 Wed @ 23:05:00 -0700 Dropped --depth from both user-repo clones +=archsetup:1462= and =:1475= now clone full history; +=tests/installer-steps/test_clone_user_repos.py= covers it with 8 cases, and +one of them asserts the AUR build clones still carry =--depth 1= so the fix +can't be over-applied by a careless repo-wide sed. Both my repos on velox were +already unshallowed by hand last session, so this is prevention rather than +repair. +*** 2026-08-19 Wed @ 23:05:00 -0700 Settled the remote-URL half and netted it +See the decision recorded above. The ISO half is a note on the secrets/ISO +task; the net is =post-rebuild-check= check 8, which ships now. ** TODO [#B] post-rebuild-check needs a reference-host mode :feature:velox:solo: :PROPERTIES: :CREATED: [2026-08-17 Mon] @@ -758,6 +780,26 @@ archangel+archsetup ISO that's already ~80% built. Two ISO modes: generic don't start the migration until the credentials are rotated. Not started. Not :solo: — repo standup and history rewrite are Craig's calls; promote to a real spec (spec-create) when work resumes. + +*Also bake the push-capable repo URLs into the personal ISO* (decided +2026-08-19). =archsetup:240= and =:245= default =archsetup_repo= and +=dotfiles_repo= to =https://git.cjennings.net/...=, the anonymous read-only +endpoint. That default is right for a stranger installing archsetup — no key on +the server — and wrong for my machines, which have to push: velox came back +from its rebuild unable to push either repo, and I only found out at a 403 four +days later. I decided against detecting an ssh key in the installer, because +archsetup never restores =~/.ssh= (I do that by hand), so key-presence at clone +time depends on ordering the installer doesn't control, and a naive "any key +means ssh" would break a stranger who happens to have one. The override already +exists and is documented — =ARCHSETUP_REPO= / =DOTFILES_REPO= in +=archsetup.conf.example= — so the personal ISO just needs to carry the ssh +form of both, alongside the secrets bundle. The generic ISO keeps the https +default untouched. + +The gap that leaves is a curl|bash or stock-ISO install, which takes the https +default straight back. =post-rebuild-check= check 8 covers that path — it flags +a working repo whose origin is the read-only endpoint — so the ISO value is the +fix and the check is the net under it. ** TODO [#B] Settings toggles reset silently at session start :bug:dotfiles: :PROPERTIES: :CREATED: [2026-07-28 Tue] -- cgit v1.2.3