#+TITLE: Audio Doctor — diagnose and repair a broken sound stack #+AUTHOR: Craig Jennings #+DATE: 2026-07-09 #+TODO: TODO | DONE #+TODO: DRAFT READY DOING | IMPLEMENTED SUPERSEDED CANCELLED * IMPLEMENTED Audio Doctor — diagnose and repair a broken sound stack :PROPERTIES: :ID: 6ce3f2ef-052e-40cb-9067-42f8e665f813 :END: - [2026-07-10 Fri] IMPLEMENTED — Phases 0 through 4 shipped (dotfiles =4d42eb3=, =01a1d80=, =76857e3=, =7223c51=, =1550ac8=, =4320255=, =bd33440=). Xruns stayed out of v1 as decided. Per-application stream routing remains a Non-Goal. The wall carries the net panel's copy + close controls, added past the spec at Craig's request; standardizing them across the other panels is tracked as its own task. - [2026-07-09 Thu] READY — all eight Decisions closed. Written from the roam-inbox ask "look into a doctor button for the audio panel: what happens if pulseaudio or its equivalent is borked? would we know? how could we test for it? what remedies would we have?" Probe set, classifier, and remedy tiers are grounded in a live survey of ratio's stack. Phase 0 shipped ahead of the rest. * IMPLEMENTED Status :PROPERTIES: :ID: 1cc2b355-9450-4ace-bdb3-b935f7051918 :END: - [2026-07-10 Fri] IMPLEMENTED — the build closed three things the Decisions did not anticipate. A tenth verdict =tooling-missing= (=remedy: None=), because an absent =pactl= is an unobserved stack rather than a broken one. A =re-probed= marker between a fix run's two row blocks, because unlabelled they read as the doctor contradicting itself. And the read-only key is named DIAGNOSE beside the engraved DOCTOR header rather than being named DOCTOR itself (Craig's call, 2026-07-10, after asking why the panel carries two keys where net's carries one; the two presses stand on the audibility decision, and net's single key auto-fixes because its remedies are cheap). Live verification on ratio replaced the planned VM run for remedies 1, 4 and 5, and immediately found two bugs no fake could surface: =pactl get-default-sink= prints the literal =@DEFAULT_SINK@= placeholder when no default is set, and remedy 1 re-probed before WirePlumber had re-created the devices, reporting a repaired stack as broken. Both fixed. - [2026-07-09 Thu] READY — Craig closed the six open Decisions, all to the recommended option: DOCTOR is a section-header key; CLI-first with the GUI as a face; hung and dead are two verdicts sharing one remedy; tiers stay Auto/Confirm/Arm as drafted; a stream-active guard refuses the audible remedies and is overridable by pressing again; xruns are out of v1. Two consequences worth naming, because they were not obvious when the questions were asked. Taking the guard is *why* remedy 4 can stay at Confirm — the danger lives in the state of the machine, not the identity of the remedy. And the guard must read its stream state from =pw-dump= rather than =pactl=, because the remedies it protects are exactly the ones you reach for when the Pulse layer is dead; a =pactl=-fed guard would go blind precisely when it is needed. - [2026-07-09 Thu] DRAFT — Phase 0 shipped (dotfiles =4d42eb3=) and its Decision closed. The first draft claimed =build_status()= had no timeout and froze the panel; driving it disproved that — =pactl.run()= already times out. The real bug was narrower: two of the four reads sat outside the degrade guard, killing waybar's audio module. The claim is corrected in place rather than quietly dropped. - [2026-07-09 Thu] DRAFT — spine complete, Decisions open. Not ready to build: the remedy tiers touch a running meeting's audio, and the classifier's precedence needs Craig's call before code. * Metadata | Field | Value | |--------+-------------------------------------------------------------------------------------------------------------| | Status | implemented | |--------+-------------------------------------------------------------------------------------------------------------| | Owner | Craig Jennings | |--------+-------------------------------------------------------------------------------------------------------------| | Repo | dotfiles (=audio/= package); archsetup owns this spec | |--------+-------------------------------------------------------------------------------------------------------------| | Kin | net panel doctor (=net/diag.py=, =classify.py=, =repair.py=, =doctor.py=) — the architecture to mirror. | | | Maintenance console (results wall, arm-to-fire) — the interaction idiom. | |--------+-------------------------------------------------------------------------------------------------------------| * Summary The audio panel shows the sound graph and lets you drive it. It has no answer for the graph being *broken*. When PipeWire, WirePlumber, or the Pulse compatibility layer fails, the panel shows an empty or stale board and the user is left to remember service names. This adds a DOCTOR key to the audio panel — read-only diagnose, then a lightest-repair-first fix — mirroring the net panel's doctor, which already solves this shape for networking. * Problem / Context Craig's questions, answered from a live survey of ratio (2026-07-09): ** "What happens if pulseaudio or its equivalent is borked?" There is no PulseAudio. The stack is three user-scope services: | Unit | Role | Failure looks like | |--------------------------+---------------------------------------------+---------------------------------------------| | =pipewire.service= | the media graph itself | no devices at all; everything silent | |--------------------------+---------------------------------------------+---------------------------------------------| | =wireplumber.service= | session manager: routing, policy, device | devices exist but nothing routes; defaults | | | profiles, default picking | unset; profiles never applied | |--------------------------+---------------------------------------------+---------------------------------------------| | =pipewire-pulse.service= | Pulse protocol compat, via | =pactl= fails; the panel goes blind while | | | =pipewire-pulse.socket= | audio may still work for others | |--------------------------+---------------------------------------------+---------------------------------------------| The third row is the trap. The panel speaks =pactl=, so a dead =pipewire-pulse= blinds the panel even though PipeWire-native clients still play. The panel's own emptiness is therefore *not* evidence that audio is broken, and the doctor must not treat it as such. ** "Would we know?" Today, no — and worse than no. Two findings from the live survey: 1. *=pactl= hangs on a hung server.* Against a refused socket it fails in ~0ms ("Connection refused"). Against a socket that accepts and never answers, it blocks indefinitely — measured: still blocked at the 5s timeout. A hung server is precisely the case the doctor exists for, so *every probe carries a timeout*. The engine's own =pactl.run()= already does this and raises =PactlTimeout=; the doctor must do the same, and must never call a bare =subprocess.run=. 2. *The panel cannot diagnose itself.* Its only data source is the layer most likely to be down. The doctor's probes therefore read =systemctl --user= (never blocks on the audio graph) and =pw-dump= / =wpctl= (PipeWire-native, alive when the Pulse layer is not) *before* they touch =pactl=. ** "How could we test for it?" Four probe tiers, cheapest and most reliable first. Each is independently useful; the classifier below reads all four. *** Tier 1 — unit state (no audio IO, cannot hang) =systemctl --user show -p ActiveState -p Result = for =pipewire=, =wireplumber=, =pipewire-pulse=, and the two sockets. Distinguishes =active= / =failed= / =inactive= / =not-found=. *** Tier 2 — graph liveness (PipeWire-native, survives a dead Pulse layer) =pw-dump= with a timeout. Answers: does the graph exist, how many nodes, are there sinks and sources at all. =wpctl status= gives the same in human form. A graph with zero sinks is a different fault from a graph that will not answer. *** Tier 3 — Pulse compat reachability (the panel's own dependency) =pactl info= under a hard timeout. Three outcomes, and they are *different faults*: answered / refused / hung. The hung case is the one that currently freezes tooling, and it is invisible to a naive "does pactl work" check because that check hangs too. *** Tier 4 — semantic health (only meaningful once 1–3 pass) Default sink and source resolve to devices that exist; at least one sink is not =SUSPENDED= when a stream is playing. Ratio's sink states today are =RUNNING= and =SUSPENDED=; =SUSPENDED= alone is normal idle, not a fault. Xruns are deliberately *not* probed in v1 — see the Decision "Do xruns belong in v1?" below. They are a quality signal with no remedy in this feature's vocabulary, and a finding nobody can act on trains the user to ignore the wall. ** "What remedies would we have?" All three units are =--user= scope. *No privilege escalation is needed anywhere in this feature*, so the maintenance console's =priv.py= verb table and its =sudo -n= machinery are dead weight here and must not be copied in. This is the single largest architectural difference from the maint console, and the reason the audio doctor mirrors the *net* panel instead. Remedies, lightest first: | # | Remedy | Blast radius | Tier | |---+-----------------------------------------------+----------------------------------------------+---------| | 1 | =systemctl --user restart wireplumber= | routing/profiles re-applied; streams survive | Auto | |---+-----------------------------------------------+----------------------------------------------+---------| | 2 | Set default sink/source to a present device | one setting | Auto | |---+-----------------------------------------------+----------------------------------------------+---------| | 3 | Unmute + restore volume on the default device | audible, but reversible | Confirm | |---+-----------------------------------------------+----------------------------------------------+---------| | 4 | =systemctl --user restart pipewire-pulse= | Pulse clients drop and reconnect | Confirm | |---+-----------------------------------------------+----------------------------------------------+---------| | 5 | =systemctl --user restart pipewire= | *every* audio client's stream drops | Arm | |---+-----------------------------------------------+----------------------------------------------+---------| Remedy 5 is the one that cuts audio out from under a live meeting. It gets the arm-to-fire treatment, and its arm line says so in words. * Goals and Non-Goals ** Goals - Answer "is my sound stack broken, and if so where" without the user knowing a service name. - Work *when the panel's own data source is down* — a doctor that needs =pactl= to answer is useless in the case it exists for. - Never hang. Every probe is bounded; a probe that times out is a *finding*, not a stall. - Repair from the lightest remedy that could plausibly fix the classified fault, exactly as the net doctor does. - Read-only by default. Diagnose without touching anything; fixing is a separate, explicit press. ** Non-Goals - Fixing a Bluetooth audio device that will not take an A2DP profile. That is the bluetooth panel's problem and it already has a doctor-shaped surface. - Per-application stream routing or a full patchbay. Helvum and qpwgraph exist. - Diagnosing hardware failure (a dead DAC, an unplugged cable). The doctor can report "sink present but never leaves SUSPENDED"; it cannot see a cable. - Sample-rate, quantum, or latency tuning. A separate concern with a separate audience. - Any use of =sudo=. If a remedy seems to need it, the remedy is wrong. * Design ** For the user A DOCTOR key on the audio panel, beside the existing console keys. Pressing it streams findings into a results wall — the same wall idiom the maintenance console uses, so the interaction is already learned: #+begin_example DOCTOR diagnose · read-only ok pipewire.service active ok wireplumber.service active FAIL pipewire-pulse.service failed (Result=exit-code) ok graph 4 sinks · 7 sources (via pw-dump) FAIL pulse compat not answering — pactl refused -- defaults not checked (pulse compat down) verdict: the Pulse compatibility layer is down. PipeWire itself is healthy — native clients still play. FIX restarts pipewire-pulse. #+end_example The verdict names one fault, in the user's terms, and names the remedy it would run. Then a FIX key applies the lightest remedy for that verdict, streaming each step and re-probing after, so the wall shows repaired truth rather than an optimistic claim. When everything passes, the wall says so and offers nothing to press. ** For the implementer Mirror the net panel's module split, which already separates these concerns cleanly: | Module | Responsibility | |---------------------+------------------------------------------------------------------------------------------------| | =audio/diag.py= | The four probe tiers. Pure IO, each bounded by a timeout. Returns a context dict; never raises | | | on a broken stack, never blocks. | |---------------------+------------------------------------------------------------------------------------------------| | =audio/classify.py= | Pure. Context dict → one verdict, by precedence. Unit-tested against fixture contexts, no | | | services involved. | |---------------------+------------------------------------------------------------------------------------------------| | =audio/repair.py= | One function per remedy. Each returns a result envelope; each re-probes the thing it claimed | | | to fix. | |---------------------+------------------------------------------------------------------------------------------------| | =audio/doctor.py= | Orchestration: gather → classify → (optionally) attempt the lightest matching remedy → | | | re-probe → emit the event stream. | |---------------------+------------------------------------------------------------------------------------------------| Three constraints that fall out of the survey and are easy to get wrong: 1. *Timeouts everywhere, and guard every read.* Bound every probe. Beyond that: a function that promises to degrade must guard *all* of its reads, not the first two. That was the Phase 0 bug. Test with a fault that answers early calls and hangs on later ones — a server that hangs on the first call masks everything behind it, and a refused socket does not reproduce the hang at all. 2. *Probe order is load-bearing.* =systemctl= first (cannot hang), then =pw-dump= (alive when Pulse is dead), then =pactl= (may hang). Reversing this makes the doctor unable to report on the case it was built for. 3. *The GTK thread never probes.* The panel already has a =bg()= worker; the doctor runs there and lands events on the main loop, as the maint console's =_stream= does. *** Classifier precedence (draft — see Decision 3) Terminal faults first, so a lower-layer failure is never reported as an upper-layer symptom: 1. =pipewire.service= not active → *graph down*. Nothing else is meaningful. 2. =wireplumber.service= not active → *no session manager*. Devices exist, routing does not. 3. =pactl= hangs → *pulse compat hung*. Distinct from dead: a restart is the remedy, but the diagnosis differs and the user should see which. 4. =pipewire-pulse= not active, or =pactl= refused → *pulse compat down*. 5. Graph healthy, zero sinks → *no output devices*. 6. Default sink/source names a device that is absent → *stale default*. 7. Everything up, default device muted or at zero volume → *silenced, not broken* (a finding, not a fault; offer to unmute). 8. Otherwise → *healthy*. *** Testing Follows the four-layer pattern the maint console established: - *Unit, fakes.* =classify.py= against fixture contexts covering every verdict, including the hung-vs-dead distinction. =diag.py= against a fake =systemctl= / =pactl= / =pw-dump= on a temp PATH, one case per outcome. Pairwise over (unit state × graph state × pulse reachability) — three parameters with three, three, and four values each is 36 exhaustive, and pairwise covers the interactions in far fewer. - *Timeout regression.* A test that stands up a Unix socket which accepts and never answers, points =PULSE_SERVER= at it, and asserts the probe returns a =hung= finding within its timeout rather than blocking. This is the test that would have caught the real behavior; write it first. - *Live read-only.* =audio doctor= (no fix) on ratio and velox, asserting a healthy verdict. - *Remedy, in a VM.* Restarting =pipewire= is not something to exercise on a live desktop. The archsetup VM harness runs remedies 1, 4, and 5 against a real stack. - *GUI smoke.* AT-SPI over the DOCTOR key and the wall, on fixtures. * Alternatives Considered ** Reuse the maintenance console's doctor rather than building one in the audio panel Rejected. The maint doctor's remedies are system-scope and route through =priv.py= and =sudo -n=. Every audio remedy is =--user= scope. Bolting audio onto that table means either giving audio remedies a privilege path they do not need, or special-casing them out of the mechanism that defines the table. The net panel already demonstrates the right shape for a user-scope doctor. ** A single RESTART AUDIO button Rejected. It is remedy 5 with no diagnosis: it drops every client's stream to fix a problem that is usually remedy 1 or 2. It also teaches nothing — the user learns to press it for every symptom, including symptoms it cannot fix, such as a stale default device. ** Probe with =pactl= only, since the panel already speaks it Rejected on evidence. =pactl= hangs against a hung server, and it is dead when the Pulse compat layer is dead — the two most interesting faults. It is the *last* probe, not the first. ** Auto-repair on detection, without asking Rejected. Remedies 3 through 5 are audible and, in a meeting, disruptive. Diagnosis is free and should be; repair is a press. * Decisions [8/8] ** DONE Where does the DOCTOR key live, and does it replace anything? CLOSED: [2026-07-09 Thu] Resolved: a section-header key, the maint console's CLEAN UP / REVIEW & FIX shape. Diagnose and fix are two presses with different consequences, and the header is where the console already teaches that distinction. The console row (INPUT / OUTPUT / PUSH TALK) stays device controls; a doctor is not one. Nothing is replaced. ** DONE Is there an =audio doctor= CLI, and is it the same code path? CLOSED: [2026-07-09 Thu] Resolved: CLI-first, GUI as a face, one code path — the maint console's architecture. =audio doctor= (read-only) and =audio doctor --fix= print the same findings the wall streams. The deciding reason: when the desktop's sound is dead, a TTY is where you will be standing. A GUI-only doctor cannot serve the case that motivates the whole feature. It also makes the classifier testable without a display. ** DONE Confirm the classifier precedence above, particularly rows 3 and 4. CLOSED: [2026-07-09 Thu] Resolved: precedence as drafted, and rows 3 and 4 stay *two verdicts sharing one remedy*. They share a fix but not a diagnosis, and the hung case is the one that freezes other tooling — collapsing them would hide the more interesting fault behind the more common one. ** DONE Does the panel's own status build get the same timeout treatment? CLOSED: [2026-07-09 Thu] Investigated and answered: it already had one. =pactl.run()= defaults to =timeout=8= and raises =PactlTimeout=, so the panel degrades rather than freezing. The draft of this spec claimed otherwise; the claim was wrong. What the investigation *did* find is a narrower, real bug, now fixed (dotfiles =4d42eb3=): =build_status()= guarded its two device-list reads but called =default_sink()= and =default_source()= outside the guard, so a server that answered the lists and then stopped answering raised =PactlTimeout= out of a function whose contract is to degrade. =waybar-audio= calls it with nothing catching it, so the bar's audio module died rather than dimming. A server that hangs on the *first* call never exposed this — =list_sinks= raised inside the guard and masked everything behind it, which is why the obvious test would have passed against broken code. ** DONE Which remedies are Auto, which Confirm, which Arm? CLOSED: [2026-07-09 Thu] Resolved: the table as drafted. Remedies 1–2 (restart wireplumber, set a present default) are Auto; 3–4 (unmute + restore volume, restart pipewire-pulse) are Confirm; 5 (restart pipewire) is Arm. Remedy 4 stays at Confirm rather than Arm *because* the stream guard below was taken. The guard, not the tier, is what catches the mid-meeting case — which is the honest place for it, since the danger is the state of the machine, not the identity of the remedy. Had the guard been declined, remedy 4 would have moved to Arm. ** DONE Should the doctor refuse to run a disruptive remedy while audio is playing? CLOSED: [2026-07-09 Thu] Resolved: yes. A stream-active guard refuses a disruptive remedy and is overridable by pressing again, mirroring the maintenance console's live-update guard. Same shape of problem: an operation that is fine when idle and destructive mid-use, where wording alone does not stop a press. Implementation note for the builder: the guard's input is "is any node running" — and it must come from *Tier 2* (=pw-dump=), not from =pactl=. The remedies the guard protects are precisely the ones you reach for when the Pulse layer is dead or hung, so a guard that asked =pactl= would go blind exactly when it is needed, and a PipeWire-native client can be mid-playback while =pactl= answers nothing at all. A cold or unreadable graph means the guard cannot see a stream. Treat that as *no stream* and let the remedy proceed: refusing on an unreadable graph would wedge the doctor shut in the worst case, which is the fault it exists to repair. Say so in the wall, so the user knows the guard abstained rather than passed. The guard applies to remedies 3, 4, and 5 — the audible ones. Its refusal names what is playing, not just the remedy. ** DONE Do xruns belong in v1? CLOSED: [2026-07-09 Thu] Resolved: left out. They are a quality signal, not a broken/working one, and they have no remedy in this feature's vocabulary. A finding you cannot act on trains you to ignore the wall — the opposite of what a doctor is for. Revisit when there is something to do about them. * Implementation phases Sequenced so each phase is independently green and independently useful. Nothing here starts until the Decisions are closed. ** DONE Phase 0 — the degrade-on-hang bug fix (prerequisite) CLOSED: [2026-07-09 Thu] Shipped ahead of the rest (dotfiles =4d42eb3=). =build_status()= now guards all four pactl reads, not just the two list reads, so a half-wedged server dims the bar's audio module instead of killing it. =fake-pactl= grew =FAKE_PACTL_SLEEP_DEFAULTS= to express "answers the lists, hangs on the defaults" — the blanket sleep knob cannot, because the first call would hang and mask the bug. ** DONE Phase 1 — =diag.py=, the four probe tiers Bounded probes, a context dict, fakes for every outcome including hung. No classifier, no GUI. =audio diag --json= prints the context. ** DONE Phase 2 — =classify.py=, the verdict Pure function, fixture-driven, pairwise over the parameter space. =audio doctor= (read-only) prints findings and a verdict in the CLI. Already useful from a TTY at this point. ** DONE Phase 3 — =repair.py= + =doctor.py=, the fix path One function per remedy, each re-probing what it claims to fix. =audio doctor --fix= applies the lightest matching remedy. VM-tested for remedies 1, 4, and 5. ** DONE Phase 4 — the DOCTOR key and the results wall CLOSED: [2026-07-10 Fri] Shipped (dotfiles =4320255=). GUI face over the CLI. Streams events to a wall, off the GTK thread. AT-SPI smoke over the key, the wall, and the fix path on fixtures. ** DONE Phase 5 — flip this spec to IMPLEMENTED CLOSED: [2026-07-10 Fri] With a history line naming what shipped and what was left out.