From 6ecd1e9bf1e3d0cdd3861077318541e193ca4532 Mon Sep 17 00:00:00 2001 From: Craig Jennings Date: Sat, 6 Jun 2026 07:49:01 -0500 Subject: docs: add the Duet design spec The full v1 design for the dual-pane file commander: architecture, the transport-backend registry, transfer execution and data-safety contracts, connection UX, diagnostics, the phased implementation plan, and a competitive-landscape appendix. --- docs/design/duet-spec.org | 842 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 842 insertions(+) create mode 100644 docs/design/duet-spec.org (limited to 'docs') diff --git a/docs/design/duet-spec.org b/docs/design/duet-spec.org new file mode 100644 index 0000000..e02e902 --- /dev/null +++ b/docs/design/duet-spec.org @@ -0,0 +1,842 @@ +#+TITLE: Design: DUET — Dual-Pane File Commander +#+AUTHOR: Craig Jennings +#+DATE: 2026-06-05 + +* Status + +READY — four review rounds plus a competitive-research pass, all folded in (external review + maintainer, 2026-06-06): round 1 (Not-ready), round 2 (Ready-with-caveats), the competitive research (Appendix B), round 3 (transfer/sync-failure + extensibility), and a final meta-review (rollback/reversibility). No blocking spec gaps remain; the open questions are non-blocking. Ready for implementation pending Craig's final pass. DUET = "DUET Unifies Endpoint Trees." + +Packaged as a standalone library at =~/code/duet= (skeleton exists: git-initialized on =main=, =*.elc= gitignored, uncommitted), with personal configuration glue at =~/.emacs.d/modules/duet-config.el=. Symbol prefix =duet-= / =duet--= (publishable namespace, not =cj/=). Publishing target MELPA or NonGNU ELPA, decided at graduation. This mirrors the pattern already used for pearl, signel, gptel, and wttrin: package in =~/code/=, config glue in =modules/-config.el=. + +* Problem + +A two-pane orthodox file manager (Midnight Commander / FileZilla style) inside Emacs, built on dirvish/dired. Left and right panes each show a directory listing on any location — local or remote — and single-key actions on the file under point (copy, move, delete, open) use the opposite pane as the implied target. Copies route through whatever transport fits the endpoint pair, including the remote-to-remote case. The point is real cross-machine file work without leaving Emacs and without hand-typing TRAMP paths. + +The durable value is not the two-pane UX alone — dired + dirvish + dired-rsync already approximate stage 1. It is the transport reach (rclone, lftp) and a community-extensible backend interface, which let DUET move files dired fundamentally cannot. + +* Non-Goals + +- Bidirectional, stateful, conflict-aware sync in v1. Deferred to stage 3 (via unison / rclone bisync). +- A directory-diff view in v1. Stage 3. +- Single-file peer-to-peer send (croc / magic-wormhole). A different gesture than directory-pair sync; see Appendix A. +- Storing credentials anywhere but the existing GPG secret store. Passwords go to =authinfo.gpg= via auth-source, never into a saved-connection record. +- Replacing dirvish. DUET builds on it as the renderer. + +* Project principles + +DUET mutates local and remote filesystems through multiple transfer tools. These are non-negotiable engineering rules; later backend work must not optimize them away. + +- *Never leave stray files on a remote server.* If a backend writes temp/partial files at the destination, DUET must know their naming pattern, log them, and verify cleanup before declaring the transfer complete or failed. If cleanup cannot be verified, surface that as a distinct final state, not as ordinary success or failure. +- *Transparency is the basis of trust.* Every running transfer is inspectable: backend, route, files, byte/progress signal when available, elapsed time, ETA when defensible, current status. Hung or stalled transfers must be visible. +- *Hung work must be killable.* A command cancels/kills a transfer, records whether the backend cleaned up, and tells the user what residue may remain. +- *No silent data loss.* Move deletes only after confirmed copy success, per source. A failed copy never deletes its source. Destructive operations require explicit confirmation unless the user has configured otherwise. +- *No silent slow-path surprises.* Remote-to-remote fallback through the local machine must be visible before or during the transfer; it can be much slower than direct host-to-host. +- *Prefer direct native transports when safe; fall back visibly.* Pick the best backend for the endpoints, but make fallback routes obvious and diagnosable. +- *User-owned configuration stays user-owned.* DUET writes only to its own connection records and =duet-= snippets unless the user explicitly confirms an external write. It never edits hand-authored ssh config blocks behind the user's back. +- *Secrets are never logged.* Logs and bug reports redact passwords, tokens, auth-source data, and private-key paths. +- *Backend behavior is a contract, not folklore.* Every backend documents temp-file behavior, cleanup semantics, progress support, cancellation behavior, error codes, and redaction rules. +- *Tests simulate the dangerous parts.* Temp-file cleanup, hung processes, cancellation, partial failure, and failed cleanup have fake-backend / fake-process tests even when live remote tests are optional. +- *The two-pane UI stays sparse.* The panes are for browsing and acting on files, not for showing diagnostics. Diagnostics live in the transfer monitor and log. Pane keys stay at commander primitives (F3/F4/F5/F6/F7/F8/F10 plus one transfer-status key); everything else is reachable by =M-x= or the monitor, so DUET never becomes a command pile. +- *Resume is visible, never silent.* A transport that can resume does so and says it is resuming; it never silently restarts a large transfer from zero (rclone's single most-cited failure). A backend that cannot resume declares that. +- *Writes are atomic.* Transfers write to a temp name and rename on completion, keep a resumable partial on interruption, and never leave a half-written file that looks complete. =--inplace= is opt-in only and forbidden in stage-3 sync (corruption from a dropped =--inplace= write propagates back over the good copy on the next sync). +- *Never build a shell string from a filename.* Every backend receives an argv / NUL-delimited file list, so spaces, newlines, quotes, and shell metacharacters in names are inert. +- *Delete defaults to the trash.* Deletion goes to the freedesktop trash by default (recoverable); permanent deletion is a deliberate, confirmed second tier. +- *Terminal states must be announced, not discovered by accident.* A transfer that finishes, fails, stalls, or lands in =cleanup-unverified= has a visible monitor state, a log entry, and a notification policy. The user should not have to notice a stale pane or inspect a log proactively to learn that a long transfer failed. +- *DUET should be the helpful error surface for the CLIs it wraps.* A user already comfortable with =rsync=, =rclone=, or =lftp= should still prefer DUET when something fails because DUET classifies the failure, explains the likely cause, shows the evidence, preserves safety state, and offers the next action without requiring them to decode raw stderr. +- *Sync is not backup.* Stage-3 sync must never imply recoverability unless versioning/trash/restore is configured and tested. Mass deletes, ransomware-like churn, ignored/selective trees, and reconnect-after-offline are high-risk sync events, not ordinary background housekeeping. + +* Assumptions + +- *Researched fact:* dired's =dired-dwim-target= offers the other visible dired window as the default copy/move target ([[https://www.gnu.org/software/emacs/manual/html_node/emacs/Operating-on-Files.html][Emacs manual: Operating on Files]]). +- *Researched fact:* dirvish has a known issue where two side-by-side dirvish panes do not reliably offer the *other* pane as the dwim target ([[https://github.com/alexluigit/dirvish/issues/36][dirvish#36]]). DUET owns other-pane resolution rather than relying on =dired-dwim-target=. +- *Researched fact:* =dired-rsync= performs async rsync from dired and handles remote-to-remote, but the remote-to-remote case requires a reverse-ssh / port-forward setup and fully-qualified TRAMP paths ([[https://github.com/stsquad/dired-rsync][stsquad/dired-rsync]]). +- *Researched fact:* TRAMP completes host names from =~/.ssh/config= (wired in =tramp-config.el= via =tramp-parse-sconfig=). The user's ssh config is a symlink into the dotfiles repo (=git@cjennings.net:dotfiles.git=), at =~/.dotfiles/common/.ssh/config=. +- *Researched fact:* =dev-fkeys.el= binds global F4 (=cj/f4-compile-and-run=) and F6 (=cj/f6-test-runner=); =dirvish-config.el= binds F11 (=dirvish-side=) and =r= (=dirvish-rsync=); =tramp-config.el= sets =tramp-default-method= "scp", adds =sshfast=, enables ssh/sshx direct-async, and disables remote auto-revert. +- *Researched fact:* Unison is an actively maintained (v2.53.8, Nov 2025), formally specified bidirectional file synchronizer in OCaml by Benjamin C. Pierce et al. ([[https://github.com/bcpierce00/unison][bcpierce00/unison]]). Used as the stage-3 sync engine, not reimplemented. +- *Researched fact:* rclone supports ~70 storage backends (S3, B2, GDrive, Dropbox, WebDAV, SFTP, FTP, ...), subsuming s3cmd, aws s3, and the MinIO client for our purposes. + +* Approaches Considered + +** Recommended: custom thin dual-pane layer over dirvish + +A =duet= entry command lays out two dirvish windows and installs a buffer-local minor mode carrying a transient keymap. DUET owns other-pane resolution (sidestepping dirvish#36) and a transport-backend registry from day one (see H2 resolution). Transport uses rsync / TRAMP backends in stage 1. + +- Pros: keeps dirvish as the front end; the #36 bug is moot because target resolution is explicit; transport policy lives in one testable registry that grows by registration, not rewrite. +- Cons: most custom code of the conventional options; DUET owns the transport-selection and remote-to-remote logic. + +** Rejected: built-ins only (dired-dwim-target + dired-rsync) + +Least code, but dirvish#36 makes other-pane targeting unreliable, and there is no commander framing or path to the backend differentiator. Its pieces (=dired-rsync=, async transfer) are reused inside the recommended approach. + +** Rejected: adopt Sunrise Commander + +The twin-pane feature already exists with ediff comparison that seeds stage 3, but it runs its own dired-derived panes, not dirvish, so it sets aside the dirvish investment. Not on MELPA. Raid its ediff model for stage 3; do not adopt wholesale. + +** Rejected (tail): wrap a TUI commander (mc/vifm/yazi) in a ghostel buffer + +Near-zero Emacs code and rides the ghostel engine, but it is not dirvish/dired — files do not open in Emacs buffers, no Emacs keymaps or marks. Defeats the intent. + +** Rejected (tail): target-by-completion (single pane, no spatial panes) + +Mark files, pick a destination from saved hosts via completing-read. Little code, scales past two locations, but loses the at-a-glance two-pane view and has no natural home for stage-3 diff. + +** Rejected (tail): rclone-mount remotes as local filesystems + +Mount each server with rclone so both panes are "local." Sidesteps remote-to-remote, but adds a mount lifecycle that fails independently and routes everything through the local machine. Superseded by treating rclone as a first-class transport backend (stage 2). + +* Roadmap + +Product stages, distinct from the build phases in "Implementation phases" below. + +** Stage 1 — Core commander (v1) + +Two-pane dirvish UX, the connect picker, the transport-backend *registry* with rsync + TRAMP registered through it, copy / move / delete / open, the buffer-local F-key scheme, the transfer log, and diagnostics. The registry is present from day one (not a later extraction); stage 1 simply registers two backends through it. + +** Stage 2 — More backends: rclone + lftp (the differentiator) + +Register rclone (cloud and ~70 protocols) and lftp (FTP/FTPS/HTTP with mirroring, parallel transfers, queueing) through the existing registry. No core rewrite — new backends are new registrations. This is where DUET does what dired cannot. + +** Stage 3 — Diff and sync (v2) + +ediff for file-to-file comparison; directory diff; bidirectional, conflict-aware sync ("make this pane match that," copy-missing-to-other) via unison and/or rclone bisync, exposed through the backend interface where it fits. + +Sync safety is non-negotiable, learned from Syncthing's most-cited failures (see Appendix B): conflicts are surfaced loudly in the diff view, never resolved silently or auto-won by timestamp — the user sees both versions and chooses, with a recoverable copy kept. Delete propagation is explicit, opt-in, dry-run-previewed, and guarded by a max-delete threshold that aborts a suspicious mass deletion. Every sync runs a dry-run plan the user can veto per file before anything applies; the first sync requires an explicit baseline; reconnection / offline-gap syncs are treated as high-risk and batched for review rather than auto-applied. =--inplace= is forbidden in sync. + +* Design + +** Package / config split and naming + +- Package =~/code/duet= (=duet.el=, growing into =duet-connect.el=, =duet-transfer.el=): the library. =duet-= commands, the minor mode, the pure logic, defcustoms with conservative portable defaults, autoloads, =Package-Requires=, =lexical-binding=, GPL, README, Makefile, developer docs. No personal opinions — no keybindings, no local paths, no =cj/=. The TDD suites live here. This is what graduates to MELPA / NonGNU ELPA. +- Config =~/.emacs.d/modules/duet-config.el=: thin glue. =use-package duet :load-path "~/code/duet" :ensure nil= during development (flip to =:ensure t= once published); the keybindings; defcustom values; the gitignored connection-store path; the dotfiles ssh-config Include dir; the first-connect opt-ins. Mirrors =signal-config.el= ↔ signel. For any custom prefix command, use =keybindings.el='s registration helpers (=cj/register-prefix-map=), not =with-eval-after-load 'keybindings= (which produced the Signal load-order bug). =.emacs.d= keeps only a config smoke test. + +** Persistence and side-effect boundaries + +The package never hardcodes a personal path and never writes a sensitive file unless a path is configured. Ownership: + +- The package exposes the *mechanism* as defcustoms/functions: =duet-connection-store-file=, =duet-ssh-config-include-file=, =duet-ssh-config-snippet-directory=, =duet-auth-source-save-enabled=, =duet-first-connect-install-key=, =duet-first-connect-save-connection=. +- *Package defaults are conservative and portable.* The connection store defaults to a file under =user-emacs-directory=; the ssh-config and authinfo writers default *off* — first-connect key install, password-to-authinfo, and ssh-config snippet writing do nothing until the config supplies paths and opts in. +- =modules/duet-config.el= supplies the Craig-specific values: the dotfiles snippet directory (=~/.dotfiles/common/.ssh/config.d/=), the Include file, and the opt-in toggles. +- *Every writer is idempotent, previewable in tests, and scoped.* The ssh-config writer touches only files matching the =duet-= prefix and refuses to modify hand-authored snippets. Writers are pure-plan + execute: the plan (what would change) is a value a test asserts on without writing. + +** Rollback and reversibility + +DUET must be easy to back out of, especially because it touches remote filesystems and optional local config: + +- *Package rollback:* disabling =duet-config.el= or uninstalling the package stops all DUET UI/keymaps and leaves ordinary dired/dirvish behavior intact. No package code is required to browse existing files after rollback. +- *Connection-store rollback:* deleting or renaming =duet-connection-store-file= removes DUET's saved overlay records. It does not remove ssh-config hosts, auth-source entries, or remote files. +- *ssh-config rollback:* DUET writes only one-file-per-connection snippets named with the =duet-= prefix. A rollback command, =duet-rollback-generated-config=, previews and removes only DUET-owned snippets and the DUET Include line if it is empty/unused. It refuses hand-authored files. +- *auth-source rollback:* DUET never silently deletes credentials. It can show the auth-source machine/user keys it wrote and provide a manual removal recipe; automated credential deletion is opt-in and confirmed. +- *authorized_keys rollback:* installed public keys are append-only and idempotent. DUET records enough metadata to show which public key was installed and gives a manual removal recipe; it does not silently edit remote =authorized_keys= to delete keys. +- *Transfer rollback:* one-way copy is not rolled back automatically after partial-batch failure; copied successes remain and failures are reported per file. Move rollback is source-preserving by design: source deletion happens only after verified per-source copy success. Delete rollback is via trash/versioning where supported; permanent delete has no rollback and is confirmed as such. +- *Stage-3 sync rollback:* before sync ships, the spec must define versioning/restore support, dry-run plans, and recovery from mass-delete/conflict mistakes. Until then, sync is not marketed as backup. + +Tests cover the rollback planner for generated config: it lists only DUET-owned files, refuses non-=duet-= snippets, and produces a no-op when nothing DUET-owned exists. + +** Architecture + +=duet= splits the frame into two side-by-side windows, each a dirvish/dired buffer, and enables the buffer-local minor mode =duet-mode= in both. The minor mode carries the transient keymap, so single-key actions fire only inside commander panes. + +Keep the pure core small and early: path classification, the backend registry, transfer-spec construction, connection-record formatting, and prompt-free conflict/move planning. The interactive layer is a thin shell over it. + +Interactive layer (=duet-*=): the entry command and per-action commands =duet-copy= / =-move= / =-delete= / =-open= / =-view= / =-mkdir= / =-refresh= / =-swap-panes= / =-connect= / =-quit= / =-cancel-transfer=. They read the file under point, resolve the target, prompt on conflicts, dispatch. + +Pure helpers (=duet--*=), each independently testable, no UI: +- =duet--other-pane= — returns the sibling pane's window and dired directory by window geometry; errors clearly with zero or three commander panes. Explicit replacement for =dired-dwim-target= (the #36 fix). +- =duet--classify-path= — path to plist (=:locality= local|remote, =:method=, =:user=, =:host=, =:port=, =:localname=), parsing TRAMP syntax. +- =duet--transfer-spec= src dst opts — classified endpoints to a concrete transfer spec via the backend registry (shape below). +- =duet--plan-conflicts= — prompt-free; returns per-file overwrite/skip/rename/apply-to-all plans. +- =duet--plan-move= — never emits a delete step for a source until that source's copy is signaled successful. +- =duet--run-transfer= — executes a spec asynchronously, streaming to the log. +- =duet--build-connection= — wizard's fields to a TRAMP path + dired directory. +- =duet--ssh-config-block= — a saved record to ssh-config Host-block text, plus the exists/overwrite/skip decision. +- =duet--pubkey-candidates= — enumerate =~/.ssh/*.pub=, mark the default. + +** Transport backends — the quartet and their division of labor + +DUET ships four transfer utilities as backends, over TRAMP as the Emacs-native substrate (universal browsing + fallback). The four do not overlap: + +- *rsync* (stage 1) — Unix-to-Unix and local. Block-level delta-transfer, zero remote install, faithful Unix metadata (permissions, ownership, symlinks, hardlinks, ACLs, xattrs via archive mode), fast and light over ssh with resume. The default for an ssh-reachable host. +- *rclone* (stage 2) — cloud and object stores plus the long protocol tail (S3, B2, GDrive, Dropbox, WebDAV, ...). The reach rsync lacks; weaker on Unix metadata and no rsync-style deltas, so breadth, not the Unix-to-Unix default. +- *lftp* (stage 2) — FTP/FTPS/HTTP with mirroring, parallel transfers, queueing. The legacy protocols rsync and ssh do not speak. +- *unison* (stage 3) — bidirectional, stateful, conflict-aware reconciliation. A different job from one-way transfer. + +One line: rclone is breadth, rsync is depth. The registry scorer encodes this — for a local↔ssh Unix pair rsync scores best; rclone wins only where rsync cannot go. + +** Data flow and the transport-backend registry + +A copy action: =duet-copy= reads the file(s) under point in the active pane (source), asks =duet--other-pane= for the destination directory, classifies both endpoints, plans conflicts, builds a transfer-spec, runs it async, refreshes the destination pane exactly once on the sentinel. Move is the same plus per-source delete-on-success. Open, view, mkdir, and delete stay local to the active pane. + +The registry is a v1 contract, not a later refactor (H2). =duet--transfer-spec= never hardcodes tools; it consults registered backends. A backend is a =cl-defstruct duet-backend= registered via =duet-register-backend= with: +- =name= (re-registering a name *replaces* the prior backend — this is how a user overrides a built-in); +- a =handles= predicate =(lambda (src dst) ...)= returning a numeric score (lower = preferred cost) or nil if it cannot handle the endpoint pair; +- a =command= builder =(lambda (src dst opts) ...)= returning a process spec; +- =capabilities= flags: =:async :resume :bidirectional :progress=; +- backend-contract metadata: temp-file naming pattern, cleanup semantics, redaction rules. + +=duet--transfer-spec= asks every registered backend to score the pair and picks the lowest-cost handler. rsync and TRAMP register through this same API in stage 1; rclone/lftp/unison are later registrations. Third parties or the user's config add backends without touching core. + +Stage-1 endpoint matrix (rsync + TRAMP backends): +- local ↔ local: the rsync backend (one uniform execution path — same logging, conflict, and cancellation semantics as every other transfer; no special-cased native copy, see Agreed decisions). +- local ↔ remote over ssh/scp/sftp: rsync over ssh, async. +- remote ↔ remote, same host: ssh in once and rsync there. +- remote ↔ remote, different hosts: route through the local machine by default (always works, slower) with a visible route notice; direct host-to-host rsync (reverse-ssh, fully-qualified paths) only when a per-connection transport override enables it. Direct mode is never automatic; if it fails, DUET falls back to round-trip with a notice and does not silently retry. +- anything the SSH backends decline: TRAMP as the universal fallback; raw FTP via TRAMP's ftp method until lftp/rclone land in stage 2. + +** Transfer execution contract + +The =duet-transfer-spec= is a plist, the pure boundary between planning and execution: +- =:sources= — a list of absolute source paths (single-file is a one-element list). +- =:destination-directory= — the target directory. +- =:destination-name= — the final name at the destination, set when a rename-on-conflict was planned, else nil (keep source basename). +- =:backend= — the chosen backend name. +- =:argv= — the full argument vector for the process. +- =:default-directory= — the process working directory (often the source or destination TRAMP root). +- =:process-environment= — any environment overrides. +- =:async= — t for a background process with sentinel; nil only for the rare synchronous path. +- =:route= — =:local=, =:local-remote=, =:remote-same-host=, =:remote-roundtrip=, or =:remote-direct= (drives the route notice). +- =:sentinel-payload= — the transfer id, source/destination, and pane references the sentinel needs to log and refresh. + +Behavior: +- *Marked files* are collected from the active pane's dired marks; with no marks, the file at point is the single source. +- *apply-to-all* is a field on the conflict plan, resolved once and applied to every remaining file in the batch. +- *Refresh* fires the destination pane exactly once when the batch's sentinel reports completion (the source pane refreshes too after a move). +- *Move* deletes each source only after that source's copy is signaled successful; a partial-batch failure leaves the already-copied files in place and the failed source intact, and the log summarizes per-file status (continue-on-error within a batch, never stop-and-rollback). +- *TRAMP fallback* has no byte-progress signal; the log records "no progress signal (TRAMP)" rather than implying a stall. +- *Resume:* when a backend advertises =:resume=, an interrupted transfer restarts from its partial (rsync =--partial=, rclone resumable uploads) and the monitor shows "resuming"; it never silently re-sends from zero. +- *File lists:* sources are passed as an argv / NUL-delimited list (rsync =--from0 --files-from=, per-backend equivalents), never interpolated into a shell command. +- *ETA:* computed from rolling recent throughput, not a lifetime average, with totals recomputed if a scan grows mid-transfer (avoiding rclone's average-speed and growing-total ETA bugs); shown only when the backend emits byte progress. +- *Verify on completion:* the sentinel confirms the expected size/checksum landed; a partial or failed item surfaces in the queue, never as success. +- *Source stability:* each source gets a preflight identity snapshot (size, mtime, inode/file id when available, checksum when cheap enough or required by backend). If the source changes during a transfer, the result is not marked clean success: DUET reports =source-changed=, preserves the source, verifies or removes the destination partial according to backend cleanup rules, and tells the user to retry. +- *Preflight capacity and writability:* before large transfers DUET checks the destination is writable and, when the backend exposes it, has enough free space/quota. If the check is unavailable, the log says so; ENOSPC/quota failures are distinct states and never collapse into a generic copy failure. + +** Connection UX + +The connect key opens a completing-read. Three fixed entries at the top: =[new ssh]=, =[raw TRAMP]=, and =[cancel]=. Below them, the user's ssh-config hosts and any saved connections, merged, saved/recently-used sorted to top. =[cancel]= aborts with no write. + +ssh-config hosts are shown by default, filtered by =duet-connect-excluded-hosts= (defcustom, default nil = show all), so a non-file host like a GitHub Enterprise remote can be pruned with one line. + +=[new ssh]= runs a short sequential minibuffer wizard — three fields, each with a stable prompt label, aborting cleanly (no write) on =C-g=: +- =DUET server: = (e.g. =cjennings.net=) +- =DUET username: = (e.g. =cjennings=) +- =DUET directory: = (e.g. =media= → =~/media=, or an absolute path) + +No password step. The wizard is a thin reader delegating to =duet--build-connection=. =[raw TRAMP]= is the escape hatch for port, jump-hosts, and non-SSH methods: read a full =/method:user@host#port:/path= string, validated before use. A per-connection transport override is kept out of the hot path — exposed only when editing a saved connection or diagnosing backend selection. + +*Connection feedback and the password prompt.* The wizard does not collect a password; authentication is TRAMP's, on demand. DUET brackets it so the prompt is always expected and attributed: before opening the remote pane it messages =DUET: connecting to — enter the server password if prompted=; TRAMP then performs the handshake and, if no ssh key, agent, cached password, or authinfo entry answers, shows its own hidden minibuffer prompt (the server's =user@host's password:=) in the usual place; on success DUET reports =DUET: connected to =, and when a saved credential answers it says =connecting (saved credential)= with no prompt. DUET does not reword TRAMP's internal prompt (that needs brittle advice); the bracketing messages carry the attribution and the "why now." Optional later refinement: a DUET-branded pre-prompt seeded into TRAMP's password cache so TRAMP never prompts — not v1. + +ssh config is the source of truth for SSH host/user/port/identity; the saved-connection store is a thin overlay keyed by host alias that adds only what ssh config cannot express (a starting path, a transport override), plus full standalone records for non-SSH targets. The overlay persists to =duet-connection-store-file=. + +** First-connect conveniences + +These are the sensitive writers. They are *designed here but sequenced late* (Implementation phase 9) and ship *disabled by default* — the core commander is trustworthy before any external-write convenience lands. Each is opt-in, confirmed, never silent, and governed by the boundaries in "Persistence and side-effect boundaries." + +On a successful connect, when enabled: + +1. =duet-first-connect-install-key= (default nil): if at least one =~/.ssh/*.pub= exists and is not already on the remote — =Install key for passwordless login? y/N: = (RET = N). On yes, a completing-read lists the public keys with the default marked (the host's ssh-config =IdentityFile=, else the default id); the chosen key is appended to the remote =authorized_keys= (ssh-copy-id), idempotent. + +2. =duet-first-connect-save-connection= (default nil): =Save this connection? y/N: = (RET = N). On yes: + - the non-secret record (server, user, directory, optional transport override) is written to =duet-connection-store-file=; + - if =duet-auth-source-save-enabled= and the key was *not* installed, DUET offers to persist the connection password to =authinfo.gpg= (pulling the just-entered value from TRAMP's password cache, or prompting once with a DUET-branded hidden prompt if not cached), keyed by host + user, then calls =auth-source-forget-all-cached= so it works immediately. If the key was installed, the password save is skipped as redundant. + - if =duet-ssh-config-snippet-directory= is set: =Save to ssh config for use outside Emacs? y/N: = (RET = N). On yes, a Host block is written as one file per connection at =/duet-.conf= (a one-time =Include= line — =duet-ssh-config-include-file= — is added to the main config). The directory lives in the dotfiles repo so it is committed and synced. If the Host alias already exists, offer to overwrite (default No); if declined, report and skip. The =duet-= filename prefix marks ours so the writer never touches hand-authored snippets. + +** Keybindings + +mc/Norton F-keys, bound in =duet-mode-map=, which is active only in commander pane buffers and takes precedence there. Global F-keys (dev-fkeys F4/F6, dirvish F11, music F10, ai-term F9) are untouched outside a DUET pane: +- F3 view, F4 edit, F5 copy, F6 move/rename, F7 mkdir, F8 delete, F10 quit-commander. + +dired's native chords (=C= copy, =R= rename/move, =D= delete, =v= view, =+= mkdir) keep working as free aliases since the panes are dired. Tests assert F4/F6 are DUET actions inside a pane and remain dev-fkeys outside. + +** Inherited dired/dirvish behavior + +DUET inherits, rather than rebuilds, the dired/dirvish surface. What it does with each: + +- *Inherited unchanged:* dired marks and marked-file operations (the source of multi-file transfers); dired's native copy/rename/delete/mkdir/view (the chord aliases); dirvish preview, attributes, history, narrow, subtree; TRAMP caching and auth-source. +- *Wrapped:* copy/move/delete on the F-keys route through DUET's transfer layer (logging, conflict planning, the registry) instead of dired's direct ops, while the dired chords remain as the unwrapped fallback. +- *Reused from config, named explicitly:* =dirvish-rsync= (bound =r=) and the remote quick-access entries in =dirvish-config.el= remain available in panes; DUET's transfer layer supersedes them for F-key actions but does not unbind them. =tramp-config.el='s scp default, =sshfast=, direct-async, and no-auto-revert settings are reused as-is. +- *Deferred affordance:* WDired for rename/edit-in-place is a future addition, not v1. +- *Open interaction:* =dired-hide-dotfiles= / omit and how hidden files interact with marked transfers — v1 treats hidden-and-marked as included (marks win), documented so it is not a surprise. +- *Archive operations (inherited now, one command in v1.1):* dired's =dired-do-compress= (compress a directory to =.tar.gz=, or unpack an archive at point) and =dired-do-compress-to= (compress marked files to a prompted archive) stay available in panes. A =duet-compress-to-other-pane= command — mark in the source pane, choose type/name, create the archive in the opposite pane (default =.tar.gz= / =.zip=, advanced formats by completion, preserving the containing directory, reusing the async/log/cancel/cleanup machinery when it runs long) — is a *v1.1* addition, kept out of v1 copy/move so archive support does not bloat the core. + +** Navigation + +Both panes always show a =..= entry to move to the parent directory, on every directory except the filesystem root. DUET relies on dired/dirvish's normal parent-directory line rather than injecting its own, so dired marks and file-at-point commands behave normally; selecting =..= moves that pane up. + +** Filesystem edge cases and data-safety + +The category's recurring data-loss traps are handled in the pure planner, testable before a byte moves: + +- *Filenames:* every backend gets an argv / NUL-delimited file list — never a shell string — so spaces, newlines, quotes, and shell metacharacters in names are inert. Encoding mismatches (NFC vs NFD, the macOS case) are normalized for comparison. +- *Symlinks and special files:* entries are classified with =lstat=, never =stat=. The planner never recurses a symlink during a delete (the classic "followed a symlink and wiped the target" trap); copy offers preserve-link vs follow-link explicitly. Hardlinks, xattrs, ACLs, and ownership are preserved only when both ends support them (rsync archive mode). +- *Same-file / dir-into-itself:* both paths are canonicalized; a copy/move where source == destination, or the destination is inside the source, is rejected before execution. +- *Cross-device move:* never a bare =rename= — copy → verify → delete, so a failed copy never loses the source. +- *Trailing-slash footgun:* paths are normalized in code (not left to shell completion), and the resolved destination is shown in the confirm step so "into" vs "contents of" is never a surprise. +- *Case-insensitive collisions:* when the destination filesystem is case-insensitive (mac / Windows / OneDrive / Box), =Foo.txt= vs =foo.txt= is detected and surfaced rather than one being silently dropped. +- *Path length / reserved names:* names that won't survive the destination (Windows =MAX_PATH=, =CON=/=NUL=, trailing dots) are flagged before transfer. +- *Trash by default:* delete sends to the freedesktop trash (recoverable); permanent delete is a deliberate, confirmed second tier. +- *Verify on completion:* a whole-file size/checksum check confirms the file landed; partial or failed items surface in the queue, never as success. + +These are the pure planner's responsibility (testable with no I/O); the backends carry them out. Each has its own test in the package suite. + +** Error handling + +Transfers run async; failures surface through the process sentinel, not a frozen Emacs. +- A nonzero exit leaves the source untouched and reports in the transfer log with the exit code and stderr tail. For a move, per-source delete fires only on that source's clean exit. +- Conflicts: =duet--plan-conflicts= checks the destination before copying; on collision, prompt overwrite / skip / rename, with apply-to-all for marked-multiple. No silent clobber. +- Stale/dead connections: a TRAMP/ssh timeout is reported with the operation, endpoint, and next action (see Error messages); the pane keeps its last good listing. The connection picker stays available. +- Locked/read-only source or destination files, permission-denied errors, quota/space failures, and backend capability mismatches are separate failure categories. They name the file and the endpoint and preserve the source. +- Remote-to-remote direct mode falls back to local round-trip with a visible route notice if the hosts cannot reach each other; no silent auto-retry. +- ssh-copy-id is idempotent. The ssh-config writer offers overwrite then skips. A malformed raw-TRAMP string is validated before use. An unhandleable endpoint pair (no backend scores it) reports "no transport for -> ". + +** Failure explainer + +Every failed/stalled/cancelled-with-residue transfer has a =duet-explain-transfer-failure= view, reachable from =duet-transfers= with =RET= or =?=. This is the "why use DUET instead of the CLI" feature. It is concise, structured, and built from the backend failure-normalizer: + +- *Verdict:* one DUET failure class, e.g. =rate-limited=, =source-vanished=, =destination-full=, =permission-denied=, =rsync-protocol-mismatch=, =remote-tool-missing=, =no-common-checksum=, =duplicate-remote-object=, =cleanup-unverified=, =backend-unknown-failure=. +- *What happened:* one sentence in user language. +- *Evidence:* exit code, signal, backend, route, and the exact stderr/log lines that triggered the classification (redacted). +- *Safety outcome:* whether the source was preserved, whether the destination was verified, whether a temp/partial file remains, and whether cleanup was verified. +- *Next action:* one or more concrete commands/actions: retry, lower transfer rate, open backend doctor, run =rclone dedupe=, install remote =rsync=, fix permissions/quota, remove a login-shell banner, trust/update an FTPS certificate, rerun with a specific backend, or open the redacted bug report. +- *CLI equivalent:* the redacted command DUET ran, plus an optional "copy command" action for users who want to reproduce outside Emacs. + +The explainer deliberately does not expose raw =-vv= logs by default. It shows a short evidence tail and links to the full redacted log. + +** Error messages + +User-facing messages name the operation, endpoint, selected backend, and next action. Representative set (tests assert substrings on the high-risk paths): + +| Situation | Message shape | +|-----------+---------------| +| Failed copy | =DUET copy failed: rsync exited 23 copying to ; see *DUET Transfers* #= | +| No backend | =DUET: no transport for -> = | +| Missing executable | =DUET: rsync not found on PATH; install it or set duet-rsync-program= | +| Invalid raw TRAMP path | =DUET: is not a valid TRAMP path= | +| Conflict refusal (skip) | =DUET: skipped (exists at destination)= | +| Skipped move delete | =DUET: kept — copy did not complete, not deleting= | +| Source changed mid-transfer | =DUET: changed while copying; kept source and marked transfer # for retry= | +| Destination full/quota | =DUET: no space/quota at ; kept source; see *DUET Transfers* #= | +| Permission/locked file | =DUET: couldn't write (permission denied or locked); source unchanged= | +| Rate limit / throttle | =DUET: is rate-limited for ; see transfer # for retry/throttle options= | +| Backend capability mismatch | =DUET: can't verify with ; see transfer # before retrying= | +| rsync protocol mismatch | =DUET: rsync protocol failed for ; remote shell may print login text. Run duet-doctor-backend= | +| Unreachable host | =DUET: couldn't reach (ssh timeout); pane unchanged= | +| Direct r2r fallback | =DUET: and can't connect directly; routing through this machine (slower)= | +| Auth failure | =DUET: authentication failed for @= | +| Refresh failure | =DUET: transfer # done; couldn't refresh (revert manually)= | +| Completion notice | =DUET transfer # complete: to ()= | + +** Transfer queue and concurrency + +DUET runs one transfer at a time by default (=duet-max-concurrent-transfers=, default 1). Additional requests enter a visible queue in the transfer monitor rather than launching in parallel. Each queued or running transfer has a stable id and a state: =queued=, =running=, =stalled=, =cancelling=, =cleanup-unverified=, =failed=, =done=. Cancellation works in any non-terminal state. Pane refresh fires once per completed transfer and is coalesced when several completed transfers target the same pane. No automatic parallelism for remote endpoints in v1 — backend-specific load and cleanup behavior must be understood first, and a single serial queue keeps log ordering, refresh, and remote load predictable. Raising the limit is a deliberate per-user choice via the defcustom. + +** Transfer monitor UX + +The log is for diagnosis; the monitor is the live status surface. =duet-transfers= opens a compact =tabulated-list-mode= buffer, one row per transfer: id, state, backend, route, source basename or count, destination, elapsed, and progress/ETA when the backend gives honest byte progress. Single-key actions: =g= refresh, =k= cancel/kill, =l= jump to the log entry, =RET= describe. Minibuffer messages stay sparse — start, completion or failure, stalled, cancellation result — so the monitor, not the echo area, is where ongoing state lives. ETA is shown only when a backend emits enough byte progress to estimate honestly (rsync =--info=progress2=, rclone stats); otherwise the row shows elapsed and last-output age, never a fabricated estimate. + +** Completion, failure, and notification policy + +Users should not have to rediscover failures by inspecting stale panes. Every transfer terminal state is detected by the queue state machine and sent through one notification path: + +- =done= — echo-area message always; desktop notification only when the transfer exceeded =duet-long-transfer-notification-threshold= or =duet-notify-on-transfer-terminal-state= is =all=. +- =failed=, =cleanup-unverified=, =stalled=, =cancelled-with-residue=, =source-changed= — echo-area message, transfer-monitor face, mode-line indicator until acknowledged, and desktop notification when =notifications-notify= is available. If desktop notifications are unavailable, fall back to =message= and keep the mode-line indicator. +- Queued/running progress stays in =duet-transfers=; the echo area is not spammed. + +Notifications include only redacted, user-useful facts: transfer id, terminal state, backend, route, source basename or count, destination host/path summary, duration, and "open =duet-transfers= / =duet-open-transfer-log= for details." They never include tokens, passwords, raw auth-source data, or private-key paths. + +Failure detection sources: +- process sentinel exit code or signal; +- backend progress parser reporting fatal output; +- stall timer crossing the backend threshold; +- source snapshot mismatch after transfer; +- destination size/checksum verification failure; +- cleanup verifier reporting residue; +- pane refresh failure after an otherwise successful transfer. + +** Responsiveness contract + +"Emacs must not block" is made measurable: +- process filters append output in bounded chunks and do minimal parsing per chunk; +- log and monitor rendering is throttled by a timer to a few updates per second, not once per output line; +- pane refresh is coalesced and never runs from a process filter; +- hung/stalled detection is timer-driven off last-output and last-progress timestamps, not a blocking wait; +- tests simulate high-volume output and assert that per-chunk filter work is bounded and that refresh is *scheduled*, not executed repeatedly; +- manual verification includes running a large transfer while typing and navigating in another buffer. + +Directory *listing* is held to the same bar, and it is the highest-risk responsiveness path: TRAMP can take many seconds to minutes on a large remote directory. Listing is async and streamed off the UI thread, the pane shows a progressive/loading state rather than freezing Emacs, =stat= is lazy, and results are cached. (Competitive note: remote-dir listing latency is DUET's most likely in-product frustration — see Appendix B — so the actual bytes of a transfer go through rsync/rclone, never TRAMP byte-streaming, and the listing path must never block.) + +** Observability and diagnostics + +Each transfer records a stable schema in =*DUET Transfers*=: transfer id, start/end time, source and destination pane paths, selected backend and the per-backend scores, route type, exact argv with secrets redacted, default-directory and environment, process id, exit status/signal, parsed failure class, evidence lines, stderr tail, full-output location, conflict decisions, refresh outcome, and final status (including the distinct "cleanup unverified" state). Async output streams into the log, not the panes; a minibuffer status line reports start and completion. + +Diagnostic commands, sequenced as the core ships first: +- v1 core: =duet-open-transfer-log=, =duet-describe-transfer-at-point=, =duet-explain-transfer-failure=, =duet-diagnose-path= (classify a path and show the plist), =duet-diagnose-backends= (show registered backends and their scores for a pair), =duet-doctor-backend= (run the selected backend's preflight/doctor for the current source/destination), =duet-report-bug= (a redacted diagnostic buffer). +- Later: =duet-copy-last-command=, =duet-test-connection=, =duet-describe-current-panes=, =duet-clear-transfer-log=. + +** Defcustom inventory + +The package exposes a coherent, documented customization surface; =duet-config.el= supplies Craig-specific values. Each defcustom states what nil means, whether it can cause an external write, and the safe default. The *load-bearing subset ships with the phase that needs it*; cosmetic knobs are added as needed. + +- Paths/identity: =duet-default-left-directory=, =duet-default-right-directory=, =duet-connection-store-file=, =duet-connect-excluded-hosts=. +- Side-effect (default off): =duet-first-connect-install-key=, =duet-first-connect-save-connection=, =duet-auth-source-save-enabled=, =duet-ssh-config-include-file=, =duet-ssh-config-snippet-directory=. +- Transport: =duet-preferred-backends=, =duet-remote-to-remote-policy=, =duet-rsync-program=, =duet-rclone-program=, =duet-lftp-program=. +- Behavior: =duet-confirm-destructive-actions= (default t), =duet-conflict-default-action=, =duet-refresh-after-transfer= (default t), =duet-long-transfer-notification-threshold=, =duet-notify-on-transfer-terminal-state= (=failures= / =long-running= / =all= / nil; default =long-running=). +- Diagnostics: =duet-transfer-log-buffer-name=, =duet-transfer-stderr-tail-lines=, =duet-debug=. + +** Backend extension API and developer contract + +=duet-register-backend= is the published seam, documented in the developer guide with a worked example. Built-in backends ship as registrations through it, and the same contract tests every built-in passes are what a third-party backend runs. + +Stable public surface (the ABI a backend author may rely on): the classified-endpoint plist, the =duet-backend= struct, the =duet-transfer-spec= plist, and the transfer-event payload the sentinel emits. + +The extension contract is tiered so plugin authors are not forced to become observability experts before their backend can run: + +- *Minimum runnable backend:* =name=, =handles= scorer, =command= builder returning argv/default-directory/env, =redaction= metadata, and conservative =capabilities=. DUET can launch it, redact logs, show generic progress/terminal states, and classify unknown failures as =backend-unknown-failure=. +- *Safe publishable backend:* add temp/partial metadata (pattern + cleanup-verifiable?), destination compatibility notes, required executable/version probe, and at least the common failure patterns listed by the CLI author or docs. This is the bar for built-in backends and for third-party backends DUET recommends. +- *Excellent backend:* add progress parser, doctor/preflight probes, cleanup verifier, cancel-command builder, retry/resume hints, and destination-specific gotcha checks. + +Required of every backend remains small: =handles=, =command=, redaction metadata, conservative capability flags, and a default =duet-backend-minimal-failure-normalizer= invocation. The minimal normalizer maps process launch failure, signal/exit, missing executable, timeout/stall, and unknown nonzero exit into useful generic explanations. Backend authors only add patterns for failures they actually know. + +Capability semantics are exact, not folklore: =:progress= means the backend emits byte-progress events (so ETA is honest); =:resume= means a restart is safe; =:bidirectional= means conflict-aware reconciliation, not merely "copies both ways." + +Failure taxonomy every backend reports against: launch failure, transfer failure, auth failure, source-vanished/source-changed, permission/locked, destination-space/quota, rate-limited/throttled, backend-capability-mismatch, cleanup-unverified, cancelled, stalled, route-fallback, verification-failed, and backend-unknown-failure. The monitor and log map each to a state. Raw exit codes are preserved as evidence but never shown as the whole answer. + +Each CLI backend owns or inherits a failure-normalizer: parse exit code, signal, stderr/stdout high-priority lines, progress/stat records, and known backend-specific messages into the taxonomy above. The normalizer also returns the user-facing explanation fields used by =duet-explain-transfer-failure=: likely cause, evidence lines, safety outcome, and next actions. Normalizers are tested with fixture logs for common failures. The helper =duet-define-cli-failure-patterns= lets authors provide a table of patterns rather than hand-writing parsers: + +| Field | Meaning | +|-------+---------| +| =:match= | regexp or predicate over exit/status/output | +| =:class= | DUET failure class | +| =:cause= | one sentence for the user | +| =:next-actions= | symbolic actions such as =retry=, =run-doctor=, =fix-quota= | +| =:evidence= | capture group or line selector | +| =:safety= | source/destination/residue statement, or =:generic= | + +Unknown cases still produce a usable explanation: "DUET did not recognize this failure; source preserved; here are the redacted evidence lines; run doctor or report bug." That keeps plugin authors honest without making the first backend impossible to write. + +Each backend may expose a doctor/preflight probe; DUET ships helpers for common probes so authors compose rather than invent: executable present, version command, endpoint reachability command, destination writability, quota/space where exposed, temp/rename support, checksum/modtime support, symlink behavior, and whether direct/server-side copy is supported or will fall back. The developer guide teaches the plugin author to add the cheapest high-signal probes first. + +The contract-test macro is tiered too: +- =duet-backend-test-minimum= — required: scoring sanity, command shape, redaction, no shell string, generic failure explainer. +- =duet-backend-test-publishable= — adds temp metadata, cleanup semantics, executable/version doctor, and fixture-based known failure patterns. +- =duet-backend-test-capability= — opt-in assertions for progress, resume, cancel, cleanup verifier, and bidirectional semantics when advertised. + +A backend that passes the minimum can be registered locally. A backend that passes publishable is safe to recommend. Community backends (unison, croc, git-annex) attach this way without patching core. + +** Backend developer experience + +The plugin developer is a DUET user too. The API must make the right thing the easy thing. + +*Scope (v1 vs stage 2).* The registry, the failure-normalizer interface, and the tiered contract-test macro are v1 — the built-in rsync and TRAMP backends use them, so they are exercised from day one. The author-facing conveniences below — =duet-scaffold-backend=, =duet-capture-failure-fixture=, the published =duet-define-cli-backend= macro, the tiered onboarding docs, and the copy-paste pattern tables — are stage-2 deliverables. They are designed now but built once the rclone backend has hardened the abstraction against a real second implementation; freezing and documenting a third-party authoring surface before the second backend exists risks locking in the wrong shape. Until then there are no third-party authors to serve. + +The full author experience, delivered in stage 2: + +- =M-x duet-scaffold-backend= creates a backend file, test file, sample failure fixture directory, and README stub from a template. +- =duet-define-cli-backend= macro covers the common CLI case: scorer, argv builder, redaction, capabilities, failure-pattern table, and doctor probes in one declarative form. +- =duet-capture-failure-fixture= saves the last failed transfer's redacted command/output as a test fixture skeleton so authors can turn real failures into regression tests. +- =duet-run-backend-contract-tests= runs the minimum/publishable/capability suites for one backend and reports which tier it currently satisfies. +- Developer docs show a "first backend in 30 minutes" path: start with minimum runnable, add two failure patterns, add one doctor probe, then publish when the publishable suite is green. +- Docs include copy-paste pattern tables for common CLI failure classes (missing executable, auth failure, permission denied, ENOSPC/quota, timeout, rate limit, checksum unavailable, source vanished, protocol mismatch). +- Docs state the line clearly: backend authors do *not* need to know every destination gotcha up front, but they must declare unknown capabilities conservatively, redact secrets, avoid shell strings, and never advertise progress/resume/bidirectional/cleanup support they do not test. + +** Project documentation and tooling + +The package's docs and Makefile are product surface, not incidental glue. V1 ships: +- =README.org= — feature summary, install (package-vc/use-package, straight, manual load-path, eventual MELPA/NonGNU), quick start (launch, connect, copy, view the log), command/key table, connection + credential policy, customization (the defcustom inventory), troubleshooting centered on the transfer log and diagnostics, development/testing pointer, history/license. +- =docs/developer-guide.org= — the backend API, the transfer-spec contract, test fixtures, adding a backend, backend tier requirements, failure-pattern tables, doctor/preflight helpers, logging/redaction requirements, rollback/reversibility expectations, and the release checklist. +- =TESTING.org= — normal tests, slow/manual tests, live transfer tests, coverage expectations. +- =Makefile= — =setup=/=deps=, =test=, =test-unit=, =test-integration=, =test-file FILE=...=, =test-name TEST=...=, =compile= (warnings as errors), =lint= (checkdoc/package-lint/elisp-lint), =coverage=, =coverage-summary=, =clean=/=clean-compiled=/=clean-tests=, =doctor= (executable + load checks), and =test-live= gated behind explicit env vars for hosts/paths. + +A =docs/user-guide.org= for end-to-end workflows is deferred until there is UI worth documenting (see Review dispositions H5). + +** Weak-point mitigations + +Where DUET would fail under real use, and the mechanism that covers it: + +| Weak point | Mitigation | +|--------------------------------+-------------------------------------------------------------------------| +| Remote residue / temp files | backend temp-file metadata + cleanup verifier + fake-backend tests | +| Hung process | last-output timer + monitor =stalled= state + cancel/kill | +| Process-output flood | throttled UI updates + bounded filter parsing (Responsiveness contract) | +| Wrong backend selected | =duet-diagnose-backends= + scorer tests | +| Raw CLI error is opaque | backend failure-normalizer + =duet-explain-transfer-failure= | +| Move data loss | per-source copy-success gate + delete-after-success tests | +| Source changes while copying | source snapshot + post-copy mismatch state + retry notice | +| Destination space/quota failure | preflight when possible + distinct ENOSPC/quota state | +| Failure unnoticed | terminal-state notification + mode-line indicator until acknowledged | +| Stale pane after transfer | refresh-once / coalesced refresh + manual-revert fallback message | +| Plugin lies about capabilities | contract tests + capability semantics in the developer contract | +| TRAMP / auth failure | actionable messages + no source mutation | +| Complexity creep | module boundaries + complexity/coverage review (below) | + +** Complexity and refactoring controls + +To keep DUET simple as backends multiply: +- command functions stay thin: read state, call a pure planner, execute and log — no transfer logic inline; +- backend-specific conditionals live inside backend modules, never in =duet-copy= or =duet--transfer-spec=; +- a soft cyclomatic budget: a function past ~8-10 branches is split or carries a written justification; +- complexity and coverage are reviewed together to find large, branchy, under-tested functions; +- a =make complexity= target is desirable once an Elisp complexity tool is chosen; until then the budget is a manual review checklist. + +** Testing + +Pure/interactive split keeps the hard logic testable without prompts or network. Tests live in the package repo (=.emacs.d= keeps one config smoke test). + +Package unit tests: +- =duet--classify-path= — local, =~= expansion, =/ssh:=, =/sshx:=, =/scp:=, host#port, multi-hop, malformed. +- =duet-register-backend= — deterministic replace-on-duplicate-name; registry ordering. +- =duet--transfer-spec= — chooses rsync over TRAMP for local/ssh pairs; TRAMP fallback for unsupported methods; every endpoint-matrix cell; different-host plans round-trip by default and direct only when the override is enabled. +- =duet--plan-conflicts= — overwrite/skip/rename/apply-to-all plans without touching files. +- =duet--plan-move= — no delete step until copy success is signaled. +- =duet--other-pane= — correct sibling in a two-window fixture; clear error with zero or three panes. +- =duet--build-connection= — fields to TRAMP path + dired dir, including =media= → =~/media=; wizard cancellation writes nothing. +- =duet--ssh-config-block= — record to Host-block text; exists/overwrite/skip; refuses non-=duet-= snippets. + +Process-boundary tests (no real transfer): simulate slow output, nonzero exit, signal exit, cancellation, stderr truncation, and prove Emacs does not block, output streams incrementally, cancellation is handled, and completion refreshes panes exactly once. + +Backend-normalizer fixture tests: rsync exit 23/24/11/12/30/35, remote rsync missing, protocol mismatch from login-shell output, rclone auth/config missing, token expired, 403/429 rate/quota, no common checksum, duplicate object, server-side-copy fallback, lftp TLS certificate failure, passive/active connection failure, resume unsupported, disk-full, and unison archive/root/conflict failures. Each fixture asserts the DUET failure class, evidence lines, safety outcome, next action, and redaction. + +Local integration tests (real =rsync=, temp directories, no remote): file copy, directory copy, conflict skip, conflict rename, failed-move-preserves-source, source-changed-during-copy detection, permission/locked destination failure when portable, ENOSPC/quota via fake backend, notification dispatch with =notifications-notify= stubbed, and stray-temp-file cleanup verification. + +Live remote and remote-to-remote tests are opt-in via =DUET_LIVE_TESTS= (or =make test-live=), never in the default suite. Coverage tooling reports line coverage and surfaces uninstrumented package files as 0% so coverage guides work rather than hiding gaps. + +Coverage is qualitative, not just a percentage: the pure core gets near-total branch coverage; the process boundary has every lifecycle state (=queued=/=running=/=stalled=/=cancelling=/=cleanup-unverified=/=failed=/=done=) covered by fakes; local integration proves real file movement in temp dirs; live remote is opt-in/manual; and every built-in backend runs the same contract tests a third-party backend will. The 80% business-logic figure is the floor, not the definition. + +* Implementation phases (TDD, green at each step) + +Each phase is a shippable deliverable reaching a clean stopping point in roughly three hours or less, and names the owning repo. Tests precede implementation within each phase. + +- *Phase 0 — Normalize skeleton + tooling* [package]. The repo is git-initialized with =*.elc= ignored; add the Makefile (targets above), the test harness, and coverage/coverage-summary. README skeleton + developer-guide stub. Commit the skeleton. +- *Phase 1 — Pure path core* [package]. =duet--classify-path= with the full Normal/Boundary/Error suite. +- *Phase 2 — Backend registry + failure-normalizer interface* [package]. =cl-defstruct duet-backend=, =duet-register-backend= (replace-on-duplicate), the registry, scoring, the backend failure-normalizer interface and failure-pattern mechanism (exercised by the built-in rsync/TRAMP backends), and the tiered contract-test macro. Tests: ordering, replacement, scoring, redaction metadata, contract-test tier behavior, failure-explanation shape. The author-facing tooling — =duet-scaffold-backend=, =duet-capture-failure-fixture=, the published =duet-define-cli-backend= macro, and the onboarding docs/pattern tables — is stage 2 (see Backend developer experience), built once the rclone backend has hardened the abstraction. +- *Phase 3 — Transfer-spec + conflict/move planning* [package]. =duet--transfer-spec= over the registry; the endpoint matrix; =duet--plan-conflicts= and =duet--plan-move= (pure, prompt-free). Tests for every cell + planning semantics. +- *Phase 4 — Pane layout + entry* [package + config smoke]. Two-window layout, =duet-mode=, =duet--other-pane= (#36 fix), the F-key map (precedence), =..= nav. Tests: sibling resolution, zero/three-pane errors, keymap precedence. +- *Phase 5 — Local transfer execution + queue* [package]. =duet--run-transfer= async, the serial transfer queue (=duet-max-concurrent-transfers= default 1) with the state machine, the log schema, coalesced refresh-once, cancellation, =duet-explain-transfer-failure=, and the responsiveness contract (bounded filters, throttled rendering). Process-boundary tests (slow output, high-volume flood, cancel, refresh-scheduling) + failure-normalizer fixture tests + local integration tests with real rsync. +- *Phase 6 — Remote transfer execution* [package]. local↔remote rsync-over-ssh; remote↔remote round-trip default + per-connection direct override with visible fallback; rsync doctor/preflight (remote executable, clean shell, version, destination writability) and exit-code/stderr normalization. Opt-in live tests. +- *Phase 7 — Connection picker + wizard* [package + config]. =[new]=/=[cancel]=/hosts/saved completing-read; the wizard; raw-TRAMP entry; the non-secret saved-connection overlay. Tests: build-connection, cancellation no-write, overlay round-trip. +- *Phase 8 — Diagnostics + transfer monitor* [package]. The full log schema; the =duet-transfers= monitor (=tabulated-list-mode=, =g=/=k=/=l=/=RET=, honest ETA); =duet-open-transfer-log=, =duet-describe-transfer-at-point=, =duet-diagnose-path=, =duet-diagnose-backends=, =duet-report-bug= (redacted), =duet-capture-failure-fixture=. +- *Phase 9 — First-connect writers* [package, disabled by default; config opts in]. ssh-copy-id with the pubkey picker; password→authinfo + cache flush; the ssh-config snippet writer (one file per connection, overwrite-offer, =duet-= prefix only). Idempotent, plan-previewable in tests, refuses non-=duet-= snippets. +- *Phase 9.5 — Rollback helpers* [package + config]. =duet-rollback-generated-config= previews/removes DUET-owned ssh-config snippets and DUET Include lines only; credential/key rollback is documented with manual recipes and confirmation gates. Tests cover no-op, DUET-owned removal, non-=duet-= refusal, and preview mode. +- *Phase 10 — Docs + manual verification* [package + config]. README, developer-guide, TESTING.org; the manual-verification matrix below. + +Manual-verification matrix (Phase 10): local/local, local/remote, same-host remote/remote, different-host fallback (route notice visible), conflict prompts (overwrite/skip/rename/apply-to-all), failed move preserves source, cancellation of a long transfer with residue report, a long successful transfer posts the configured completion notification, a failed transfer posts a failure notification and leaves an acknowledged monitor row, a stray-file cleanup check on a remote destination, a large transfer while typing/navigating in another buffer (responsiveness), and a queued second transfer showing in the monitor. + +* Acceptance criteria + +- Two panes launch; =..= navigates up; F-keys act in-pane and do not shadow globals outside a DUET pane. +- Copy/move/delete/open work local↔local, local↔remote, and same-host remote↔remote; different-host defaults to round-trip with a visible route notice. +- A failed copy never deletes its source; move deletes per-source only after that source's copy succeeds; partial-batch failure leaves copied files in place and the log shows per-file status. +- Conflicts prompt overwrite/skip/rename with apply-to-all; no silent clobber. +- Every transfer is logged with the full schema, is cancellable, and refreshes the destination pane exactly once. +- Every non-success transfer has a failure explainer that names the likely cause, evidence, safety outcome, and next action; common CLI failure fixtures map to DUET states rather than raw stderr alone. +- No stray remote temp files left unaccounted: cleanup is verified or surfaced as the distinct "cleanup unverified" state. +- Secrets are redacted in the log and in =duet-report-bug=. +- Package tests green; coverage ≥80% on business logic (reported by =make coverage-summary=, not hard-gated initially); the config smoke test loads without a live remote. +- First-connect writers are disabled by default, act only via config-set paths, are idempotent, and never touch non-=duet-= ssh snippets. +- DUET-generated config is reversible: rollback previews/removes only DUET-owned snippets; credential and authorized-key removal are documented and confirmed, never silent. +- One transfer runs at a time by default; additional requests queue visibly; cancellation works in any non-terminal state. +- A large transfer does not block typing or navigation in another buffer; UI updates are throttled, not per output line. +- The =duet-transfers= monitor shows live state; ETA appears only when a backend emits honest byte progress. +- Long successful transfers notify according to =duet-notify-on-transfer-terminal-state=; failures/stalls/cleanup-unverified/source-changed states always remain visible until acknowledged and never require the user to discover them from stale panes. +- Every built-in backend passes the same contract tests a third-party backend runs. + +* Agreed decisions + +Settled inputs, moved out of open questions. + +- *Name:* DUET; package =~/code/duet=; config =modules/duet-config.el=; symbol prefix =duet-=. +- *Registry timing:* the backend registry is v1 (stage 1), with rsync + TRAMP registered through =duet-register-backend= from day one. Stage 2 adds backends, not the registry. +- *local↔local transport:* the rsync backend, one uniform execution path (no special-cased native =copy-file= for small files), so every transfer shares logging, conflict, and cancellation semantics. A native fast-path is a possible later optimization, not v1. +- *Direct remote-to-remote:* never automatic; enabled only by a per-connection transport override; on failure it falls back to round-trip with a visible notice and no silent retry. +- *Keybindings:* mc F-keys in =duet-mode-map=, buffer-local, precedence inside panes only; dired chords as aliases. +- *Connect picker:* all ssh-config hosts shown, =duet-connect-excluded-hosts= prunes; =[new]=/=[cancel]= fixed at top; raw-TRAMP escape hatch retained. +- *ssh-config persistence:* one file per connection, =duet-= prefix, Include directory in the dotfiles repo, overwrite-offer-then-skip. +- *Password persistence:* to =authinfo.gpg= via auth-source (when enabled), never into the connection record; cache flushed after write. +- *First-connect writers:* designed now, sequenced to Phase 9, shipped disabled by default behind config-supplied paths. +- *Rollback:* DUET-generated config has a previewable rollback helper; credentials and authorized-key cleanup require explicit/manual confirmation; transfer rollback is source-preserving, not automatic deletion of copied successes. +- *Coverage:* ≥80% business-logic target, reported not gated for v1. +- *Concurrency:* one transfer at a time for v1 (=duet-max-concurrent-transfers= default 1); raising it is a deliberate per-user choice; no auto-parallelism for remotes in v1. +- *Transfer monitor:* a =duet-transfers= =tabulated-list-mode= buffer is the live status surface, separate from the diagnostic log; ETA only from honest backend byte-progress. +- *Notifications:* failures, stalls, cleanup-unverified, cancelled-with-residue, and source-changed terminal states always notify and remain visible until acknowledged; successful completion notifies for long-running transfers by default, all transfers if configured. +- *Connection wizard:* =[new ssh]= asks host/user/directory only — no password step; TRAMP/auth-source authenticates on demand, bracketed by DUET connect/connected messages so the prompt is attributed and timely. Top entries =[new ssh]= / =[raw TRAMP]= / =[cancel]=; transport override out of the hot path. Supersedes the earlier four-field wizard (Craig, 2026-06-06). +- *Archive:* dired's compress/extract inherited in panes now; =duet-compress-to-other-pane= is v1.1, not v1. +- *Pane UI stays sparse:* diagnostics live in the monitor and log, never the panes. + +* Open Questions + +- [ ] Publishing target: MELPA vs NonGNU ELPA. Decide at graduation; does not affect structure. Skip GNU ELPA (FSF copyright assignment). +- [ ] External =view= / =edit= tool for F3/F4: reuse dired's defaults, or a duet-specific viewer for remote files. +- [ ] lftp vs rclone overlap for FTP/FTPS in stage 2: register both and let scoring choose, or pick one as the FTP owner. +- [ ] Per-backend stall threshold: the elapsed-time / no-output value that marks a transfer hung, and whether it is configurable per backend (needs real backend behavior to calibrate). +- [ ] Elisp complexity tool for the eventual =make complexity= target — no standard one exists; a manual cyclomatic-budget checklist holds until one is chosen. +- [ ] Stage-2 rclone backend: are duplicate cloud objects with the same display name auto-diagnosed, refused, or routed to a "run rclone dedupe first" message? (non-blocking; decide when the rclone backend lands.) +- [ ] Stage-3 sync: the explicit versioning/restore story (what a user can recover, and how) before sync is trustworthy as more than bidirectional reconciliation. Dropbox/Syncthing recovery is a loved feature DUET's sync mode must answer. + +* Next Steps + +- The "Implementation phases" list is the task source. After the spec-review rounds finish, splice one todo child per phase under the DUET parent task and move this spec into =~/code/duet=. +- Commit the package skeleton (Phase 0). +- Open questions → =arch-decide= where a recorded decision helps. + +* Appendix A: Tools Considered, Not Adopted + +The quartet DUET implements is rsync (stage 1, over the TRAMP substrate), rclone + lftp (stage 2), and unison (stage 3) — see "Transport backends — the quartet" for each one's distinct value. Tools weighed and left out, with why: + +- *croc / magic-wormhole* — peer-to-peer single-file transfer. They move a particular file to a particular person, not keep two directory trees in agreement. A different gesture that wants a different UX than the two-pane directory model; better as a "send this file" command than a pane-to-pane transport. Deferred as a possible separate feature, not part of the commander. +- *s3cmd / aws s3 / MinIO client (mc)* — object-store CLIs. Subsumed by rclone, which speaks S3 and dozens of backends through one interface. +- *scp / sftp (the CLI)* — basic SSH transfer. Subsumed by rsync (efficient, resumable, async) over the same transport and by TRAMP for browsing. +- *git-annex* — content-addressed file management with its own model. Niche, and its workflow does not map onto a directory-pair commander. Left to the extension API. +- *restic / borg / rdiff-backup* — backup tools. A different problem from interactive sync (snapshots, retention, dedup); out of scope. +- *fpsync / parsyncfp* — parallel rsync wrappers. A tuning of rsync, not a new capability; belongs inside the rsync backend's options if parallelism matters. +- *Unison as the package name* — the perfect-meaning name is already the canonical bidirectional synchronizer; using it would mislead. Adopted unison as the stage-3 sync *engine* instead, and took DUET as the name. + +* Appendix B: Competitive landscape and product validation + +Research across the orthodox commanders (mc, Total Commander, Far, Sunrise Commander), modern TUI managers (ranger, lf, yazi, nnn, vifm), Syncthing, rclone + transfer GUIs (RcloneBrowser, Cyberduck, Mountain Duck, WinSCP, FileZilla), and category-wide failure modes. Sentiment is from HN, Reddit, project forums, and issue trackers — genuine reception, not marketing. + +** What the category loves — and DUET's coverage + +| Loved | Seen in | DUET | +|-------+---------+------| +| Two-pane direct manipulation (copy/move between visible panes, no path retyping) | mc, FileZilla, Total Commander | Core — the whole UX | +| Async, non-blocking transfer + live monitor with cancel | yazi (the literal ranger→yazi migration driver) | Yes — queue + monitor + cancel; the headline, not a v2 nicety | +| Cloud / many-protocol reach | rclone (70+ backends) | Yes — rclone backend (stage 2) | +| Reuse the system ssh stack (ssh-config, keys, agent) | the absence of it in mc's SFTP is the anti-pattern | Yes — TRAMP + ssh-config + the wizard | +| Native rendering, no terminal-graphics fragility | yazi's chronic sixel/kitty preview bugs | Yes — GUI Emacs sidesteps the whole problem | +| Filtering / fuzzy-find | nnn (pulled defectors back from yazi) | Yes — free via dired/consult | +| Local-first, no account or subscription | Syncthing | Yes — local, no cloud account | +| Batteries-included defaults | lf's "assemble it yourself" is the anti-pattern | Yes — sensible transport/preview defaults | + +** What the category hates — and whether DUET avoids it + +| Hated | Seen in | DUET | +|-------+---------+------| +| Synchronous transfers freeze the UI | mc, ranger | Resolved — async + queue + monitor | +| Bolted-on remote auth ignores ssh_config | mc SFTP | Resolved — TRAMP / ssh-config reuse | +| Silent restart-from-zero, no resume | rclone (most-cited) | Resolved — visible per-backend resume contract | +| Opaque errors buried at =-vv= | rclone | Resolved — error table + monitor surfaces retry/restart | +| Inaccurate ETA (avg speed, growing totals) | rclone | Resolved — rolling-speed ETA, recomputed totals | +| Leftover temp/partial residue | rclone, category-wide | Resolved — cleanup-verifier + =cleanup-unverified= state | +| Config / flag overload | rclone | Resolved — registry hides flags, wizard hides TRAMP syntax | +| Silent conflicts / timestamp auto-win / auto-delete-propagation | Syncthing (the trust-killers) | Resolved (stage 3) — loud conflicts, no auto-win, explicit + dry-run + max-delete deletes | +| Adware / bundleware trust hit | FileZilla | N/A — GPL, MELPA, no monetization | +| Heavyweight derived mode breaks TRAMP/omit/icons, then orphaned | Sunrise Commander | Resolved — thin layer on dirvish that coexists; backend API invites contributors | +| Data-loss traps (symlink-delete, =--inplace= corruption, case collisions, cross-device move) | category-wide | Resolved — see "Filesystem edge cases and data-safety" | + +** Where DUET ranks + +- *Unique, no real competitor:* native remote-to-remote transfer (no TUI manager does it; FileZilla is local↔remote only; mc's remote is the thing people flee); an Emacs-native two-pane commander with a pluggable multi-backend transport; rsync depth + rclone breadth + lftp + unison sync under one UX. +- *At parity with the best:* two-pane direct manipulation (mc/FileZilla), async monitor (yazi), cloud reach (rclone), filtering (nnn). +- *Weak spots:* Emacs-only audience (the addressable market, not a flaw, but a ceiling); TRAMP large/remote-dir listing latency; external-CLI dependencies (handled by =make doctor= + graceful warnings); single-maintainer survivability risk (mitigated by building on dirvish so a stall doesn't rot the user's dired, and by the contributor-friendly backend API — the two failure modes that killed Sunrise). + +** What users will like least — and is it a dealbreaker? + +- *Most likely top frustration: remote-directory listing latency through TRAMP.* This is the one thing that could send users back to a terminal + rsync. It is an execution risk, not a design dealbreaker: the responsiveness contract requires async/streamed listing that never freezes Emacs, and transfers move bytes through rsync/rclone, never TRAMP byte-streaming. If that contract is met, the frustration is bounded; if it is not, it is the make-or-break. +- *The Emacs-only ceiling* caps reach but is not a dealbreaker for the target user — it is the market. +- *Learning curve* (Emacs + F-keys + connection model) — mitigated by which-key, sensible defaults, and a real quick-start; the target audience likes keybindings. + +** Validation verdict + +DUET occupies a real, unoccupied niche: the only async, multi-backend, remote-to-remote two-pane commander, and the only one living in Emacs where filtering, preview, and editing are free and terminal-graphics fragility is absent. It directly resolves the #1 complaint of its closest analogs (blocking transfers — mc/ranger) and the #1 complaint of its transport engine (silent no-resume — rclone), while avoiding the two traps that killed its closest predecessor (Sunrise's heavyweight-derived-mode fragility and loss of a maintainer). The product is well-justified; the principal risk is execution on remote-listing responsiveness, not product-market fit. + +* Appendix C: Transfer and sync failure-mode research + +Third research pass, 2026-06-06. Sources: Syncthing docs on block verification, temp files, conflicts, case sensitivity, versioning, ignored deletes, and missing folder markers; Dropbox help on sync icons/status, conflicted copies, selective-sync conflicts, stuck sync, locked/read-only files, symlinks, and recovery; rclone docs on cloud backend capability differences and bisync =--inplace= corruption risk; current community reports about stuck sync, opaque file lists, conflicts, and reappearing files. Full review matrix: [[file:duet-spec-review.org][duet-spec-review.org]]. + +** General transfer/sync edge cases + +| Case | DUET disposition | +|------+------------------| +| Partial destination after interruption | Covered: atomic temp writes, verify, cleanup verifier. | +| =--inplace= corruption | Covered: forbidden in stage-3 sync; opt-in only outside sync. | +| Source changes during copy | Covered: source snapshot + =source-changed= terminal state. | +| Destination full / quota exceeded | Covered: preflight when possible + distinct quota/space failure. | +| Locked/read-only files | Covered: distinct locked/permission failure. | +| Permission/owner/ACL/xattr mismatch | Covered: preserve only when both ends support; backend contract declares support. | +| Symlink traversal / special files | Covered: =lstat=, never recurse symlink on delete, unsupported specials surfaced. | +| Case collisions / restricted names / invalid encodings | Covered: preflight destination compatibility checks. | +| Duplicate remote objects with same display name | Stage-2 rclone concern: diagnose and refuse or route to dedupe before sync. | +| Same-file / dir-into-itself / trailing-slash ambiguity | Covered: canonicalization and resolved-destination display. | +| Move after partial copy | Covered: delete source only after verified per-source copy success. | +| Modify-vs-delete sync conflict / deleted file reappears | Avoided in v1; stage 3 must surface both sides and never auto-win by timestamp. | +| Selective/ignored folder conflicts | Avoided in v1; stage 3 treats ignored/selective trees as preflight state. | +| Missing sync root / unmounted drive interpreted as mass deletion | Avoided in v1; stage 3 baseline/root checks refuse until confirmed. | +| Watcher delay / missed filesystem event | Avoided in v1; stage 3 dry-run plan is the trusted surface, not hidden watcher state. | +| Clock skew / unreliable mtime | Avoided in v1 transfer; stage 3 uses content/version metadata where possible. | +| Network drop / auth / route failure | Covered: async failure states, route notices, source unchanged. | +| Hung transfer | Covered: stall timer, monitor state, cancel/kill. | +| Failure unnoticed | Covered: terminal-state notification + monitor/mode-line visibility. | +| Too many files / huge directory | Covered as highest-risk implementation area: async listing, lazy stat, cached results, throttled UI. | + +** Syncthing and Dropbox lessons + +What people love and DUET's parity: + +| Loved behavior | DUET answer | +|----------------+-------------| +| Continuous sync across machines | Stage 3 only; explicitly not v1. | +| Local-first / no cloud account | Yes for v1 local/SSH flows. | +| Privacy and user-owned storage | Yes: SSH/auth-source, no service account. | +| Status surface | Yes: =duet-transfers=, log, notifications. | +| Recovery/versioning | Partial v1 via trash; stage 3 needs explicit versioning/restore story. | +| Conflict copies preserve both sides | V1 prompts before overwrite; stage 3 surfaces both versions. | +| Fast local/LAN/block transfer | Yes via rsync for Unix/local/SSH pairs. | +| Cloud breadth | Stage 2 via rclone. | +| Simple mental model | Yes: explicit pane-to-pane commands, no background magic in v1. | + +Top pain points and whether DUET avoids/resolves them: + +| Pain point | DUET answer | +|------------+-------------| +| Stuck "syncing X files" with no list | Resolved: monitor rows and log identify work. | +| Conflicted copies multiply | Avoided in v1; stage 3 explicit conflict UI. | +| Deleted files reappear | Avoided in v1; stage 3 no silent modify/delete winner. | +| Selective-sync conflict folders | Avoided in v1; no selective background sync. | +| Mass delete propagation | Stage 3 dry-run + max-delete threshold. | +| Partial/corrupt files after interruption | Resolved: atomic temp + verify + no =--inplace= sync. | +| Temp/residue files left behind | Resolved: cleanup verifier and =cleanup-unverified=. | +| Vague errors hidden behind icons/log levels | Resolved: actionable messages + diagnostics. | +| Source/destination changed during run | Covered: source snapshot; stage 3 snapshot/dry-run model. | +| Online-only placeholder confusion | Mostly avoided; stage-2 cloud backend surfaces capability limits. | +| Symlink surprises | Covered by explicit symlink policy. | +| Case/restricted filename mismatch | Covered by compatibility preflight. | +| Locked/open/read-only file | Covered by distinct failure state. | +| Huge file counts bog down client | Covered by async listing and responsiveness tests, but remains the main implementation risk. | +| Completion/failure missed | Resolved by notification policy and non-success mode-line indicator. | + +* Appendix D: CLI and destination gotcha matrix + +DUET relies heavily on external CLIs. Success means a user who hit a confusing =rsync=, =rclone=, =lftp=, or =unison= failure would choose DUET for the better explanation, not merely for pane convenience. Sources: rsync exit values and diagnostics ([[https://rsync.samba.org/ftp/rsync/rsync.1.html][rsync(1)]]); rclone docs on checksums, partials, retries, logging, exit codes, backend capabilities, and bisync =--inplace= risk ([[https://rclone.org/docs/][rclone docs]], [[https://rclone.org/overview/][overview]], [[https://rclone.org/bisync/][bisync]]); lftp manual on retry/resume, passive mode, TLS verification, temp files, disk-full handling, and mirror options ([[https://lftp.yar.ru/lftp-man.pdf][lftp(1)]]); Unison troubleshooting on archives, root identity, conflicts, hostname changes, and debug output ([[https://alliance.seas.upenn.edu/~bcpierce/wiki/index.php?n=Main.UnisonFAQTroubleshooting][Unison troubleshooting]]). + +** Backend gotchas + +| Backend | Common failures / gotchas | DUET handling | +|---------+---------------------------+---------------| +| rsync over ssh | exit 23 partial transfer, exit 24 vanished source, file I/O, protocol stream, timeout, permission denied, no space, remote =rsync= missing, login shell prints banner text and breaks protocol, symlink policy surprises, FAT/Windows mtime granularity causing repeats. | Failure-normalizer maps exit/stderr to =partial=, =source-vanished=, =permission=, =space=, =protocol=, =timeout=, =remote-tool-missing=. Backend doctor checks remote =rsync=, clean shell, version, and writability. Use tracked =--partial-dir=, explicit symlink policy, and mtime tolerance note for FAT-like destinations. | +| rclone | config/auth missing, token expired, 403/429 rate/quota, storage quota, transfer cap, no shared checksum, modtime unsupported/upload-time-only, duplicate cloud objects, case/restricted name mapping, crypt/hash limitations, server-side copy fallback, huge listing backlog, low-level retries make progress look hung, =--inplace= corruption risk. | Doctor checks =rclone version=, =listremotes=, auth/config, =about= where supported, and backend capabilities. Parse exit codes and high-priority log lines into named states. Prefer JSON/stats output; show retry-after/throttle guidance; refuse/dedupe duplicate objects; record verification basis when no common hash; never use =--inplace= for sync. | +| lftp / FTP / FTPS / HTTP | passive/active NAT failures, TLS certificate errors, old LIST/NLST behavior, REST/resume unsupported, disk full waiting instead of aborting, hidden files omitted, FXP direct server-to-server often fails, server charset mismatch. | Default passive mode; keep certificate verification on and expose explicit trust/fingerprint workflow; parse lftp-prefixed errors; force disk-full fatal where possible; record temp/rename capability; log FXP fallback; expose hidden-listing and charset policy in doctor. | +| TRAMP fallback | slow listings, auth prompts, no honest byte progress, stale cache, remote process quirks. | Use mainly for browsing/fallback; prefer rsync/rclone/lftp for bytes. Monitor says "no progress signal (TRAMP)"; listing path is async/lazy/cached. | +| unison stage 3 | archive missing/corrupt, hostname/root changes create new archive identities, missing root/unmounted drive, conflicts, temp commit-log permission errors, verbose debug needed to isolate one file. | Stage 3 only. Always dry-run first; refuse missing roots; expose archive/root identities in doctor; never delete archives automatically; route conflicts to diff UI; capture redacted debug bundle. | + +** Destination gotchas + +| Destination class | Gotchas | DUET handling | +|-------------------+---------+---------------| +| Local POSIX filesystem | Permissions, ownership, xattrs/ACL support, sparse files, symlinks, device files, ENOSPC. | Preflight writability/space; preserve metadata only when supported; special-file policy explicit. | +| FAT/exFAT/NTFS/mac case-insensitive volumes | Case collisions, path length, reserved names, mtime granularity, symlink limitations. | Destination compatibility scan; mtime tolerance; symlink follow/preserve prompt; reject impossible names before transfer. | +| SSH Unix host | Remote =rsync= missing/version mismatch, shell startup output, sudo/root-owned files, remote disk quota. | Doctor checks remote executable and clean shell; permission/quota states; no hidden sudo escalation. | +| NAS/NFS/SMB mounts | Mounted-over directories can look empty, stale file handles, POSIX metadata lies, symlink/ACL behavior varies. | Root identity preflight; source snapshot; metadata capability warnings; no source delete until verified. | +| FTP/FTPS | Passive/NAT, TLS certs, LIST quirks, no atomic rename on some servers, weak metadata. | lftp diagnostics; temp/rename support recorded; certificate trust explicit; metadata expectations lowered. | +| S3/object stores | Directories are not real directories, ETags are not always MD5, eventual consistency, no POSIX metadata, server-side-copy limitations. | rclone capability probe; checksum kind logged; no POSIX metadata promise; fallback visible. | +| Google Drive / shared drives | API quotas/rate limits, duplicate names, shortcuts/Google Docs pseudo-files, server-side copy surprises. | Rate/quota state with retry guidance; duplicate-object refusal/dedupe guidance; unsupported pseudo-files documented. | +| Dropbox via rclone | Path/case restrictions, symlink behavior, modtime/checksum differences from desktop client. | Backend defaults respected; verification basis shown; symlink policy explicit. | +| OneDrive/SharePoint | Path length/reserved names, quota, throttling, case-insensitivity, occasional size/reporting oddities. | Compatibility preflight; quota/throttle states; verification fallback recorded. | +| WebDAV | Weak checksum/metadata, server-specific locking, slow listings. | Capability probe; conservative verification; locked-file state. | + +** DUET-over-CLI success bar + +DUET should be worth opening when a CLI transfer is failing. It must provide: + +- preflight checks before surprise stderr: executable, auth/config, destination writability, quota when available, backend capabilities, route/fallback, and dangerous sync/delete plan; +- normalized terminal states instead of raw exit codes; +- the failure explainer: likely cause, evidence, safety outcome, and next action; +- visible queue/progress/ETA/stall/cancel instead of terminal scrollback; +- redacted logs and bug-report bundles instead of asking users to paste secrets from verbose CLI output; +- retry/resume/cancel semantics that name what residue remains; +- destination-specific warnings before damage, not after; +- one-key answers to "why this backend?", "what command ran?", "what failed?", and "what should I do next?" + +If these are not implemented, a user comfortable with =rsync= or =rclone= has little reason to choose DUET beyond pane convenience. + +* Review dispositions + +Modified and rejected recommendations, each with a reason. Everything else from the 2026-06-06 external review was accepted as written and woven into the body above. + +- *H5 (docs/tooling) — modified (scope/timing).* Accepted that README, developer-guide, Makefile, and TESTING.org are v1 product surface. Modified the doc split: =docs/user-guide.org= is deferred until there is UI worth documenting; front-loading a workflow guide for an unbuilt UI is premature, and the review itself raised the split as an open question. +- *H6 (diagnostics) — modified (phasing).* Accepted the full log schema as a v1 requirement. Phased the diagnostic command set: a core subset ships with the transfer flow (Phase 8); =duet-copy-last-command=, =duet-test-connection=, =duet-describe-current-panes=, =duet-clear-transfer-log= follow. The schema is the contract; the convenience commands grow. +- *M1 (stale scaffolding facts) — modified (already addressed).* The review's specifics are stale: =~/code/duet= is already git-initialized on =main=, =*.elc= is gitignored, and no =duet.elc= remains. Accepted the framing change ("normalize the existing skeleton," not "create from nothing"); the git-init and =.elc= concerns are already resolved this session. +- *M6 (defcustom inventory) — modified (phasing).* Accepted the full inventory as the documented target. Modified delivery: the load-bearing subset ships with the phase that needs it rather than landing twenty knobs in Phase 1; cosmetic knobs (=stderr-tail-lines=, =conflict-default-action=) arrive as the behavior they tune does. +- *"v1 requires everything" framing — modified across H5/H6/M6.* The full product-grade surface (exhaustive defcustoms, the complete diagnostic suite, the doc trilogy, the contract-test macro, a coverage gate) is treated as *graduation/publish* requirements, timed across the phases, so they sharpen the public release without blocking the first trustworthy internal build. The core commander ships and earns trust first. + +No round-1 recommendation was rejected outright; the review was strong and its safety-oriented findings (H3, H4, H7) materially improved the spec. + +Round 2 (2026-06-06): accepted H8-H11, M7, and M10-M12 as written and folded them into the body (Transfer queue and concurrency, Transfer monitor UX, Responsiveness contract, the expanded Backend developer contract, Complexity and refactoring controls, the Weak-point mitigations table, the sparse-pane principle, and qualitative coverage). Modified: + +- *M8 (archive) — modified (scope/timing).* Accepted dired's compression as inherited in panes; deferred =duet-compress-to-other-pane= to v1.1 so archive support does not bloat v1 copy/move. +- *M9 (connection UX) — modified; overrides a prior decision, confirmed with Craig.* Accepted dropping the explicit password step. The wizard now asks host/user/directory and TRAMP authenticates on demand, bracketed by DUET connect/connected messages so the prompt stays attributed and timely; the passwordless-reconnect convenience still arrives via the Phase-9 authinfo save. Craig confirmed this supersedes his earlier four-field wizard (he asked specifically that the prompt be timely and attributed, which the bracketing messages address). + +No round-2 recommendation was rejected outright; the verdict moved from Not-ready to Ready-with-caveats, and the caveats are now folded in. + +Round 3 (2026-06-06): three further reviewer passes (final-readiness = Ready; transfer/sync-failure research; backend extensibility) were authored and folded in. The responder verified the fold against the current spec and confirmed correct phasing — the new failure states (=source-changed=, =destination-full=, permission/locked), the Failure explainer, the Completion/failure/notification policy, and the expanded error-message table are genuine v1 safety (they hit rsync/TRAMP too), while the backend-specific normalizers and destination gotchas correctly live in Appendices C/D as reference for the stage where each backend lands. Accepted as written, with one modification: + +- *Backend-author tooling — modified (phasing).* The third-party-plugin authoring surface (=duet-scaffold-backend=, =duet-capture-failure-fixture=, the published =duet-define-cli-backend= macro, the "first backend in 30 minutes" onboarding, and the copy-paste pattern tables) was scoped from v1/Phase 2 to stage 2. v1 keeps the registry, the failure-normalizer interface, and the contract-test macro (the built-ins exercise these); the author-facing DX is built once the rclone backend has hardened the abstraction against a real second implementation. Freezing and documenting an extension surface before a second backend exists, with no third-party authors yet to serve, is the same premature-surface trap the round-1/2 phasing discipline guards against. + +The two non-blocking open questions the research raised (duplicate cloud objects; stage-3 versioning/restore) were added to Open Questions. No round-3 recommendation was rejected outright; the verdict is =Ready=. + +Final meta-review (2026-06-06): a process-focused pass (review-as-risk-reduction, separate blockers from preferences, require rollout/rollback for state-changing surfaces). Its one concrete finding — reversibility for generated config and first-connect side effects — was folded-in as the "Rollback and reversibility" section, =duet-rollback-generated-config=, Phase 9.5, and matching acceptance criteria. Responder verified the fold: correctly scoped (rollback for the Phase-9 writers, sequenced as Phase 9.5; append-only keys and manual+confirmed credential removal respect the safety principles), accepted as written, nothing to modify or reject. Verdict: =Ready=. + +* Review and iteration history + +** 2026-06-06 Sat @ 01:24:53 -0500 — External reviewer — reviewer + +- *What changed or was recommended:* Renamed the file from =duet.org= to =duet-spec.org= for spec-workflow compliance and wrote a companion implementation-readiness review. Verdict: =Not ready=. Recommended workflow-grade implementation phases and acceptance criteria, making the minimal backend registry part of v1, defining persistence/side-effect ownership, specifying the transfer execution contract, and adding project principles, product-grade documentation, Makefile/coverage tooling, diagnostics, a defcustom inventory, dired/dirvish inheritance, and error-message contracts. +- *Why:* The draft gave a coherent product direction but left key build contracts to the implementer — phase boundaries, registry timing, file/auth writer ownership, safe copy/move semantics, remote residue cleanup, contributor tooling, and failure diagnosis. +- *Artifacts:* =duet-spec-review.org= (consumed and deleted by the response pass). + +** 2026-06-06 Sat @ 02:05:00 -0500 — Maintainer — responder + +- *What changed:* Folded the full external review in. Accepted H1-H4 and H7 wholesale: added Project principles, Persistence and side-effect boundaries, the Transfer execution contract, Implementation phases (0-10, TDD, repo-scoped, ≤3h), and Acceptance criteria. Resolved the H2 registry contradiction in favor of a v1 registry with rsync + TRAMP registered through =duet-register-backend=. Added Inherited dired/dirvish behavior (M3/M4), an Error-messages table (M5), the Defcustom inventory (M6), a v1 diagnostics core (H6), keymap-precedence detail and tests (M2), and Documentation/tooling scope (H5). Moved settled items into Agreed decisions, including the local↔local-uses-rsync uniform-path call and the deferral of the first-connect writers to Phase 9 behind disabled-by-default defcustoms. +- *Why:* The review's safety findings (side-effect ownership, safe move semantics, remote residue, secrets redaction) were the high-value content for a filesystem-mutating tool and were accepted in full. The modifications were timing/scope: keep the core commander shippable by treating exhaustive product surface (full diagnostics, doc trilogy, defcustom set) as graduation requirements phased across the build rather than gates on the first working version. See Review dispositions. +- *Artifacts:* This spec; the deleted =duet-spec-review.org=; the DUET parent task in =todo.org=. + +** 2026-06-06 Sat @ 02:30:00 -0500 — External reviewer — reviewer + +- *What changed or was recommended:* Ran a second readiness pass after the first review was folded in. Verdict: =Ready with caveats=. Recommended adding transfer concurrency/queueing, a compact transfer monitor UX, measurable responsiveness under heavy process output, a more precise backend developer contract, complexity/refactoring controls, archive-operation opportunities based on dired's built-in compression support, a weak-point mitigation table, a simpler v1 connection hot path, sparse-pane UX principles, and qualitative coverage expectations beyond the 80% target. +- *Why:* The spec is now implementable, but DUET's hardest production risks are responsiveness under load, backend extensibility, long-running transfer visibility, and complexity creep. These caveats make those risks explicit before code hardens around accidental choices. +- *Artifacts:* =duet-spec-review.org= (round 2; consumed and deleted by this response pass). + +** 2026-06-06 Sat @ 02:35:00 -0500 — Maintainer — responder + +- *What changed:* Folded round 2 in. Accepted H8-H11 + M7 + M10-M12 wholesale: added Transfer queue and concurrency (serial, default 1, state machine), Transfer monitor UX (=duet-transfers= tabulated buffer, honest ETA), the Responsiveness contract (bounded filters, throttled rendering, timer-driven hung detection), the expanded Backend developer contract (ABI, capability semantics, failure taxonomy, contract-test macro), Complexity controls, the Weak-point mitigations table, the sparse-pane principle, and qualitative coverage. Wove the new work into Implementation phases 2/5/8/10 and Acceptance criteria. Resolved the round-2 open questions into Agreed decisions (concurrency default 1, monitor = tabulated-list-mode, ETA only from byte-progress, archive-to-other-pane = v1.1). Modified M8 (archive deferred to v1.1) and M9 (wizard simplified to host/user/dir, password via TRAMP on demand) — M9 with Craig's explicit confirmation and the connection-feedback bracketing he asked for. +- *Why:* Round 2's caveats targeted the real production risks — responsiveness under load, long-running-transfer visibility, extension-surface misuse, and complexity creep — so they were accepted in full. The one decision returned to Craig (M9) overrode his earlier wizard design; he chose the simpler hot path and asked that the resulting TRAMP prompt be timely and attributed, which the bracketing-message design delivers. +- *Artifacts:* This spec; the deleted round-2 =duet-spec-review.org=; the DUET tracking tasks in =todo.org=. + +** 2026-06-06 Sat @ 02:55:57 -0500 — Maintainer — research + author + +- *What changed:* Folded in a competitive-research pass (Appendix B): parallel research streams on the orthodox commanders (mc/Total Commander/Far/Sunrise), modern TUI managers (ranger/lf/yazi/nnn/vifm), Syncthing, rclone + transfer GUIs, and category-wide failure modes, sourced from HN/Reddit/forums/issue trackers. Added Appendix B (loved/hated tables with DUET's coverage, ranking, what-users-like-least, validation verdict). The research forced new build content: a "Filesystem edge cases and data-safety" section (argv/NUL file lists, =lstat= symlink classification, same-file/dir-into-itself rejection, cross-device copy-verify-delete, case-insensitive collisions, path-length/reserved names, trash-by-default, verify-on-completion); four new Project principles (visible resume, atomic writes, never-shell-strings, trash-default); resume/ETA/argv/verify additions to the transfer contract; async-streamed directory *listing* in the responsiveness contract (the highest-risk path); and stage-3 sync-safety rules drawn from Syncthing's conflict/delete-propagation failures (loud conflicts, no timestamp auto-win, explicit + dry-run + max-delete deletes, baseline-required first sync). +- *Why:* Craig asked for competitive validation — do we avoid what people hate, match what they love, and where do we rank. The answer: DUET sits in an unoccupied niche (async, multi-backend, remote-to-remote, Emacs-native) and resolves the #1 complaints of mc/ranger (blocking) and rclone (no resume) while avoiding Sunrise's fragility. The research's highest-value output was the category data-loss traps the prior spec hadn't covered — now closed — and naming remote-listing latency (TRAMP) as the make-or-break execution risk rather than a design flaw. +- *Artifacts:* This spec (Appendix B + the data-safety section + principle/contract additions); five research-agent reports (sources cited inline in Appendix B's framing). + +** 2026-06-06 Sat @ 04:32:35 -0500 — External reviewer — final readiness reviewer + +- *What changed or was recommended:* Re-ran the spec-review readiness gate after the response and research passes. Verdict: =Ready=. No new blocking recommendations. The remaining Open Questions are intentionally non-blocking: publishing target is a graduation decision; F3/F4 can reuse dired defaults for v1; lftp/rclone FTP ownership is stage 2; stall thresholds need backend behavior to calibrate; and the complexity tool can remain a manual checklist until one is selected. +- *Why:* The spec now has the implementation contracts needed to start: phased TDD work, transport registry, side-effect boundaries, transfer execution semantics, data-safety rules, queue/monitor/responsiveness contracts, backend extension API, diagnostics/logging, documentation/tooling, coverage expectations, acceptance criteria, and explicit weak-point mitigations. +- *Artifacts:* This spec; stale round-2 task in =todo.org= marked complete. + +** 2026-06-06 Sat @ 04:52:13 -0500 — External reviewer — researcher + reviewer + +- *What changed or was recommended:* Ran a third online-research pass focused on transfer/sync failures and Syncthing/Dropbox user sentiment. Added terminal-state notifications, source-mutated-during-transfer detection, quota/space and locked/read-only failure categories, notification defcustoms, expanded tests/manual verification, and Appendix C's covered/avoided/deferred matrix. Wrote a companion review matrix at =duet-spec-review.org=. +- *Why:* Craig asked whether common and edge transfer/sync failures were fully covered or avoided, whether DUET should notify on completion/failure, and whether users would discover failures without help. The research showed v1 avoids most continuous-sync trust failures by not doing background sync, but ordinary transfer still needs explicit terminal-state surfacing and a few more filesystem/backend failure states. +- *Artifacts:* This spec; [[file:duet-spec-review.org][duet-spec-review.org]]; updated spec-review workflow and rulesets inbox note. + +** 2026-06-06 Sat @ 05:07:48 -0500 — External reviewer — extensibility reviewer + +- *What changed or was recommended:* Tightened the backend extension model around plugin-author usability. Added minimum runnable / safe publishable / excellent backend tiers; =duet-define-cli-backend=; =duet-define-cli-failure-patterns=; =duet-scaffold-backend=; =duet-capture-failure-fixture=; =duet-run-backend-contract-tests=; tiered contract tests; and developer-doc guidance that authors can start small while declaring capabilities conservatively. +- *Why:* Craig pointed out that DUET's explanatory-backend promise affects future CLI plugin authors. The spec now treats plugin authors as end-users of DUET's developer tooling: they need clear required-vs-optional information, pattern helpers instead of hand-written parsers, and a path that does not make the first useful plugin too complex to write. +- *Artifacts:* This spec; [[file:duet-spec-review.org][duet-spec-review.org]]; updated spec-review workflow and rulesets inbox note. + +** 2026-06-06 Sat @ 05:23:25 -0500 — External reviewer — meta-review applier + +- *What changed or was recommended:* Applied the final meta-review guidance about rollout/rollback and author-usable, risk-focused review. Added Rollback and reversibility, =duet-rollback-generated-config=, rollback acceptance criteria, and Phase 9.5. No new blocker remains. +- *Why:* Online review-practice research reinforced that good spec reviews should avoid churn and focus on implementation risk. DUET's remaining concrete gap was reversibility for generated config and first-connect side effects. +- *Artifacts:* This spec; [[file:duet-spec-review.org][duet-spec-review.org]]; updated spec-review workflow and rulesets inbox note. + +** 2026-06-06 Sat @ 05:14:12 -0500 — Maintainer — responder + +- *What changed:* Ran spec-response over the round-3 review (the transfer/sync-failure pass) and verified the three review passes (final-readiness, failure research, extensibility) against the current spec. Confirmed phasing held — v1 stayed scoped to rsync/TRAMP with the failure-explainer framework; backend-specific normalizers and destination gotchas live in Appendices C/D as stage-2/3 reference, not v1 gates. One modification: scoped the third-party-plugin authoring tooling (scaffold, capture-fixture, published declarative macro, onboarding docs, pattern tables) from v1/Phase 2 to stage 2, keeping only the registry + normalizer interface + contract-test macro in v1. Added the two non-blocking open questions (duplicate cloud objects, stage-3 versioning/restore) and recorded a round-3 dispositions block. Deleted the round-3 review file. +- *Why:* The failure-research additions are high-value safety the prior spec lacked (terminal-state notification, source-changed/quota/locked states, the diagnostic explainer that justifies DUET over the raw CLI), so they were accepted in full. The one modify continues the established discipline: don't freeze and document an extension surface before a second backend has revealed the right shape, when there are no outside authors yet to serve. +- *Artifacts:* This spec; the deleted round-3 =duet-spec-review.org=; the DUET tracking tasks in =todo.org=. + +** 2026-06-06 Sat @ 05:27:25 -0500 — Maintainer — responder + +- *What changed:* Ran spec-response over the final meta-review. Verified the folded-in rollback/reversibility addition against the spec: the "Rollback and reversibility" section, =duet-rollback-generated-config=, Phase 9.5, and the rollback acceptance criteria all landed and are correctly scoped — rollback covers the Phase-9 generated-config writers and is sequenced right after them, append-only key handling and manual+confirmed credential removal respect the "user-owned config" and "no silent data loss" principles, and stage-3 sync rollback correctly defers to the versioning/restore open question. Accepted as written; nothing to modify or reject. Deleted the final review file. Verdict: =Ready=. +- *Why:* Reversibility for state-changing first-connect writers is the right complement to those writers and pairs cleanly with the existing safety principles, so it was accepted in full. This closes the review cycle: four rounds plus competitive research, all folded, no open blockers. +- *Artifacts:* This spec; the deleted final =duet-spec-review.org=; the Duet tracking task in =todo.org=. Flag: the reviewer also edited =.ai/workflows/spec-review.org= in-project — a template-synced file that reverts on next startup unless the change is committed to rulesets. -- cgit v1.2.3