test(net): add the doctor control-plane repair scenarios and runner

net Phase 1's three privileged fixes (unmask-nm, disable-rival, chmod-keyfile) can't be verified against fakes past the point where real sudo and a running NetworkManager matter, and the broken states they repair are too dangerous to stage on a daily driver. I added the live-verification harness they need.

Three scenario files encode the break/fix/assert for each: mask NetworkManager, enable a rival dhcpcd, and chmod a profile keyfile to 0644, then run the real net doctor --fix and assert the repair. Each also names the read-only diagnose verdict it expects, so a run that breaks catches whether the fault is in the diagnosis or the fix. run-net-scenarios.sh rsyncs the net and panelkit trees into a target VM over ssh and drives the scenarios there.

The target is a booted VM, not a container: NetworkManager needs a real kernel and network stack to actually run, so an nspawn container can only show the filesystem-level half (the mask symlink, the keyfile mode) and not "NM is active". A disposable VM you revert is the right environment.

The scenario break/fix/assert logic is the reviewed part. The ssh transport plumbing hasn't been exercised against a live target yet, so it's marked first-draft and may need a shakeout on the first real run. The by-hand equivalent already lives in the manual-testing checklist.

1 files changed, 115 insertions, 0 deletions

diff --git a/scripts/testing/run-net-scenarios.sh b/scripts/testing/run-net-scenarios.sh
new file mode 100755
index 0000000..7197e37
--- /dev/null
+++ b/scripts/testing/run-net-scenarios.sh
@@ -0,0 +1,115 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Run net-doctor control-plane repair scenarios against a booted VM.
+# Author: Craig Jennings <craigmartinjennings@gmail.com>
+# License: GNU GPLv3
+#
+# Live verification for net Phase 1's three privileged fixes (unmask-nm,
+# disable-rival, chmod-keyfile). Each scenario breaks the target in a known
+# way, runs the real `net doctor --fix` inside it, and asserts the post-state.
+# These states are dangerous on a daily driver (masking NM kills the network),
+# so they run against a DISPOSABLE VM you can revert, never ratio/velox.
+#
+# The target must be a booted VM (a real kernel + network stack — NetworkManager
+# does not run reliably in an nspawn container) reachable over ssh as root, with
+# NetworkManager and dhcpcd installed. The runner rsyncs the net + panelkit
+# source trees in and runs the doctor from them via a small wrapper (no install
+# needed), matching the stowed shim. jq is used host-side to read the JSON.
+#
+# STATUS: first-draft harness. The scenario break/fix/assert logic is the
+# reviewed part; the ssh transport plumbing has not been exercised against a
+# live target yet and may need a shakeout on the first real run.
+#
+# Usage: run-net-scenarios.sh --target root@HOST [--list] [--profile NAME]
+#   --target   ssh destination of the booted VM (required unless --list)
+#   --profile  a saved NM profile name for the keyfile scenario
+#              (else the keyfile scenario is skipped)
+#   --list     print the scenario plan and exit
+#
+# Reverting the VM to a clean snapshot between runs is the caller's job; each
+# scenario also restores what it broke on the way out.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SCENARIO_DIR="${NET_SCENARIO_DIR:-$SCRIPT_DIR/net-scenarios}"
+DOTFILES="${DOTFILES:-$HOME/.dotfiles}"
+T_NET="/root/net-scenario-tree"   # where the source trees land in the target
+
+TARGET=""
+LIST_ONLY=false
+export NET_SCENARIO_PROFILE="${NET_SCENARIO_PROFILE:-}"
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --target)  TARGET="$2"; shift 2 ;;
+        --profile) NET_SCENARIO_PROFILE="$2"; shift 2 ;;
+        --list)    LIST_ONLY=true; shift ;;
+        *) echo "usage: $0 --target root@HOST [--list] [--profile NAME]" >&2; exit 1 ;;
+    esac
+done
+
+mapfile -t S_FILES < <(find "$SCENARIO_DIR" -maxdepth 1 -name '*.sh' | sort)
+
+if [ "$LIST_ONLY" = true ]; then
+    echo "net doctor scenario plan:"
+    for f in "${S_FILES[@]}"; do
+        ( . "$f"; printf '  %-24s %s\n' "$(basename "$f" .sh)" "$SCENARIO_DESC" )
+    done
+    exit 0
+fi
+
+[ -n "$TARGET" ] || { echo "--target root@HOST is required" >&2; exit 1; }
+command -v jq >/dev/null || { echo "jq is required host-side" >&2; exit 1; }
+
+SSH() { ssh -o BatchMode=yes "$TARGET" "$@"; }
+
+# ── target helpers (available to the scenario scripts) ───────────────────
+# nexec runs a payload in the target as root; the doctor runs through the
+# /root/net-run wrapper synced below, so no fragile inline quoting is needed.
+nexec()        { SSH "bash -lc $(printf '%q' "$*")"; }
+nfix()         { nexec "/root/net-run doctor --fix --json"; }
+ndoctor_json() { nexec "/root/net-run doctor --json" | jq -er "$1"; }
+
+info() { printf '  %s\n' "$*"; }
+pass() { printf '  PASS  %s\n' "$*"; }
+fail() { printf '  FAIL  %s\n' "$*"; }
+
+# ── sync the source trees + the doctor wrapper into the target ───────────
+echo "==> syncing net + panelkit into $TARGET:$T_NET"
+SSH "mkdir -p $T_NET/net $T_NET/panelkit"
+rsync -a -e "ssh -o BatchMode=yes" "$DOTFILES/net/src" "$TARGET:$T_NET/net/"
+rsync -a -e "ssh -o BatchMode=yes" "$DOTFILES/panelkit/src" "$TARGET:$T_NET/panelkit/"
+# The wrapper: run as root, so NET_SUDO empty (commands run directly) and
+# PANELKIT_SUDO=sudo (root's `sudo -n true` succeeds -> the model resolves RUN).
+SSH "cat > /root/net-run" <<EOF
+#!/bin/bash
+exec env PYTHONPATH=$T_NET/net/src:$T_NET/panelkit/src NET_SUDO= PANELKIT_SUDO=sudo \\
+    python3 -c 'import sys; from net.cli import main; sys.exit(main(sys.argv[1:]))' "\$@"
+EOF
+SSH "chmod +x /root/net-run"
+
+# ── run each scenario ────────────────────────────────────────────────────
+fails=0
+for f in "${S_FILES[@]}"; do
+    name="$(basename "$f" .sh)"
+    # shellcheck disable=SC1090
+    ( . "$f"
+      case "$name" in
+          *keyfile*) [ -n "$NET_SCENARIO_PROFILE" ] || { info "skip $name (no --profile)"; exit 0; } ;;
+      esac
+      echo "== $name — $SCENARIO_DESC"
+      scenario_break || { fail "$name: break failed"; exit 1; }
+      if declare -F scenario_diagnose_expect >/dev/null; then
+          if scenario_diagnose_expect; then pass "$name: diagnose named it"
+          else fail "$name: diagnose did NOT name it (inspect net doctor --json)"; fi
+      fi
+      scenario_fix || info "$name: net doctor --fix returned non-zero"
+      if scenario_assert; then pass "$name: repaired"
+      else fail "$name: NOT repaired"; exit 1; fi
+    ) || fails=$((fails + 1))
+done
+
+echo
+[ "$fails" -eq 0 ] && echo "all scenarios passed" || echo "$fails scenario(s) failed"
+exit "$fails"