1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
|
#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup
#+AUTHOR: Craig Jennings
#+DATE: 2026-08-13
Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon
890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry,
and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the
pool. Decision: full reinstall via archangel + archsetup, run deliberately as
a disaster-recovery test of the ISO and scripts before the Sunday flight.
Recent backup in hand; ratio available as the working machine.
Fallback ordering if the test finds a real gap:
- Before partitioning starts: the old system is intact — the ZBM repair
route (efibootmgr entry pointing at the ZBM loader on the ESP, then
amd-ucode swap in a chroot) is still available.
- After partitioning: the floor is a manual Arch install; the backup makes
that survivable.
* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk)
- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates
the microcode vendor-detection fix (archsetup, 2026-08-08) and was built
without ARCHSETUP_DIR at all.
#+begin_src sh
cd ~/code/archangel && sudo ARCHSETUP_DIR=~/code/archsetup ./build.sh
#+end_src
- [ ] build.sh fixes before the final rebuild (archangel repo):
- rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the
credential audit — off the portable USB stick).
- copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile
is on the ISO at =/root/velox-zfs.conf=.
- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=),
=/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs
spot-check.
- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy
drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16
archangel ISOs removed. Boot the stick and pick
=archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu.
* Phase 1 — UEFI setup on velox (BIOS screen, before any boot)
- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned;
the new board ships with it enforced by factory default.
- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01;
a wrong clock breaks TLS and pacman signature checks in the live env.
Rough accuracy is fine — NTP tightens it once networked.
- [ ] While you're in setup: check boot-order UI shows the USB.
* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED
NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is
DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke
around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio
and mybitch backed up today). Everything since Jul 6 exists only on the old
SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO
backup at all. This pass also keeps the repair fallback alive until
partitioning starts.
- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=.
- [ ] Import the old pool read-only and unlock:
#+begin_src sh
zpool import -N -o readonly=on -R /mnt zroot
zfs load-key zroot # passphrase prompt
zfs mount zroot/ROOT/default
zfs mount -a 2>/dev/null # home datasets etc.; ignore failures
#+end_src
- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout
the backup job uses (etc + home), into a clearly-named one-off dir:
#+begin_src sh
rsync -aHAX --info=progress2 /mnt/etc /mnt/home \
truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/
#+end_src
(=/usr= is in the regular backups but is all reinstallable — skip unless
paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.)
- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest
files in =~/documents= and =~/downloads=.
- [ ] Check for uncommitted repo work and either push or note it:
=~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge),
=~/.dotfiles=, anything under =~/code=.
- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=.
* Phase 3 — Install (the actual DR test)
- [ ] Review the profile, then run the installer:
#+begin_src sh
less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme
archangel --config-file /root/velox-zfs.conf
#+end_src
Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome=
placeholders — fine for install; both change post-install (=zfs change-key
zroot= for the pool, =passwd= for root).
- [ ] Record every rough edge as a DR-test finding — that's the point of
running it this way. Anything that needs a manual nudge gets a todo entry
in archangel or archsetup afterward.
- [ ] Reboot into ZBM → boot the new environment.
* Phase 4 — archsetup (first boot of the installed system)
- [ ] Log in as root, network up, then verify the clock synced.
- [ ] Get archsetup — two paths, test the offline one since this is a DR
drill (the online curl path is the everyday alternative):
#+begin_src sh
# offline: mount the install USB and copy the baked tree
mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt
cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup
./archsetup
#+end_src
- [ ] Expected on the new board: =install_cpu_microcode= detects
AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests).
Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all
in the installer now — none need manual application afterward.
- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel
assumptions (swept 2026-08-13); maint's capability probe runtime-detects
amd-pstate.
* Phase 5 — Post-install restore + verification
- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The
secrets-bundle-in-ISO design is not built yet — manual restore is the
known gap, not a test failure.
- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=;
re-import the NM profile (autoconnect off, as before).
- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=.
- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known
idle instability (Panel Self Refresh hangs/reboots the machine; hit during
the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the
installed system's kernel command line — velox boots via ZBM, so set it on
the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default=
(keep the existing args; append). Revisit after a BIOS update ≥3.05 or a
kernel that fixes PSR on Strix Point — track via the Framework issue
tracker (SoftwareFirmwareIssueTracker #110).
- [ ] New-hardware spot-checks:
- =journalctl -k | grep -i microcode= — amd-ucode applied.
- =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect
amd-pstate(-epp).
- wifi + bluetooth up (new board radios), touchpad behavior, camera.
- =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV.
- [ ] Fresh clones automatically carry the post-purge rewritten git history —
closes the clone-reconcile rider from 2026-08-11 without action.
- [ ] Fix and verify the backup timer on the fresh install — it was silently
broken since ~Jul 6. After the first manual run succeeds, confirm a new
DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it
broke (timer unit dead? mount failure? credential?) if the old journal
survives in the salvage copy.
- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate),
both daily drivers AMD. Fix the stale =intel_pstate= comment in
=airplane-mode= line 6 while at it (cosmetic).
- [ ] File every DR-test finding in the owning project's todo.
* Timing
Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or
Friday, leaving Saturday as pure buffer. If the install stalls past Friday
evening, cut losses to the manual-install floor.
|
