aboutsummaryrefslogtreecommitdiff
path: root/working/velox-reinstall
diff options
context:
space:
mode:
Diffstat (limited to 'working/velox-reinstall')
-rw-r--r--working/velox-reinstall/velox-reinstall-runbook.org161
-rw-r--r--working/velox-reinstall/velox-uefi-boot-entry-reference.org108
-rw-r--r--working/velox-reinstall/wttrin-bf0457f.bundlebin1826602 -> 0 bytes
3 files changed, 0 insertions, 269 deletions
diff --git a/working/velox-reinstall/velox-reinstall-runbook.org b/working/velox-reinstall/velox-reinstall-runbook.org
deleted file mode 100644
index 0c14946..0000000
--- a/working/velox-reinstall/velox-reinstall-runbook.org
+++ /dev/null
@@ -1,161 +0,0 @@
-#+TITLE: Velox Reinstall Runbook — DR Test of archangel + archsetup
-#+AUTHOR: Craig Jennings
-#+DATE: 2026-08-13
-
-Context: velox's mainboard swapped Intel → AMD (Ryzen AI 9 HX 370, Radeon
-890M, 96GB RAM). Old SSD intact but the new board's NVRAM has no boot entry,
-and velox is ZFS root + ZFSBootMenu, so a stock Arch USB can't even read the
-pool. Decision: full reinstall via archangel + archsetup, run deliberately as
-a disaster-recovery test of the ISO and scripts before the Sunday flight.
-Recent backup in hand; ratio available as the working machine.
-
-Fallback ordering if the test finds a real gap:
-- Before partitioning starts: the old system is intact — the ZBM repair
- route (efibootmgr entry pointing at the ZBM loader on the ESP, then
- amd-ucode swap in a chroot) is still available.
-- After partitioning: the floor is a manual Arch install; the backup makes
- that survivable.
-
-* Phase 0 — Preflight on ratio (agent-driven, done before you leave the desk)
-
-- [ ] Rebuild the ISO with archsetup baked in: the 2026-08-02 ISO predates
- the microcode vendor-detection fix (archsetup, 2026-08-08) and was built
- without ARCHSETUP_DIR at all.
- #+begin_src sh
- cd ~/code/archangel && sudo ARCHSETUP_DIR="$HOME/code/archsetup" ./build.sh
- #+end_src
- Two traps in that one line, and either alone silently produces a bare ISO
- with archsetup absent (archangel, 2026-08-20). The =VAR=value= form is
- required because sudo's =env_reset= discards an exported variable. And
- =$HOME= is required because zsh does not expand a tilde on the right-hand
- side of an assignment — the earlier =ARCHSETUP_DIR=~/code/archsetup= here
- passed the literal string. build.sh now warns and reports baked/not-baked
- in its closing summary, so the failure is visible rather than silent.
-- [ ] build.sh fixes before the final rebuild (archangel repo):
- - rsync exclude for =.ai= (keeps =archsetup/.ai/private-design/= — the
- credential audit — off the portable USB stick).
- - copy =installer/velox-*.conf= to =airootfs/root/= so the machine profile
- is on the ISO at =/root/velox-zfs.conf=.
-- [ ] Verify the ISO carries: =/code/archsetup= (with =install_cpu_microcode=),
- =/root/velox-zfs.conf=, no =.ai/private-design=. Loop-mount or unsquashfs
- spot-check.
-- [X] USB ready (done 2026-08-13 15:25): the new ISO was copied to the Ventoy
- drive, sha256-verified against the source, and the 2026-04-09 + 2026-06-16
- archangel ISOs removed. Boot the stick and pick
- =archangel-2026-08-13-vmlinuz-6.18.43-lts-x86_64.iso= from the Ventoy menu.
-
-* Phase 1 — UEFI setup on velox (BIOS screen, before any boot)
-
-- [ ] Disable Secure Boot. Mandatory — the ZFS kernel modules are unsigned;
- the new board ships with it enforced by factory default.
-- [ ] Set the system clock. The board swap reset the RTC to 2025-01-01;
- a wrong clock breaks TLS and pacman signature checks in the live env.
- Rough accuracy is fine — NTP tightens it once networked.
-- [ ] While you're in setup: check boot-order UI shows the USB.
-
-* Phase 2 — Salvage pass (live ISO, BEFORE running the installer) — REQUIRED
-
-NOT optional insurance. Verified 2026-08-13: velox's newest truenas backup is
-DAILY.0 = 2026-07-06 — five weeks stale. The backup timer on velox broke
-around Jul 6 (truenas itself only went dark Jul 24, and it's back now; ratio
-and mybitch backed up today). Everything since Jul 6 exists only on the old
-SSD — including =wolf.conf.gpg= (created Jul 29), which is therefore in NO
-backup at all. This pass also keeps the repair fallback alive until
-partitioning starts.
-
-- [ ] Network up (=nmtui= or ethernet), then confirm clock: =timedatectl=.
-- [ ] Import the old pool read-only and unlock:
- #+begin_src sh
- zpool import -N -o readonly=on -R /mnt zroot
- zfs load-key zroot # passphrase prompt
- zfs mount zroot/ROOT/default
- zfs mount -a 2>/dev/null # home datasets etc.; ignore failures
- #+end_src
-- [ ] Push a full fresh backup to truenas over the LAN — mirror the layout
- the backup job uses (etc + home), into a clearly-named one-off dir:
- #+begin_src sh
- rsync -aHAX --info=progress2 /mnt/etc /mnt/home \
- truenas:/mnt/vault/backups/velox/pre-reinstall-2026-08-13/
- #+end_src
- (=/usr= is in the regular backups but is all reinstallable — skip unless
- paranoid. The 96GB-RAM board will not be the bottleneck; the LAN is.)
-- [ ] Spot-check the copy landed: =wolf.conf.gpg=, =.ssh=, =.gnupg=, newest
- files in =~/documents= and =~/downloads=.
-- [ ] Check for uncommitted repo work and either push or note it:
- =~/.emacs.d= (known: the auto-dim-other-buffers.el unresolved merge),
- =~/.dotfiles=, anything under =~/code=.
-- [ ] Export cleanly: =cd /; zfs unmount -a; zpool export zroot=.
-
-* Phase 3 — Install (the actual DR test)
-
-- [ ] Review the profile, then run the installer:
- #+begin_src sh
- less /root/velox-zfs.conf # FILESYSTEM=zfs, HOSTNAME=velox, single nvme
- archangel --config-file /root/velox-zfs.conf
- #+end_src
- Note: the profile's ZFS_PASSPHRASE / ROOT_PASSWORD are the =welcome=
- placeholders — fine for install; both change post-install (=zfs change-key
- zroot= for the pool, =passwd= for root).
-- [ ] Record every rough edge as a DR-test finding — that's the point of
- running it this way. Anything that needs a manual nudge gets a todo entry
- in archangel or archsetup afterward.
-- [ ] Reboot into ZBM → boot the new environment.
-
-* Phase 4 — archsetup (first boot of the installed system)
-
-- [ ] Log in as root, network up, then verify the clock synced.
-- [ ] Get archsetup — two paths, test the offline one since this is a DR
- drill (the online curl path is the everyday alternative):
- #+begin_src sh
- # offline: mount the install USB and copy the baked tree
- mount /dev/disk/by-label/ARCHANGEL* /mnt 2>/dev/null || mount /dev/sdX1 /mnt
- cp -r /mnt/code/archsetup /root/archsetup && cd /root/archsetup
- ./archsetup
- #+end_src
-- [ ] Expected on the new board: =install_cpu_microcode= detects
- AuthenticAMD and installs amd-ucode (verified 2026-08-13, 7/7 tests).
- Podman socket, camera udev rule, tlp radio state, ZFS /tmp mask are all
- in the installer now — none need manual application afterward.
-- [ ] archsetup clones + stows dotfiles. The velox host tier has no Intel
- assumptions (swept 2026-08-13); maint's capability probe runtime-detects
- amd-pstate.
-
-* Phase 5 — Post-install restore + verification
-
-- [ ] Restore from backup (credentials, ssh keys, gpg, user data). The
- secrets-bundle-in-ISO design is not built yet — manual restore is the
- known gap, not a test failure.
-- [ ] WireGuard: decrypt + re-place =wolf.conf.gpg= at =~/.config/wireguard/=;
- re-import the NM profile (autoconnect off, as before).
-- [ ] Change the placeholder passwords: =passwd=, =zfs change-key zroot=.
-- [ ] PSR workaround — REQUIRED on this board. The Ryzen AI 300 has a known
- idle instability (Panel Self Refresh hangs/reboots the machine; hit during
- the live session 2026-08-13). Add =amdgpu.dcdebugmask=0x610= to the
- installed system's kernel command line — velox boots via ZBM, so set it on
- the pool: =zfs set org.zfsbootmenu:commandline="... amdgpu.dcdebugmask=0x610" zroot/ROOT/default=
- (keep the existing args; append). Revisit after a BIOS update ≥3.05 or a
- kernel that fixes PSR on Strix Point — track via the Framework issue
- tracker (SoftwareFirmwareIssueTracker #110).
-- [ ] New-hardware spot-checks:
- - =journalctl -k | grep -i microcode= — amd-ucode applied.
- - =cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver= — expect
- amd-pstate(-epp).
- - wifi + bluetooth up (new board radios), touchpad behavior, camera.
- - =glxinfo -B= / =vulkaninfo --summary= — Radeon 890M on RADV.
-- [ ] Fresh clones automatically carry the post-purge rewritten git history —
- closes the clone-reconcile rider from 2026-08-11 without action.
-- [ ] Fix and verify the backup timer on the fresh install — it was silently
- broken since ~Jul 6. After the first manual run succeeds, confirm a new
- DAILY.0 appears under =truenas:/mnt/vault/backups/velox/=. Diagnose why it
- broke (timer unit dead? mount failure? credential?) if the old journal
- survives in the salvage copy.
-- [ ] Update the machine-identity memory: velox is now AMD (amd-pstate),
- both daily drivers AMD. Fix the stale =intel_pstate= comment in
- =airplane-mode= line 6 while at it (cosmetic).
-- [ ] File every DR-test finding in the owning project's todo.
-
-* Timing
-
-Today is Thursday; the flight is Sunday. Target: Phases 0–4 tonight or
-Friday, leaving Saturday as pure buffer. If the install stalls past Friday
-evening, cut losses to the manual-install floor.
diff --git a/working/velox-reinstall/velox-uefi-boot-entry-reference.org b/working/velox-reinstall/velox-uefi-boot-entry-reference.org
deleted file mode 100644
index 53e6511..0000000
--- a/working/velox-reinstall/velox-uefi-boot-entry-reference.org
+++ /dev/null
@@ -1,108 +0,0 @@
-#+TITLE: Velox UEFI Boot Entry — Recovery Reference
-#+AUTHOR: Craig Jennings
-#+DATE: 2026-08-15
-
-Captured 2026-08-15 before a BIOS update (03.05 → 04.02) as insurance against
-the update clearing NVRAM. Velox's mainboard swap on 2026-08-13 left exactly
-this kind of empty NVRAM, which is what forced the reinstall — so a cleared
-boot entry is the specific failure worth being able to undo in one command
-rather than reconstruct.
-
-* Before recreating anything: check Secure Boot first
-
-The 04.02 update landed on 2026-09-12 (staged via fwupd, flashed on the next
-reboot). It did NOT clear NVRAM: Boot0001 survived with its command line
-intact. What it did was re-enable Secure Boot (Enforce Secure Boot =
-Enabled), so the unsigned ZBM loader was rejected and the Framework BIOS
-reported it as "Default Boot Device Missing / no bootable drive" rather than
-a security violation. The symptom is indistinguishable from the NVRAM wipe
-this document was written for.
-
-The tell: booting the Ventoy stick shows shim's MOK management screen.
-
-Fix: F2 → Security → Secure Boot → Enforce Secure Boot = Disabled, F10. The
-machine then boots straight into ZBM. No efibootmgr needed.
-
-So when velox says no bootable device after a firmware update, check Secure
-Boot before touching the boot entries. Only if Secure Boot is already off
-and =efibootmgr -v= (from the stick) shows Boot0001 gone does the recreate
-below apply.
-
-Before any future firmware update, record both =efibootmgr -v= and the
-Secure Boot state so the post-reboot diagnosis is a comparison, not a guess.
-The pre-firmware-update checklist in
-[[file:../../docs/workflows/system-health-check.org][docs/workflows/system-health-check.org]]
-(Phase 3) carries the steps.
-
-Still open: velox's ESP has no removable-media fallback (=/efi/EFI/BOOT=
-does not exist), so a real NVRAM wipe would still need the stick. Copying
-=zfsbootmenu.efi= to =/efi/EFI/BOOT/BOOTX64.EFI= would let it boot unaided.
-That belongs to archangel's ZBM install, where it is filed as [#C]
-"Installed systems have no removable-media boot fallback on the ESP"
-(2026-09-12) and ships with the next ISO rebuild after it lands.
-
-* State at capture
-
-- BIOS: 03.05 (2025-10-30)
-- BootCurrent: 0001
-- BootOrder: 2001,0001,2002,2003 (USB ahead of ZBM — why the Ventoy stick
- boots when it's inserted)
-- Timeout: 0 seconds
-
-* The entry that matters
-
-=Boot0001* ZFSBootMenu=
-
-| field | value |
-|----------------+----------------------------------------------|
-| ESP part GUID | 8e51b680-f90a-444f-8da5-7e4f93625775 |
-|----------------+----------------------------------------------|
-| partition | 1 (GPT), start 0x800, size 0x100000 |
-|----------------+----------------------------------------------|
-| loader path | =\EFI\ZBM\zfsbootmenu.efi= |
-|----------------+----------------------------------------------|
-| cmdline (data) | =spl_hostid=0x22f8a7a1 zbm.timeout=3= |
-| | =zbm.prefer=zroot zbm.import_policy=hostid= |
-|----------------+----------------------------------------------|
-
-The =data= field is that command line in UTF-16LE, which is how efibootmgr
-passes it as optional data. Recreate with =-u= and the plain string; efibootmgr
-does the encoding.
-
-* Recreating it
-
-From a booted system (or the archangel ISO), with the ESP identified as
-=/dev/nvme0n1p1= or whatever it enumerates as:
-
-#+begin_src bash
-efibootmgr --create \
- --disk /dev/nvme0n1 --part 1 \
- --label "ZFSBootMenu" \
- --loader '\EFI\ZBM\zfsbootmenu.efi' \
- --unicode 'spl_hostid=0x22f8a7a1 zbm.timeout=3 zbm.prefer=zroot zbm.import_policy=hostid'
-#+end_src
-
-Confirm the disk/part against =lsblk -o NAME,PARTUUID,PARTTYPENAME= first —
-the partition GUID above is the authoritative identifier, not the device name,
-which can enumerate differently.
-
-Then set the order so ZBM is reachable:
-
-#+begin_src bash
-efibootmgr --bootorder 0001,2001,2002,2003
-#+end_src
-
-(The original order put USB first. Keep whichever you prefer; what matters is
-that the ZBM entry exists and is in the list.)
-
-* Other entries (firmware-generated, recreate themselves)
-
-| Boot2001 | EFI USB Device |
-|----------+----------------|
-| Boot2002 | EFI DVD/CDROM |
-|----------+----------------|
-| Boot2003 | EFI Network |
-|----------+----------------|
-
-These are stock firmware entries and come back on their own. Only Boot0001
-carries anything unique.
diff --git a/working/velox-reinstall/wttrin-bf0457f.bundle b/working/velox-reinstall/wttrin-bf0457f.bundle
deleted file mode 100644
index 5921866..0000000
--- a/working/velox-reinstall/wttrin-bf0457f.bundle
+++ /dev/null
Binary files differ