summaryrefslogtreecommitdiff
path: root/assets/outbox
diff options
context:
space:
mode:
Diffstat (limited to 'assets/outbox')
-rw-r--r--assets/outbox/2025-11-08-keyring-fix-next-steps.org75
-rw-r--r--assets/outbox/2025-11-08-test-failure-analysis.org222
-rw-r--r--assets/outbox/2026-01-17-gvfs-smb-feature-request.txt6
-rw-r--r--assets/outbox/2026-01-17-security-and-hardening-recommendations.txt119
-rw-r--r--assets/outbox/2026-01-17-zfs-sanoid-feature-request.txt202
-rw-r--r--assets/outbox/2026-01-19-remove-zfs-scripts-request.md29
-rw-r--r--assets/outbox/2026-01-20-console-display-issues.txt112
-rw-r--r--assets/outbox/2026-01-21-grub-timeout-request.txt4
-rw-r--r--assets/outbox/2026-01-21-syncthing-service-conflict.org72
-rw-r--r--assets/outbox/2026-01-23-avahi-mdns-fixes.org125
10 files changed, 966 insertions, 0 deletions
diff --git a/assets/outbox/2025-11-08-keyring-fix-next-steps.org b/assets/outbox/2025-11-08-keyring-fix-next-steps.org
new file mode 100644
index 0000000..e478ac3
--- /dev/null
+++ b/assets/outbox/2025-11-08-keyring-fix-next-steps.org
@@ -0,0 +1,75 @@
+#+TITLE: GNOME Keyring Authentication Fix - Next Steps
+#+DATE: 2025-11-08
+#+AUTHOR: Craig Jennings & Claude
+
+* URGENT: Complete Keyring Fix on Next Login
+
+** What Was Done
+- ✅ Identified the problem: Apps (nm-applet, signal-desktop, protonmail-bridge, insync) need keyring access
+- ✅ Updated =/etc/pam.d/system-login= to unlock keyring on console login
+- ✅ Created backup: =/etc/pam.d/system-login.backup=
+
+** What Needs to Be Done
+
+The PAM fix only works if your keyring password matches your login password.
+
+*** Option 1: Delete Old Keyring (RECOMMENDED - Fresh Start)
+
+#+begin_src bash
+# Backup the old keyring
+mv ~/.local/share/keyrings/login.keyring ~/.local/share/keyrings/login.keyring.backup
+
+# On next login, a new keyring will be created automatically with your login password
+# The PAM integration will unlock it automatically
+#+end_src
+
+*Caveat:* You'll need to re-enter saved passwords:
+- WiFi passwords (nm-applet)
+- Signal encryption password
+- ProtonMail Bridge credentials
+- Insync cloud sync credentials
+
+*** Option 2: Change Keyring Password to Match Login Password
+
+#+begin_src bash
+# Install seahorse if needed
+sudo pacman -S seahorse
+
+# Launch seahorse GUI
+seahorse
+
+# Then: Right-click "Login" keyring → "Change Password"
+# Set it to your current login password
+#+end_src
+
+** Testing the Fix
+
+After choosing one option above:
+
+1. Logout completely (=logout= or Ctrl+D)
+2. Login at console (PAM will now unlock the keyring using your password)
+3. Type =startx=
+4. The "Authentication Required" dialog should NOT appear
+
+** Rollback Instructions (If Needed)
+
+If something goes wrong:
+
+#+begin_src bash
+# Restore original PAM configuration
+sudo cp /etc/pam.d/system-login.backup /etc/pam.d/system-login
+#+end_src
+
+** Files Modified
+
+- =/etc/pam.d/system-login= - Added pam_gnome_keyring.so integration
+- =/etc/pam.d/system-login.backup= - Original backup
+
+** PAM Changes Made
+
+Added two lines to =/etc/pam.d/system-login=:
+
+#+begin_example
+auth optional pam_gnome_keyring.so # Line 5 - unlock keyring with login password
+session optional pam_gnome_keyring.so auto_start # Line 14 - start daemon if needed
+#+end_example
diff --git a/assets/outbox/2025-11-08-test-failure-analysis.org b/assets/outbox/2025-11-08-test-failure-analysis.org
new file mode 100644
index 0000000..56453c3
--- /dev/null
+++ b/assets/outbox/2025-11-08-test-failure-analysis.org
@@ -0,0 +1,222 @@
+#+TITLE: Test Failure Analysis - VM Test Run 20251108-204202
+#+AUTHOR: Craig Jennings & Claude
+#+DATE: 2025-11-08
+
+* Test Overview
+
+Test ID: 20251108-204202
+Date: 2025-11-08 21:16:11
+VM: archsetup-test-20251108-204202
+Result: **FAILED** (archsetup exited 0 but validation failed)
+
+* Critical Findings
+
+** PRIMARY ROOT CAUSE: Disk Space Exhausted
+
+The 20GB VM disk ran out of space during package installation:
+
+#+begin_example
+error: Partition / too full: 90773 blocks needed, 9323 blocks free
+error: not enough free disk space
+error: failed to commit transaction (not enough free disk space)
+#+end_example
+
+This caused cascading failures of ~100+ packages after initial packages filled the disk.
+
+*Impact:* Most package installation failures
+*Severity:* CRITICAL
+*Resolution:* ✅ FIXED - Increased VM disk size to 50GB (was 20GB)
+
+** SECONDARY ROOT CAUSE: git.cjennings.net Server Unavailable
+
+DWM, dmenu, and st failed to build due to 504 Gateway Timeout errors:
+
+#+begin_example
+Cloning into '/home/cjennings/.local/src/dwm'...
+fatal: unable to access 'https://git.cjennings.net/dwm.git/': The requested URL returned error: 504
+ERROR: cloning source code for dwm failed with error code 0
+#+end_example
+
+*Impact:* DWM validation check failed (critical)
+*Severity:* HIGH
+*Resolution:* ✅ RESOLVED - git.cjennings.net is working (verified 2025-11-08, transient 504 errors)
+
+** VALIDATION FAILURE: DWM Not Found
+
+Test validation checks:
+- ✅ yay is installed
+- ❌ DWM not found at /usr/local/bin/dwm
+
+*Cause:* git.cjennings.net 504 errors prevented DWM build
+*Impact:* Test marked as FAILED
+
+* Error Summary
+
+Total errors: 134
+
+** Error Categories
+
+*** Git Repository Access (3 errors)
+- dwm clone/pull failed (504 error)
+- dmenu clone/pull failed (504 error)
+- st clone partially succeeded (permission warning)
+
+*** Package Installation Failures (~100+ errors)
+All caused by disk space exhaustion after initial packages installed.
+
+Examples:
+- emacs
+- code (VS Code)
+- virtualbox
+- Many AUR packages (obsidian, warpinator, etc.)
+- Standard packages (aspell, imagemagick, ffmpegthumbnailer, etc.)
+
+*** Configuration Failures (2 errors)
+- Dotfile restoration failed (error 128)
+- Boot menu regeneration failed
+- Blue light filter configuration failed
+
+*** Other Errors
+- prep to workaround tidal-dl issue failed
+
+* Timeline of Failure
+
+1. **20:44** - Dotfile restoration error (early warning sign)
+2. **20:46** - Boot menu regeneration failed
+3. **20:47-20:49** - git.cjennings.net 504 errors (DWM/dmenu/st)
+4. **20:56** - First package failures start (nitrogen)
+5. **21:03** - adwaita-color-schemes fails
+6. **21:11** - Major package failures begin (disk full):
+ - emacs
+ - code
+ - virtualbox
+ - exercism-bin
+ - And ~100+ more packages
+7. **21:16** - archsetup completes (exit 0)
+8. **21:16** - Validation fails (DWM not found)
+
+* Affected Components
+
+** Window Manager (Critical)
+- ❌ DWM - Not built (git server error)
+- ❌ dmenu - Not built (git server error)
+- ⚠️ st - Partially built? (permission warning)
+
+** Development Tools
+- ❌ emacs
+- ❌ code (VS Code)
+- ❌ virtualbox
+- ❌ exercism-bin
+- ❌ libvips
+- ❌ isync
+
+** Desktop Applications
+- ❌ obsidian
+- ❌ warpinator
+- ❌ valent
+- ❌ nitrogen (wallpaper setter)
+- ❌ foliate
+- ❌ mcomix
+- ❌ nsxiv
+
+** System Utilities
+- ❌ aspell / aspell-en
+- ❌ imagemagick
+- ❌ ffmpegthumbnailer
+- ❌ 7zip
+- ❌ fd
+- ❌ And many more...
+
+* Resolution Plan
+
+** Immediate Actions (Before Next Test)
+
+1. **✅ DONE - Increase VM Disk Size**
+ - ✅ Changed from 20GB → 50GB
+ - ✅ Updated create-base-vm.sh
+ - ✅ Updated lib/vm-utils.sh
+ - ✅ Updated scripts/testing/README.org
+ - ✅ Updated docs/testing-strategy.org
+ - ⏳ TODO: Re-create base VM
+
+2. **✅ DONE - Verify git.cjennings.net Access**
+ - ✅ Server is working (dwm cloned successfully)
+ - ✅ 504 errors were transient network issues
+
+3. **TODO - Re-run Test**
+ - Re-create base VM with 50GB disk: ./scripts/testing/create-base-vm.sh
+ - Run full test: ./scripts/testing/run-test.sh
+ - Expected: Much fewer errors, all critical components should build
+
+** Long-term Improvements
+
+1. **Disk Space Monitoring**
+ - Add disk usage checks during archsetup run
+ - Warn if disk space < 5GB free
+ - Fail fast if insufficient space detected early
+
+2. **Repository Fallbacks**
+ - Mirror critical repos to GitHub
+ - Auto-fallback if primary git server unavailable
+ - Document required repositories
+
+3. **Better Error Reporting**
+ - Distinguish "disk full" from "package doesn't exist"
+ - Report root cause clearly
+ - Group related failures
+
+4. **Test Scenarios**
+ - Add "minimum disk space" test
+ - Add "offline installation" test (local package cache)
+ - Add "repository unavailable" resilience test
+
+* Lessons Learned
+
+1. **20GB is insufficient** for full archsetup with all packages
+ - Base system: ~3-5GB
+ - Package downloads: ~5-10GB
+ - AUR builds: ~5-10GB (tmpfs in VM?)
+ - Installed packages: ~10-15GB
+ - **Total needed: 40-50GB minimum**
+ - **✅ FIXED: Increased to 50GB**
+
+2. **External dependencies are fragile**
+ - git.cjennings.net unavailability blocked critical components
+ - Need fallback mechanisms
+ - Consider hosting mirrors
+
+3. **Cascading failures mask root cause**
+ - Disk full caused 100+ package errors
+ - Easy to miss the root cause in noise
+ - Better error aggregation needed
+
+4. **Validation checks are essential**
+ - archsetup exited 0 (success) but system was broken
+ - Validation caught DWM failure
+ - Need more validation checks
+
+* Next Test Expectations
+
+After increasing disk to 50GB (git server was working, just transient 504s):
+
+** Expected Results (with 50GB disk)
+- ✅ archsetup exits with code 0
+- ✅ User 'cjennings' created
+- ✅ Dotfiles are stowed
+- ✅ yay is installed
+- ✅ DWM is built and installed
+- ✅ Most/all packages installed successfully
+- ✅ No disk space errors
+
+** Acceptable Failures
+- Some deprecated AUR packages may still fail
+- Some optional packages may have build issues
+- These should be < 10 errors, not 134
+
+* Files Referenced
+
+- Test report: [[file:../test-results/20251108-204202/test-report.txt]]
+- Test log: [[file:../test-results/20251108-204202/test.log]]
+- archsetup log: [[file:../test-results/20251108-204202/archsetup-2025-11-08-20-42-27.log]]
+- Base VM creation: [[file:../test-results/create-base-vm-20251108-182022.log]]
+- Auto-install script: [[file:../vm-images/auto-install.sh]]
diff --git a/assets/outbox/2026-01-17-gvfs-smb-feature-request.txt b/assets/outbox/2026-01-17-gvfs-smb-feature-request.txt
new file mode 100644
index 0000000..79892f7
--- /dev/null
+++ b/assets/outbox/2026-01-17-gvfs-smb-feature-request.txt
@@ -0,0 +1,6 @@
+Install gvfs-smb for Thunar SMB network browsing
+
+Package: gvfs-smb
+Install: sudo pacman -S gvfs-smb
+
+Without this package, Thunar cannot browse SMB/CIFS network shares.
diff --git a/assets/outbox/2026-01-17-security-and-hardening-recommendations.txt b/assets/outbox/2026-01-17-security-and-hardening-recommendations.txt
new file mode 100644
index 0000000..22a0c53
--- /dev/null
+++ b/assets/outbox/2026-01-17-security-and-hardening-recommendations.txt
@@ -0,0 +1,119 @@
+# Security and Hardening Recommendations for archsetup
+
+These recommendations come from the install-archzfs base install.
+The base system is minimal - archsetup should handle hardening.
+
+## SSH Hardening (Priority: High)
+
+If SSH was enabled during install (for headless servers), it uses password auth.
+archsetup should:
+
+1. Install and configure fail2ban
+ - pacman -S fail2ban
+ - Enable sshd jail
+ - Configure ban times (suggested: 10m first offense, escalating)
+ - Consider integration with firewalld/nftables
+
+2. Switch to key-based authentication
+ - Prompt user for SSH public key or generate keypair
+ - Disable password authentication in /etc/ssh/sshd_config:
+ PasswordAuthentication no
+ PermitRootLogin prohibit-password (or 'no' for desktop)
+
+3. Consider changing default SSH port (optional, security through obscurity)
+
+## Firewall (Priority: High)
+
+Base install has no firewall configured. Options:
+
+1. firewalld (recommended for most users)
+ - pacman -S firewalld
+ - systemctl enable --now firewalld
+ - Default zone should block incoming except SSH
+
+2. nftables (for advanced users)
+ - Already installed as iptables backend
+ - Needs manual configuration
+
+3. ufw (simpler alternative)
+ - pacman -S ufw
+ - Good for users coming from Ubuntu
+
+## ZFS-Specific Recommendations
+
+1. Sanoid/Syncoid for automated snapshots
+ - pacman -S sanoid
+ - Configure /etc/sanoid/sanoid.conf for automatic snapshot retention
+ - Suggested policy: hourly for 24h, daily for 7d, monthly for 12m
+
+2. ZFS scrub timer
+ - systemctl enable zfs-scrub-weekly.timer
+ - Or create monthly timer for large pools
+
+3. ZED (ZFS Event Daemon) email alerts
+ - Configure /etc/zfs/zed.d/zed.rc
+ - Set ZED_EMAIL_ADDR for pool health notifications
+
+4. Consider zfs-auto-snapshot as alternative to sanoid
+
+## User Account Setup
+
+Base install only has root. archsetup should:
+
+1. Create primary user account with sudo access
+2. Lock root account for direct login (sudo only)
+3. Configure sudo timeout and logging
+
+## Package Manager Hardening
+
+1. Enable pacman hooks for security
+ - Verify package signatures (already default)
+
+2. Consider enabling reflector timer
+ - Keeps mirrorlist updated with fastest/most recent mirrors
+
+3. Install pacman-contrib for paccache
+ - Configure paccache.timer to clean old package cache
+
+## Automatic Updates (Optional)
+
+For servers that need unattended security updates:
+- Consider pacman-auto-update or similar
+- ZFS pre-pacman snapshots (already in install-archzfs) make this safer
+
+## AppArmor/SELinux (Optional, Advanced)
+
+For high-security environments:
+- AppArmor is easier: pacman -S apparmor
+- Requires kernel parameter: lsm=apparmor
+
+## Misc Recommendations
+
+1. Install and enable systemd-timesyncd or chrony for NTP
+
+2. Configure journald retention
+ - /etc/systemd/journald.conf
+ - SystemMaxUse=500M (or appropriate for system)
+
+3. Disable core dumps for security (optional)
+ - /etc/security/limits.conf: * hard core 0
+
+4. Install lynis for security auditing
+ - pacman -S lynis
+ - Run: lynis audit system
+
+## Desktop-Specific (if applicable)
+
+1. Consider firejail for sandboxing applications
+2. Install a password manager (pass, keepassxc)
+3. Configure automatic screen lock
+
+## Server-Specific (if applicable)
+
+1. Install and configure logwatch or logrotate
+2. Consider setting up centralized logging
+3. Install monitoring (prometheus node_exporter, netdata, etc.)
+
+---
+Generated by install-archzfs build system
+These are recommendations - implement based on your security requirements.
diff --git a/assets/outbox/2026-01-17-zfs-sanoid-feature-request.txt b/assets/outbox/2026-01-17-zfs-sanoid-feature-request.txt
new file mode 100644
index 0000000..87207f2
--- /dev/null
+++ b/assets/outbox/2026-01-17-zfs-sanoid-feature-request.txt
@@ -0,0 +1,202 @@
+ZFS Detection and Sanoid Installation
+======================================
+
+When archsetup runs, it should detect if the system is on ZFS and install sanoid.
+
+Detection:
+- Check if root filesystem is ZFS: `findmnt -n -o FSTYPE /` returns "zfs"
+- Or check if zpool exists: `zpool list -H 2>/dev/null`
+
+If ZFS detected:
+1. Install sanoid from AUR: `yay -S sanoid`
+2. Create /etc/sanoid/sanoid.conf (see below)
+3. Enable the timer: `systemctl enable --now sanoid.timer`
+4. Create the syncoid replication script and systemd units (see below)
+
+Context:
+- install-archzfs can't install sanoid (AUR package)
+- archsetup already has AUR helper setup, so it's the right place to install it
+- syncoid (for TrueNAS replication) comes with the sanoid package
+
+Added: 2026-01-17
+
+================================================================================
+SANOID CONFIGURATION (/etc/sanoid/sanoid.conf)
+================================================================================
+
+# Sanoid configuration for ZFS snapshots
+# Less aggressive - TrueNAS handles long-term backups
+
+#############################
+# Templates
+#############################
+
+[template_production]
+ # Local rollback capability
+ hourly = 6
+ daily = 7
+ weekly = 2
+ monthly = 1
+ autosnap = yes
+ autoprune = yes
+
+[template_backup]
+ # Less frequent for large/static data
+ hourly = 0
+ daily = 3
+ weekly = 2
+ monthly = 1
+ autosnap = yes
+ autoprune = yes
+
+[template_none]
+ autosnap = no
+ autoprune = yes
+
+#############################
+# Datasets
+#############################
+
+[zroot/ROOT/default]
+ use_template = production
+
+[zroot/home]
+ use_template = production
+ recursive = yes
+
+[zroot/media]
+ use_template = backup
+
+[zroot/vms]
+ use_template = backup
+
+[zroot/var/log]
+ use_template = production
+
+[zroot/var/lib/pacman]
+ use_template = production
+
+[zroot/var/cache]
+ use_template = none
+
+[zroot/var/tmp]
+ use_template = none
+
+[zroot/tmp]
+ use_template = none
+
+================================================================================
+SYNCOID REPLICATION SCRIPT (/usr/local/bin/zfs-replicate)
+================================================================================
+
+#!/bin/bash
+# zfs-replicate - Replicate ZFS datasets to TrueNAS
+#
+# Usage:
+# zfs-replicate # Replicate all configured datasets
+# zfs-replicate [dataset] # Replicate specific dataset
+
+set -e
+
+# TrueNAS Configuration
+# Try local network first, fall back to tailscale
+TRUENAS_LOCAL="truenas.local"
+TRUENAS_TAILSCALE="truenas"
+TRUENAS_USER="root"
+TRUENAS_POOL="vault"
+BACKUP_PATH="backups" # TODO: Configure actual path
+
+# Datasets to replicate
+DATASETS="zroot/ROOT/default zroot/home zroot/media zroot/vms"
+
+# Colors
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+RED='\033[0;31m'
+NC='\033[0m'
+
+info() { echo -e "${GREEN}[INFO]${NC} $1"; }
+warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
+error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }
+
+command -v syncoid >/dev/null 2>&1 || error "syncoid not found. Install sanoid package."
+
+# Determine which host to use
+determine_host() {
+ if ping -c 1 -W 2 "$TRUENAS_LOCAL" &>/dev/null; then
+ echo "$TRUENAS_LOCAL"
+ elif ping -c 1 -W 2 "$TRUENAS_TAILSCALE" &>/dev/null; then
+ echo "$TRUENAS_TAILSCALE"
+ else
+ error "Cannot reach TrueNAS at $TRUENAS_LOCAL or $TRUENAS_TAILSCALE"
+ fi
+}
+
+TRUENAS_HOST=$(determine_host)
+info "Using TrueNAS host: $TRUENAS_HOST"
+
+# Single dataset mode
+if [[ -n "$1" ]]; then
+ dataset="$1"
+ dest="$TRUENAS_USER@$TRUENAS_HOST:$TRUENAS_POOL/$BACKUP_PATH/${dataset#zroot/}"
+ info "Replicating $dataset -> $dest"
+ syncoid --recursive "$dataset" "$dest"
+ exit 0
+fi
+
+# Full replication
+info "Starting ZFS replication to $TRUENAS_HOST"
+echo ""
+
+for dataset in $DATASETS; do
+ dest="$TRUENAS_USER@$TRUENAS_HOST:$TRUENAS_POOL/$BACKUP_PATH/${dataset#zroot/}"
+ info "Replicating $dataset -> $dest"
+
+ if syncoid --recursive "$dataset" "$dest"; then
+ info " Success"
+ else
+ warn " Failed (will retry next run)"
+ fi
+ echo ""
+done
+
+info "Replication complete."
+
+================================================================================
+SYSTEMD SERVICE (/etc/systemd/system/zfs-replicate.service)
+================================================================================
+
+[Unit]
+Description=ZFS Replication to TrueNAS
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/zfs-replicate
+User=root
+
+[Install]
+WantedBy=multi-user.target
+
+================================================================================
+SYSTEMD TIMER (/etc/systemd/system/zfs-replicate.timer)
+================================================================================
+
+[Unit]
+Description=Run ZFS replication nightly
+
+[Timer]
+OnCalendar=*-*-* 02:00:00
+RandomizedDelaySec=1800
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+
+================================================================================
+ENABLE REPLICATION
+================================================================================
+
+After SSH key auth is set up to TrueNAS:
+ systemctl enable --now zfs-replicate.timer
diff --git a/assets/outbox/2026-01-19-remove-zfs-scripts-request.md b/assets/outbox/2026-01-19-remove-zfs-scripts-request.md
new file mode 100644
index 0000000..f67aa47
--- /dev/null
+++ b/assets/outbox/2026-01-19-remove-zfs-scripts-request.md
@@ -0,0 +1,29 @@
+# Task: Remove zfssnapshot and zfsrollback from archsetup
+
+## Summary
+Remove the `zfssnapshot` and `zfsrollback` scripts from archsetup's dotfiles. These scripts are now provided by the archzfs ISO and installed to `/usr/local/bin/` during `install-archzfs`.
+
+## Files to Remove
+- `dotfiles/system/.local/bin/zfssnapshot`
+- `dotfiles/system/.local/bin/zfsrollback`
+
+## Reason for Change
+These scripts need to be available immediately after a fresh install from the archzfs ISO, before archsetup runs. Key use cases:
+
+1. **Rescue scenarios**: Rolling back from live USB when the installed system won't boot
+2. **Genesis rollback**: If archsetup fails mid-run, user can rollback to genesis and retry
+3. **Script availability**: The scripts themselves must survive a genesis rollback (they're now part of genesis snapshot)
+
+By including them in the ISO and `install-archzfs`, they're guaranteed to be present from first boot, with fzf also installed as a dependency.
+
+## Changes Made in archzfs
+- Added `custom/zfssnapshot` and `custom/zfsrollback`
+- `build.sh` copies them to `/usr/local/bin/` on the ISO
+- `install-archzfs` installs `fzf` to target system (required by zfsrollback)
+- fzf was already in ISO package list
+
+## Note: Keep fzf in archsetup
+Archsetup should continue to install `fzf` in its package list. Archsetup can run on vanilla Arch installs with ext4 or btrfs (not just ZFS from archzfs ISO), where `install-archzfs` would not have run and fzf wouldn't be present.
+
+## Date
+2026-01-19
diff --git a/assets/outbox/2026-01-20-console-display-issues.txt b/assets/outbox/2026-01-20-console-display-issues.txt
new file mode 100644
index 0000000..f8dc710
--- /dev/null
+++ b/assets/outbox/2026-01-20-console-display-issues.txt
@@ -0,0 +1,112 @@
+Console Display Issues - Potential Causes in archsetup
+======================================================
+Date: 2026-01-20
+Source: archzfs testing on ratio - console not showing after install
+
+SUMMARY
+-------
+After running install-archzfs and archsetup on ratio, the console stopped
+displaying. The system boots but shows no console output. These are the
+suspected culprits in archsetup.
+
+SUSPECTED ISSUES
+----------------
+
+1. Console Font Configuration (boot_ux, lines 1574-1579)
+
+ File: archsetup
+ Lines: 1574-1579
+
+ Code:
+ if grep -q "^FONT=" /etc/vconsole.conf 2>/dev/null; then
+ sed -i 's/^FONT=.*/FONT=ter-132n/' /etc/vconsole.conf
+ else
+ echo "FONT=ter-132n" >> /etc/vconsole.conf
+ fi
+
+ Problem: Sets console font to ter-132n (Terminus 32pt). If the font
+ is missing, corrupted, or incompatible with the framebuffer, the
+ console may fail to display anything.
+
+ Fix: Verify terminus-font package is installed and font exists before
+ setting. Add fallback handling.
+
+2. mkinitcpio Hook Change (boot_ux, lines 1581-1583)
+
+ File: archsetup
+ Lines: 1581-1583
+
+ Code:
+ sed -i '/^HOOKS=/ s/\budev\b/systemd/' /etc/mkinitcpio.conf
+ mkinitcpio -P
+
+ Problem: Changes mkinitcpio from 'udev' to 'systemd' hook and
+ regenerates ALL initramfs images. This is a significant change that
+ affects early boot. If the systemd hook isn't properly configured
+ or conflicts with other hooks, boot may fail or console may not
+ initialize properly.
+
+ Fix: Ensure all required systemd-related hooks are present. Consider
+ whether this change is necessary or could be made optional.
+
+3. GRUB Quiet Boot Settings (boot_ux, line 1624)
+
+ File: archsetup
+ Line: 1624
+
+ Code:
+ sed -i "s/.*GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT=\"rw loglevel=2 rd.systemd.show_status=auto rd.udev.log_level=2 nvme.noacpi=1 mem_sleep_default=deep nowatchdog quiet splash\"/g" /etc/default/grub
+
+ Problem: Adds 'quiet splash' and sets loglevel=2, which suppresses
+ most boot messages. If something goes wrong during boot, you won't
+ see any output. The 'splash' option may also interfere with console.
+
+ Fix: Consider removing 'splash' or making quiet boot optional.
+ For debugging, temporarily remove 'quiet splash' from GRUB.
+
+4. Kernel Message Suppression (boot_ux, lines 1571-1572)
+
+ File: archsetup
+ Lines: 1571-1572
+
+ Code:
+ echo "kernel.printk = 3 3 3 3" >/etc/sysctl.d/20-quiet-printk.conf
+
+ Problem: Suppresses kernel messages to console. Combined with other
+ quiet settings, this could hide important boot information.
+
+ Fix: For debugging, remove or adjust this setting.
+
+5. Xorg VT Switching Disabled (xorg, lines 1102-1107)
+
+ File: archsetup
+ Lines: 1102-1107
+
+ Code:
+ cat << EOF > /etc/X11/xorg.conf.d/00-no-vt-or-zap.conf
+ Section "ServerFlags"
+ Option "DontVTSwitch" "True"
+ Option "DontZap" "True"
+ EndSection
+ EOF
+
+ Problem: Disables VT switching when X is running. If X starts
+ automatically, you cannot switch to a text console with Ctrl+Alt+F2.
+ This is a security feature but makes debugging harder.
+
+ Note: This only affects post-X boot, not early console display.
+
+DEBUGGING STEPS
+---------------
+1. Boot with 'nomodeset' kernel parameter to rule out GPU/framebuffer issues
+2. Remove 'quiet splash' from GRUB temporarily
+3. Check if ter-132n font exists: ls /usr/share/kbd/consolefonts/ter-*
+4. Review mkinitcpio.conf HOOKS line for conflicts
+5. Check journalctl -b for boot errors
+
+RECOMMENDED CHANGES
+-------------------
+- Make quiet boot optional or add a debug boot menu entry
+- Verify font exists before setting in vconsole.conf
+- Document the udev->systemd hook change and its implications
+- Consider adding a recovery boot option that skips quiet settings
diff --git a/assets/outbox/2026-01-21-grub-timeout-request.txt b/assets/outbox/2026-01-21-grub-timeout-request.txt
new file mode 100644
index 0000000..fa03f62
--- /dev/null
+++ b/assets/outbox/2026-01-21-grub-timeout-request.txt
@@ -0,0 +1,4 @@
+* TODO Increase GRUB_TIMEOUT to 2 seconds
+Currently setting GRUB_TIMEOUT=0 which doesn't give users time to access GRUB menu.
+Change to GRUB_TIMEOUT=2 for a reasonable delay while keeping boot fast.
+
diff --git a/assets/outbox/2026-01-21-syncthing-service-conflict.org b/assets/outbox/2026-01-21-syncthing-service-conflict.org
new file mode 100644
index 0000000..7f86b39
--- /dev/null
+++ b/assets/outbox/2026-01-21-syncthing-service-conflict.org
@@ -0,0 +1,72 @@
+#+TITLE: Syncthing Service Conflict Issue
+#+DATE: 2026-01-21
+
+* Problem
+
+archsetup enables the system service:
+#+begin_src bash
+systemctl enable "syncthing@$username.service"
+#+end_src
+
+However, the user service can also get enabled (either by default or manually):
+#+begin_src bash
+systemctl --user enable syncthing.service
+#+end_src
+
+When BOTH services are enabled, they fight over the same lock file:
+=~/.local/state/syncthing/syncthing.lock=
+
+This causes one or both to fail with:
+: Failed to acquire lock: is another Syncthing instance already running?
+
+* Symptoms
+
+- Syncthing fails to start or keeps crashing
+- Lock file errors in journalctl
+- Two syncthing processes running with different parent services
+- Config changes don't persist (one service overwrites the other)
+
+* Recommendation
+
+Standardize on ONE service type. Options:
+
+** Option A: User Service (recommended for desktops)
+
+Runs when user logs in. Cleaner for desktop use.
+
+Change archsetup from:
+#+begin_src bash
+systemctl enable "syncthing@$username.service"
+#+end_src
+
+To:
+#+begin_src bash
+# Enable user service (requires user session)
+sudo -u "$username" systemctl --user enable syncthing.service
+#+end_src
+
+Note: User services require lingering or an active session:
+#+begin_src bash
+loginctl enable-linger "$username"
+#+end_src
+
+** Option B: System Service (recommended for headless/servers)
+
+Runs at boot without user login. Better for servers.
+
+Keep current archsetup config, but ensure user service is disabled:
+#+begin_src bash
+systemctl enable "syncthing@$username.service"
+# Explicitly disable user service to prevent conflicts
+sudo -u "$username" systemctl --user disable syncthing.service 2>/dev/null || true
+#+end_src
+
+* Resolution on ratio (2026-01-21)
+
+Disabled system service, kept user service:
+#+begin_src bash
+sudo systemctl stop syncthing@cjennings.service
+sudo systemctl disable syncthing@cjennings.service
+systemctl --user enable syncthing.service
+systemctl --user start syncthing.service
+#+end_src
diff --git a/assets/outbox/2026-01-23-avahi-mdns-fixes.org b/assets/outbox/2026-01-23-avahi-mdns-fixes.org
new file mode 100644
index 0000000..89b005e
--- /dev/null
+++ b/assets/outbox/2026-01-23-avahi-mdns-fixes.org
@@ -0,0 +1,125 @@
+#+TITLE: Avahi/mDNS Configuration Fixes
+#+DATE: 2026-01-23
+
+* Problem Summary
+
+On velox, mDNS hostname resolution was not working correctly from other machines on the LAN (e.g., ratio). Attempting to access =http://velox.local:8384= (Syncthing web UI) failed, while accessing via IP address worked.
+
+* Issues Identified
+
+** Issue 1: Hostname Conflict (velox-3.local)
+
+*Symptom:* Avahi was running as =velox-3.local= instead of =velox.local=
+
+*Cause:* Avahi was publishing on multiple network interfaces including virtual ones:
+- =enp0s13f0u3= (physical LAN - correct)
+- =docker0= (Docker bridge)
+- =virbr0= (libvirt bridge)
+- =vnet0= (VM virtual NIC)
+- =tailscale0= (Tailscale VPN)
+
+Each interface was effectively registering as a separate host, causing mDNS hostname conflicts with itself.
+
+*Solution:* Restrict Avahi to only the physical LAN interface.
+
+#+begin_src conf
+# /etc/avahi/avahi-daemon.conf
+[server]
+allow-interfaces=enp0s13f0u3
+#+end_src
+
+** Issue 2: IPv6-Only Resolution
+
+*Symptom:* =velox.local= resolved to IPv6 link-local address (=fe80::...=) only, but Syncthing was listening on IPv4 only (=0.0.0.0:8384=).
+
+*Cause:* Default Avahi configuration does not publish A records (IPv4) in response to AAAA queries (IPv6).
+
+*Solution:* Enable =publish-a-on-ipv6= to ensure IPv4 addresses are returned.
+
+#+begin_src conf
+# /etc/avahi/avahi-daemon.conf
+[publish]
+publish-a-on-ipv6=yes
+#+end_src
+
+** Issue 3: Conflicting mDNS Stacks
+
+*Symptom:* Avahi logged warning: "Detected another IPv4 mDNS stack running on this host"
+
+*Cause:* Both =avahi-daemon= and =systemd-resolved= were configured to handle mDNS:
+
+#+begin_src conf
+# /etc/systemd/resolved.conf (before fix)
+[Resolve]
+MulticastDNS=yes
+#+end_src
+
+*Solution:* Disable mDNS in systemd-resolved, let Avahi handle it exclusively.
+
+#+begin_src conf
+# /etc/systemd/resolved.conf
+[Resolve]
+Domains=~local
+MulticastDNS=no
+#+end_src
+
+* Complete Fix Applied
+
+** Files Modified
+
+*** /etc/avahi/avahi-daemon.conf
+
+Changes made:
+#+begin_src diff
+-#allow-interfaces=eth0
++allow-interfaces=enp0s13f0u3
+
+-#publish-a-on-ipv6=no
++publish-a-on-ipv6=yes
+#+end_src
+
+*** /etc/systemd/resolved.conf
+
+Changes made:
+#+begin_src diff
+-MulticastDNS=yes
++MulticastDNS=no
+#+end_src
+
+** Services Restarted
+
+#+begin_src bash
+sudo systemctl restart systemd-resolved
+sudo systemctl restart avahi-daemon
+#+end_src
+
+* Verification
+
+After fixes:
+- Avahi runs as =velox.local= (not =velox-3.local=)
+- No mDNS stack conflict warning
+- From ratio: =avahi-resolve -n velox.local= returns =192.168.86.42=
+- From ratio: =curl http://velox.local:8384/= returns HTTP 200
+
+* Notes for archsetup
+
+These configurations should be added to the Arch setup scripts:
+
+1. Install avahi: =pacman -S avahi nss-mdns=
+
+2. Configure =/etc/avahi/avahi-daemon.conf=:
+ - Set =allow-interfaces= to physical LAN interface (determine dynamically or prompt user)
+ - Set =publish-a-on-ipv6=yes=
+
+3. Configure =/etc/systemd/resolved.conf=:
+ - Set =MulticastDNS=no= to avoid conflict with Avahi
+
+4. Enable and start avahi-daemon:
+ #+begin_src bash
+ systemctl enable --now avahi-daemon
+ #+end_src
+
+5. Ensure =/etc/nsswitch.conf= has mdns in hosts line:
+ #+begin_src conf
+ hosts: mymachines mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] files dns
+ #+end_src