path: root/custom/RESCUE-GUIDE.txt

================================================================================
                         ARCHZFS RESCUE GUIDE
================================================================================

This guide covers common rescue and recovery scenarios. For quick command
reference, use: tldr <command>

Table of Contents:
  1. ZFS Recovery
  2. Data Recovery
  3. Boot Repair
  4. Windows Recovery
  5. Hardware Diagnostics
  6. Disk Operations
  7. Network Troubleshooting
  8. Encryption & GPG

================================================================================
1. ZFS RECOVERY
================================================================================

QUICK REFERENCE
---------------
  tldr zfs          # ZFS filesystem commands
  tldr zpool        # ZFS pool commands
  man zfs           # Full ZFS manual
  man zpool         # Full zpool manual

SCENARIO: Import a pool from another system
-------------------------------------------
List pools available for import:

  zpool import

Import a specific pool:

  zpool import poolname

If the pool was not cleanly exported (e.g., system crash):

  zpool import -f poolname

Import with a different name (to avoid conflicts):

  zpool import oldname newname


SCENARIO: Pool won't import - "pool may be in use"
--------------------------------------------------
Force import (use when you know it's safe):

  zpool import -f poolname

If that fails, try recovery mode:

  zpool import -F poolname

Last resort - import read-only to recover data:

  zpool import -o readonly=on poolname


SCENARIO: Check pool health and repair
--------------------------------------
Check pool status:

  zpool status poolname

Start a scrub (checks all data, can take hours):

  zpool scrub poolname

Check scrub progress:

  zpool status poolname

Clear transient errors after fixing hardware:

  zpool clear poolname


SCENARIO: Recover from snapshot / Rollback
------------------------------------------
List all snapshots:

  zfs list -t snapshot

Rollback to a snapshot (destroys changes since snapshot):

  zfs rollback poolname/dataset@snapshot

For snapshots with intermediate snapshots, use -r:

  zfs rollback -r poolname/dataset@snapshot


SCENARIO: Copy data from ZFS pool
---------------------------------
Mount datasets if not auto-mounted:

  zfs mount -a

Or mount specific dataset:

  zfs set mountpoint=/mnt/recovery poolname/dataset
  zfs mount poolname/dataset

Copy with rsync (preserves permissions, shows progress):

  rsync -avP --progress /mnt/recovery/ /destination/


SCENARIO: Send/Receive snapshots (backup/migrate)
-------------------------------------------------
Create a snapshot first:

  zfs snapshot poolname/dataset@backup

Send to a file (local backup):

  zfs send poolname/dataset@backup > /path/to/backup.zfs

Send with progress indicator:

  zfs send poolname/dataset@backup | pv > /path/to/backup.zfs

Send to another pool locally:

  zfs send poolname/dataset@backup | zfs recv newpool/dataset

Send to remote system over SSH:

  zfs send poolname/dataset@backup | ssh user@remote zfs recv pool/dataset

With progress and buffering for network transfers:

  zfs send poolname/dataset@backup | pv | mbuffer -s 128k -m 1G | \
    ssh user@remote "mbuffer -s 128k -m 1G | zfs recv pool/dataset"


SCENARIO: Encrypted pool - unlock and mount
-------------------------------------------
Load the encryption key (will prompt for passphrase):

  zfs load-key poolname

Or for all encrypted datasets:

  zfs load-key -a

Then mount:

  zfs mount -a


SCENARIO: Replace failed drive in mirror/raidz
----------------------------------------------
Check which drive failed:

  zpool status poolname

Replace the drive (assuming /dev/sdc is new drive):

  zpool replace poolname /dev/old-drive /dev/sdc

Monitor resilver progress:

  zpool status poolname


SCENARIO: See what's using a dataset (before unmount)
-----------------------------------------------------
Check what processes have files open:

  lsof /mountpoint

Or for all ZFS mounts:

  lsof | grep poolname


USEFUL ZFS COMMANDS
-------------------
  zpool status              # Pool health overview
  zpool list                # Pool capacity
  zpool history poolname    # Command history
  zfs list                  # All datasets
  zfs list -t snapshot      # All snapshots
  zfs get all poolname      # All properties
  zdb -l /dev/sdX           # Low-level pool label info


================================================================================
2. DATA RECOVERY
================================================================================

QUICK REFERENCE
---------------
  tldr ddrescue     # Clone failing drives
  tldr testdisk     # Partition/file recovery
  tldr photorec     # Recover deleted files by type
  tldr smartctl     # Check drive health

FIRST: Assess drive health before recovery
------------------------------------------
Check if drive is failing (SMART data):

  smartctl -H /dev/sdX              # Quick health check
  smartctl -a /dev/sdX              # Full SMART report

Key things to look for:
  - "PASSED" vs "FAILED" health status
  - Reallocated_Sector_Ct - bad sectors remapped (increasing = dying)
  - Current_Pending_Sector - sectors waiting to be remapped
  - Offline_Uncorrectable - sectors that couldn't be read

If SMART shows problems, STOP and use ddrescue immediately.
Do not run fsck or other tools that write to a failing drive.


SCENARIO: Clone a failing drive (CRITICAL - do this first!)
------------------------------------------------------------
Golden rule: NEVER work directly on a failing drive.
Clone it first, then recover from the clone.

Clone to an image file (safest):

  ddrescue -d -r3 /dev/sdX /path/to/image.img /path/to/logfile.log

  -d    = direct I/O, bypass cache
  -r3   = retry bad sectors 3 times
  logfile = allows resuming if interrupted

Clone to another drive:

  ddrescue -d -r3 /dev/sdX /dev/sdY /path/to/logfile.log

Monitor progress (ddrescue shows its own progress, but for pipes):

  ddrescue -d /dev/sdX - 2>/dev/null | pv > /path/to/image.img

Resume an interrupted clone:

  ddrescue -d -r3 /dev/sdX /path/to/image.img /path/to/logfile.log

The log file tracks what's been copied. Same command resumes.

If drive is very bad, do a quick pass first, then retry bad sectors:

  ddrescue -d -n /dev/sdX image.img logfile.log     # Fast pass, skip errors
  ddrescue -d -r3 /dev/sdX image.img logfile.log   # Retry bad sectors


SCENARIO: Recover deleted files (PhotoRec)
------------------------------------------
PhotoRec recovers files by their content signatures, not filesystem.
Works even if filesystem is damaged or reformatted.

Run PhotoRec (included with testdisk):

  photorec /dev/sdX            # From device
  photorec image.img           # From disk image

Interactive steps:
  1. Select the disk/partition
  2. Choose filesystem type (usually "Other" for FAT/NTFS/exFAT)
  3. Choose "Free" (unallocated) or "Whole" (entire partition)
  4. Select destination folder for recovered files
  5. Wait (can take hours for large drives)

Recovered files are named by type (e.g., f0001234.jpg) in recup_dir.*/


SCENARIO: Recover lost partition / Fix partition table
------------------------------------------------------
TestDisk can find and recover lost partitions.

Run TestDisk:

  testdisk /dev/sdX            # From device
  testdisk image.img           # From disk image

Interactive steps:
  1. Select disk
  2. Select partition table type (usually Intel/PC for MBR, EFI GPT)
  3. Choose "Analyse" to scan for partitions
  4. "Quick Search" finds most partitions
  5. "Deeper Search" if quick search misses any
  6. Review found partitions, select ones to recover
  7. "Write" to save new partition table (or just note the info)

TestDisk can also:
  - Recover deleted files from FAT/NTFS/ext filesystems
  - Repair FAT/NTFS boot sectors
  - Rebuild NTFS MFT


SCENARIO: Recover specific file types (Foremost)
------------------------------------------------
Foremost carves files based on headers/footers.
Useful when PhotoRec doesn't find what you need.

Basic usage:

  foremost -t all -i /dev/sdX -o /output/dir
  foremost -t all -i image.img -o /output/dir

Specific file types:

  foremost -t jpg,png,gif -i image.img -o /output/dir
  foremost -t pdf,doc,xls -i image.img -o /output/dir

Supported types: jpg, gif, png, bmp, avi, exe, mpg, wav, riff,
wmv, mov, pdf, ole (doc/xls/ppt), doc, zip, rar, htm, cpp, all


SCENARIO: Can't mount filesystem - try repair
----------------------------------------------
WARNING: Only run fsck on a COPY, not the original failing drive!

For ext2/ext3/ext4:

  fsck.ext4 -n /dev/sdX        # Check only, no changes (safe)
  fsck.ext4 -p /dev/sdX        # Auto-repair safe problems
  fsck.ext4 -y /dev/sdX        # Say yes to all repairs (risky)

For NTFS:

  ntfsfix /dev/sdX             # Fix common NTFS issues

For XFS:

  xfs_repair -n /dev/sdX       # Check only
  xfs_repair /dev/sdX          # Repair

For FAT32:

  fsck.fat -n /dev/sdX         # Check only
  fsck.fat -a /dev/sdX         # Auto-repair


SCENARIO: Mount a disk image for file access
---------------------------------------------
Mount a full disk image (find partitions first):

  fdisk -l image.img           # List partitions and offsets

Note the "Start" sector of the partition you want, multiply by 512:

  mount -o loop,offset=$((START*512)) image.img /mnt/recovery

Or use losetup to set up loop devices for all partitions:

  losetup -P /dev/loop0 image.img
  mount /dev/loop0p1 /mnt/recovery

For NTFS images:

  mount -t ntfs-3g -o loop,offset=$((START*512)) image.img /mnt/recovery


SCENARIO: Low-level recovery from very bad drives (safecopy)
------------------------------------------------------------
Safecopy is more aggressive than ddrescue for very damaged media.
Use when ddrescue can't make progress.

  safecopy /dev/sdX image.img

With multiple passes (increasingly aggressive):

  safecopy --stage1 /dev/sdX image.img    # Quick pass
  safecopy --stage2 /dev/sdX image.img    # Retry errors
  safecopy --stage3 /dev/sdX image.img    # Maximum recovery


DATA RECOVERY TIPS
------------------
1. STOP using a failing drive immediately - every access risks more damage
2. Clone first, recover from clone - never work on original
3. Keep the log file from ddrescue - allows resuming
4. Recover to a DIFFERENT drive - never same drive
5. For deleted files on working drive, unmount immediately to prevent
   overwriting the deleted data
6. If drive makes clicking/grinding noises, consider professional recovery
7. For SSDs, TRIM may have already zeroed deleted blocks - recovery harder

================================================================================
3. BOOT REPAIR
================================================================================

QUICK REFERENCE
---------------
  tldr grub-install     # Install GRUB bootloader
  tldr efibootmgr       # Manage UEFI boot entries
  tldr arch-chroot      # Chroot into installed system
  man mkinitcpio        # Rebuild initramfs

FIRST: Identify your boot mode
------------------------------
Check if system is UEFI or Legacy BIOS:

  ls /sys/firmware/efi       # If exists, you're in UEFI mode

If booting from this rescue USB in UEFI mode, you need to fix UEFI.
If booting in Legacy mode, you need to fix MBR/Legacy boot.


SCENARIO: Chroot into broken system (preparation for most repairs)
------------------------------------------------------------------
This is the foundation for most boot repairs.

1. Find your partitions:

  lsblk -f                    # Shows filesystems and labels

2. Mount the root filesystem:

  mount /dev/sdX2 /mnt        # Replace with your root partition

   For ZFS root:

     zpool import -R /mnt zroot
     zfs mount -a

3. Mount required system directories:

  mount /dev/sdX1 /mnt/boot   # EFI partition (if separate)
  mount --bind /dev /mnt/dev
  mount --bind /proc /mnt/proc
  mount --bind /sys /mnt/sys
  mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars

   Or use arch-chroot (handles mounts automatically):

     arch-chroot /mnt

4. Now you can run commands as if booted into the system.


SCENARIO: Reinstall GRUB (UEFI)
-------------------------------
After chrooting into the system:

  grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB

If EFI partition is mounted elsewhere:

  grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

Regenerate GRUB config:

  grub-mkconfig -o /boot/grub/grub.cfg


SCENARIO: Reinstall GRUB (Legacy BIOS/MBR)
------------------------------------------
After chrooting into the system:

  grub-install --target=i386-pc /dev/sdX    # Note: device, not partition

Regenerate GRUB config:

  grub-mkconfig -o /boot/grub/grub.cfg


SCENARIO: Fix UEFI boot entries
-------------------------------
List current boot entries:

  efibootmgr -v

Delete a broken entry (replace XXXX with boot number):

  efibootmgr -b XXXX -B

Create a new boot entry:

  efibootmgr --create --disk /dev/sdX --part 1 --label "Arch Linux" \
    --loader /EFI/GRUB/grubx64.efi

Change boot order (comma-separated boot numbers):

  efibootmgr -o 0001,0002,0003

Set next boot only:

  efibootmgr -n 0001


SCENARIO: Rebuild initramfs (kernel panic, missing modules)
-----------------------------------------------------------
After chrooting into the system:

List available presets:

  ls /etc/mkinitcpio.d/

Rebuild for specific kernel:

  mkinitcpio -p linux          # Standard kernel
  mkinitcpio -p linux-lts      # LTS kernel

Rebuild all:

  mkinitcpio -P

Check mkinitcpio.conf for ZFS:

  grep "^HOOKS" /etc/mkinitcpio.conf

For ZFS, HOOKS should include 'zfs':
  HOOKS=(base udev autodetect modconf block zfs filesystems keyboard fsck)


SCENARIO: GRUB not detecting Windows (dual-boot)
------------------------------------------------
After chrooting into the system:

Enable os-prober in GRUB config:

  echo 'GRUB_DISABLE_OS_PROBER=false' >> /etc/default/grub

Mount the Windows EFI partition if not already mounted.

Regenerate GRUB config:

  grub-mkconfig -o /boot/grub/grub.cfg

os-prober should find Windows and add it to the menu.


SCENARIO: Restore Windows MBR (remove GRUB, restore Windows boot)
-----------------------------------------------------------------
If you need to remove Linux and restore Windows-only MBR:

  ms-sys -w /dev/sdX           # Write Windows 7+ MBR

Other options:
  ms-sys -7 /dev/sdX           # Windows 7 MBR specifically
  ms-sys -i /dev/sdX           # Show current MBR type


SCENARIO: Install syslinux (lightweight alternative to GRUB)
------------------------------------------------------------
For Legacy BIOS:

  syslinux-install_update -i -a -m

For UEFI, copy the EFI binary:

  cp /usr/lib/syslinux/efi64/* /boot/EFI/syslinux/

Create /boot/syslinux/syslinux.cfg with boot entries.


SCENARIO: Can't boot - kernel panic with ZFS
--------------------------------------------
Common causes:
1. ZFS module not in initramfs - rebuild with mkinitcpio
2. Pool name changed - check zpool.cache
3. hostid mismatch - regenerate hostid

After chrooting:

Check if ZFS hook is present:

  grep zfs /etc/mkinitcpio.conf

Regenerate hostid if needed:

  zgenhostid $(hostid)

Rebuild initramfs:

  mkinitcpio -P


SCENARIO: Emergency boot from GRUB command line
-----------------------------------------------
If GRUB loads but config is broken, press 'c' for command line:

For Linux (non-ZFS):

  set root=(hd0,gpt2)
  linux /boot/vmlinuz-linux root=/dev/sda2
  initrd /boot/initramfs-linux.img
  boot

For Linux with ZFS root:

  set root=(hd0,gpt1)
  linux /vmlinuz-linux-lts root=ZFS=zroot/ROOT/default
  initrd /initramfs-linux-lts.img
  boot

Tab completion works in GRUB command line!


BOOT REPAIR TIPS
----------------
1. Always backup your current EFI partition before making changes
2. Use 'efibootmgr -v' to see full paths and verify entries
3. Some UEFI firmwares are picky about the bootloader path -
   try /EFI/BOOT/BOOTX64.EFI as a fallback
4. If all else fails, most UEFI has a boot menu (F12, F8, Esc at POST)
5. GRUB reinstall usually fixes most boot issues
6. For ZFS, the initramfs must include the zfs hook

================================================================================
4. WINDOWS RECOVERY
================================================================================

QUICK REFERENCE
---------------
  tldr chntpw         # Reset Windows passwords
  tldr ntfs-3g        # Mount NTFS filesystems
  man dislocker       # Access BitLocker drives
  man hivexregedit    # Edit Windows registry

FIRST: Identify and mount the Windows partition
-----------------------------------------------
Find Windows partition:

  lsblk -f            # Look for "ntfs" filesystem
  fdisk -l            # Look for "Microsoft basic data" type

Check if BitLocker encrypted:

  lsblk -f            # Will show "BitLocker" instead of "ntfs"

Mount NTFS partition (read-write):

  mkdir -p /mnt/windows
  mount -t ntfs-3g /dev/sdX1 /mnt/windows

If Windows wasn't shut down cleanly (hibernation/fast startup):

  mount -t ntfs-3g -o remove_hiberfile /dev/sdX1 /mnt/windows

Read-only mount (safer):

  mount -t ntfs-3g -o ro /dev/sdX1 /mnt/windows


SCENARIO: Reset forgotten Windows password
------------------------------------------
Mount the Windows partition first (see above).

Navigate to the SAM database:

  cd /mnt/windows/Windows/System32/config

List all users:

  chntpw -l SAM

Reset password for a specific user (interactive):

  chntpw -u "Username" SAM

In the interactive menu:
  1. Clear (blank) user password  <-- Recommended
  2. Unlock and enable user account
  3. Promote user to administrator
  q. Quit

After making changes, type 'q' to quit, then 'y' to save.

Alternative - blank ALL passwords:

  chntpw -i SAM      # Interactive mode, select options


SCENARIO: Unlock disabled/locked Windows account
------------------------------------------------
  cd /mnt/windows/Windows/System32/config
  chntpw -u "Username" SAM

Select option 2: "Unlock and enable user account"


SCENARIO: Promote user to Administrator
---------------------------------------
  cd /mnt/windows/Windows/System32/config
  chntpw -u "Username" SAM

Select option 3: "Promote user (make user an administrator)"


SCENARIO: Access BitLocker encrypted drive
------------------------------------------
You MUST have either:
  - The BitLocker password, OR
  - The 48-digit recovery key

Find your recovery key:
  - Microsoft account: account.microsoft.com/devices/recoverykey
  - Printed/saved during BitLocker setup
  - Active Directory (for domain-joined PCs)

Decrypt with password:

  mkdir -p /mnt/bitlocker-decrypted /mnt/windows
  dislocker -V /dev/sdX1 -u -- /mnt/bitlocker-decrypted
  # Enter password when prompted

Decrypt with recovery key:

  dislocker -V /dev/sdX1 -p123456-789012-345678-901234-567890-123456-789012-345678 -- /mnt/bitlocker-decrypted

Now mount the decrypted volume:

  mount -t ntfs-3g /mnt/bitlocker-decrypted/dislocker-file /mnt/windows

When done:

  umount /mnt/windows
  umount /mnt/bitlocker-decrypted


SCENARIO: Copy files from Windows that won't boot
-------------------------------------------------
Mount the Windows partition (see above), then:

Copy specific files/folders:

  cp -r "/mnt/windows/Users/Username/Documents" /destination/

Copy with rsync (shows progress, preserves attributes):

  rsync -avP "/mnt/windows/Users/Username/" /destination/

Common locations for user data:
  /mnt/windows/Users/Username/Desktop/
  /mnt/windows/Users/Username/Documents/
  /mnt/windows/Users/Username/Downloads/
  /mnt/windows/Users/Username/Pictures/
  /mnt/windows/Users/Username/AppData/  (hidden app data)


SCENARIO: Edit Windows Registry
-------------------------------
The registry is stored in several hive files:

  SYSTEM    - Hardware, services, boot config
  SOFTWARE  - Installed programs, system settings
  SAM       - User accounts (password hashes)
  SECURITY  - Security policies
  DEFAULT   - Default user profile
  NTUSER.DAT - Per-user settings (in each user's profile)

View registry contents:

  hivexregedit --export /mnt/windows/Windows/System32/config/SYSTEM '\' > system.reg

Merge changes from a .reg file:

  hivexregedit --merge /mnt/windows/Windows/System32/config/SOFTWARE changes.reg

Interactive registry shell:

  hivexsh /mnt/windows/Windows/System32/config/SYSTEM
  # Commands: cd, ls, lsval, cat, exit


SCENARIO: Fix Windows boot (from Linux)
---------------------------------------
Sometimes you can fix Windows boot issues from Linux:

Rebuild BCD (Windows Boot Configuration Data):
  - This usually requires Windows Recovery Environment
  - From Linux, you can backup/restore the BCD file:

  cp /mnt/windows/Boot/BCD /mnt/windows/Boot/BCD.backup

Restore Windows bootloader to MBR (if GRUB overwrote it):

  ms-sys -w /dev/sdX     # Write Windows 7+ compatible MBR

For UEFI systems, Windows boot files are in:
  /mnt/efi/EFI/Microsoft/Boot/


SCENARIO: Scan Windows for malware (offline scan)
-------------------------------------------------
Update ClamAV definitions first (requires internet):

  freshclam

Scan the Windows partition:

  clamscan -r /mnt/windows              # Basic scan
  clamscan -r -i /mnt/windows           # Only show infected files
  clamscan -r --move=/quarantine /mnt/windows   # Quarantine infected

Scan common malware locations:

  clamscan -r "/mnt/windows/Users/*/AppData"
  clamscan -r "/mnt/windows/Windows/Temp"
  clamscan -r "/mnt/windows/ProgramData"

Note: ClamAV detection isn't as comprehensive as commercial AV.
Best for known malware; may miss new/sophisticated threats.


SCENARIO: Disable Windows Fast Startup (to mount NTFS read-write)
-----------------------------------------------------------------
Windows 8+ uses "Fast Startup" (hybrid shutdown) by default.
This leaves NTFS in a "dirty" state, preventing safe writes from Linux.

Option 1: Force mount (may cause issues):

  mount -t ntfs-3g -o remove_hiberfile /dev/sdX1 /mnt/windows

Option 2: Boot Windows and disable Fast Startup:
  - Control Panel > Power Options > "Choose what the power buttons do"
  - Click "Change settings that are currently unavailable"
  - Uncheck "Turn on fast startup"
  - Shutdown (not restart) Windows

Option 3: Via registry from Linux:

  hivexregedit --merge /mnt/windows/Windows/System32/config/SYSTEM << 'EOF'
  Windows Registry Editor Version 5.00

  [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Power]
  "HiberbootEnabled"=dword:00000000
  EOF


WINDOWS RECOVERY TIPS
---------------------
1. Always try mounting read-only first to assess the situation
2. Windows Fast Startup/hibernation prevents safe NTFS writes
3. BitLocker recovery key is essential - no key = no access
4. chntpw blanks passwords; it cannot recover/show old passwords
5. Back up registry hives before editing them
6. If Windows is bootable but locked out, just reset the password
7. For serious Windows issues, Windows Recovery Environment may be needed
8. Some antivirus/security software may re-lock accounts on next boot

================================================================================
5. HARDWARE DIAGNOSTICS
================================================================================

QUICK REFERENCE
---------------
  tldr smartctl       # Check drive health
  tldr lshw           # List hardware
  tldr hdparm         # Disk info and benchmarks
  man memtester       # Memory testing
  man stress-ng       # Stress testing

SCENARIO: Check if a drive is failing (SMART)
---------------------------------------------
Quick health check:

  smartctl -H /dev/sdX

Full SMART report:

  smartctl -a /dev/sdX

For NVMe drives:

  smartctl -a /dev/nvme0n1
  nvme smart-log /dev/nvme0n1

Key SMART attributes to watch:
  - Reallocated_Sector_Ct: Bad sectors remapped (increasing = dying)
  - Current_Pending_Sector: Sectors waiting to be remapped
  - Offline_Uncorrectable: Unreadable sectors
  - UDMA_CRC_Error_Count: Cable/connection issues
  - Wear_Leveling_Count: SSD wear (lower = more worn)

Run a self-test:

  smartctl -t short /dev/sdX    # Quick test (~2 min)
  smartctl -t long /dev/sdX     # Thorough test (~hours)

Check test results:

  smartctl -l selftest /dev/sdX


SCENARIO: Test RAM for errors
-----------------------------
Option 1: Memtest86+ (from boot menu)
  - Restart and select "Memtest86+" from the boot menu
  - Most thorough test, runs before OS loads
  - Let it run for at least 1-2 passes (can take hours)

Option 2: memtester (from running system)
  - Tests available RAM while system is running
  - Can't test RAM used by kernel/programs

Test 1GB of RAM (adjust based on free memory):

  free -h                       # Check available memory
  memtester 1G 1                # Test 1GB, 1 iteration
  memtester 2G 5                # Test 2GB, 5 iterations

Note: memtester can only test free RAM. For thorough testing,
use Memtest86+ from the boot menu.


SCENARIO: Monitor temperatures, fans, voltages
----------------------------------------------
First, detect and load sensor modules:

  sensors-detect --auto         # Auto-detect sensors

Then view readings:

  sensors                       # Show all sensor data

Continuous monitoring:

  watch -n 1 sensors            # Update every second

If sensors shows nothing, modules may need loading:

  modprobe coretemp             # Intel CPU temps
  modprobe k10temp              # AMD CPU temps
  modprobe nct6775              # Common motherboard chip


SCENARIO: Stress test hardware (verify stability)
-------------------------------------------------
Useful for:
  - Testing used/refurbished hardware
  - Verifying overclocking stability
  - Burn-in testing before deployment
  - Reproducing intermittent issues

CPU stress test:

  stress-ng --cpu $(nproc) --timeout 300s     # All cores, 5 min

Memory stress test:

  stress-ng --vm 2 --vm-bytes 1G --timeout 300s

Combined CPU + memory:

  stress-ng --cpu $(nproc) --vm 2 --vm-bytes 1G --timeout 600s

Disk I/O stress:

  stress-ng --hdd 2 --timeout 300s

Monitor during stress test (in another terminal):

  watch -n 1 sensors            # Watch temperatures
  htop                          # Watch CPU/memory usage


SCENARIO: Get detailed hardware information
-------------------------------------------
Full hardware report:

  lshw                          # All hardware (verbose)
  lshw -short                   # Summary view
  lshw -html > hardware.html    # HTML report

Specific components:

  lshw -class processor         # CPU info
  lshw -class memory            # RAM info
  lshw -class disk              # Disk info
  lshw -class network           # Network adapters

BIOS/motherboard info:

  dmidecode                     # All DMI tables
  dmidecode -t bios             # BIOS info
  dmidecode -t system           # System/motherboard
  dmidecode -t memory           # Memory slots and modules
  dmidecode -t processor        # CPU socket info

Quick system overview:

  inxi -Fxz                     # If inxi is installed
  cat /proc/cpuinfo             # CPU details
  cat /proc/meminfo             # Memory details


SCENARIO: Test disk speed / benchmark
-------------------------------------
Basic read speed test:

  hdparm -t /dev/sdX            # Buffered read speed
  hdparm -T /dev/sdX            # Cached read speed

More accurate test (run 3 times, average):

  hdparm -tT /dev/sdX
  hdparm -tT /dev/sdX
  hdparm -tT /dev/sdX

Get drive information:

  hdparm -I /dev/sdX            # Detailed drive info

For NVMe drives:

  nvme list                     # List NVMe drives
  nvme id-ctrl /dev/nvme0n1     # Controller info
  nvme smart-log /dev/nvme0n1   # SMART/health data


SCENARIO: Check for bad blocks (surface scan)
---------------------------------------------
WARNING: This is read-only but takes a long time on large drives.

  badblocks -sv /dev/sdX

For faster progress indication:

  badblocks -sv -b 4096 /dev/sdX

Note: For modern drives, SMART is usually more informative.
badblocks is useful for older drives without good SMART support.


SCENARIO: Identify unknown hardware / find drivers
--------------------------------------------------
List PCI devices:

  lspci                         # All PCI devices
  lspci -v                      # Verbose (with drivers)
  lspci -k                      # Show kernel drivers

List USB devices:

  lsusb                         # All USB devices
  lsusb -v                      # Verbose

Find what driver a device is using:

  lspci -k | grep -A3 "Network"    # Network adapter driver
  lspci -k | grep -A3 "VGA"        # Graphics driver


HARDWARE DIAGNOSTICS TIPS
-------------------------
1. Run SMART checks regularly - drives often show warning signs
2. Memtest86+ (from boot menu) is more thorough than memtester
3. Stress test new/used hardware before trusting it with data
4. High temperatures during stress test = cooling problem
5. Random crashes/errors often indicate RAM or power issues
6. SMART "Reallocated Sector Count" increasing = drive dying
7. Back up immediately if SMART shows any warnings
8. SSDs have limited write cycles - check Wear_Leveling_Count

================================================================================
6. DISK OPERATIONS
================================================================================

QUICK REFERENCE
---------------
  tldr partclone      # Filesystem-aware partition cloning
  tldr fsarchiver     # Backup/restore filesystems to archive
  man nwipe           # Secure disk wiping (DBAN replacement)
  tldr parted         # Partition management
  tldr mkfs           # Create filesystems

FIRST: Understand your options for disk copying
-----------------------------------------------
Different tools for different situations:

  dd / ddrescue    - Byte-for-byte copy (use for failing drives)
  partclone        - Filesystem-aware, only copies used blocks (faster)
  fsarchiver       - Creates compressed archive (smallest, most flexible)
  partimage        - Legacy imaging (for restoring old partimage backups)

Rule of thumb:
  - Failing drive? Use ddrescue (section 2)
  - Clone partition quickly? Use partclone
  - Backup for long-term storage? Use fsarchiver
  - Restore old .img.gz from partimage? Use partimage


SCENARIO: Clone a partition (partclone - faster than dd)
--------------------------------------------------------
Partclone only copies used blocks. A 500GB partition with 50GB used
takes ~50GB to clone instead of 500GB.

Clone ext4 partition to image:

  partclone.ext4 -c -s /dev/sdX1 -o partition.img

Clone with compression (recommended):

  partclone.ext4 -c -s /dev/sdX1 | gzip -c > partition.img.gz

  -c = clone mode
  -s = source
  -o = output

Restore from image:

  partclone.ext4 -r -s partition.img -o /dev/sdX1

Restore from compressed image:

  gunzip -c partition.img.gz | partclone.ext4 -r -s - -o /dev/sdX1

Supported filesystems:

  partclone.ext4      partclone.ext3      partclone.ext2
  partclone.ntfs      partclone.fat32     partclone.fat16
  partclone.xfs       partclone.btrfs     partclone.exfat
  partclone.f2fs      partclone.dd        (dd mode for any fs)


SCENARIO: Create a full system backup (fsarchiver)
--------------------------------------------------
Fsarchiver creates compressed, portable archives. Archives can be
restored to different-sized partitions.

Backup a filesystem:

  fsarchiver savefs backup.fsa /dev/sdX1

Backup with compression level and progress:

  fsarchiver savefs -v -z7 backup.fsa /dev/sdX1

  -v  = verbose
  -z7 = compression level (1-9, higher = smaller but slower)

Backup multiple filesystems to one archive:

  fsarchiver savefs backup.fsa /dev/sdX1 /dev/sdX2 /dev/sdX3

List contents of archive:

  fsarchiver archinfo backup.fsa

Restore to a partition:

  fsarchiver restfs backup.fsa id=0,dest=/dev/sdX1

  id=0 = first filesystem in archive (0, 1, 2...)

Restore to different-sized partition (will resize):

  fsarchiver restfs backup.fsa id=0,dest=/dev/sdY1


SCENARIO: Restore a legacy partimage backup
-------------------------------------------
Partimage is legacy software but you may have old backups to restore.

Restore partimage backup:

  partimage restore /dev/sdX1 backup.img.gz

Interactive mode:

  partimage

Note: partimage cannot create images of ext4, GPT, or modern filesystems.
Use fsarchiver for new backups.


SCENARIO: Securely wipe a drive (nwipe)
---------------------------------------
DANGER: This PERMANENTLY DESTROYS all data. Triple-check the device!

Interactive mode (recommended - shows all drives, select with space):

  nwipe

Wipe specific drive with single zero pass (usually sufficient):

  nwipe --method=zero /dev/sdX

Wipe with DoD 3-pass method:

  nwipe --method=dod /dev/sdX

Wipe with verification:

  nwipe --verify=last /dev/sdX

Available wipe methods:

  zero      - Single pass of zeros (fastest, usually sufficient)
  one       - Single pass of ones
  random    - Random data
  dod       - DoD 5220.22-M (3 passes)
  dodshort  - DoD short (3 passes)
  gutmann   - Gutmann 35-pass (overkill for modern drives)

For SSDs, use the drive's built-in secure erase instead:

  # Set a temporary password
  hdparm --user-master u --security-set-pass Erase /dev/sdX
  # Trigger secure erase (password is cleared after)
  hdparm --user-master u --security-erase Erase /dev/sdX

For NVMe SSDs:

  nvme format /dev/nvme0n1 --ses=1    # Cryptographic erase


SCENARIO: Work with XFS filesystems
-----------------------------------
Create XFS filesystem:

  mkfs.xfs /dev/sdX1
  mkfs.xfs -L "mylabel" /dev/sdX1     # With label

Repair XFS (must be unmounted):

  xfs_repair /dev/sdX1
  xfs_repair -n /dev/sdX1             # Check only, no changes

Grow XFS filesystem (while mounted):

  xfs_growfs /mountpoint

Note: XFS cannot be shrunk, only grown.

Show XFS info:

  xfs_info /mountpoint


SCENARIO: Work with Btrfs filesystems
-------------------------------------
Create Btrfs filesystem:

  mkfs.btrfs /dev/sdX1
  mkfs.btrfs -L "mylabel" /dev/sdX1   # With label

Check Btrfs (must be unmounted):

  btrfs check /dev/sdX1
  btrfs check --repair /dev/sdX1      # Repair (use with caution!)

Scrub (online integrity check - safe):

  btrfs scrub start /mountpoint
  btrfs scrub status /mountpoint

Show filesystem info:

  btrfs filesystem show
  btrfs filesystem df /mountpoint
  btrfs filesystem usage /mountpoint

List/manage subvolumes:

  btrfs subvolume list /mountpoint
  btrfs subvolume create /mountpoint/newsubvol
  btrfs subvolume delete /mountpoint/subvol


SCENARIO: Work with F2FS filesystems (Flash-Friendly)
-----------------------------------------------------
F2FS is optimized for flash storage (SSDs, SD cards, USB drives).
Common on Android devices.

Create F2FS filesystem:

  mkfs.f2fs /dev/sdX1
  mkfs.f2fs -l "mylabel" /dev/sdX1    # With label

Check/repair F2FS:

  fsck.f2fs /dev/sdX1
  fsck.f2fs -a /dev/sdX1              # Auto-repair


SCENARIO: Work with exFAT filesystems
-------------------------------------
exFAT is common on USB drives and SD cards (>32GB).
Cross-platform compatible (Windows, Mac, Linux).

Create exFAT filesystem:

  mkfs.exfat /dev/sdX1
  mkfs.exfat -L "LABEL" /dev/sdX1     # With label (uppercase recommended)

Check/repair exFAT:

  fsck.exfat /dev/sdX1
  fsck.exfat -a /dev/sdX1             # Auto-repair


SCENARIO: Partition a disk
--------------------------
Interactive partition editors:

  parted /dev/sdX                     # Works with GPT and MBR
  gdisk /dev/sdX                      # GPT-specific (recommended for UEFI)
  fdisk /dev/sdX                      # Traditional (MBR or GPT)

Create GPT partition table:

  parted /dev/sdX mklabel gpt

Create partitions (example: 512MB EFI + rest for Linux):

  parted /dev/sdX mkpart primary fat32 1MiB 513MiB
  parted /dev/sdX set 1 esp on
  parted /dev/sdX mkpart primary ext4 513MiB 100%

View partition layout:

  parted /dev/sdX print
  lsblk -f /dev/sdX
  fdisk -l /dev/sdX


DISK OPERATIONS TIPS
--------------------
1. partclone is 5-10x faster than dd for partially-filled partitions
2. fsarchiver archives can restore to different-sized partitions
3. For SSDs, nwipe is less effective than ATA/NVMe secure erase
4. Always verify backups can be restored before wiping originals
5. XFS cannot be shrunk, only grown - plan partition sizes carefully
6. Btrfs check --repair is risky; try without --repair first
7. Keep partition tables aligned to 1MiB boundaries for SSD performance
8. exFAT is best for cross-platform USB drives >32GB
9. F2FS is optimized for flash but less portable than ext4

================================================================================
7. NETWORK TROUBLESHOOTING
================================================================================

QUICK REFERENCE
---------------
  tldr ip             # Network interface configuration
  tldr nmcli          # NetworkManager CLI
  tldr ping           # Test connectivity
  tldr ss             # Socket statistics (netstat replacement)
  tldr curl           # Transfer data from URLs

FIRST: Check basic network connectivity
---------------------------------------
Is the interface up?

  ip link show
  ip a                            # Show all addresses

Is there an IP address?

  ip addr show dev eth0           # Replace eth0 with your interface
  ip addr show dev wlan0          # For WiFi

Can you reach the gateway?

  ip route                        # Show default gateway
  ping -c 3 $(ip route | grep default | awk '{print $3}')

Can you reach the internet?

  ping -c 3 1.1.1.1               # Test IP connectivity
  ping -c 3 google.com            # Test DNS resolution


SCENARIO: Configure network with NetworkManager
-----------------------------------------------
List connections:

  nmcli connection show

Show WiFi networks:

  nmcli device wifi list

Connect to WiFi:

  nmcli device wifi connect "SSID" password "password"

Show current connection details:

  nmcli device show

Restart networking:

  systemctl restart NetworkManager


SCENARIO: Configure network manually (no NetworkManager)
--------------------------------------------------------
Bring up interface:

  ip link set eth0 up

Get IP via DHCP:

  dhclient eth0
  # or
  dhcpcd eth0

Set static IP:

  ip addr add 192.168.1.100/24 dev eth0
  ip route add default via 192.168.1.1

Set DNS:

  echo "nameserver 1.1.1.1" > /etc/resolv.conf


SCENARIO: Mount remote filesystem over SSH (sshfs)
--------------------------------------------------
Access files on a remote system as if they were local.
Useful for copying data to/from a working machine during recovery.

Mount remote directory:

  mkdir -p /mnt/remote
  sshfs user@hostname:/path/to/dir /mnt/remote

Mount with password prompt (if no SSH keys):

  sshfs user@hostname:/home/user /mnt/remote -o password_stdin

Mount remote root filesystem:

  sshfs root@192.168.1.100:/ /mnt/remote

Common options:

  sshfs user@host:/path /mnt/remote -o reconnect           # Auto-reconnect
  sshfs user@host:/path /mnt/remote -o port=2222           # Custom SSH port
  sshfs user@host:/path /mnt/remote -o IdentityFile=~/.ssh/key  # SSH key

Copy files to/from mounted remote:

  cp /mnt/remote/important-file.txt /local/backup/
  rsync -avP /local/data/ /mnt/remote/backup/

Unmount when done:

  fusermount -u /mnt/remote
  # or
  umount /mnt/remote

Why use sshfs instead of scp/rsync?
  - Browse remote files interactively before deciding what to copy
  - Run local tools on remote files (grep, diff, etc.)
  - Easier than remembering rsync syntax for quick operations


SCENARIO: Transfer files over SSH
---------------------------------
Copy file to remote:

  scp localfile.txt user@host:/path/to/destination/

Copy file from remote:

  scp user@host:/path/to/file.txt /local/destination/

Copy directory recursively:

  scp -r /local/dir user@host:/remote/path/

With progress and compression:

  rsync -avzP /local/path/ user@host:/remote/path/


SCENARIO: Test network speed and latency
----------------------------------------
Ping with timing:

  ping -c 10 hostname              # 10 pings with statistics

Traceroute (find network path):

  traceroute hostname
  traceroute -I hostname           # Use ICMP (may work better)

Test bandwidth (if iperf3 server available):

  iperf3 -c server-ip              # Test to iperf3 server


SCENARIO: Debug DNS issues
--------------------------
Check current DNS servers:

  cat /etc/resolv.conf

Test DNS resolution:

  host google.com
  dig google.com
  nslookup google.com

Test specific DNS server:

  dig @1.1.1.1 google.com
  dig @8.8.8.8 google.com

Temporarily use different DNS:

  echo "nameserver 1.1.1.1" > /etc/resolv.conf


SCENARIO: Check what's listening on ports
-----------------------------------------
Show all listening ports:

  ss -tlnp                         # TCP
  ss -ulnp                         # UDP
  ss -tulnp                        # Both

Check if specific port is open:

  ss -tlnp | grep :22              # SSH
  ss -tlnp | grep :80              # HTTP

Check what process is using a port:

  ss -tlnp | grep :8080


SCENARIO: Download files
------------------------
Download with curl:

  curl -O https://example.com/file.iso
  curl -L -O https://example.com/file    # Follow redirects

Download with wget:

  wget https://example.com/file.iso
  wget -c https://example.com/file.iso   # Resume partial download

Download and verify checksum:

  curl -O https://example.com/file.iso
  curl -O https://example.com/file.iso.sha256
  sha256sum -c file.iso.sha256


NETWORK TROUBLESHOOTING TIPS
----------------------------
1. If no IP, check cable/wifi and try dhclient or dhcpcd
2. If IP but no internet, check gateway with ip route
3. If gateway reachable but no internet, check DNS
4. Use ping 1.1.1.1 to test IP connectivity without DNS
5. sshfs is great for browsing before deciding what to copy
6. rsync -avzP is better than scp for large transfers (resumable)
7. Check firewall if services aren't reachable: iptables -L
8. For WiFi issues, check rfkill: rfkill list

================================================================================
8. ENCRYPTION & GPG
================================================================================

QUICK REFERENCE
---------------
  tldr gpg            # GNU Privacy Guard
  tldr cryptsetup     # LUKS disk encryption
  man gpg             # Full GPG manual

FIRST: Understand encryption types you may encounter
----------------------------------------------------
Common encryption scenarios in recovery:

  GPG symmetric     - Password-protected files (gpg -c)
  GPG asymmetric    - Public/private key encrypted files
  LUKS              - Full disk/partition encryption (Linux standard)
  BitLocker         - Windows disk encryption (see section 4)
  ZFS encryption    - ZFS native encryption (see section 1)

This section covers GPG and LUKS. For BitLocker, see section 4.
For ZFS encryption, see section 1.


SCENARIO: Decrypt a password-protected file (GPG symmetric)
-----------------------------------------------------------
Files encrypted with `gpg -c` use a password only, no keys needed.

Decrypt to original filename:

  gpg -d encrypted-file.gpg > decrypted-file

Decrypt (GPG auto-detects output name if .gpg extension):

  gpg encrypted-file.gpg

You'll be prompted for the password.

Decrypt with password on command line (less secure, visible in history):

  gpg --batch --passphrase "password" -d file.gpg > file


SCENARIO: Decrypt a file encrypted to your GPG key
--------------------------------------------------
Files encrypted with `gpg -e -r yourname@email.com` require your private key.

If your private key is on this system:

  gpg -d encrypted-file.gpg > decrypted-file

If you need to import your private key first:

  gpg --import /path/to/private-key.asc
  gpg -d encrypted-file.gpg > decrypted-file

You'll be prompted for your key's passphrase.


SCENARIO: Import GPG keys (public or private)
---------------------------------------------
Import a public key (to verify signatures or encrypt to someone):

  gpg --import public-key.asc

Import from a keyserver:

  gpg --keyserver keyserver.ubuntu.com --recv-keys KEYID

Import your private key (for decryption):

  gpg --import private-key.asc

List keys on the system:

  gpg --list-keys               # Public keys
  gpg --list-secret-keys        # Private keys


SCENARIO: Verify a signed file or ISO
-------------------------------------
Verify a detached signature (.sig or .asc file):

  gpg --verify file.iso.sig file.iso

If you don't have the signer's public key:

  # Find the key ID in the error message, then:
  gpg --keyserver keyserver.ubuntu.com --recv-keys KEYID
  gpg --verify file.iso.sig file.iso

Verify an inline-signed message:

  gpg --verify signed-message.asc


SCENARIO: Encrypt a file for safe transfer
------------------------------------------
Symmetric encryption (password only - recipient needs password):

  gpg -c sensitive-file.txt
  # Creates sensitive-file.txt.gpg

With specific cipher and compression:

  gpg -c --cipher-algo AES256 sensitive-file.txt

Asymmetric encryption (to someone's public key):

  gpg -e -r recipient@email.com sensitive-file.txt

Encrypt to multiple recipients:

  gpg -e -r alice@example.com -r bob@example.com file.txt


SCENARIO: Unlock a LUKS-encrypted partition
-------------------------------------------
LUKS is the standard Linux disk encryption.

Check if a partition is LUKS-encrypted:

  cryptsetup isLuks /dev/sdX1 && echo "LUKS encrypted"
  lsblk -f    # Shows "crypto_LUKS" for encrypted partitions

Open (decrypt) a LUKS partition:

  cryptsetup open /dev/sdX1 decrypted
  # Enter passphrase when prompted
  # Creates /dev/mapper/decrypted

Mount the decrypted partition:

  mount /dev/mapper/decrypted /mnt/recovery

When done, unmount and close:

  umount /mnt/recovery
  cryptsetup close decrypted


SCENARIO: Open LUKS with a key file
-----------------------------------
If LUKS was set up with a key file instead of (or in addition to) password:

  cryptsetup open /dev/sdX1 decrypted --key-file /path/to/keyfile

Key file might be on a USB drive:

  mount /dev/sdb1 /mnt/usb
  cryptsetup open /dev/sdX1 decrypted --key-file /mnt/usb/luks-key


SCENARIO: Recover data from damaged LUKS header
-----------------------------------------------
If LUKS header is damaged, you need a header backup (hopefully you made one).

Restore LUKS header from backup:

  cryptsetup luksHeaderRestore /dev/sdX1 --header-backup-file header-backup.img

If no backup exists and header is damaged, data is likely unrecoverable.
This is why LUKS header backups are critical:

  # How to create a header backup (do this BEFORE disaster):
  cryptsetup luksHeaderBackup /dev/sdX1 --header-backup-file header-backup.img


SCENARIO: Access eCryptfs encrypted home directory
--------------------------------------------------
Ubuntu's legacy home encryption uses eCryptfs.

Mount an eCryptfs-encrypted home:

  # You need the user's login password
  ecryptfs-recover-private

Or manually:

  mount -t ecryptfs /home/.ecryptfs/username/.Private /mnt/recovery


ENCRYPTION TIPS
---------------
1. GPG symmetric encryption (gpg -c) only needs the password to decrypt
2. GPG asymmetric encryption requires the private key - no key = no access
3. Always keep LUKS header backups separate from the encrypted drive
4. BitLocker recovery keys are often in Microsoft accounts
5. ZFS encryption keys are derived from passphrase - no separate key file
6. eCryptfs wrapped passphrase is in ~/.ecryptfs/wrapped-passphrase
7. If you forget encryption passwords and have no backups, data is gone
8. Hardware security keys (YubiKey) may be required for some GPG keys

================================================================================
                              END OF GUIDE
================================================================================