diff options
Diffstat (limited to 'assets/2026-02-03-yt-sync-improvements-obsolete.md')
| -rw-r--r-- | assets/2026-02-03-yt-sync-improvements-obsolete.md | 100 |
1 files changed, 100 insertions, 0 deletions
diff --git a/assets/2026-02-03-yt-sync-improvements-obsolete.md b/assets/2026-02-03-yt-sync-improvements-obsolete.md new file mode 100644 index 0000000..0db1161 --- /dev/null +++ b/assets/2026-02-03-yt-sync-improvements-obsolete.md @@ -0,0 +1,100 @@ +# yt-sync.sh Improvements + +## Problem +- Current scan takes 1-2 hours +- Scans 200 videos × 15 channels = 3000 metadata fetches +- Most videos are skipped (already downloaded or too old) +- Cron running hourly causes overlap + +## Speed Improvements for yt-dlp + +### 1. Break on existing (RECOMMENDED) +```bash +--break-on-existing +``` +Stops scanning when it hits a video already in archive. Since playlists are chronological, once we hit an old video, all subsequent are old too. + +### 2. Break on date reject +```bash +--break-on-reject +``` +Stops when hitting a video outside the --dateafter range. Combined with chronological order, stops at first old video. + +### 3. Reduce playlist scan depth +```bash +--playlist-end 50 # Instead of 200 +``` +Most channels don't post 50 videos in 30 days. + +### 4. Track last sync timestamp +Store last successful sync time and use tighter --dateafter: +```bash +LAST_SYNC_FILE="$YOUTUBE_DIR/.last_sync" +if [[ -f "$LAST_SYNC_FILE" ]]; then + LAST_SYNC=$(cat "$LAST_SYNC_FILE") + DATE_AFTER="--dateafter $LAST_SYNC" +else + DATE_AFTER="--dateafter $(date -d '30 days ago' '+%Y%m%d')" +fi +# After successful sync: +date '+%Y%m%d' > "$LAST_SYNC_FILE" +``` + +### 5. Parallel channel downloads (aggressive) +Use GNU parallel to download multiple channels simultaneously: +```bash +parallel -j 3 yt-dlp [opts] ::: "${CHANNELS[@]}" +``` +Risk: More likely to trigger rate limiting. + +## Scheduling Options + +### Option A: Systemd timer (prevents overlap) +```ini +# ~/.config/systemd/user/yt-sync.timer +[Unit] +Description=YouTube Sync Timer + +[Timer] +OnCalendar=*-*-* 00,06,12,18:00:00 +Persistent=true + +[Install] +WantedBy=timers.target +``` + +```ini +# ~/.config/systemd/user/yt-sync.service +[Unit] +Description=YouTube Sync + +[Service] +Type=oneshot +ExecStart=/home/cjennings/.local/bin/yt-sync.sh all +ExecStartPost=/home/cjennings/.local/bin/yt-sync.sh sync +``` + +Systemd won't start a new run if previous is still running. + +### Option B: Lock file wrapper +```bash +#!/bin/bash +LOCKFILE="/tmp/yt-sync.lock" +exec 200>"$LOCKFILE" +flock -n 200 || { echo "Already running"; exit 1; } +# ... run sync ... +``` + +### Option C: Longer cron interval +```cron +# Every 4 hours during off-peak +0 0,4,20 * * * /home/cjennings/.local/bin/yt-sync.sh all && yt-sync.sh sync +``` + +## Recommended Changes + +1. Add `--break-on-existing` to YT_OPTS (biggest win) +2. Add `--break-on-reject` to YT_OPTS +3. Reduce `--playlist-end` to 50 +4. Use systemd timer instead of cron +5. Optionally track last sync date for tighter filtering |
