diff options
Diffstat (limited to 'working')
| -rw-r--r-- | working/fsrs-spec/fsrs-spec.org | 354 | ||||
| -rw-r--r-- | working/stats-dashboard/stats-dashboard.org | 419 |
2 files changed, 0 insertions, 773 deletions
diff --git a/working/fsrs-spec/fsrs-spec.org b/working/fsrs-spec/fsrs-spec.org deleted file mode 100644 index d0f0bc1..0000000 --- a/working/fsrs-spec/fsrs-spec.org +++ /dev/null @@ -1,354 +0,0 @@ -#+TITLE: FSRS scheduler for org-drill — v0 design spec -#+AUTHOR: Craig Jennings -#+DATE: 2026-05-27 - -* Status - -Draft v0 — pending decisions marked =DECIDE:= inline. The companion todo -entry is =todo.org=:125 [#B] "FSRS (Free Spaced Repetition Scheduler) -Algorithm". No implementation has started. - -* Context and motivation - -org-drill ships three SuperMemo-family schedulers — SM2, SM5, and Simple8. -They share an interval/repeats/EF/failures/meanq/total-repeats state model -inherited from the SuperMemo 2 paper and refined since. The card-state -struct refactor (#147) bundles those fields into =org-drill-card-state= and -makes plugging a new scheduler into the =cl-case= dispatch a one-arm change -in =org-drill-smart-reschedule= and =org-drill-hypothetical-next-review-date=. - -FSRS (Free Spaced Repetition Scheduler) is the algorithm Anki has migrated -to since 2024. It uses the DSR memory model (Difficulty, Stability, -Retrievability) and a 17-to-21-parameter formula family fit on hundreds of -millions of real reviews. Published evaluations show materially better -retention than SM* at the same review load. Adding it as a fourth option -to =org-drill-spaced-repetition-algorithm= lets a user opt in without -disturbing existing SM-family decks. - -* Goals - -- A fourth value =fsrs= for =org-drill-spaced-repetition-algorithm=. -- The =fsrs= arm of the =cl-case= in =smart-reschedule= and - =hypothetical-next-review-date= calls =org-drill-determine-next-interval-fsrs= - with the same shape the SM-family arms now use (a state value + the - rating quality + algorithm-specific extras). -- Per-card FSRS state persisted to org properties. Existing SM-style - properties stay untouched on cards that haven't been switched. -- Test coverage at parity with the SM* test files: Normal / Boundary / - Error per public function, plus reference-vector tests checking the - algorithm output against the canonical FSRS implementation. - -* Non-goals (v1) - -- *No parameter optimizer.* Anki's FSRS shines because of parameter - fitting on each user's review history. v1 ships fixed default - parameters only; the optimizer becomes its own todo and ticket. Once - the algorithm is in the tree, plugging an optimizer in front of it is - additive. -- *No retro-migration heuristic.* A card switched from SM* to FSRS - cold-starts on its next review — FSRS treats it as a fresh first review. - Anki has heuristics that estimate initial =D= and =S= from prior review - history; we explicitly defer that as a follow-on. -- *No bulk rescheduling pass.* Switching algorithms does not retroactively - rewrite SCHEDULED dates on existing cards. Cards keep their current - SCHEDULED until their next review. - -* Version pin - -=DECIDE:= FSRS-4.5 (recommended) vs FSRS-6.x. - -The current stable upstream is FSRS-6.3.1 (March 2026) with 21 -parameters. FSRS-4.5 is the last release whose generic defaults provide -meaningful retention improvement over SM-family scheduling without -optimization. The FSRS community's own guidance — quoting Expertium's -algorithm explainer, March 2026 — is that "generic v6 defaults offer -minimal improvement over v4.5" because v6's gains come from optimizing -=w[17]..w[20]= on a user's data. - -Since v1 ships without an optimizer (above), the pragmatic pin is -*FSRS-4.5*: 17 parameters, well-documented defaults, the most-published -test vectors, and the version with the largest delta over SM* in the -no-optimizer regime. Upgrading to v6 becomes a separate ticket alongside -the optimizer one — by then there'd be a real reason to take v6's -additional surface. - -Reference parameters (FSRS-4.5 defaults, source: =py-fsrs= up to v4 and -the original =fsrs4anki= release notes): - -#+begin_src elisp -;; w[0..16], FSRS-4.5 -(0.4 0.6 2.4 5.8 4.93 0.94 0.86 0.01 1.49 0.14 0.94 2.18 0.05 0.34 1.26 0.29 2.61) -#+end_src - -(=DECIDE:= confirm this exact tuple before code lands. The 4.5 defaults -were tuned twice during 4.x's life; pinning the final 4.5-stable tuple is -a one-line lookup against the =py-fsrs= 4.x release tag, deferred to -implementation.) - -* Algorithm shape (v4.5) - -The four-rating scale FSRS uses internally: - -| Rating | Name | Meaning | -|--------+-------+------------------------| -| 1 | Again | failed recall | -| 2 | Hard | recalled with struggle | -| 3 | Good | recalled normally | -| 4 | Easy | trivial recall | - -org-drill uses a 0–5 quality scale. The mapping respects -=org-drill-failure-quality= (default 2): - -| org-drill quality | FSRS rating | -|-------------------+-------------| -| 0, 1, 2 | Again (1) | -| 3 | Hard (2) | -| 4 | Good (3) | -| 5 | Easy (4) | - -=DECIDE:= mapping table. Above is the natural reading of -=org-drill-failure-quality=, but org-drill historically lets users -remap session keys, and the mapping should respect that or be its -own defcustom. Simplest: pin the mapping above for v1 and add -=org-drill-fsrs-quality-mapping= as a future enhancement. - -** State model - -Per-card persisted state: - -| Property | Type | Meaning | -|-----------------------+--------+--------------------------------------| -| =DRILL_FSRS_STABILITY= | float | =S=, the memory-stability estimate | -| =DRILL_FSRS_DIFFICULTY= | float | =D=, the difficulty estimate (1..10) | -| =DRILL_FSRS_REVIEWS= | int | review count under FSRS (≥ 0) | -| =DRILL_FSRS_LAPSES= | int | failures (rating=Again) under FSRS | -| =DRILL_LAST_REVIEWED= | time | reused from the existing SM property | - -The last-reviewed timestamp is already written by the SM path -(=org-drill-store-item-data= writes it via =org-set-property= elsewhere in -the flow); FSRS reads the same property. - -A virgin FSRS card has all four =DRILL_FSRS_*= properties absent; the -scheduler treats absence as "first review." - -** Update equations (v4.5 form) - -First review (no prior FSRS state) with rating R ∈ {1..4}: - -#+begin_src elisp -S0 = w[R-1] ; initial stability per rating -D0 = clamp (- w[4] (* w[5] (- R 2))) 1 10 -#+end_src - -Subsequent review at time =elapsed-days= since last review, with current -stability =S=, difficulty =D=, and rating =R=: - -#+begin_src elisp -;; Retrievability when the card is shown -R-retrieval = (expt (1+ (/ elapsed-days (* 9 S))) -1) - -;; Difficulty update (linear with mean reversion toward w[4]) -D-delta = (* w[6] (- (- R 3))) ; positive on Again, neutral on Good -D-new-raw = (+ D D-delta) -D-new = (+ (* w[7] w[4]) (* (- 1 w[7]) D-new-raw)) ; mean reversion -D-new = (clamp D-new 1 10) - -;; Stability update — success (R ≥ 2) -S-new = S * (1 + - (exp w[8]) * - ((11 - D) / 9) * - (S ^ (-w[9])) * - ((exp ((1 - R-retrieval) * w[10])) - 1) * - (R == Hard ? w[15] : 1) * - (R == Easy ? w[16] : 1)) - -;; Stability update — lapse (R = 1, Again) -S-new = w[11] * - D^(-w[12]) * - ((S + 1)^w[13] - 1) * - exp((1 - R-retrieval) * w[14]) -#+end_src - -Next interval given desired retention =DR= and new stability =S=: - -#+begin_src elisp -I = (* 9 S (- (expt DR -1) 1)) ; days, rounded down (floor) -#+end_src - -Default desired retention is *0.9* (a common Anki default). Pinned as -=org-drill-fsrs-desired-retention= defcustom. - -=DECIDE:= the exact form of the v4.5 update equations above is paraphrased -from Expertium's algorithm walkthrough (March 2026, v6-oriented) and the -=py-fsrs= reference implementation. Implementation will cross-check the -exact constants and grouping against a tagged =py-fsrs= v4.x release -before writing the function body. - -** Default parameters (org-drill defcustoms) - -#+begin_src elisp -(defcustom org-drill-fsrs-parameters - '(0.4 0.6 2.4 5.8 4.93 0.94 0.86 0.01 1.49 0.14 0.94 2.18 0.05 0.34 1.26 0.29 2.61) - "17-tuple FSRS-4.5 default parameters. See =docs/fsrs-spec.org=.") - -(defcustom org-drill-fsrs-desired-retention 0.9 - "Target retention for FSRS scheduling. Higher = shorter intervals.") -#+end_src - -* Integration points - -** Public function - -#+begin_src elisp -(defun org-drill-determine-next-interval-fsrs (state quality) - "Return next-interval (in days) plus updated FSRS state for STATE after -QUALITY (0-5). STATE here is a 5-tuple (S D reviews lapses last-reviewed), -not an `org-drill-card-state' — FSRS stores its own state and treats the SM -slots as opaque.") -#+end_src - -The shape =(state quality)= matches the post-#147 scheduler signature -convention. =state= for FSRS is a *separate* struct =org-drill-fsrs-state= -(or a plist — =DECIDE:= which), populated from the =DRILL_FSRS_*= -properties at the call site. - -=DECIDE:= the return shape. Options: - -1. Same shape as the SM* schedulers (a list with =next-interval= - first plus updated state). Consistent but the trailing slots are - irrelevant to FSRS. -2. A small =org-drill-fsrs-result= struct (=next-interval=, - =new-stability=, =new-difficulty=, =new-reviews=, =new-lapses=). - Cleaner for FSRS but introduces a return-shape special case in the - =cl-case= caller. - -Recommended: option 2. =cl-case= already special-cases =new-ofmatrix= -for SM5; one more is fine and the FSRS state shape is genuinely different. - -** Caller integration - -In =org-drill-smart-reschedule=: - -#+begin_src elisp -;; sketch — exact destructure shape TBD per the return-shape DECIDE -(fsrs - (let ((fsrs-state (org-drill-get-fsrs-state))) - (org-drill-fsrs-apply-result - (org-drill-determine-next-interval-fsrs fsrs-state quality)))) -#+end_src - -Two new helpers parallel the existing item-data round-trip: - -- =org-drill-get-fsrs-state= — reads the =DRILL_FSRS_*= properties, returns - the state struct or nil for a virgin card. -- =org-drill-store-fsrs-state= (or =org-drill-fsrs-apply-result=) — writes - the updated state back. The =SCHEDULED= timestamp is set via - =org-schedule= using the returned =next-interval=, the same way - =smart-reschedule= already does for SM*. - -Same in =hypothetical-next-review-date= (read-only, no store). - -** Defcustom entry - -=org-drill-spaced-repetition-algorithm= grows a fourth =:option= line: - -#+begin_src elisp -- FSRS :: Free Spaced Repetition Scheduler, the DSR-based algorithm - used in Anki since 2024. See `docs/fsrs-spec.org` for the algorithm - and the state model. -#+end_src - -* Backward compatibility - -A card last reviewed under SM* has =DRILL_LAST_INTERVAL=, -=DRILL_REPEATS_SINCE_FAIL=, etc., but no =DRILL_FSRS_*=. On the first -FSRS review after a switch: - -- The FSRS scheduler sees absent =DRILL_FSRS_*= and treats this as a - virgin first review (initial =S₀=, =D₀= from the rating). -- The SM-style properties are left in place. If the user switches back, - the SM scheduler reads them as before. -- The =DRILL_FSRS_*= properties are written from this point forward. - Cards drilled under both algorithms accumulate both property sets. - -The cold-start tradeoff is a known one-time loss of the SM-derived -estimate of card difficulty. Anki's migration heuristic estimates -initial =D= and =S= from the SM2 ease factor and interval; pulling that in -is its own ticket (=todo:= "FSRS migration heuristic from SM history"). - -=DECIDE:= whether to also surface a one-line "first FSRS review on this -card — cold-starting" message during the prompt for the first FSRS review. -Probably not — users don't need to know. - -* Test strategy - -Three categories per public function, matching the existing scheduler -test files: - -1. =tests/test-org-drill-determine-next-interval-fsrs.el= — Normal / - Boundary / Error coverage of =org-drill-determine-next-interval-fsrs=. - *Reference-vector tests* check =(state, rating) -> (next-interval, - new-state)= against pre-computed outputs from =py-fsrs= at a tagged - v4.x release. About a dozen vectors covering the four ratings × first - review / nth review / lapse / long-elapsed-time, plus boundary ones - (=elapsed-days=0=, =D= and =S= at clamp limits). - -2. =tests/test-org-drill-fsrs-state-roundtrip.el= — get-fsrs-state / - store-fsrs-state round-trip parity, mirroring the SM - item-data-roundtrip test. - -3. =tests/test-org-drill-fsrs-integration.el= — end-to-end through - =smart-reschedule= with =org-drill-spaced-repetition-algorithm= set to - =fsrs=, covering: virgin card first review, returning card under - FSRS, switched-from-SM card cold-starts cleanly, both algorithms can - coexist on the same buffer. - -Mapping helpers (org-drill quality 0–5 → FSRS rating 1–4) and the -defcustoms get their own small tests. - -The reference vectors get committed alongside the tests as a -machine-checkable expected-output table, generated once from =py-fsrs= -and pinned. The generation script lives at =tests/fixtures/fsrs-vectors.py= -so re-generation is reproducible. - -* Effort estimate - -Multi-day, plausibly spanning sessions: - -- Algorithm function + helpers: 1 day. -- State round-trip + integration in =smart-reschedule=/=hypothetical=: 0.5 - day. -- Test scaffolding + reference vectors + Normal/Boundary/Error: 1 day. -- Documentation (manual entry, README option list, defcustom docstrings, - this spec ratified): 0.5 day. - -Realistic: a session to implement-and-test the algorithm + state -round-trip with reference-vector tests; a follow-up session for the -integration + docs. No optimizer. - -* Open decisions index - -Pinned in one place for the implementation gate: - -- =DECIDE:= version pin (FSRS-4.5 vs newer). Recommended: 4.5. -- =DECIDE:= the exact FSRS-4.5 default-parameter tuple. One-line lookup - against the =py-fsrs= 4.x final release. -- =DECIDE:= quality-mapping table (0–5 → 1–4). Recommended: above. -- =DECIDE:= scheduler return shape — list (SM-shape) or - =org-drill-fsrs-result= struct. Recommended: struct. -- =DECIDE:= whether to surface a "first FSRS review on this card" - message. Recommended: no. -- =DECIDE:= the exact form of the update equations after cross-checking - =py-fsrs= 4.x source. - -* References - -- [[https://github.com/open-spaced-repetition/py-fsrs][py-fsrs]] — the canonical Python reference implementation. Tagged - v4.x releases pin FSRS-4.5; v6.x is the current line. -- [[https://github.com/open-spaced-repetition/fsrs4anki][fsrs4anki]] — the Anki integration; its wiki has user-facing notes on - algorithm history and migration. -- [[https://expertium.github.io/Algorithm.html][Expertium: A technical explanation of FSRS]] (March 2026, v6-oriented) - — the most accessible English-language walkthrough of the formulas - and parameter roles. -- [[https://help.remnote.com/en/articles/9124137-the-fsrs-spaced-repetition-algorithm][RemNote: The FSRS Spaced Repetition Algorithm]] — third-party - implementer's overview, useful as a sanity check on terminology. diff --git a/working/stats-dashboard/stats-dashboard.org b/working/stats-dashboard/stats-dashboard.org deleted file mode 100644 index 8b195cb..0000000 --- a/working/stats-dashboard/stats-dashboard.org +++ /dev/null @@ -1,419 +0,0 @@ -#+TITLE: Comprehensive Statistics Dashboard for org-drill — v0 design spec -#+AUTHOR: Craig Jennings -#+DATE: 2026-05-27 - -* Status - -*Ratified 2026-05-27* — all 10 open decisions accepted as recommended. -The companion todo entry is =todo.org=:20 [#A] "Comprehensive -Statistics Dashboard". No implementation has started; the spec is now -the implementation gate. - -See =Ratified decisions= at the bottom of this file for the locked -choice on each question; inline =Decided:= blocks repeat the choice -next to the section it affects. - -* Context and motivation - -org-drill's end-of-session report is =org-drill-final-report= -(=org-drill.el=:3128) — a minibuffer prompt with the pass percentage, -qualities histogram, and the new/mature/failed counts for the *single -just-completed session*. Nothing about that report persists. Once the -user presses a key to dismiss it, the only retained learning data is -what each card carries in its own =DRILL_*= properties. - -The currently-retained per-card data is real and usable: - -| Property | Source | -|----------+--------| -| =DRILL_LAST_INTERVAL= | scheduler output, updated every review | -| =DRILL_REPEATS_SINCE_FAIL= | review counter resetting on a lapse | -| =DRILL_TOTAL_REPEATS= | lifetime review count for the card | -| =DRILL_FAILURE_COUNT= | lifetime lapses | -| =DRILL_AVERAGE_QUALITY= | running mean of the 0–5 quality scale | -| =DRILL_EASE= | SM-family easiness factor | -| =DRILL_LAST_QUALITY= | quality at the most recent review | -| =DRILL_LAST_REVIEWED= | inactive timestamp of the most recent review | -| =DATE_ADDED= | inactive timestamp of card creation | - -What's missing is the *temporal axis*: there's no record that the user -reviewed N cards on a given day, or that the pass percentage trended -upward over the last month, or that a particular card has been failed -three times in the last two weeks. Those questions need a per-review -or per-session event log that survives Emacs restarts. - -This spec sketches a dashboard that combines (a) per-card aggregates -read live from existing properties (no new storage) with (b) a new -persisted session/review history (org-persist, separate file) that -gives the time-series view. - -* Goals - -- New command =org-drill-statistics= opens an interactive dashboard - buffer summarizing review history across all known drill files. -- Per-card aggregates read live from the existing =DRILL_*= properties - — no migration, no new per-card properties. -- A new persisted session log records, at the end of every session, - what landed (date, files scoped, cards reviewed, qualities histogram, - duration, pass percentage). Survives Emacs restarts. -- Dashboard renders trends, distributions, and "needs attention" lists - by reading the session log + the per-card properties. -- CSV export for users who want to bring the data into a spreadsheet - or a notebook. -- No-op for users who don't open the dashboard — collection cost on - session end is bounded (single =persist-save= write), and the - dashboard command is lazy-loaded. - -* Non-goals (v1) - -- *No graphical charting.* The dashboard renders in plain org-mode - text — tables, unicode-block sparklines, and minibuffer summaries. - A Vega/SVG/PNG output stays out of v1; users who want charts use - CSV export. -- *No per-review event log.* The persisted record is per-session, not - per-card-review. A full per-review log is more expensive to write, - more expensive to read, and crosses the line into "we are now a - learning-analytics platform." Per-card lifetime aggregates already - cover most card-level questions. -- *No multi-machine sync.* The persist file is local; users who want - cross-machine continuity sync it themselves (org-roam-style, - Syncthing, whatever). Defining a sync protocol is out of scope. -- *No retroactive backfill.* Sessions before this feature lands have - no history; the dashboard starts from the first session after the - upgrade. Per-card properties carry their own retroactive baseline. -- *No forecasting.* "How many cards will be due tomorrow" is one of - the dashboard's panels (cheap — sum over SCHEDULED dates), but - predictive forecasting of retention or workload is FSRS territory - and stays there. - -* Data model - -** Live (no new storage) - -Read at dashboard-open time, no persistence: - -- Per-card aggregates by walking =org-map-entries= over the user's - configured =org-drill-scope= or an explicit scope argument. For each - drill entry, the existing accessors (=org-drill-entry-total-repeats=, - =-failure-count=, =-average-quality=, =-last-reviewed=, =-last-interval=, - =-ease=, =-last-quality=, =-days-since-creation=) yield the per-card - view without touching disk beyond reading the file. -- Card status counts (new / young-mature / old-mature / overdue / - lapsed / due-tomorrow) by reusing =org-drill-entry-status= over the - scope. This is what the session opener already does in =org-drill= - for the initial counts. - -** Persisted (new storage) - -A single new persisted variable, the session log: - -#+begin_src elisp -(persist-defvar org-drill-session-log nil - "List of completed-session records, newest first. - -Each entry is an `org-drill-session-record' struct.") -#+end_src - -=org-drill-session-record= is a =cl-defstruct=: - -| Slot | Type | Meaning | -|------+------+---------| -| =start-time= | float | =float-time= at session start | -| =end-time= | float | =float-time= at session end | -| =scope= | symbol or list | =org-drill-scope= value at session start | -| =algorithm= | symbol | =org-drill-spaced-repetition-algorithm= at start | -| =qualities= | vector of int | every quality 0–5 entered, in order | -| =pass-percent= | int | qualities > =org-drill-failure-quality=, / total | -| =new-count= | int | size of =(oref session new-entries)= at end | -| =mature-count= | int | =young-mature= + =old-mature= entries at end | -| =failed-count= | int | =failed-entries= at end | -| =cram-mode= | bool | =cram-mode= at session start | - -A single record is small (a few hundred bytes). At one session per -day, the log holds a year of history in well under 100 KB. At three -sessions per day for ten years, still under 4 MB. No pruning needed -for v1. - -=Decided:= persistence shape = =persist-defvar=. Integrates with the -=persist= dependency already in =Package-Requires=, lives at -=~/.emacs.d/persist/org-drill-session-log= by default, mirrors the -=org-drill-sm5-optimal-factor-matrix= precedent including its -=condition-case= wrapper for corrupt-load recovery. Plain elisp file -and org-mode log file declined — the former is what =persist-defvar= -already is under the hood, the latter trades structured I/O for -parse-and-rewrite failure modes. - -=Decided:= corrupted-load recovery = log a single warning, start with -a fresh empty log, rename the corrupt file to =.corrupt-YYYY-MM-DD= -before next save so it isn't overwritten. Mirrors the SM5-matrix -path verbatim. History loss is bounded; the precedent makes this -consistent. - -** Out-of-scope storage - -Explicit non-storage to keep scope tight: - -- No per-review event log (a row per card-rating). -- No per-card review history (a list of past timestamps + qualities - per card). =DRILL_AVERAGE_QUALITY= and =DRILL_LAST_REVIEWED= cover - the dashboard's needs; full per-card history is what FSRS would - need, and FSRS owns that question. -- No per-deck (per-file) aggregate cache. Walking org files on - dashboard open is fast enough for the file counts org-drill users - actually have; if a user has 10 000 files this becomes an issue — - cross that bridge if it shows up. - -* UI shell - -The dashboard is a single command =org-drill-statistics= that opens a -read-only org-mode buffer named =*Org Drill Statistics*= with the -sections below. =Decided:= keymap = =q= bury, =g= refresh, =e= -export-csv, =s= scope, =r= range, =a= algorithm-filter, =RET= follow -the card link at point. None conflict with read-only org-mode -bindings. - -** Section 1 — Overview - -A four-column summary table: - -#+begin_example -| Total cards | New | Mature | Lapsed | -|-------------+-----+--------+--------| -| 412 | 18 | 367 | 27 | -#+end_example - -Plus a one-line "last session" recap reading the most recent record -from =org-drill-session-log= (date, duration, cards reviewed, pass %). - -** Section 2 — Trends - -Two unicode-block sparklines: - -- Reviews per day (last 90 days). X axis = day, Y axis = card count. -- Pass rate per day (last 90 days). Same X axis, Y axis = 0..100. - -Range and bar count are defcustoms. Default 90 days = roughly a -quarter, fits on one line at 1 cell/day. - -Below the sparklines, a small table of weekly aggregates for the last -12 weeks (reviews, pass %, average duration). - -=Decided:= sparkline character set = quadrant blocks (▁▂▃▄▅▆▇█), -8 levels. Emacs renders them fine by default; users on a font without -them are rare enough to handle by documentation rather than a runtime -fallback. - -** Section 3 — Distribution - -Quality histogram across all sessions in the log (or scoped — see the -range filter below). A horizontal bar per quality 0..5 with the -absolute count and the percentage. - -** Section 4 — Needs attention - -Three tables: - -- *Leech candidates* — cards with =DRILL_FAILURE_COUNT= ≥ - =org-drill-leech-failure-threshold= and =DRILL_AVERAGE_QUALITY= below - =org-drill-statistics-leech-quality-threshold= (=Decided:= default - *2.5*, below the Hard boundary of 3). Link to card. -- *Long-overdue* — cards with =DRILL_LAST_REVIEWED= ≥ - =org-drill-lapse-threshold-days= ago. Sorted most-overdue first. -- *Forgotten new* — cards with =DATE_ADDED= ≥ 14 days ago but - =DRILL_TOTAL_REPEATS= = 0 (or absent). Useful for catching cards - that never got into rotation. - -Each table caps at =org-drill-stats-attention-row-limit= rows -(defcustom, default 10) with a "+N more" footer. - -** Section 5 — Forecast - -A one-line table for the next 7 days: - -#+begin_example -| Today | +1 | +2 | +3 | +4 | +5 | +6 | -|-------+----+----+----+----+----+----+ -| 34 | 18 | 22 | 9 | 41 | 12 | 6 | -#+end_example - -Computed from SCHEDULED timestamps across the scope. No prediction — -just count of cards already scheduled for each day. - -** Range filter - -A line at the top of the buffer carrying the active filters: - -#+begin_example -Scope: file (~/notes/drill.org) Range: last 90d Algorithm: simple8 -#+end_example - -Filters are interactive at the buffer header (=s= cycles scope, =r= -cycles range, =a= filters algorithm). Defaults: =org-drill-scope=, "last -90d", "all algorithms". - -=Decided:= single buffer-wide filter for v1. Per-section filters -multiply the UI surface and most users want the same window across -sections. - -* Export - -=M-x org-drill-statistics-export-csv= writes one CSV per requested -view to a user-chosen directory: - -- =sessions.csv= — one row per session record in the log. Columns - match the struct slots. -- =cards.csv= — one row per drill entry in the active scope, with - every =DRILL_*= property plus the computed status. -- =daily.csv= — one row per day in the active range, with reviews, - passes, fails, pass-percent, duration-minutes. - -=Decided:= column delimiter = =,= (CSV) with proper quoting via -=csv-mode='s writer if available, else a hand-rolled =csv-quote= -helper. Users who want TSV can run a one-line sed pipe. - -* Performance - -The expensive paths and their bounds: - -| Path | Cost | Mitigation | -|------+------+------------| -| Session log save | one =persist-save= per session | already wrapped in =condition-case=; cost is a few KB write | -| Scope walk on dashboard open | =org-map-entries= over scope | same cost as a session open; cached for the dashboard's lifetime, refreshes on =g= | -| CSV export | one walk + one file write | one-off; user-triggered | -| Sparkline rendering | bucket the in-memory log by day | log is bounded; bucket-by-day is linear in log length | - -The dashboard does *not* run at session open or close — only when the -user invokes =M-x org-drill-statistics=. Session-end pays one =persist-save= -write. Idle Emacs pays nothing. - -=Decided:= sync dashboard open for v1. =org-map-entries= over a -typical scope is well under a second. Async refresh -(=run-with-idle-timer=, status line in the buffer) is a follow-on -ticket if anyone reports >2 s on a large scope. - -* Integration points - -** New code - -- =org-drill-session-record= struct (=cl-defstruct=). -- =org-drill-session-log= persistent variable. -- =org-drill-record-session= — called once from the end-of-session - finalizer (=org-drill-finalize-session= or wherever - =org-drill-final-report= currently sits) when the session was not - aborted. Appends a record, =persist-save='s the log. -- =org-drill-statistics= — interactive command, opens the dashboard. -- =org-drill-statistics-mode= — minor-mode-like keymap on top of an - org-mode buffer (=q g e s r a=). -- =org-drill-statistics--render-*= — one helper per section (overview, - trends, distribution, attention, forecast). -- =org-drill-statistics-export-csv= — interactive command. - -** Existing code touched - -- =org-drill-final-report= grows a single call to =org-drill-record-session= - before the read-char-exclusive dismisses the report (so the record - lands even if the user dismisses immediately). -- A defcustom group =org-drill-statistics= added as a sibling group - next to =org-drill-session= (=Decided:= sibling, not nested — the - dashboard isn't session-state and a separate group keeps Customize's - tree readable). - -** Aborted-session handling - -A session ended via =C-g= or quit doesn't reach =org-drill-final-report=. -=Decided:= record nothing for these — an =unwind-protect= path to -salvage partial sessions is deferred. The "abort discards" semantics -matches =org-drill-on-timeout-action= =discard-current= already, and a -partial record's pass percentage misrepresents what the user -experienced. - -* Defcustoms (proposed list) - -#+begin_src elisp -(defcustom org-drill-statistics-trend-days 90 - "Number of days the trends section spans.") - -(defcustom org-drill-statistics-forecast-days 7 - "Number of days ahead the forecast section spans.") - -(defcustom org-drill-statistics-attention-row-limit 10 - "Maximum rows in each `Needs attention' table.") - -(defcustom org-drill-statistics-leech-quality-threshold 2.5 - "Cards with `DRILL_AVERAGE_QUALITY' below this and at least -`org-drill-leech-failure-threshold' failures appear in `Leech candidates'.") - -(defcustom org-drill-statistics-export-directory - (expand-file-name "org-drill-stats/" user-emacs-directory) - "Default directory for CSV exports.") -#+end_src - -* Test strategy - -Three test files, matching the existing file-per-area convention: - -1. =tests/test-org-drill-session-record.el= — struct construction, - round-trip through =persist-save=/=persist-load=, log appending, - newest-first ordering, corrupted-load recovery. -2. =tests/test-org-drill-statistics-aggregates.el= — given a fixture - session log + a fixture org file with known =DRILL_*= properties, - each of the dashboard's section-render helpers produces the - expected table. Aggregation math (pass %, weekly buckets, - sparkline buckets) is tested here with deterministic input. -3. =tests/test-org-drill-statistics-integration.el= — end-to-end: - simulate three completed sessions via =org-drill-record-session= - against a fixture file, open the dashboard, assert the buffer - contents and the export-CSV output. - -Normal / Boundary / Error per public function on the helpers. -Boundary cases worth pinning: empty log (no sessions yet), single -session, range filter that selects zero days, scope that contains -zero drill entries, the day-bucket histogram on the day-boundary -edge (a session that crosses midnight). - -* Effort estimate - -Multi-day, plausibly spanning sessions: - -- Session record + persist round-trip + recording hook: 0.5 day. -- Dashboard renderer (5 sections) + minor-mode keymap + range filter: - 1.5 days. Each section is a small helper; the time goes to layout - polish and the sparkline math. -- CSV export: 0.5 day. -- Test coverage at parity (the three files above, ~40 tests): 1 day. -- Documentation (manual entry, README option list, defcustom - docstrings, this spec ratified): 0.5 day. - -Realistic: a session for the persist + recording layer with full -tests, a session for the dashboard renderer, a follow-up session for -the export + docs + polish. - -* Ratified decisions (2026-05-27) - -All 10 open questions resolved as recommended. Implementation can -proceed against this spec. - -| # | Question | Resolution | -|---+----------+------------| -| 1 | Persistence shape | =persist-defvar=, mirroring the SM5 matrix | -| 2 | Corrupted-load recovery | warn, fresh-start, rename to =.corrupt-YYYY-MM-DD= | -| 3 | Sparkline character set | quadrant blocks (▁▂▃▄▅▆▇█) | -| 4 | Filter scope | single buffer-wide filter | -| 5 | CSV delimiter | =,= with proper quoting | -| 6 | Dashboard-open mode | sync | -| 7 | Aborted-session recording | record nothing; =unwind-protect= deferred | -| 8 | Dashboard keymap | =q g e s r a RET= | -| 9 | Leech-quality threshold default | 2.5 | -| 10 | Defcustom group placement | sibling group =org-drill-statistics= | - -* References - -- =org-drill.el= around line 729 — the =org-drill-session= EIEIO class. - Source of the per-session in-memory state that becomes a record. -- =org-drill.el= around line 3128 — =org-drill-final-report=. Hook - site for =org-drill-record-session=. -- =org-drill.el= around line 540 — the existing =persist-defvar= use - for the SM5 matrix. Template for the session-log persist + the - =condition-case= wrapper for corrupt-load recovery. -- =working/fsrs-spec/fsrs-spec.org= — sister v0 spec, same DECIDE-marker - convention. |