aboutsummaryrefslogtreecommitdiff
path: root/.ai/workflows
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2026-05-30 15:46:00 -0500
committerCraig Jennings <c@cjennings.net>2026-05-30 15:46:00 -0500
commitb80a9ceb3fc9cdca9798b48fbc4f9ab9c1592b57 (patch)
treebe0f328c6be0ddb8d999730cc6fa07277ebc2ec7 /.ai/workflows
parent968a39bb3978e6ad499447ce173e2265dee772a2 (diff)
downloadrulesets-b80a9ceb3fc9cdca9798b48fbc4f9ab9c1592b57.tar.gz
rulesets-b80a9ceb3fc9cdca9798b48fbc4f9ab9c1592b57.zip
fix(drill-deck): cut leakage false positives and codify source/date conventions
Health ran the new leakage check on a 43-card deck and hit two false-positive classes. The check read the whole card body, so a =Source: <label> — <url>= citation line inflated the front/back overlap whenever the URL slug repeated the question's words. Range/category cards ("What are the HbA1c ranges across normal, prediabetes, and diabetes?") tripped it too, because the question's categories echo in the answer even though the recalled content is the numbers. drill-deck-stats.py now routes leakage through an is_leaky helper. It strips =Source:= and created-date lines before computing overlap, and exempts a card when the answer carries a numeric range or threshold the question lacks. leakage_ratio itself is unchanged, so the genuine-restatement case still flags. Two body conventions now hold: a =Source:= citation goes at the end of a card after two blank lines, and no created/added date goes on a card. drill-to-anki.py now strips =Created:= / =:CREATED:= lines from the back as a backstop, and the workflow's Phase C removes them from the source during the rewrite. I added tests for the source-strip, the numeric carve-out, and the created-line strip, and documented all of it in drill-deck-review.org.
Diffstat (limited to '.ai/workflows')
-rw-r--r--.ai/workflows/drill-deck-review.org13
1 files changed, 10 insertions, 3 deletions
diff --git a/.ai/workflows/drill-deck-review.org b/.ai/workflows/drill-deck-review.org
index fe12f3c..390f296 100644
--- a/.ai/workflows/drill-deck-review.org
+++ b/.ai/workflows/drill-deck-review.org
@@ -103,6 +103,8 @@ The =drill-deck-stats.py= helper recognizes both =?=-form and imperative-verb fo
- *Body opens by naming the topic.* "Air Force Research Laboratory. Air Force's R&D arm." or "Vrezh Mikayelyan. Armenia-based, full-time as of ..." The Anki back shows this directly under the front question; restating the topic makes the back read as a complete answer.
- *PROPERTIES drawer stays.* Org-drill needs the =:ID:=, =:DRILL_LAST_INTERVAL:=, =:DRILL_EASE:= etc. for SRS state. The Anki output strips it (see the script change).
- *=SCHEDULED:= / =DEADLINE:= planning lines stay.* Same reason. The Anki output strips them.
+- *Source citation goes at the very end, after two blank lines.* When a card cites a source, put a =Source: <label> — <url>= line at the end of the body, separated from the answer by two blank lines (two empty paragraphs) so it reads as a footer, not part of the answer. =drill-deck-stats.py= ignores =Source:= lines when checking for answer leakage, since a URL slug often repeats the question's words.
+- *No created/added date on the card.* Don't stamp a card with the date it was written. If a card body carries a =Created:= line (or a =:CREATED:= line outside the drawer), remove it during the rewrite. The Anki output strips =Created:= lines as a backstop, but they shouldn't be in the source either. Volatile facts get dated in the answer prose itself ("full-time as of April 2026"), never via a card-level timestamp.
* Card Authoring Principles
@@ -110,7 +112,7 @@ The canonical shapes above are the house style; these are the reasons behind the
- *One fact per card (minimum information principle).* A card should test a single retrievable connection. A back that bundles several independent facts gets partially recalled and burns repetitions on the parts you already know. When a body covers unrelated attributes, split it into separate cards. =drill-deck-stats.py= flags long backs as a non-blocking NOTE.
-- *Demand recall, not recognition (effortful retrieval).* Pulling the answer from memory is what strengthens it, so the question must not let you infer the answer from its own wording. This is why person headings never name the person, and why a question that restates its own answer is a defect. =drill-deck-stats.py= flags high front/back word overlap as answer leakage.
+- *Demand recall, not recognition (effortful retrieval).* Pulling the answer from memory is what strengthens it, so the question must not let you infer the answer from its own wording. This is why person headings never name the person, and why a question that restates its own answer is a defect. =drill-deck-stats.py= flags high front/back word overlap as answer leakage — excluding =Source:= citation lines, and exempting range/category cards whose answer recalls numbers the question doesn't give away.
- *Avoid binary prompts.* "Is X true?" and "A or B?" allow a coin-flip guess and produce shallow understanding. Reformulate open-ended — "How does X affect Y?" beats "Does X affect Y?" Flagged as a non-blocking NOTE.
@@ -188,6 +190,8 @@ Rewrite shape per card:
Drop the =*** Answer= sub-header entirely. The body that was under =*** Answer= becomes the body of the card. If the original body had a question above =*** Answer= (the pre-rewrite norm), drop that question — the new heading carries it.
+Two body conventions to apply during the rewrite: remove any =Created:= / created-date line (no card-level timestamps), and if the card cites a source, put the =Source:= line at the end of the body after two blank lines.
+
For the file as a whole, use a single =Write= rather than per-card =Edit= calls. One pass through the source, one write back. Per-card edits multiply tool calls by N and risk drift.
** Phase D: Regenerate the Anki deck
@@ -244,7 +248,7 @@ The core converter. Reads an org-drill source file, emits a stable-ID Anki =.apk
** =drill-deck-stats.py=
-Inventory + authoring-quality checks for a single deck source. Counts cards, PROPERTIES drawers, =*** Answer= sub-headers, cards missing =:ID:=, and cards whose heading is neither =?=-form nor an imperative-verb prompt. It also checks authoring quality: answer leakage (front/back content-word overlap) and duplicate / near-duplicate fronts are blocking WARNs; overloaded backs, list-shaped backs, and binary prompts are non-blocking NOTEs. Exits 0 when no blocking warning is present, 1 otherwise, so it gates =drill-deck-sync=.
+Inventory + authoring-quality checks for a single deck source. Counts cards, PROPERTIES drawers, =*** Answer= sub-headers, cards missing =:ID:=, and cards whose heading is neither =?=-form nor an imperative-verb prompt. It also checks authoring quality: answer leakage (front/back content-word overlap) and duplicate / near-duplicate fronts are blocking WARNs; overloaded backs, list-shaped backs, and binary prompts are non-blocking NOTEs. Exits 0 when no blocking warning is present, 1 otherwise, so it gates =drill-deck-sync=. The leakage check ignores =Source:= and created-date lines and exempts range/category cards whose answer recalls numbers the question doesn't give away.
Imperative-verb allowlist: Spell, Describe, Explain, Name, List, Give, Show, Tell, Define, Compare, Identify, Outline, Introduce, Walk, State, Recite, Recall, Summarize.
@@ -269,7 +273,7 @@ drill-deck-sync <source.org> --diff-against <previous-version.org>
The =drill-to-anki.py= script has these contracts that this workflow depends on:
1. *Strips =:PROPERTIES:= drawers* from the card body before rendering. Org-drill needs them in source; Anki cards shouldn't show them.
-2. *Strips =SCHEDULED:= / =DEADLINE:= / =CLOSED:= planning lines* from the card body. Same reason.
+2. *Strips =SCHEDULED:= / =DEADLINE:= / =CLOSED:= planning lines and =Created:= / =:CREATED:= date lines* from the card body. Same reason — and a created date never belongs on a card.
3. *Does NOT strip =*** Answer= sub-headers.* If the source still has them, the Anki cards will show them. This workflow's Phase C removes them at the source. =drill-deck-stats.py= flags any remaining as a workflow violation.
4. *Front of each Anki card* = the heading text without the =:drill:= tag.
5. *Back of each Anki card* = the cleaned body (after #1 and #2), joined with =<br>= and HTML-escaped.
@@ -318,3 +322,6 @@ Craig noticed the Anki deck name still showed as "DeepSat Org-Drill Flashcards"
*** 2026-05-30: Authoring-quality checks + Card Authoring section (same day)
Researched flashcard / spaced-repetition best practices (Wozniak's twenty rules, Matuschak's prompt-writing guide, Nielsen, the Anki manual, the FSRS docs) and folded the findings in. =drill-deck-stats.py= gained answer-leakage and duplicate-front checks (blocking), plus non-blocking NOTEs for overloaded backs, list-shaped backs, and binary prompts. Added a "Card Authoring Principles" section (the why behind the canonical shapes), a person-card splitting path, a Phase B cost-benefit-removal + leech-feedback disposition, and a scheduling-is-Anki-side note in the Overview. Deliberately not adopted, with reasons: cloze cards (would need a second note type and an authoring convention), per-card tractability targeting and FSRS-retention encoding (Anki-side telemetry that never flows back to the source), on-face source-stamping (the converter strips those drawers by design; provenance stays in the org layer).
+
+*** 2026-05-30: Leakage false-positive fixes + source/created-date conventions (same day)
+Health ran the leakage check on a 43-card deck and hit two false-positive classes. Fixed both in =drill-deck-stats.py=: =Source:= citation lines are stripped before the overlap is computed (a URL slug repeats the question's words), and range/category cards whose answer carries numeric ranges or thresholds the question lacks are exempted (the recalled content is the numbers, which aren't given away). Codified two body conventions: a =Source:= citation sits at the end of the card after two blank lines, and no created/added date goes on a card. =drill-to-anki.py= now strips =Created:= / =:CREATED:= lines from the back as a backstop, and Phase C removes them from the source during the rewrite.