Many changes:2.0

- Item drill data is stored in a clearer format, each value has its own property, the old 'org-learn'
  LEARN_DATA property has been dropped. Items that contain LEARN_DATA are silently rolled over to
  the new format.
- For SM5, matrix of optimal factors is now correctly *global* rather than per-item, and is saved
  between drill sessions as an emacs customization variable.
- We store interval data as floating point numbers rather than rounding to the nearest integer.
- New experimental algorithm: 'simple8', based on SuperMemo SM8.
- New options:
  - org-drill-forgetting-index: define the percentage of items that it is "acceptable" to fail.
    A warning message is shown after the drill session if more items than this were failed.
  - org-drill-learn-fraction: governs how fast time intervals increase with each successful
    repetition of an item.
  - org-drill-adjust-intervals-for-early-and-late-repetitions-p: if true, alter next scheduled date
    if an item is reviewed earlier or later than it was supposed to be.
- Almost all variables have been declared safe for file-local reassignment.
- We now ignore org-learn-always-reschedule, regarding it as always true.
- Post-session report now shows a count of dormant items (those not yet due for review).

1 files changed, 234 insertions, 106 deletions

diff --git a/README.org b/README.org
index 1fbc98a..77688c2 100755
--- a/README.org
+++ b/README.org
@@ -22,72 +22,21 @@ student in different ways.
 
 For more on the spaced repetition algorithm, and examples of other programs
 that use it, see:
-- [[http://supermemo.com/index.htm][SuperMemo]] (the SM5 algorithm is discussed [[http://www.supermemo.com/english/ol/sm5.htm][here]])
+- [[http://supermemo.com/index.htm][SuperMemo]] (see descriptions of the SM2, SM5 and SM8 algorithms)
 - [[http://ichi2.net/anki/][Anki]]
 - [[http://mnemosyne-proj.org/index.php][Mnemosyne]]
 
 
-* Installation and Customisation
+* Installation
 
 
 Put the following in your =.emacs=. You will also need to make sure that Org's
 "contrib/lisp" directory is in the emacs load-path.
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC elisp
 (require 'org-drill)
-#+END_EXAMPLE
+#+END_SRC
 
-I also recommend the following, so that items are always eventually retested,
-even when you remember them well.
-
-#+BEGIN_EXAMPLE
-(setq org-learn-always-reschedule t)
-#+END_EXAMPLE
-
-If you want cloze-deleted text to show up in a special font within Org mode
-buffers, also add:
-
-#+BEGIN_EXAMPLE
-(setq org-drill-use-visible-cloze-face-p t)
-#+END_EXAMPLE
-
-Item headings may contain information that "gives away" the answer to the item,
-either in the heading text or in tags. If you want item headings to be made
-invisible while each item is being tested, add:
-
-#+BEGIN_EXAMPLE
-(setq org-drill-hide-item-headings-p t)
-#+END_EXAMPLE
-
-Org-Drill supports two different spaced repetition algorithms -- SM5 (the
-default, implemented by =org-learn=) and SM2. SM2 is an earlier algorithm which
-remains very popular -- Anki and Mnemosyne, two of the most popular spaced
-repetition programs, use SM2.
-
-If you want Org-Drill to use the SM2 algorithm, put the following in your
-=.emacs=:
-
-#+BEGIN_EXAMPLE
-(setq org-drill-spaced-repetition-algorithm 'sm2)
-#+END_EXAMPLE
-
-The intervals generated by the SM2 and SM5 algorithms are pretty
-deterministic. If you tend to add items in large, infrequent batches, the lack
-of variation in interval scheduling can lead to the problem of "lumpiness" --
-one day a large batch of items are due for review, the next there is almost
-nothing, a few days later another big pile of items is due.
-
-This problem can be ameliorated by adding some random "noise" to the interval
-scheduling algorithm. The author of SuperMemo actually recommends this approach
-for the SM5 algorithm, and Org-Drill's implementation uses [[http://www.supermemo.com/english/ol/sm5.htm][his code]].
-
-To enable random "noise" for item intervals, set the variable
-=org-drill-add-random-noise-to-intervals-p= to true by putting the following in
-your =.emacs=:
-
-#+BEGIN_EXAMPLE
-(setq org-drill-add-random-noise-to-intervals-p t)
-#+END_EXAMPLE
 
 
 * Demonstration
@@ -104,8 +53,9 @@ drill topics are written.
 
 
 Org-Drill uses org mode topics as 'drill items'. To be used as a drill item,
-the topic must have a tag that matches =org-drill-question-tag=. This is
-=:drill:= by default. Any other org topics will be ignored.
+the topic must have a tag that matches the value of
+=org-drill-question-tag=. This is =:drill:= by default. Any other org topics
+will be ignored.
 
 You don't need to schedule the topics initially.  However =org-drill= *will*
 recognise items that have been scheduled previously with
@@ -113,28 +63,41 @@ recognise items that have been scheduled previously with
 memorisation.
 
 How should 'drill topics' be structured? Any org topic is a legal drill topic
--- it will simply be shown with subheadings collapsed. After pressing a
-key, any hidden subheadings will be revealed, and you will be asked to rate
-your "recall" of the item.
+-- it will simply be shown with all subheadings collapsed, so thta only the
+material beneath the main item heading is visible. After pressing a key, any
+hidden subheadings will be revealed, and you will be asked to rate your
+"recall" of the item.
 
 This will be adequate for some items, but usually you will want to write items
 where you have more control over what information is hidden from the user for
-recall purposes.
+recall purposes. For this reason, some other card types are defined, including:
+- [[Two-sided cards]]
+- [[Multi-sided cards]]
+- [[Multicloze cards]]
+- [[User-defined card types]]
+
+*A note about comments:* In org mode, comment lines start with '#'. The rest of
+the line is ignored by Org (apart from some special cases). You may sometimes
+want to put material in comments which you do not want to see when you are
+being tested on the item. For this reason, comments are always rendered
+invisible while items are being tested.
+
 
 ** Simple topics
 
+
 The simplest drill topic has no special structure. When such a topic is
 presented during a drill session, any subheadings are "collapsed" with their
 contents hidden. So, you could include the question as text beneath the main
 heading, and the answer within a subheading. For example:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 * Item                                   :drill:
 What is the capital city of Estonia?
 
 ** The Answer
 Tallinn.
-#+END_EXAMPLE
+#+END_SRC
 
 When this item is presented for review, the text beneath the main heading will
 be visible, but the contents of the subheading ("The Answer") will be hidden.
@@ -142,6 +105,7 @@ be visible, but the contents of the subheading ("The Answer") will be hidden.
 
 ** Cloze deletion
 
+
 Cloze deletion can be used in any drill topic regardless of whether it is
 otherwise 'simple', or is one of the specialised topic types discussed
 below. To use cloze deletion, one or more parts of the body of the topic is
@@ -150,10 +114,11 @@ so]. When the topic is presented for review, the text within square brackets
 will be obscured. The text is then revealed after the user presses a key. For
 example:
 
-#+BEGIN_EXAMPLE
+
+#+BEGIN_SRC org
 * Item                                   :drill:
 The capital city of Estonia is [Tallinn].
-#+END_EXAMPLE
+#+END_SRC
 
 During review, the user will see:
 
@@ -171,10 +136,10 @@ the rest of the text is hidden.
 
 Example:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 Type 1 hypersensitivity reactions are mediated by [immunoglobulin E|molecule]
 and [mast cells|cell type].
-#+END_EXAMPLE
+#+END_SRC
 
 #+BEGIN_QUOTE
 Type 1 hypersensitivity reactions are mediated by
@@ -186,6 +151,7 @@ and @<font style="background-color: blue;" color="cyan">
 
 
 ** Two-sided cards
+# <<Two-sided cards>>
 
 The remaining topic types all use the topic property, =DRILL_CARD_TYPE=. This
 property tells =org-drill= which function to use to present the topic during
@@ -202,7 +168,7 @@ A two sided card can have more than 2 subheadings, but all subheadings after
 the first two are considered as "notes" and will always be hidden during topic
 review.
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 * Noun                                               :drill:
     :PROPERTIES:
     :DRILL_CARD_TYPE: twosided
@@ -219,7 +185,7 @@ the woman
 ** Example sentence
 ¿Quién fue esa mujer?
 Who was that woman?
-#+END_EXAMPLE
+#+END_SRC
 
 In this example, the user will be shown the main text -- "Translate this word"
 -- and either 'la mujer', /or/ 'the woman', at random. The section 'Example
@@ -228,12 +194,14 @@ not one of the first two 'sides' of the topic.
 
 
 ** Multi-sided cards
+# <<Multi-sided cards>>
+
 
 The =multisided= card type is similar to =twosided=, except that any
 subheading has a chance of being presented during the topic review. One
 subheading is always shown and all others are always hidden.
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 * Noun                                               :drill:
     :PROPERTIES:
     :DRILL_CARD_TYPE: multisided
@@ -249,28 +217,31 @@ the table
 
 ** Picture
 [[file:table.jpg][PICTURE]]
-#+END_EXAMPLE
+#+END_SRC
 
 The user will be shown the main text and either 'la mujer', /or/ 'the woman',
 /or/ a picture of a table.
 
+
 ** Multi-cloze cards
+# <<Multi-cloze cards>>
+
 
 Often, you will wish to create cards out of sentences that express several
 facts, such as the following:
 
-#+BEGIN_QUOTE
+#+BEGIN_SRC org
 The capital city of New Zealand is Wellington, which is located in the
 South Island and has a population of about 400,000.
-#+END_QUOTE
+#+END_SRC
 
 There is more than one fact in this statement -- you could create a single
 'simple' card with all the facts marked as cloze text, like so:
 
-#+BEGIN_QUOTE
+#+BEGIN_SRC org
 The capital city of [New Zealand] is [Wellington], which is located in
-[the South Island] and has a population of about [400,000].
-#+END_QUOTE
+the [North|North/South] Island and has a population of about [400,000].
+#+END_SRC
 
 But this card will be difficult to remember. If you get just one of the 4
 hidden facts wrong, you will fail the card. A card like this is likely to
@@ -280,7 +251,7 @@ A better way to express all these facts using 'simple' cards is to create
 several cards, with one fact per card. You might end up with something
 like this:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 * Fact
 The capital city of [New Zealand] is Wellington, which has a population of
 about 400,000.
@@ -295,16 +266,16 @@ about [400,000].
 
 * Fact
 The capital city of [New Zealand] is Wellington, which is located in the
-the South Island.
+the North Island.
 
 * Fact
 The capital city of New Zealand is [Wellington], which is located in
-the South Island.
+the North Island.
 
 * Fact
 The capital city of New Zealand is Wellington, which is located in
-[the South Island].
-#+END_EXAMPLE
+the [North|North/South] Island.
+#+END_SRC
 
 However, this is really cumbersome. The 'multicloze' card type exists for this
 situation. Multicloze cards behave like 'simple' cards, except that when there
@@ -317,19 +288,22 @@ but change its card type to 'multicloze'. Each time the card is presented for
 review, one of 'New Zealand', 'Wellington', 'the South Island' or '400,000'
 will be hidden.
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 * Fact
   :PROPERTIES:
   :DRILL_CARD_TYPE: multicloze
   :END:
 
 The capital city of [New Zealand] is [Wellington], which is located in
-[the South Island] and has a population of about [400,000].
-#+END_EXAMPLE
+the [North|North/South] Island and has a population of about [400,000].
+#+END_SRC
+
 
-** User-defined topic types
+** User-defined card types
+# <<User-defined card types>>
 
-Finally, you can write your own elisp functions to define new kinds of
+
+Finally, you can write your own emacs lisp functions to define new kinds of
 topics. Any new topic type will need to be added to
 =org-drill-card-type-alist=, and cards using that topic type will need to have
 it as the value of their =DRILL_CARD_TYPE= property. For an example, see the
@@ -338,13 +312,6 @@ function =org-drill-present-spanish-verb=, which defines the new topic type
 
 See the file [[file:spanish.org][spanish.org]] for a full set of example material.
 
-** Comments
-
-In org mode, comment lines start with '#'. The rest of the line is ignored by
-Org (apart from some special cases). You may sometimes want to put material in
-comments which you do not want to see when you are being tested on the
-item. For this reason, comments are always rendered invisible while items are
-being tested.
 
 
 * Running the drill session
@@ -384,6 +351,19 @@ answer, or 'e' to finish the drill and jump to the current topic for editing
 (your progress up to that point will be saved).
 
 
+* Cram mode
+
+
+There are some situations, such as before an exam, where you will want to
+revise all of your cards regardless of when they are next due for review.
+
+To do this, run a /cram session/ with the =org-drill-cram= command (=M-x
+org-drill-cram RET=). This works the same as a normal drill session, except
+that all items are considered due for review unless you reviewed them within
+the last 12 hours (you can change the number of hours by customising the
+variable =org-drill-cram-hours=).
+
+
 * Leeches
 # <<leeches>>
 
@@ -416,17 +396,164 @@ card.
 See [[http://www.supermemo.com/help/leech.htm][the SuperMemo website]] for more on leeches.
 
 
-* Cram mode
+* Customisation
 
 
-There are some situations, such as before an exam, where you will want to
-revise all of your cards regardless of when they are next due for review.
+Org-Drill has several settings which you change using
+=M-x customize-group org-drill <RET>=. Alternatively you can change these
+settings by adding elisp code to your configuration file (=.emacs=).
 
-To do this, run a /cram session/ with the =org-drill-cram= command (=M-x
-org-drill-cram RET=). This works the same as a normal drill session, except
-that all items are considered due for review unless you reviewed them within
-the last 12 hours (you can change the number of hours by customising the
-variable =org-drill-cram-hours=).
+
+** Appearance of items during drill sessions
+
+
+If you want cloze-deleted text to show up in a special font within Org mode
+buffers, add this to your .emacs:
+
+#+BEGIN_SRC elisp
+(setq org-drill-use-visible-cloze-face-p t)
+#+END_SRC
+
+Item headings may contain information that "gives away" the answer to the item,
+either in the heading text or in tags. If you want item headings to be made
+invisible while each item is being tested, add:
+
+#+BEGIN_SRC elisp
+(setq org-drill-hide-item-headings-p t)
+#+END_SRC
+
+
+** Duration of drill sessions
+
+
+By default, a drill session will end when either 30 items have been
+successfully reviewed, or 20 minutes have passed. To change this behaviour, use
+the following settings.
+
+#+BEGIN_SRC elisp
+(setq org-drill-maximum-items-per-session 40)
+(setq org-drill-maximum-duration 30)   ; 30 minutes
+#+END_SRC
+
+If either of these variables is set to nil, then item count or elapsed time
+will not count as reasons to end the session. If both variables are nil, the
+session will not end until /all/ outstanding items have been reviewed.
+
+
+** Spaced repetition algorithm
+
+
+*** Choice of algorithm
+
+
+Org-Drill supports three different spaced repetition algorithms, all based on
+SuperMemo algorithms. These are:
+- [[http://www.supermemo.com/english/ol/sm2.htm][SM2]] :: an early algorithm, used in SuperMemo 2.0 (1988), which remains very
+  popular -- Anki and Mnemosyne, two of the most popular spaced repetition
+  programs, use SM2. This algorithm stores an 'ease factor' for each item,
+  which is modified each time you rate your recall of the item.
+- [[http://www.supermemo.com/english/ol/sm5.htm][SM5]] (default) :: used in SuperMemo 5.0 (1989). This algorithm uses 'ease
+     factors' but also uses a persistent, per-user 'matrix of optimal factors'
+     which is also modified after each item repetition.
+- Simple8 :: an experimental algorithm based on the [[http://www.supermemo.com/english/algsm8.htm][SM8]] algorithm. SM8 is used
+             in SuperMemo 8.0 (1998) and is almost identical to SM11 which is
+             used in SuperMemo 2002. Like SM5, it uses a matrix of optimal
+             factors. Simple8 differs from SM8 in that it does not adapt the
+             matrix to the individual user, though it does adapt each item's
+             'ease factor'.
+
+
+If you want Org-Drill to use the =SM2= algorithm, put the following in your
+=.emacs=:
+
+#+BEGIN_SRC elisp
+(setq org-drill-spaced-repetition-algorithm 'sm2)
+#+END_SRC
+
+
+*** Random variation of repetition intervals
+
+
+The intervals generated by the SM2 and SM5 algorithms are pretty
+deterministic. If you tend to add items in large, infrequent batches, the lack
+of variation in interval scheduling can lead to the problem of "lumpiness" --
+one day a large batch of items are due for review, the next there is almost
+nothing, a few days later another big pile of items is due.
+
+This problem can be ameliorated by adding some random "noise" to the interval
+scheduling algorithm. The author of SuperMemo actually recommends this approach
+for the SM5 algorithm, and Org-Drill's implementation uses [[http://www.supermemo.com/english/ol/sm5.htm][his code]].
+
+To enable random "noise" for item intervals, set the variable
+=org-drill-add-random-noise-to-intervals-p= to true by putting the following in
+your =.emacs=:
+
+#+BEGIN_SRC elisp
+(setq org-drill-add-random-noise-to-intervals-p t)
+#+END_SRC
+
+
+*** Adjustment for early or late review of items
+
+
+Reviewing items earlier or later than their scheduled review date may affect
+how soon the next review date should be scheduled. Code to make this adjustment
+is also presented on the SuperMemo website. It can be enabled with:
+
+#+BEGIN_SRC elisp
+(setq org-drill-adjust-intervals-for-early-and-late-repetitions-p t)
+#+END_SRC
+
+This will affect both early and late repetitions if the Simple8 algorithm is
+used. For the SM5 algorithm it will affect early repetitions only. It has no
+effect on the SM2 algorithm.
+
+
+*** Adjusting item difficulty globally
+
+
+The =learn fraction= is a global value which affects how quickly the intervals
+(times between each retest of an item) increase with successive repetitions,
+for /all/ items. The default value is 0.5, and this is the value used in
+SuperMemo. For some collections of information, you may find that you are
+reviewing items too often (they are too easy and the workload is too high), or
+too seldom (you are failing them too often). In these situations, it is
+possible to alter the learn fraction from its default in order to increase or
+decrease the frequency of repetition of items over time. Increasing the value
+will make the time intervals grow faster, and lowering it will make them grow
+more slowly. The table below shows the growth in intervals (in days) with some
+different values of the learn fraction (F). The table assumes that the item is
+successfully recalled each time, with an average quality of just under 4.
+
+
+| Repetition | F=0.3 | F=0.4 | *F=0.5* | F=0.6 | F=0.7 |
+|------------+-------+-------+---------+-------+-------|
+| 1st        |     2 |     2 |       2 |     2 |     2 |
+| 2nd        |     7 |     7 |       7 |     7 |     7 |
+| 5th        |    26 |    34 |      46 |    63 |    85 |
+| 10th       |    85 |   152 |     316 |   743 |  1942 |
+| 15th       |   233 |   501 |    1426 |  5471 | 27868 |
+
+To alter the learn fraction, put the following in your .emacs:
+
+#+BEGIN_SRC elisp
+(setq org-drill-learn-fraction 0.45)   ; change the value as desired
+#+END_SRC
+
+
+** Per-file customisation settings
+
+
+Most of the above settings are safe as file-local variables. This means you can
+include a commented section like this at the end of your .org file to apply
+special settings when running a Drill session using that file:
+
+#+BEGIN_SRC org
+# Local Variables:
+# org-drill-maximum-items-per-session:    50
+# org-drill-spaced-repetition-algorithm:  simple8
+# End:
+#+END_SRC org
 
 
 * Incremental reading
@@ -454,7 +581,7 @@ An example of using Org-Drill for incremental reading is given below. First,
 and most importantly, we need to define a couple of =org-capture= templates for
 captured facts.
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC elisp
 (setq org-capture-templates
        `(("u"
          "Task: Read this URL"
@@ -475,7 +602,7 @@ captured facts.
          :immediate-finish t)
         ;; ...other capture templates...
     ))
-#+END_EXAMPLE
+#+END_SRC
 
 Using these templates and =org-protocol=, you can set up buttons in your web
 browser to:
@@ -527,12 +654,12 @@ your "fact" file in Emacs. You should see that each piece of text you selected
 has been turned into a drill item. Continuing the above example, you would see
 something like:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC org
 ** Fact: 'Tuberculosis - Wikipedia, the Free Encyclopedia'        :drill:
 
 Prevention relies on screening programs and vaccination, usually with Bacillus
 Calmette-Guérin vaccine.
-#+END_EXAMPLE
+#+END_SRC
 
 You need to edit this fact so it makes sense independent of its context, as
 that is how it will be presented to you in future. The easiest way to turn the
@@ -548,7 +675,8 @@ templates, each of which might send its fact to a different file or subheading,
 or give it different tags or properties, for example.
 
 
-* Still to do
+* Still to do                                                         :noexport:
+
 
 - =org-drill-question-tag= should use a tag match string, rather than a
   single tag? Can use =org-make-tag-matcher=.