summaryrefslogtreecommitdiff
path: root/modules/lorem-optimum.el
AgeCommit message (Collapse)Author
45 hoursperf(lorem-optimum): fix O(n²) tokenization algorithmCraig Jennings
The tokenizer was creating substring copies on every iteration: - (substring text pos (1+ pos)) for whitespace check - (substring text pos) for regex matching - copies ALL remaining text This caused 10K word tokenization to take 727ms instead of 6ms. Fix: Use string-match with start position parameter and check characters directly with aref instead of creating substrings. Performance improvement: - Tokenize 10K words: 727ms → 6ms (120x faster) - Learn 10K words: 873ms → 15ms (59x faster) - Learn 100K words: 70s → 208ms (341x faster)
2025-10-26feat:text-generation: improve and rename lorem generatorCraig Jennings
Rename `lorem-generator.el` to `lorem-optimum.el` for fun. Enhance text tokenization, Markov chain learning, and text generation functions. Introduce new configuration variables for training files and improve efficiency with vectorized access. Add comprehensive benchmarks and unit tests under `tests/`. This improves performance and lays groundwork for further extensions.