devdocs/elisp/smie-lexer.html


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

 <h4 class="subsubsection">Defining Tokens</h4>   <p>SMIE comes with a predefined lexical analyzer which uses syntax tables in the following way: any sequence of characters that have word or symbol syntax is considered a token, and so is any sequence of characters that have punctuation syntax. This default lexer is often a good starting point but is rarely actually correct for any given language. For example, it will consider <code>"2,+3"</code> to be composed of 3 tokens: <code>"2"</code>, <code>",+"</code>, and <code>"3"</code>. </p> <p>To describe the lexing rules of your language to SMIE, you need 2 functions, one to fetch the next token, and another to fetch the previous token. Those functions will usually first skip whitespace and comments and then look at the next chunk of text to see if it is a special token. If so it should skip the token and return a description of this token. Usually this is simply the string extracted from the buffer, but it can be anything you want. For example: </p>
<div class="example"> <pre class="example">(defvar sample-keywords-regexp
  (regexp-opt '("+" "*" "," ";" "&gt;" "&gt;=" "&lt;" "&lt;=" ":=" "=")))
</pre>
<pre class="example">(defun sample-smie-forward-token ()
  (forward-comment (point-max))
  (cond
   ((looking-at sample-keywords-regexp)
    (goto-char (match-end 0))
    (match-string-no-properties 0))
   (t (buffer-substring-no-properties
       (point)
       (progn (skip-syntax-forward "w_")
              (point))))))
</pre>
<pre class="example">(defun sample-smie-backward-token ()
  (forward-comment (- (point)))
  (cond
   ((looking-back sample-keywords-regexp (- (point) 2) t)
    (goto-char (match-beginning 0))
    (match-string-no-properties 0))
   (t (buffer-substring-no-properties
       (point)
       (progn (skip-syntax-backward "w_")
              (point))))))
</pre>
</div> <p>Notice how those lexers return the empty string when in front of parentheses. This is because SMIE automatically takes care of the parentheses defined in the syntax table. More specifically if the lexer returns <code>nil</code> or an empty string, SMIE tries to handle the corresponding text as a sexp according to syntax tables. </p><div class="_attribution">
  <p class="_attribution-p">
    Copyright &copy; 1990-1996, 1998-2022 Free Software Foundation, Inc. <br>Licensed under the GNU GPL license.<br>
    <a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/SMIE-Lexer.html" class="_attribution-link">https://www.gnu.org/software/emacs/manual/html_node/elisp/SMIE-Lexer.html</a>
  </p>
</div>