diff options
Diffstat (limited to 'devdocs/python~3.12/library%2Ftoken.html')
| -rw-r--r-- | devdocs/python~3.12/library%2Ftoken.html | 216 |
1 files changed, 216 insertions, 0 deletions
diff --git a/devdocs/python~3.12/library%2Ftoken.html b/devdocs/python~3.12/library%2Ftoken.html new file mode 100644 index 00000000..798b567b --- /dev/null +++ b/devdocs/python~3.12/library%2Ftoken.html @@ -0,0 +1,216 @@ + <span id="token-constants-used-with-python-parse-trees"></span><h1>token — Constants used with Python parse trees</h1> <p><strong>Source code:</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.12/Lib/token.py">Lib/token.py</a></p> <p>This module provides constants which represent the numeric values of leaf nodes of the parse tree (terminal tokens). Refer to the file <code>Grammar/Tokens</code> in the Python distribution for the definitions of the names in the context of the language grammar. The specific numeric values which the names map to may change between Python versions.</p> <p>The module also provides a mapping from numeric codes to names and some functions. The functions mirror definitions in the Python C header files.</p> <dl class="py data"> <dt class="sig sig-object py" id="token.tok_name"> +<code>token.tok_name</code> </dt> <dd> +<p>Dictionary mapping the numeric values of the constants defined in this module back to name strings, allowing more human-readable representation of parse trees to be generated.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="token.ISTERMINAL"> +<code>token.ISTERMINAL(x)</code> </dt> <dd> +<p>Return <code>True</code> for terminal token values.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="token.ISNONTERMINAL"> +<code>token.ISNONTERMINAL(x)</code> </dt> <dd> +<p>Return <code>True</code> for non-terminal token values.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="token.ISEOF"> +<code>token.ISEOF(x)</code> </dt> <dd> +<p>Return <code>True</code> if <em>x</em> is the marker indicating the end of input.</p> </dd> +</dl> <p>The token constants are:</p> <dl class="py data"> <dt class="sig sig-object py" id="token.ENDMARKER"> +<code>token.ENDMARKER</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NAME"> +<code>token.NAME</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NUMBER"> +<code>token.NUMBER</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.STRING"> +<code>token.STRING</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NEWLINE"> +<code>token.NEWLINE</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.INDENT"> +<code>token.INDENT</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DEDENT"> +<code>token.DEDENT</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LPAR"> +<code>token.LPAR</code> </dt> <dd> +<p>Token value for <code>"("</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RPAR"> +<code>token.RPAR</code> </dt> <dd> +<p>Token value for <code>")"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LSQB"> +<code>token.LSQB</code> </dt> <dd> +<p>Token value for <code>"["</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RSQB"> +<code>token.RSQB</code> </dt> <dd> +<p>Token value for <code>"]"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.COLON"> +<code>token.COLON</code> </dt> <dd> +<p>Token value for <code>":"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.COMMA"> +<code>token.COMMA</code> </dt> <dd> +<p>Token value for <code>","</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.SEMI"> +<code>token.SEMI</code> </dt> <dd> +<p>Token value for <code>";"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.PLUS"> +<code>token.PLUS</code> </dt> <dd> +<p>Token value for <code>"+"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.MINUS"> +<code>token.MINUS</code> </dt> <dd> +<p>Token value for <code>"-"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.STAR"> +<code>token.STAR</code> </dt> <dd> +<p>Token value for <code>"*"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.SLASH"> +<code>token.SLASH</code> </dt> <dd> +<p>Token value for <code>"/"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.VBAR"> +<code>token.VBAR</code> </dt> <dd> +<p>Token value for <code>"|"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.AMPER"> +<code>token.AMPER</code> </dt> <dd> +<p>Token value for <code>"&"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LESS"> +<code>token.LESS</code> </dt> <dd> +<p>Token value for <code>"<"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.GREATER"> +<code>token.GREATER</code> </dt> <dd> +<p>Token value for <code>">"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.EQUAL"> +<code>token.EQUAL</code> </dt> <dd> +<p>Token value for <code>"="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DOT"> +<code>token.DOT</code> </dt> <dd> +<p>Token value for <code>"."</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.PERCENT"> +<code>token.PERCENT</code> </dt> <dd> +<p>Token value for <code>"%"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LBRACE"> +<code>token.LBRACE</code> </dt> <dd> +<p>Token value for <code>"{"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RBRACE"> +<code>token.RBRACE</code> </dt> <dd> +<p>Token value for <code>"}"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.EQEQUAL"> +<code>token.EQEQUAL</code> </dt> <dd> +<p>Token value for <code>"=="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NOTEQUAL"> +<code>token.NOTEQUAL</code> </dt> <dd> +<p>Token value for <code>"!="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LESSEQUAL"> +<code>token.LESSEQUAL</code> </dt> <dd> +<p>Token value for <code>"<="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.GREATEREQUAL"> +<code>token.GREATEREQUAL</code> </dt> <dd> +<p>Token value for <code>">="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.TILDE"> +<code>token.TILDE</code> </dt> <dd> +<p>Token value for <code>"~"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.CIRCUMFLEX"> +<code>token.CIRCUMFLEX</code> </dt> <dd> +<p>Token value for <code>"^"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LEFTSHIFT"> +<code>token.LEFTSHIFT</code> </dt> <dd> +<p>Token value for <code>"<<"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RIGHTSHIFT"> +<code>token.RIGHTSHIFT</code> </dt> <dd> +<p>Token value for <code>">>"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DOUBLESTAR"> +<code>token.DOUBLESTAR</code> </dt> <dd> +<p>Token value for <code>"**"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.PLUSEQUAL"> +<code>token.PLUSEQUAL</code> </dt> <dd> +<p>Token value for <code>"+="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.MINEQUAL"> +<code>token.MINEQUAL</code> </dt> <dd> +<p>Token value for <code>"-="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.STAREQUAL"> +<code>token.STAREQUAL</code> </dt> <dd> +<p>Token value for <code>"*="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.SLASHEQUAL"> +<code>token.SLASHEQUAL</code> </dt> <dd> +<p>Token value for <code>"/="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.PERCENTEQUAL"> +<code>token.PERCENTEQUAL</code> </dt> <dd> +<p>Token value for <code>"%="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.AMPEREQUAL"> +<code>token.AMPEREQUAL</code> </dt> <dd> +<p>Token value for <code>"&="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.VBAREQUAL"> +<code>token.VBAREQUAL</code> </dt> <dd> +<p>Token value for <code>"|="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.CIRCUMFLEXEQUAL"> +<code>token.CIRCUMFLEXEQUAL</code> </dt> <dd> +<p>Token value for <code>"^="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.LEFTSHIFTEQUAL"> +<code>token.LEFTSHIFTEQUAL</code> </dt> <dd> +<p>Token value for <code>"<<="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RIGHTSHIFTEQUAL"> +<code>token.RIGHTSHIFTEQUAL</code> </dt> <dd> +<p>Token value for <code>">>="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DOUBLESTAREQUAL"> +<code>token.DOUBLESTAREQUAL</code> </dt> <dd> +<p>Token value for <code>"**="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DOUBLESLASH"> +<code>token.DOUBLESLASH</code> </dt> <dd> +<p>Token value for <code>"//"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.DOUBLESLASHEQUAL"> +<code>token.DOUBLESLASHEQUAL</code> </dt> <dd> +<p>Token value for <code>"//="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.AT"> +<code>token.AT</code> </dt> <dd> +<p>Token value for <code>"@"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.ATEQUAL"> +<code>token.ATEQUAL</code> </dt> <dd> +<p>Token value for <code>"@="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.RARROW"> +<code>token.RARROW</code> </dt> <dd> +<p>Token value for <code>"->"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.ELLIPSIS"> +<code>token.ELLIPSIS</code> </dt> <dd> +<p>Token value for <code>"..."</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.COLONEQUAL"> +<code>token.COLONEQUAL</code> </dt> <dd> +<p>Token value for <code>":="</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.EXCLAMATION"> +<code>token.EXCLAMATION</code> </dt> <dd> +<p>Token value for <code>"!"</code>.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.OP"> +<code>token.OP</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.AWAIT"> +<code>token.AWAIT</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.ASYNC"> +<code>token.ASYNC</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.TYPE_IGNORE"> +<code>token.TYPE_IGNORE</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.TYPE_COMMENT"> +<code>token.TYPE_COMMENT</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.SOFT_KEYWORD"> +<code>token.SOFT_KEYWORD</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.FSTRING_START"> +<code>token.FSTRING_START</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.FSTRING_MIDDLE"> +<code>token.FSTRING_MIDDLE</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.FSTRING_END"> +<code>token.FSTRING_END</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.COMMENT"> +<code>token.COMMENT</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NL"> +<code>token.NL</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.ERRORTOKEN"> +<code>token.ERRORTOKEN</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.N_TOKENS"> +<code>token.N_TOKENS</code> </dt> <dd></dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.NT_OFFSET"> +<code>token.NT_OFFSET</code> </dt> <dd></dd> +</dl> <p>The following token type values aren’t used by the C tokenizer but are needed for the <a class="reference internal" href="tokenize#module-tokenize" title="tokenize: Lexical scanner for Python source code."><code>tokenize</code></a> module.</p> <dl class="py data"> <dt class="sig sig-object py"> <span class="sig-prename descclassname">token.</span><span class="sig-name descname">COMMENT</span> +</dt> <dd> +<p>Token value used to indicate a comment.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py"> <span class="sig-prename descclassname">token.</span><span class="sig-name descname">NL</span> +</dt> <dd> +<p>Token value used to indicate a non-terminating newline. The <a class="reference internal" href="#token.NEWLINE" title="token.NEWLINE"><code>NEWLINE</code></a> token indicates the end of a logical line of Python code; <code>NL</code> tokens are generated when a logical line of code is continued over multiple physical lines.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="token.ENCODING"> +<code>token.ENCODING</code> </dt> <dd> +<p>Token value that indicates the encoding used to decode the source bytes into text. The first token returned by <a class="reference internal" href="tokenize#tokenize.tokenize" title="tokenize.tokenize"><code>tokenize.tokenize()</code></a> will always be an <code>ENCODING</code> token.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py"> <span class="sig-prename descclassname">token.</span><span class="sig-name descname">TYPE_COMMENT</span> +</dt> <dd> +<p>Token value indicating that a type comment was recognized. Such tokens are only produced when <a class="reference internal" href="ast#ast.parse" title="ast.parse"><code>ast.parse()</code></a> is invoked with <code>type_comments=True</code>.</p> </dd> +</dl> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.5: </span>Added <a class="reference internal" href="#token.AWAIT" title="token.AWAIT"><code>AWAIT</code></a> and <a class="reference internal" href="#token.ASYNC" title="token.ASYNC"><code>ASYNC</code></a> tokens.</p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.7: </span>Added <a class="reference internal" href="#token.COMMENT" title="token.COMMENT"><code>COMMENT</code></a>, <a class="reference internal" href="#token.NL" title="token.NL"><code>NL</code></a> and <a class="reference internal" href="#token.ENCODING" title="token.ENCODING"><code>ENCODING</code></a> tokens.</p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.7: </span>Removed <a class="reference internal" href="#token.AWAIT" title="token.AWAIT"><code>AWAIT</code></a> and <a class="reference internal" href="#token.ASYNC" title="token.ASYNC"><code>ASYNC</code></a> tokens. “async” and “await” are now tokenized as <a class="reference internal" href="#token.NAME" title="token.NAME"><code>NAME</code></a> tokens.</p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>Added <a class="reference internal" href="#token.TYPE_COMMENT" title="token.TYPE_COMMENT"><code>TYPE_COMMENT</code></a>, <a class="reference internal" href="#token.TYPE_IGNORE" title="token.TYPE_IGNORE"><code>TYPE_IGNORE</code></a>, <a class="reference internal" href="#token.COLONEQUAL" title="token.COLONEQUAL"><code>COLONEQUAL</code></a>. Added <a class="reference internal" href="#token.AWAIT" title="token.AWAIT"><code>AWAIT</code></a> and <a class="reference internal" href="#token.ASYNC" title="token.ASYNC"><code>ASYNC</code></a> tokens back (they’re needed to support parsing older Python versions for <a class="reference internal" href="ast#ast.parse" title="ast.parse"><code>ast.parse()</code></a> with <code>feature_version</code> set to 6 or lower).</p> </div> <div class="_attribution"> + <p class="_attribution-p"> + © 2001–2023 Python Software Foundation<br>Licensed under the PSF License.<br> + <a href="https://docs.python.org/3.12/library/token.html" class="_attribution-link">https://docs.python.org/3.12/library/token.html</a> + </p> +</div> |